Skip to content

Structuring the review focus: top-down, then bottom-up

Picture of Yusuf Moolla
TL;DR
• Don’t default to a “standard” or “comprehensive” package.
• Start with a simple top-down focus so the review stays tied to the objective.
• Then do a bottom-up check to catch important issues the top-down view might miss. Include bottom-up areas only when there’s a clear, context-driven reason.

 

When defining the focus for an algorithm integrity review, it’s tempting to treat the range of possible areas like a menu and order one of everything.

This doesn’t always work in practice:

  1. If you take on too much, the review loses sight of the objective.
  2. Doing everything can cost more than it needs to.
  3. Some areas may not be relevant, creating confusion and frustration.

We want focused, efficient reviews that don’t miss important matters.

This article outlines one approach to making that happen. It works in most cases, but a bottom-up “check everything” approach may suit compliance reviews in highly regulated contexts, while an iterative approach may work better for new or poorly documented systems.

 

Top-down: driven by the objective(s)

The first and most important step is to define exactly why you want the review. Are you after an independent perspective for your board, a fairness check, an accuracy review, or something else?

Start with “why are we doing this review?” Make it about what you need, not what the “standard package” includes.

Example objective: “We want confidence that our claims triage is accurate and fair.”

Pick a small set of focus areas that match this:

  • Accuracy: our rules, models, and weights are working
  • Fairness: we’re targeting behaviour, not protected characteristics
  • Explainability: we can explain why claims were prioritised.

 

Bottom-up: fill in key gaps

Top-down can leave gaps. So we check carefully for anything important the top-down view missed. In most cases, we either add secondary areas, add a few targeted checks, or disregard.

We ask “is there any reason to include other areas, even partially?”

Sometimes a deeper look will surface an aspect you initially left out, like:

  • Alignment with objectives: even if the system is accurate, it might be optimising low-value cases and not serving the intended purpose.
  • Ethics and training: to ensure ongoing fairness.

 

A brief recap

Start with the core aspects that you know are directly relevant to your objective.

Then consider the remaining areas and test whether any are material for your use case, even if only partially. Expand only where you have a clear, context-driven reason.

This keeps the review focused, covers what matters, and avoids bloat.


Disclaimer: The information in this article does not constitute legal advice. It may not be relevant to your circumstances. It was written for specific algorithmic contexts within banks and insurance companies, may not apply to other contexts, and may not be relevant to other types of organisations.