When defining the focus for an algorithm integrity review, it’s tempting to treat the range of possible areas like a menu and order one of everything.
This doesn’t always work in practice:
We want focused, efficient reviews that don’t miss important matters.
This article outlines one approach to making that happen. It works in most cases, but a bottom-up “check everything” approach may suit compliance reviews in highly regulated contexts, while an iterative approach may work better for new or poorly documented systems.
The first and most important step is to define exactly why you want the review. Are you after an independent perspective for your board, a fairness check, an accuracy review, or something else?
Start with “why are we doing this review?” Make it about what you need, not what the “standard package” includes.
Example objective: “We want confidence that our claims triage is accurate and fair.”
Pick a small set of focus areas that match this:
Top-down can leave gaps. So we check carefully for anything important the top-down view missed. In most cases, we either add secondary areas, add a few targeted checks, or disregard.
We ask “is there any reason to include other areas, even partially?”
Sometimes a deeper look will surface an aspect you initially left out, like:
Start with the core aspects that you know are directly relevant to your objective.
Then consider the remaining areas and test whether any are material for your use case, even if only partially. Expand only where you have a clear, context-driven reason.
This keeps the review focused, covers what matters, and avoids bloat.
Disclaimer: The information in this article does not constitute legal advice. It may not be relevant to your circumstances. It was written for specific algorithmic contexts within banks and insurance companies, may not apply to other contexts, and may not be relevant to other types of organisations.