How reliable is the algorithm review that you have commissioned?

One common issue with audits and reviews is undue reliance.

Can you rely on the report to tell you what you need to know?

Could you be relying on it too much?

 Listen to the audio (human) version of this article - Episode 1 of Algorithm Integrity Matters

Why is this a problem?

There are several reasons:

Undue reliance can create a false sense of security.

If you think that the report means that everything is ok, when it's not, nothing will change.

In this case, you may be better off without the review. A false sense of security can make us complacent.

Not knowing what our blind spots are can make us more vigilant in trying to find them.

But if a report gives us a clean bill of health, we are lulled into thinking we don't have to worry.

This is dangerous.

An review may not identify improvements that we can make.

And some of these could be easy fixes.

We want to know about these and resolve them straight away.

We also want to know the longer-term opportunities, and how to plan for or build towards them.

We may not be meeting our regulatory compliance expectations, or obligations.

No explanation needed here.

We're not doing what we can to protect our customers from disadvantage or harm.

We think that we are. We smile at our customers, safe in the knowledge that we're looking after them.

But - inadvertently - we're not. And when we find out that this is the case, it will make us really uncomfortable.

How can we prevent this from happening?

The order of this list may vary, but it is worth considering each of these:

1. Make sure the auditor is independent

An independent auditor is essential for an unbiased and objective audit.

By ensuring your auditor is independent, you enhance the credibility and reliability of the work. 

Here’s what to look for:

• External Party: the auditor is typically an external party, not employed by your organisation. This helps ensure objectivity and impartiality.

• Revenue Sources: the auditor should not derive more revenue from non-audit work than audit work for your organisation. They should focus on audit work and prioritise it over consulting or advisory services. If this balance is not maintained, their judgment could be compromised.

• No Prior Involvement: the auditor should not have been involved in the design of the subject matter. An auditor marking their own work is a conflict of interest. They need to remain objective.

• Internal Audit: an internal audit function can be independent, but not all internal audit teams are independent. It's important to assess the factors that contribute to independence. They include:

  • Reporting Lines: IA reports (functionally) to an audit committee or board, not to management.

  • Quality Assurance: regular reviews, including adherence to professional standards.

  • No Prior Involvement: not involved in the design of the subject matter. 

2. Be clear about what is in scope

ForHumanity refers to this as the "Target of Evaluation". The definition is quite comprehensive.

If you're commissioning an audit under a ForHumanity scheme, you'll need to cover all aspects they outline.

Even if you're not commissioning such an audit, include enough to make the scope unambiguous.

A good auditor will insist on this.

Here’s a few key considerations:

• Specify Objectives: state the specific objectives of the audit. What are you aiming to achieve? For example, compliance verification, process improvement, or making sure customers are not harmed or disadvantaged.
• Define the Boundaries: clearly outline what areas, products, brands, processes, systems, etc. are included. This helps avoid any confusion about what will be examined.
• Confirm the Deliverables: what these will be, whether there will be drafts produced, what the deliverables (typically reports) will contain, perhaps even what level of detail these will have.
• Document the Scope: keep a written record. This serves as a reference point and is valuable.
• Optional: detail the specific models or types of models that will be included.

By being clear about what is in scope, you set the stage for a focused and effective audit, making sure everyone involved knows exactly what to expect.

3. Be clear about what is not in scope

This sounds like it should be included in the previous point about scope.

It should.

But we get this wrong so often that it's worth calling out separately.

Here's why this matters:

• Clarity on Findings: knowing what's not in scope helps you interpret the audit findings correctly. You won't mistakenly assume certain areas were reviewed when they weren't.
• Risk Assessment: understanding exclusions allows you to more easily determine whether any critical areas are being overlooked. You might want to renegotiate.
• Expectation Management: it prevents misunderstandings with stakeholders about what the audit will (and won't) cover. This is crucial for managing expectations about the audit's outcomes.
• Future Planning: knowing what's not covered can inform your planning for future audits.

Understanding what's not in scope is just as important as knowing what is in scope.

It provides a more complete picture of the audit's boundaries and limitations.

4. Understand how the audit is being conducted

A good auditor will not mind showing you exactly how they are performing the audit.

Be careful with this, because you might get very bored. A good auditor could go on for hours about the specific way they selected a sample for testing, how they merged two datasets and filtered for ... blah blah blah.But most good auditors are also just humans, and you can ask them politely to summarise.

Ask about things like these:

• How they are testing: which in most cases should involve more than just discussions, and may need to go beyond control testing alone.

• How they are varying their testing: this includes different approaches like observing processes, reviewing documentation, reviewing code, or testing entire populations.

• How deep they are going: are they testing individual models or data fields, checking calculations at a granular level, or just skimming the surface?

• What the bases for expectations are (what they are testing against): this could be vendor / industry guidance for a specific control, a recognised standard, your internal policy, etc.

• Whether they are testing design, effectiveness or both: in testing controls, are they setting a baseline by testing the design, and then covering a period by testing effectiveness?

5. Appreciate that an audit may not pick up all issues

Audits are not designed to find every potential issue.

An audit that provides 100% confidence is extremely rare (if not non-existent) and would be prohibitively expensive.

However, good audits:

• Focus on Key Improvements: a good audit will try to identify the most significant opportunities for improvement.

• Are Well-Planned: a good audit is thoroughly planned, and can get close to finding all the critical matters.

The Goal: Doing Right by Our Customers

Ultimately, our aim is to serve our customers well.

This means:

• We want to know the truth.
• We need to be aware when there's a risk that we might not meet our obligations.
• We want to identify issues so we can fix them.

The Need for Robust Audits

This is why we need robust audits that we can largely rely on to point out what's working well. 

And, importantly, highlight what isn't working.

While an audit might not catch everything, a well-conducted audit is a valuable tool for better serving our customers.

It's about finding a balance between thoroughness and practicality, with the end goal of enhancing our service and keeping our customers safe.