If our algorithmic system was a human employee, would we keep them on the payroll?
Algorithmic systems are not just tech products.
Regulators are formalising this too, including the EBA's AI mapping (Europe), MAS (Singapore) with the FEAT principles, FAR and CPS230 in Australia, the UK’s SMCR, the OCC (U.S.) model risk guidance, and Canada's OSFI guidelines.
roughly translates to:
If a model drives a business outcome, the business head owns the result.
It’s not an IT problem, and certainly not a third-party provider’s problem.
It’s a line-of-business responsibility.
So, models making underwriting/credit or claims decisions aren’t the tech team’s responsibility. They fall under your (line of business) remit. Of course, depending on org size and structure, other teams will help with many aspects: platform, security, privacy, broad frameworks, guidance, etc. But you own the decisions. That means if the algorithmic system isn't delivering to your standard, you have the authority to stop using it until it’s fixed.
And evaluated it like one?
Now, I know that we don’t treat models or systems like people; I’m not a fan of anthropomorphic fluff. So, consider this a simple thought experiment. Of course, the details will differ from how we manage a person.
If a human underwriter:
Declined customers purely because they lived in the "wrong" postcode…
Couldn’t explain why they rejected a loan application (or gave an answer so complex it was useless to the customer)…
Made inconsistent decisions, affected by their mood for that day or hour…
Used information that had nothing to do with the matter at hand, potentially breaching privacy obligations…
We wouldn’t open a Jira ticket. We’d put them on a performance improvement plan.
Why should a model get a free pass?
Forget the validation reports (that mainly focus on technical accuracy) for a moment.
Look at the decisions our model made last month.
If a human made those exact same calls, would we promote them, or performance manage them?
Disclaimer: The info in this article is not legal advice. It may not be relevant to your circumstances. It was written for specific contexts within banks and insurers, may not apply to other contexts, and may not be relevant to other types of organisations.