Checklists – useful, but use with care

For about as long as I’ve been working, I have used checklists in one form or another.
They help make sure that all required items are completed, particularly when we do the same things regularly.

For algorithm integrity, checklists are helpful for designing and operating controls. They are also very useful in preparing for reviews, and in conducting audits. To check that nothing obvious is missed.

But I have a bit of love-hate relationship with them.
They have their limits. And when we rely on them too much, we can miss important issues.

The Comfort of a List

When we’re scoping an algorithm integrity review, we use these 10 key aspects of algorithm integrity. We check each item for relevance. This helps us structure the approach, and make sure that we don’t forget important areas.

Checklists also help with consistency. When different people are involved, or when processes are repeated over time, a checklist can make sure everyone is focusing on the same things. It’s a simple way to keep the basics front of mind.

The Limits of Ticking Boxes

But checklists can also be hazardous.

They can create constraints. To extend on the scoping example, if we stick to those 10 items only, we could miss important areas and fall short of the objective of the review. So, after we go through the 10 items, we review the scope again to confirm that the objective will be met.

Checklist items are typically binary. Not everything can be reduced to a simple yes/no answer:

• “Is the model documented?” is clear-cut.
• “Does the documentation include all the required sections?” can be answered with specific checklist items, if we detail them.
• But then there’s “is the documentation accurate and sufficient?” This probably needs more thought and discussion, rather than just a tick or cross.

A tool, not a rulebook

This is why it’s important to treat a checklist as a tool, or guide.
Not a rulebook. And not something that should limit us; we often need to go beyond it.

They work best when they prompt questions and encourage thought and discussion.
And, sometimes, the most important insights come from the questions that aren’t on the list at all.

Disclaimer: The information in this article does not constitute legal advice. It may not be relevant to your circumstances. It was written for specific algorithmic contexts within banks and insurance companies, may not apply to other contexts, and may not be relevant to other types of organisations.