Algorithm Integrity Docs - Getting Started | Risk Insights Article

Documentation makes it easier to consistently maintain algorithm integrity.

This is well known.

But there are lots of types of documents to prepare, and often the first hurdle is just thinking about where to start.

So this simple guide is meant to help do exactly that – get going.

Algorithm Integrity Documentation – starting points

Here are 5 key items to get started on if you don't yet have anything in place.

If you have already started, consider whether you have each of these.

1. Algorithm inventory
2. Risk assessment
3. Policies, procedures, standards, and controls
4. Design and development documentation
5. Technical documentation

Let’s explore each in a bit more detail.

1. Algorithm Inventory

A list of the algorithms in use and/or planned.

Includes a clear, documented statement of each algorithm's objective - purpose and intended use.

This provides an overview of your algorithmic/AI landscape.

2. Risk Assessment

A document outlining the identified risks, and how these will be managed.

If the algorithm affects customers, we may want to make this a risk and impact assessment, adding impact to reflect the importance of ensuring customer safety and fairness.

This helps prioritise efforts, ensure responsible use and keep track of the key matters to focus on to maintain customer safety.

3. Policies, Procedures, Standards and Controls

The principles and expectations for all stages of the algorithm lifecycle, together with step-by-step instructions, mandatory rules, and then measures to ensure the above are all upheld.

For these, we set a regular review cycle – for example, at least annually - to ensure they remain current.

These help ensure consistency and reduce ambiguity.

4. Design and Development Documentation

Key decisions, methodologies, assumptions and rationales during algorithm design and development.

Note: algorithm design and development is not a one-and-done, as the algorithms will change regularly.

This ensures transparency and aids in future improvements.

5. Technical Documentation

Describes the AI system's design, specifications, algorithms, data sources, testing methods, performance metrics, known limitations, and workarounds.

Ideally, this will include a version history, and audit trail reflecting key changes.

This supports maintainability and compliance efforts.

A few important points to note

This is certainly not a comprehensive list. It is just a starting point.

Involving relevant stakeholders in creating and maintaining these can help ensure coverage and buy-in.

These should be living documents, regularly reviewed, and updated as your algorithms evolve.

Aim for accessibility and readability. There is value in the documents being understandable to both technical and non-technical stakeholders, where relevant. In some cases, the detail (that is also important) may mean that the audience is technical folk, so they shouldn’t be diluted either.

Store them in an accessible location, but without risking security, privacy, and confidentiality.

There will be other documentation; what exactly these are will depend on several factors. The factors determining what these are include organisational expectations, the nature, context and purpose of each algorithmic system, and compliance obligations.

Disclaimer: The information in this article does not constitute legal advice. It may not be relevant to your circumstances. It may not be appropriate for high-risk use cases (e.g., as outlined in The Artificial Intelligence Act - Regulation (EU) 2024/1689, a.k.a. the EU AI Act). It was written for consideration in certain algorithmic contexts within banks and insurance companies, may not apply to other contexts, and may not be relevant to other types of organisations.