Skip to content

Curated Insight: what to keep on file for a model that affects customers

Picture of Yusuf Moolla

 

When a customer disputes an automated decision, or a regulator asks how a model reached its outcome, the answer is either in your records or it isn't. By then, it might be too late to backfill the information; even if the information can be found, getting it can be painful and stressful.

EIOPA's expert group on digital ethics published a set of AI governance principles for insurers in 2021. These curated insights are usually based on more recent guides, but the EIOPA report is still relevant, with clear explanations and useful detail. EIOPA is the European Insurance and Occupational Pensions Authority.

In its section on data governance, there's a page that helps answer the question “what do we need to keep a record of for the AI models we use?”. A plain checklist of 11 things to record so a decision can be traced, audited, and defended later.

I've reproduced that page below (for reference, it's page 59 of the report).

Insurance firms should keep appropriate records of the data and the modelling methodologies to ensure their reproducibility / traceability

Considering the difficulty of understanding the functioning of an AI system (the so-called black-box effect), and thus the associated difficulties in auditing them, insurance firms should keep relevant records of the data used in the AI models as well as the modelling methodologies. This would allow, on the one hand, to trace back decisions and verify decisions in case they could eventually be disputed, and on the other hand, avoid misuse of models by inattention.

In this context, and duly taking into account the principle of proportionality, for those high-impact AI applications (for which it is recommended to have repositories with all deployed models in the organisation), the main attributes of the model, regardless of whether it is developed internally or if it is outsourced from third parties, should be recorded as described in the table below:

Figure 17 – Record keeping requirements for high impact AI applications

Record

Description

Reasons for using AI

Explanation of the business objective / task pursued by using AI and its consistency with corporate strategies / objectives. Explanation how this was implemented into the AI system. This would help avoid misusage of the AI system and enable its audit and independent review.

Integration into IT infrastructure

Description of how the model is integrated in the current IT system of the organisation and document any significant changes that could eventually take place.

Staff involved in the design and implementation of the AI model

Identify all the roles and responsibilities of the staff involved in the design and implementation of the AI model as well as their training needs. This would allow to ensure accountability of the responsible persons.

Data collection

Document how the ground truth was built including how consideration was given to identifying and removing potential bias in the data. This would include explaining how input data was selected, collected and labelled.

Data preparation

Records of the data used for training the AI model, i.e. the variables with their respective domain range. This would include defining the construction of training, test and prediction dataset. For built (engineered) features, records should exist on how the feature was built and the associated intention.

Data post processing

Description of processes in place to operationalize the use of data and to achieve continuous improvement (including addressing potential bias). Records should specify the timing and frequency of data improvement actions.

Technical choices / arbitration

Document why a specific type of AI algorithm was chosen and not others, as well as the associated libraries with exact references. The limitation / constraints of the AI model should be documented and how they are being optimised alongside their supporting rationale. Ethical, transparency and explainability trade-offs that may apply together with their rationale should also be recorded.

Code and data

Record the code used to build any AI model which goes to production / exploitation. Exclusively for high impact applications, insurance firms should record the training data used to build the AI model and all the associated hyper parameters, including pseudo-random seeds. If this requirement proved to be too burdensome, insurance firms may put in place alternative measures that ensure the auditability of the AI model and the accountability of the firm using them.

Model performance

Explanations should include, inter alia, how performance is measured (KPIs) and what level of performance is deemed satisfactory, including scenario analysis and timing and frequency of reviews and / or retraining of the model. Ethical, transparency and explainability trade-offs that may apply together with their rationale should also be recorded.

Model security

Describe mechanisms in place (or make reference to) to ensure the model is protected from outside attacks and more subtle attempts to manipulate data or algorithms themselves: how robust is the model to manipulation attacks (especially important in auto ML models).

Ethics and trustworthy assessment

Description of the AI use case impact assessment i.e. the potential impact on consumers and / or insurance firms of the concrete AI use case. Explain how the governance measures put in place throughout the AI systems lifecycle address the risks included in the AI use case impact assessment and ensure ethical and trustworthy AI systems.

Source: EIOPA Consultative Expert Group on Digital Ethics in insurance, "Artificial Intelligence Governance Principles: Towards Ethical and Trustworthy Artificial Intelligence in the European Insurance Sector" (2021), page 59, Figure 17. © EIOPA, 2021. (Reproduction is authorised provided the source is acknowledged.)

 

Takeaway

Pick one model that affects customers and see:

  • how many of the 11 you could produce today
  • how many you could produce for previous versions of the model
  • how far back you can go.

The ones you can't produce are your starting list, for the current version and, ideally, the ones before it.

 

Disclaimer: The info in this article is not legal advice. It may not be relevant to your circumstances. It was written for specific contexts within banks and insurers, may not apply to other contexts, and may not be relevant to other types of organisations.