Algorithmic System Integrity: Explainability (Part 2) - Complexity

In this previous article, we explored the concept of explainability, its importance, and four challenges.

In this article, we explore some solutions to the first challenge: complex algorithms.

Challenge Recap

Complex models can operate as "black boxes."

Their internal processes are so complex that even their developers may struggle to explain how inputs lead to specific outputs.

This poses several problems:

• Trust: Customers and other stakeholders are wary of automated decisions that they can't understand.
• Compliance: Regulatory frameworks and standards are now calling for explainability, especially for decisions with significant human impact.
• Errors: When we can't see how a model works, it's harder to spot and fix mistakes.

Solutions

Banks and insurers can use several methods to address this problem.

Here are three sets of commonly used methods, each of which work better when cross-functional teams are involved:

1. Simplification

Strike a balance between accuracy and simplicity.

Simpler models can be easier to explain, but may not be as accurate as the more complex models.

Consider the performance (accuracy) vs explainability diagram below - Figure 1.

In general, the more complex models (that often produce more accurate results) are harder to explain.

Figure 1: Model explainability vs. model performance comparison, reproduced from Zhang et al. (2021), licensed under CC BY 4.0 [i].

Notes: Not an exact science, the diagram is indicative; other diagrammatic representations may look different. Based on the nature of the research, we have interpreted performance as being synonymous with accuracy/reliability, rather than processing efficiency.

Simplification can include:

• Establishing clear thresholds: determine what an acceptable trade-off between complexity and explainability is. Use this to choose the model that suits.
• Implementing validation: more complex isn’t always more accurate, so don’t miss the opportunity to use a model that is easier to explain and equally/more accurate.
• Developing parallel models: in some situations, you may want to create simpler, interpretable models that approximate the complex ones. If you can get them close enough, you may be able to use them to explain.

2. Data selection

If the data going into the model is not relevant, it can make it difficult to determine how decisions are reached.

There are situations where irrelevant data can result in more “accurate” predictions. But if the data is not relevant, then this accuracy gain might be a technical improvement, not a real one.

We also need to be careful not to eliminate relevant data. And this can vary, depending on the use case and the nature of the system.

For both internal data and external data, carefully choose relevant data, and use data for its intended purpose.

There's also a discussion about eliminating protected attributes. We'll explore this in a future article, it is a long story.

3. Visibility Techniques

These methods work best when combined with proactive complexity management from the previous sections.

There are several techniques that we can use to interpret model processing.

This list is not exhaustive, and what to use will depend on the use case:

• Scenario Testing: Conduct "what-if" analyses to demonstrate how changing inputs affect outcomes.
• SHAP (SHapley Additive exPlanations): Calculate feature importance scores. This method is used to explain the output of machine learning models by assigning each feature a value for a particular prediction.
• LIME (Local Interpretable Model-agnostic Explanations): Create linear explanations of complex decisions. This method is used to explain individual predictions by approximating the model locally with an interpretable model.
• PDP (Partial Dependence Plots) and ICE (Individual Conditional Expectation) Plots: Create population level and instance-specific feature impacts. These methods are used to visualise how features affect predictions on average (PDP) and for specific instances (ICE).

In determining which method(s) to use, consider regulatory requirements (e.g., GDPR's "right to explanation") and operational constraints (e.g., computational resources required).

Next

The next article in this series will focus on the second challenge: complicated processes.

[i] Zhang, Y., Liu, D., Cai, W., Chen, H., Heng, P., Li, Q., ... & Wang, S. (2021). Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond. ResearchGate. https://www.researchgate.net/publication/353621660 (CC BY 4.0 https://creativecommons.org/licenses/by/4.0/).

Disclaimer: The information in this article does not constitute legal advice. It may not be relevant to your circumstances. It was written for specific algorithmic contexts within banks and insurance companies, may not apply to other contexts, and may not be relevant to other types of organisations.