Our Approach

An algorithmic system includes various components.

A "system" is not purely about tech.  An algorithmic system includes:

• manual processes, governance mechanisms, human oversight of decisions, etc.
• data, data flows, algorithms - rules based programs, machine learning algorithms, AI components, etc.

When we review algorithm integrity, we focus on accuracy and/or fairness.

You may decide to include any of the other key aspects of algorithm integrity.

When scoping the audit, based on your objective for the audit, we will work with you to determine which of the other aspects to include.

In some cases, other aspects may have to form part of the audit. For example, "alignment with objectives" is often required. If this is the case, we will discuss this with you upfront and include it. 

Evaluating algorithmic systems to help eliminate bias

Examples include: fair algorithmic systems for loan origination, insurance premium calculations or claims fraud triage.

Approach

1. Scope definition: work with you to clearly define the audit objectives and scope.
2. Base understanding: explore your governance mechanisms, data flow diagrams and how decisions are made.
3. Information collection: gather relevant data and documentation, interview key stakeholders.
4. Deep dive: review data inputs, models, other system components and outputs to identify potential sources of bias.
5. Validation: confirm the sources(s) of potential bias and identify potential ways to resolve or mitigate these.
6. Reporting: you receive a comprehensive report with findings and suggestions for resolution or mitigation.

What We Typically Look For

• Equitability
• Proxies
• Unapproved data sources (e.g., external data that is not needed)
• If relevant, an inability to explain how the decisions/outputs are produced.