Algorithm Audit Guidance: some aren't relevant to your context

AI and algorithm audits help ensure ethical and accurate data processing, preventing harm and disadvantage.

However, the guidelines are not yet mature, and quite disparate.

This can make the audit process confusing, and quite daunting - how do you wade through it all to find the information that you need, in deciding how to commission your audit?

Fortunately, there is a solution - narrowing the guidelines down, based on relevance.

Not all existing guidelines are universally applicable. This can vary based on your situation, including:

1. The specific context of your industry

2. The nature of your deployment

3. The characteristics of the system being audited

4. Whether the audit is internal or external

5. Who produces the guidance and for whom it is intended

This article will help you distinguish between audit guidance that applies to your situation and guidance that may not be relevant to your industry, deployment, or organisational needs.

 Listen to the audio (human) version of this article - Episode 3 of Algorithm Integrity Matters

Industry Context

Your industry significantly influences the applicability of audit guidance. There are varying disclosure requirements and risk profiles:

1. Disclosure: Public sector organisations and publicly listed companies often need to disclose audit outcomes. This transparency helps to maintain public confidence. Private companies, however, may opt to keep audit results private to safeguard business confidentiality and strategic interests.

2. Risk: The level of risk, and the nature of risks, vary by industry. For example, the aviation industry, where safety is paramount, may require more rigorous audit standards than some other industries. 

Nature of the Deployment

The specific approach you take towards acquiring and implementing AI or algorithmic systems significantly impacts the relevance of audit guidance.

1. Off-the-Shelf Systems: off-the-shelf solutions may benefit from audit guidance focused on evaluating vendor reliability and understanding pre-built model limitations, etc.

2. Internally Built Systems: for organisations that develop their own AI systems, audit guidance may focus on the entire development lifecycle, including design, testing, and deployment. 

3. Adapted Systems: When organisations buy AI systems and then adapt them, such as through retraining or fine-tuning, audit guidance may include focus on the adaptation process. This includes evaluating the quality and relevance of the new data used for training, the effectiveness of the adaptation in meeting specific goals, and the potential risks introduced by modifications.

Nature of the System

The specific characteristics of the system being audited also impact the relevance of audit guidance.

1. Complexity and Scale: Systems with high complexity or large scale may require specialised audit techniques and guidance that address specific technical challenges, such as model interpretability and data integration.

2. Purpose and Functionality: The intended use of the system can dictate the focus of the audit. For example, systems used for critical decision-making, like autonomous vehicles, may require more stringent audits compared to systems used for marketing analytics.

3. Lifecycle Stage: The stage of the system's lifecycle—whether it is in development, deployment, or maintenance—can influence the type of audit guidance needed. Early-stage systems might focus on design and testing, mature systems may extend this to include performance monitoring.

Internal vs. External Audits

Whether an audit is conducted internally or externally can affect the applicability of guidance.

1. Frequency:

   • Internal Audits: can be conducted more frequently as part of ongoing monitoring and improvement efforts - e.g., monthly or quarterly.

   • External Audits: typically less frequent - e.g., annually.

2. Audit Process:

   • Internal Audits: tends to be more integrated with operations, with flexibility to adjust the scope and focus based on emerging risks or strategic changes. 

   • External Audits: follow a more structured and standardised process, often guided by established frameworks and methodologies. A more formal, independent perspective.

3. Type of Report Produced:

   • Internal Audits: generally more detailed and operational, providing specific recommendations for improvement. Sometimes used for internal decision-making and strategic planning.

   • External Audits: often concise and focused on high-level findings, compliance, and assurance. They are designed to provide stakeholders with an independent assessment. Sometimes these are able to be shared with external parties, such as regulators or investors. Where appropriate (e.g., public sector or publicly traded) the reports may be produced to the public.

Source and Audience

Understanding who produces the guidance, and for whom it is intended, helps determine applicability.

1. Regulatory Bodies: focuses on compliance and legal requirements. 

2. Industry Associations: reflects industry specific standards and better practices. 

3. Consulting Firms and Experts: can offer tailored solutions and innovative practices. However, these need careful consideration, to ensure alignment with your specific context and needs. Finding out who the firm/expert serves will help clarify how relevant the guidance is to you: for example, a firm that typically works with public sector entities will typically produce guidance that includes an expectation that audit reports are made public, but this may not be relevant to you.

4. Academic and Research Institutions: may provide insights into emerging trends and theoretical frameworks. While valuable, this may not always be practical for immediate implementation.

One size does not fit all

Guidelines taken out of context can create unnecessary complexity and frustration.

By carefully evaluating the relevance of audit guidance, you can focus on what truly matters.

This ensures that the audits you commission deliver the insights and assurance you need to move forward with confidence.