Deprovisioning User Access to Maintain Algorithm Integrity

The integrity of algorithmic systems goes beyond core algorithm accuracy and fairness.

In a previous article, we outlined key aspects of algorithm integrity.

Number 5 in that list (not in order of importance) is Security: the algorithmic system needs to be protected from unauthorised access, manipulation and exploitation.

In this article, we'll explore an important sub-component of this: deprovisioning user access.

 Listen to the audio (human) version of this article - Episode 11 of Algorithm Integrity Matters

Understanding deprovisioning

Revoking user access helps maintain security and, by extension, algorithm integrity.

When users retain access that they no longer need or should no longer have, it can lead to unwanted activity, including unauthorised transactions, access to confidential data, IP theft – e.g., access to confidential models, and exploitation by malicious actors – e.g., denial of service attacks.

We'll focus on two triggers for deprovisioning - termination (e.g., of employment) and dormant accounts.

Other triggers exist (like changes in job roles), but termination and dormancy are typical, and fairly risky.

Here are two examples that demonstrate this:

1. External attacks: the U.S. National Coordinator for Critical Infrastructure Security and Resilience (CISA) released this advisory recently. 
   It explains that malicious actors have targeted dormant/terminated accounts.
2. Internal attacks: are a little bit different. 
   What we often see is that the access is by an existing employee, using a password left behind. 
   The result: unauthorised changes to algorithmic parameters or data inputs, or privacy breaches. 
   Internal attacks are not often exposed, for several reasons. They’re also sometimes harder to detect because they come from within the trusted environment. And they are internal “attacks”, as harsh as that may sound. The activity could be considered fraudulent.

Back to the two triggers, and what we can do to reduce the risk.

Termination

When users leave, retaining their access can lead to serious security risks that directly reduce integrity.

• Unauthorised activity: if accounts assigned to former employees are retained, ther accounts might be used to initiate unauthorised activity. A former employee is usually not accountable for such actions after they leave. It's the organisation's responsibility to remove their access. 
• Bypassing segregation of duties (SoD) controls: where there are maker-checker controls, if a current employee has access to a former employee's account, they could log in as themselves to initiate a transaction and then use the other account to approve it. Or, for algorithm change control, they could use one account to make a code change and use the other account to migrate that change to production. Those controls exist to prevent fraud, and also errors - so an errant line of code could go unnoticed, resulting in reduced algorithmic integrity. 
• Data breaches or IP theft: accounts belonging to former employees might be used to view or extract sensitive data or models, resulting in confidentiality breaches. This can include customer information, or proprietary models.

The solution is simple:

1. Revoke access when an employee or contractor (including vendor personnel) leaves
2. Disable user access on or before the date of termination
3. Set up automated deprovisioning processes to revoke access on time
4. Conduct regular reviews to verify that terminated users no longer have access to systems.

In practice, this can take some effort, but the benefits generally outweigh the costs.

Dormant Accounts

Accounts that aren't accessed regularly are risky.

They’re prime for exploitation, as they're often not actively monitored.

This could lead to similar risks to those outlined for termination earlier.

For dormant accounts:

1. Determine what a reasonable period of inactivity is - for example, if people typically log in once a month, perhaps a threshold of 90 days
2. Check account usage regularly, to identify dormant accounts (accounts that exceed our threshold of inactivity)
3. Deactivate accounts that are no longer needed
4. Consider strengthening other known dormant accounts – for example, if certain service accounts are used less frequently, but need to be active, implement mitigating controls like more complex passwords and actively monitor their usage
5. Consider automated tools to identify and deactivate dormant accounts.

Additional considerations

If automated deprovisioning is not possible or not feasible, and you use a manual process, exercise extra caution, and emphasise regular reviews.

For automated deprovisioning, perform regular checks to make sure the automation functions correctly. These things can, and do, break.

When using single sign-on (SSO) systems – where you login once and don’t have to enter passwords again for other applications – check for termination and dormancy at the application level too. For example, if you use Active Directory or Google Identity Services, it is tempting to disable access there and assume that all is well; because that controls authentication everywhere. This can be risky. Applications sometimes also allow direct (non-SSO) logins. Employees return, sometimes to different roles, and don’t need the access they had before – reactivating their SSO credentials automatically gives them their previous access. It is safer to disable/revoke access everywhere.

Consider additional measures when systems, processes or behaviours change. This includes changes like:

• If you implement multifactor authentication (MFA), and users are forced to enable MFA when they next log in, dormant accounts will remain unprotected. Have a plan for dealing with these.
• If you automate processes, and login activity is more frequent, perhaps the dormancy threshold should be reduced. For example, if timesheets were manual and are now processed through the HR/payroll system, employees may need to login fortnightly. If you were using 90 days as the threshold, maybe reduce it to 30 days.

Wrapping Up

Proper deprovisioning of user access contributes to algorithm integrity. We need to make sure that only current and authorised users have access.

Getting this right is not difficult. It does take some effort, and the effort needs to be sustained. But without it, we risk losing integrity.

Disclaimer: The information in this article does not constitute legal advice. It may not be relevant to your circumstances. It may not be appropriate for high-risk use cases (e.g., as outlined in The Artificial Intelligence Act - Regulation (EU) 2024/1689, a.k.a. the EU AI Act). It was written for consideration in certain algorithmic contexts within banks and insurance companies, may not apply to other contexts, and may not be relevant to other types of organisations.