Sometimes the error is logic, not system

System error is a convenient label. It makes it sound accidental, like a server overheated or a cable came loose, or some rogue code appeared out of nowhere.

But the problem often isn’t hardware, or even software. It's business logic, or business rules that are no longer understood, or perfect rules that simply got lost in translation to code.

Three examples

These aren't just coding defects. They show what happens when business logic drifts.

1. Late last year, The Australian Securities and Investment Commission (a regulator) admitted to a fee calculation error that went unnoticed for 14 years. It affected nearly two million entities. The cause wasn't a "bug" in the traditional sense;  it came from how indexation was interpreted and implemented after a rule change.

2. Then there is the Income Security Integrated System, a 40-year-old mainframe system used for welfare payments in Australia. A recent federal audit found an error that overpaid/underpaid pensioners by $5 billion over 3 years. The report revealed that staff were "trained to trust system calculations" and had "limited training on conducting manual calculations". Because the system was so old and complex, the agency admitted it didn't even have documentation for some of the business rules anymore. The report traced the issue to a mix of system configuration, business rules and administrative practice, rather than a single coding defect.

3. In the UK, major insurer Direct Line was reported to have overcharged customers, resulting in refunds of approximately £30m. When the regulator introduced new rules banning loyalty penalties (charging existing customers more than new ones), the pricing algorithm didn't fully align with the new requirement. The insurer was then stuck with a large remediation bill because of a system that hadn't kept up with the rules outside it.

Of course, in systems this big, incorrect payments rarely have a single cause. Disclosure issues, process gaps and policy interpretation all play a part. But, in each example, the system logic was at least partly out of step with the business rules it was meant to apply.

Why this keeps happening

There are many reasons for these failures. Among them are:

1. Blind trust

Errors can go undetected when we don’t question the machine. When a claims officer sees a number on the screen, they assume it’s right because "the system calculates it." We have effectively removed the human check, and replaced it with passive trust.

2. Complexity

Legacy systems can have years of rule changes layered on top of each other (40 years in the case of the welfare system). When we patch a patch of a patch, we lose sight of the original logic, and the code becomes the “truth”.

3. Inadequate testing

We test for what should happen when everything goes right. Edge cases, like what happens to indexation logic when it runs for 14 years straight without a reset or check, can easily slip through. Or project delays squeeze the testing window, forcing us to rush through QA to meet the deadline. We've all seen this scenario before: teams schedule their UAT work, but then the system isn't ready for them in time; they have other work to do, so then have to somehow fit it in; can we really blame them for any quality slippage?

How we can fix it

We don't always need a new IT system to fix accuracy. But we generally need better controls.

Not every rounding difference or edge case justifies a big fix. But when they're material, or disadvantage customers, they become a problem. These controls are aimed at that kind of high‑impact logic, not every line of code.

1. Implement human checks

If we can't calculate a commission or an interest payment manually, we have a problem. At minimum, we need to be able to manually verify at least some realistic examples of our key payments.

A Fix: Pick a sample of accounts, take the raw inputs, and ask a human to calculate the result using the policy document (not the system). If they can't match the system's number, we have a logic gap.

Another approach involves some automation to recalculate all the results. We determine the rules from the underlying PDS or contract document (with some internal tweaks, as needed). We then compare the recalculated results to the actuals, and investigate the discrepancies.

2. Document the rules

The welfare entity has now agreed to document the business rules for its aged pension calculations. It sounds basic, but many banks and insurers are in the same boat; running code where the original documentation is lost.

A Fix: Reverse-engineer our critical calculations. Write them down in plain English. If the logic is too complex to write down, maybe it’s too complex to be safe, and we need to find a way to make it simpler. 

3. Test for integrity, not just function

In a previous article, we said that testing is often treated as a tick-box exercise. To catch errors, we need to test for accuracy specifically.

A Fix: Don't just check if the system runs. Check if the output is valid against the intent. Test the weird dates, the complex scenarios, the 14-year timelines, etc.

If we can't explain how the number was calculated, we shouldn't be paying it.

Disclaimer: The info in this article is not legal advice. It may not be relevant to your circumstances. It was written for specific contexts within banks and insurers, may not apply to other contexts, and may not be relevant to other types of organisations.