Automated Decisions, Human Consequences: Who Is Accountable When Technology Gets It Wrong
A plan gets generated. A payment gets flagged. A review gets triggered. All of it can happen without a single person actually reading the full story first. Automated systems are moving deeper into health, disability and government services, quietly doing work that used to sit with someone who could be asked a question and was expected to answer it.
I don’t think efficiency is the enemy here. But when something goes wrong and there’s no one left who can explain why, that’s not efficiency anymore. That’s just a decision nobody can reach.
Where this is already happening
This isn’t a future problem, it’s already here. Disability representative groups have been calling for transparency on computer-generated NDIS plans, and the reforms currently moving through parliament include new requirements around safeguards and how automated administration gets used. Plan drafting, funding calculations, payment checks, reassessment triggers, automation is turning up in all of it, often well before anyone’s told what’s actually happened.
How bias ends up built into a tool
A tool trained on past decisions carries the patterns of those decisions with it, including the ones that were never fair to begin with. Groups that were historically under-funded, needs that were historically waved off as less urgent, a system trained on that history can end up repeating the same pattern, just faster and dressed up as something neutral. “Automated” doesn’t mean unbiased. It usually just means the bias is harder to spot.
Explanations that actually mean something
A decision about a plan, a payment or a support isn’t accountable if the only explanation on offer is “that’s what the system generated.” A real explanation says what was considered, what wasn’t, and what would need to change for a different result. Anything less isn’t really an explanation. It’s a locked door with a sign on it.
Why human review still matters
A system can sort and calculate faster than any person ever could. What it can’t do is sit with the full context of someone’s actual life the way a person can. Human review can’t just be a box ticked after the automation’s already decided, it has to be a genuine second look, one that can still change the outcome if the outcome was wrong. Take that away and review is just theatre in front of a decision that was already made.
Who stays accountable when the technology fails
“The system generated it” can’t be where the conversation ends when something goes wrong. Someone built that tool. Someone chose what data it learned from. Someone decided it was fit to rely on. That’s a decision made by people, and the responsibility for it doesn’t vanish just because a computer handled the last step. Real accountability means there’s a person or organisation who can be named, questioned, and held to the outcome, not a system that quietly soaks up the blame and moves on.
The bit that matters most
Efficiency cannot become an excuse for decisions nobody can explain or challenge.
Technology can carry a lot of the workload. It should never be allowed to carry the accountability. When a decision affects someone’s life, there should always be a person behind it who can say why, and mean it.
Madeleine x