Medicare's AI breach reads like a governance failure

In brief

• In June, an AI agent got into a Services Australia portal, and nobody told the government for nearly three months. It reached aggregate health statistics and internal file names, not patient records.
• Nobody can yet say what Australia's AI bill will demand. The work looks much the same under every model: know where you use AI, and be able to show that you checked it.
• Board oversight of AI, and truthful claims about what your AI does, are already expected today. From December, your privacy policy must also say where software uses personal information to make, or substantially shape, decisions that could significantly affect someone's rights or interests.
• We do not know if a new AI regulator will come. For now, the people who will ask are the regulators, buyers and auditors you already deal with.
• Start with a named owner and one list of where you use AI. Take both to the next board meeting.

The wrong question

Last month I attended a round table on Australian AI regulation for the infrastructure sector. An expert on European AI regulation made the case for the EU's AI Act, complexity and all. At one point, the discussion turned to how light a touch of it Australia would end up adopting. As one participant put it, "Australia has had a history of implementing soft-law versions of regulatory frameworks from other jurisdictions".

In June, an AI agent got into a Services Australia portal. The government did not find out for nearly three months. The Prime Minister called it the first known case of AI hacking a government system. No patient records were accessed. The agent reached only aggregate health statistics and internal file names. OpenAI's own account is that its models "took action we did not intend".

Read as a governance report, it shows:

1. A system did something unplanned, even unauthorised.
2. The portal's owner did not detect it.
3. The portal's owner heard about it from someone else, by email.

That's three control failures, and not one of them turns on which law applies. Which regulation Australia borrows matters to policymakers. For businesses, it can wait. Whichever model Australia copies, the work is the same.

Sooner or later, a board member will ask what your position on AI is, if they have not already. The honest answer, in many organisations, is a list still sitting in the to-do pile.

Australian AI regulation is still being shaped

Countries have split three ways on regulating AI: one big AI law, rules by risk level, or the regulators they already have. None of them have really settled. The EU is pushing its hardest rules back. Britain dropped its AI bill from this year's programme. Japan passed a law with no penalties in it.

Australia, too, changed its mind twice in a year. First it dropped its planned rules for high-risk AI. Then it announced national AI standards and a new central office. A bill is promised for early next year, with little detail so far.

It's a moving target. Whichever way it lands, three steps will hold for businesses that want to start early.

Three steps that hold under any law

1. Know what you have

   Give AI an owner, a named person rather than a committee. Then keep one list of where you use AI. Include the AI inside the products you bought. That is also where many organisations find they are further along than they thought.

2. Check each new use of AI before you start using it, and say what it decides

   Write down the risk check: what could go wrong, who it would affect, and what you would do about it. Note which decisions your software makes about people (shortlisting job applicants, approving credit, assessing a claim, etc), and give those people one way to object.

   From 10 December, the privacy rules make part of this public. The duty applies when software uses personal information to make, or substantially shape, a decision that could significantly affect someone's rights or interests. Your privacy policy must then name the kinds of decisions and the kinds of personal information used.

3. Put the same expectations in your supplier contracts

   Ask suppliers what they do when their own systems misbehave, and how quickly they must notify you. Services Australia had no contract with OpenAI, so it had no say in when it heard. It found out nearly three months later, by email. With your own suppliers, you do have a say. Write the notice period in, through a policy annexed to the contract if that is easier, and decide who handles that report.

The above is not new advice; it is shaped from Australia's ten voluntary guardrails, the international standard for managing AI, the American risk framework the suppliers quote, and the directors' guide from the AICD. They agree because they address the same problems.

Who will ask you for it

We do not know if an AI regulator will come. The central office coordinates the work across ministers and has no powers of its own. The national AI safety institute advises rather than compels.

The Medicare breach is the first real test for both. The taskforce runs out of the Prime Minister's department, pulling in the signals directorate, the safety institute and the Office of AI. A body with no powers is likely to have limited leverage over a foreign AI company. Your leverage over your own suppliers is the contract, which is why step three above matters.

The people who will ask are the ones you already deal with: the privacy regulator, the consumer regulator, the online safety regulator, your buyers and your own auditors. The online safety regulator is already acting on AI products, without waiting for an AI law. The privacy regulator has done the same. Bunnings was allowed to use facial recognition in its stores, and still lost for not telling customers and not writing down a risk check.

Not all of them are being resourced for the work. The privacy regulator takes on the new disclosure duty with a smaller budget than last year. That cuts both ways - less capacity means fewer knocks at the door. It also means the first case in your sector is likely to set the expectation for everyone else.

The dates below show what applies, and when.

Is that enough?

Is the approach too weak, or does it overreach? Both arguments are live, and neither changes the work in front of an organisation. Three things would change it: the parliamentary committee's report, the bill early next year, and whatever the Medicare taskforce recommends. An incident like this one has the potential to move timelines, and the government has already changed course twice in a year.

Until then, the first step is small, and nobody needs a law to take it. If your AI governance is at an early stage, name an owner and ask them to start the list. Take both to the next board meeting.

The lesson from June is simple. An unwatched system can do things nobody authorised. An organisation that cannot see its own AI, its operational limits and its risks may find out from somebody else. In this case - that took nearly three months.

This is the work we do. Our services page sets out how we run it.