AI agents are taking action, can boards keep them in check?

AI agents can do more than generate answers. They can access systems, execute tasks and take actions their operators may not anticipate. The Medicare AI incident has brought the governance challenge into focus: How can organisations ensure these systems stay within their authority? 

An OpenAI AI agent tasked with researching public medicine spending gained unauthorised access to non-public files on a Medicare statistics portal in June. The government said there was no evidence that individual Medicare details had been accessed, but the incident raised questions about how the agent had accessed restricted information.

The breach highlights the security challenges that emerge when AI moves from generating answers to taking action. 

Agentic AI introduces security risks, including prompt injection (LLM or AI agent tricked into ignoring original instructions and executing unauthorised commands), data poisoning and adversarial manipulation (data altered to trick AI model into making a wrong decision or output), requiring strong governance and human oversight, says a spokesperson for the Australian Signals Directorate (ASD).

The ASD recently released guidance advising organisations to restrict agents’ access to the data and tools they need, require human approval for sensitive actions and establish clear accountability for AI-driven outcomes.

“Organisations should also protect, retain and review records of the agent’s activity and approvals,” says the ASD. “These records support security monitoring, incident response, investigations and accountability.”

Dealing with an AI breach

Imagine an AI agent with access to an organisation’s systems identifies what it believes to be a vulnerability and starts acting on it.

The security team detects unusual activity. Who is alerted? Who can stop the agent? What evidence is captured, and who ultimately answers to the board?

“This scenario speaks to the broader importance of AI governance,” says Andrew Amos, Vice President, APAC at Diligent.

“AI agents should only operate within defined boundaries, with actions subject to appropriate authorisation and human oversight.” 

Organisations need clear processes for monitoring activity, escalating issues and maintaining audit trails, he says. Boards must have confidence that the right governance structures are in place to manage AI responsibly.

“Ultimately, AI doesn’t change who is accountable. People remain responsible for the decisions made and actions taken.”

The board-level test is straightforward, says Amos. Can the organisation articulate the system’s boundaries, name the executives accountable for its use and explain how a material issue would be identified and resolved?

Don’t assume there’s a fail-safe 

AI agents can take unexpected routes to achieve an assigned goal, making their actions and consequences difficult to predict, says Valeska Bloch, Partner and Head of Cyber at Allens.

Delegating execution does not remove management accountability or board oversight, she says. Organisations need to extend AI governance beyond model governance to rigorous agent governance.

That means naming an executive accountable for an agent’s use, explicitly defining its purpose and permissions, setting operational, network and data boundaries it cannot cross, and monitoring how it pursues its assigned tasks.

But even strong controls cannot guarantee that every consequence can be reversed.

“A kill switch can’t undo information already disclosed or every transaction already completed,” says Bloch.

The implication for boards is that preventing an incident is only part of the challenge. Organisations also need to understand what an agent has done, establish whether its actions can be reversed and determine who is responsible for responding.

That becomes more difficult when the records used to investigate an incident cannot be relied upon. What can be recorded, who can access the records and how reliably they capture an agent’s activity will vary across models, use cases and organisations, cautions Bloch.

“The challenge we’re facing is that the chain of thought that has historically been used to understand what has happened is becoming increasingly unreliable,” she says. 

“We also know that in some cases agents are removing or tampering with logs. That said, I expect regulators will increasingly expect tamper-proof logging where that’s possible.”

Bloch adds that effective AI governance must address both AI safety and cybersecurity.

“AI safety tries to make agents behave as intended, whereas cybersecurity assumes they won’t. Put another way, an instruction to ‘act safely’ is no substitute for technically enforced restrictions. Both need to be in place.”

Resilience starts with the basics

While agentic AI creates new governance challenges, established cybersecurity measures remain essential.

Rapid advances in frontier AI capabilities aren’t a reason for panic, says the ASD. Attack surface management, patching, defence in depth and a modern, defensible architecture remain highly effective.

Strong cyber resilience also depends on timely threat sharing, transparent incident reporting and organisations learning from incidents collectively rather than in isolation.

Organisations can report cyber incidents and suspicious activity through cyber.gov.au or by contacting the Australian Cyber Security Hotline on 1300 CYBER 1.

Agentic AI for good

Agentic AI is not solely a source of risk. It can also help organisations coordinate workflows, reduce administrative tasks and identify emerging issues, says Amos.

Diligent’s Diligent One platform, for example, uses specialised agents to help risk teams move from periodic assessments towards continuous risk management. Agents can monitor risk data, clean records, coordinate assessments, follow up on overdue actions, escalate issues and surface emerging risks.

“They can also help internal audit teams plan and execute audits faster by identifying relevant risks, mapping them to audit objectives, linking existing controls and supporting evidence collection, testing and findings development,” says Amos.

The potential value lies in connecting these activities across risk and audit workflows, giving teams a more current view of what is changing across the organisation and where attention is needed.

Diligent’s approach uses cited, traceable and overridable outputs, with approval gates for consequential actions. Agents can undertake work, but people retain authority over decisions that could materially affect the organisation.

“Anything that changes the organisation’s risk position, alters a record, commits the company or publishes an official report requires explicit human approval,” says Amos, adding that AI should remove the administrative burden around decisions, not the decisions themselves.