25 October 2026 · Conversational AI

Audit Logs for Chatbots: What to Record and How to Test

What a chatbot audit log should capture, how to keep it privacy-safe under Australian rules, and the test cases that prove it works before an incident does.

A chatbot audit log is a time-stamped, access-controlled record of who or what did what, to which data, with what outcome. Saving chat transcripts is not enough. A useful log also captures tool calls, data access, configuration changes, errors and exports, and it masks personal information by default. Then you test it, so that it answers a regulator’s or customer’s question when it matters.

This guide covers what to record, how to design logs for Australian privacy obligations, how to retrieve logs from the platforms you already use, and a set of test cases you can run before go-live.

TL;DR

Log three layers: what the bot received, what it decided or did (including tool calls), and what it said back. Transcripts alone cover only the last.
Record administrative activity too: role changes, prompt edits, exports and configuration changes.
Minimise and mask. Under Australian privacy guidance, personal information in chatbot inputs and outputs is still personal information, so your log is part of your privacy risk.
Retention and visibility are product-specific. Confirm limits, delays and omissions with each vendor.
Test the log like a feature: write test cases, run them, and check each event appears with the right fields, the right masking and the right access.

Why a transcript is not an audit trail
What a chatbot audit-log entry should contain
What to log, and what to leave out
The trade-off: content versus accountability
Designing for Australian privacy obligations
How to get audit logs from the platforms you already use
Test cases for chatbot audit logging
Our perspective
See it working in your environment
FAQ
Sources

Why a transcript is not an audit trail

Most teams start by keeping chat histories. That answers “what did the customer see?” It does not answer the questions an investigator, auditor or complaints officer will ask:

Which CRM record did the agent read before answering?
Which policy rule allowed or blocked an action?
Who changed the system prompt last Tuesday?
Did a staff member export this conversation, and when?

Guidance for engineers describes the goal as moving from guessing what an agent was “thinking” to holding a verifiable record of each decision. We agree, and it is why we separate the record into layers rather than one big text field.

There are two kinds of event to capture:

Conversation and agent events: messages, tool calls, tool results, data accessed, errors.
Administrative events: logins, role and permission changes, configuration and prompt edits, exports, deletions.

A log that covers only the first cannot tell you whether a bad answer came from the model or from someone editing the prompt.

What a chatbot audit-log entry should contain

A practical entry includes a unique event ID, the actor (human, bot or service), the action, the affected entity and its ID, a timestamp, the outcome, and a link to a trace. One vendor’s audit console documentation shows this pattern in practice: it records user messages, assistant responses, tool calls, tool results, PDF exports and error events.

For an AI agent, extend the basic entry with the user message (or a redacted hash of it), the assistant response, the tool call and its result, the data accessed, any error, model or provider metadata, and any export event.

Here is an illustrative entry for an agent that looks up a case in a CRM:

Field	Example value
event_id	evt_000482
actor	agent:receptionist-01
action	TOOL_CALL
entity	CRM_CASE / 10293
timestamp_utc	2026-10-23T02:14:09Z
trace_id	tr_77af
tool	crm.lookup_case
redacted_arguments	{name: “[MASKED]”, case: “10293”}
policy_decision	allowed
result	success
external_reference	appointment ref only

Three design choices matter more than the field list:

Use UTC and one clock. Voice, SMS, email and chat events from the same customer need to sort into a single timeline.
Link everything by trace ID. One customer interaction may produce a call, a CRM read, an SMS confirmation and a follow-up email. The trace ID turns four logs into one story.
Log the policy decision, not just the action. “Blocked: identity not verified” is the most valuable line in an investigation.

What to log, and what to leave out

Recording everything is not the safe option. The audit log can itself become a store of sensitive data that needs protecting.

The OAIC’s guidance on commercially available AI products applies to personal information entered into AI systems and to personal information the systems generate or infer. It also says organisations should consider whether using personal information with an AI system is necessary. Apply the same thinking to logs.

Event type	Log it?	Handling
Authentication and role changes	Yes	Full detail, long retention
Prompt, variable and model setting edits	Yes	Who, when, before and after
Tool calls and results	Yes	Mask arguments and results by default
Data accessed (record type and ID)	Yes	Reference the record, don’t copy it
User message and bot response	Where lawful and needed	Redact or hash; restrict access
Errors and failed policy checks	Yes	Include the reason code
Exports and deletions	Yes	Actor, scope, destination
Health, financial or identity details in free text	Avoid storing in clear	Redact at ingestion

Masking by default is a control worth copying. The audit console documentation above describes masking of messages, tool arguments and results, IP address and user agent by default, so administrators can investigate without every operator seeing raw customer content. If your platform cannot do this, treat the log as sensitive data and restrict it accordingly. Our guide to keeping PII onshore with redaction goes deeper on the mechanics.

The trade-off: content versus accountability

Some logs deliberately leave out prompt and response text. Microsoft’s documentation for its Sales agent audit records which user interacted, when, and which CRM data was accessed, but does not include prompts or full generated responses. That reduces exposure of sensitive content. It also means the log alone may not settle a dispute about what the bot actually said.

The answer is to decide on purpose. If you need to resolve complaints, keep a conversation record under tight access control and a defined retention period, and keep the audit log of access and actions separately. If you cannot justify keeping content, rely on references and accept the limits. Either way, write the decision down.

The trade-off: content versus accountability — overview diagram

Designing for Australian privacy obligations

Audit logging supports privacy governance. It does not replace it. The OAIC guidance recommends:

product due diligence before adoption;
transparency that a public-facing chatbot is an AI;
human oversight built into the process;
regular monitoring of performance;
policies covering permitted and prohibited uses.

Each of those produces something you can log. Disclosure shown to the customer can be recorded as an event. Escalations to a human are events. Monitoring reviews can be signed off against log extracts. The OAIC also says the work continues across the life of the system, so a log you review once at launch falls short.

Two practical consequences for IT and operations managers:

Where the log lives matters. A log containing personal information that is stored or processed offshore creates the same exposure as the chatbot itself. Our guide to preventing offshore data transfers covers the contract and architecture questions.
Retention needs a reason. Decide how long each event class is kept, and why. Our 30-day chatbot retention plan sets out an OAIC and NAA-aligned approach.

For the lifecycle side, a 90 to 180 day AI governance framework shows how logging fits into ownership, review and escalation. For why auditing decisions matters in the first place, see why audit AI system decisions.

How to get audit logs from the platforms you already use

The general process is the same across products:

Define the scope. Configuration changes, conversations, tool execution, data access or security events. Each may live in a different source.
Assign least-privilege roles. Not everyone who runs the bot should read its audit trail.
Enable or verify the audit source. Don’t assume it is on.
Filter by actor, action, entity and time.
Export or forward to your approved SIEM, so the records outlive the platform’s own retention.

Take Microsoft Purview as an example. You assign an Audit Manager or Audit Reader role, open Audit and Search, select the activity, run the query and export the results. Copilot and AI-app activity is logged automatically once auditing is enabled.

Other platforms differ. In a Discord server, viewing the audit log needs a specific permission, so not everyone can see it, and entries are stored for 45 days. For a command-driven chat integration, CloudWatch logging for Amazon Q Developer in chat applications shows that command-execution audit events are always enabled while broader event logging is optional.

The lesson: defaults vary. Find out what is on, what is optional and who can see it.

Delay and retention are real constraints

For the Sales agent, records can take 30 minutes to 2 hours to appear, and standard retention is 180 days, per Microsoft’s documentation. Audit Premium can provide up to one year for appropriately licensed users, and retention policies may extend selected records, according to the licensing guidance. These figures depend on service, licence and policy. They are not universal.

So a log that appears two hours late is no use for real-time fraud detection. And a 180-day window is no use for a complaint that arrives in month eight. Match the product to your investigation and regulatory timelines, not the reverse.

Test cases for chatbot audit logging

An audit log you have never tested is an assumption. Treat it as a feature with acceptance criteria. Run these before go-live and again after every significant change to the bot.

Completeness tests

Happy path: run a normal conversation that triggers one tool call. Check that the message, tool call, tool result and outcome all appear under one trace ID.
Failure path: force a tool error (for example, a CRM timeout). Check that the error, the reason and the bot’s fallback response are recorded.
Escalation: trigger a handover to a human. Check the reason, the time and the receiving team or person are recorded.
Channel coverage: repeat across voice, SMS, email and chat. Check each channel writes the same core fields.

Policy and safety tests

Blocked action: ask the bot to do something it should refuse, such as disclosing another customer’s details. Check the log shows the policy decision, not just silence.
Unverified identity: try to access account information without verification. Check the denial is logged.
Disclosure: confirm the AI disclosure event is recorded at the start of public-facing interactions.

Privacy and masking tests

Masking: enter a name, phone number, address and card number. Check that logs, tool arguments and results show them masked.
Access control: sign in with a low-privilege role. Confirm you cannot read raw content or change logs.
Deletion and export: request an export and a deletion. Check both are logged with actor and scope.

Integrity and operations tests

Admin changes: edit the system prompt, then check who, when, and before and after values are logged.
Clock and ordering: check events from different channels sort correctly in UTC.
Forwarding: confirm records reach your SIEM and note the delay.
Retention: confirm older records are purged or retained as the policy says, and that the purge itself is logged.

Keep the results. The test record is evidence that you monitor the system, which is what the OAIC’s guidance on regular monitoring asks of you.

Our perspective

We build conversational AI for Australian organisations in regulated sectors, and the same pattern repeats in procurement. Buyers ask whether the bot is accurate. Their risk and privacy teams ask whether they can prove what it did.

We think the second question should lead. A voice agent that books an appointment, a chat agent that reads a CRM record and an SMS agent that sends a follow-up each act on behalf of your organisation. Each action needs an owner and a record.

Three convictions shape how we approach it:

Audit trails belong where the data lives. Our platform is hosted entirely in Australia, and our positioning is that data residency and control are not negotiable for healthcare, finance and professional services. A log held offshore undoes that.
The CRM and the agent should share one story. Because our agents integrate natively with CRM systems, the interaction and the record it touched can be traced together rather than reconciled by hand.
Logging is never finished. Agents change, prompts change, regulations change. Reporting and analytics should feed review, not sit in a drawer.

If a vendor cannot tell you what is logged, for how long, who can read it and where it is stored, treat that as your answer.

— Sowrabh

See it working in your environment

If you are evaluating voice, SMS, email or live chat agents and want to see how interactions, CRM actions and reporting fit together under Australian hosting, explore the Conversational AI platform. For regulated teams, our healthcare and banking and finance pages show how we approach those sectors, and our security playbook sets out a nine-step approach to chatbot security.

FAQ

What should a chatbot audit log include?

At minimum: a unique event ID, actor, action, affected entity, timestamp, outcome and trace ID. For AI agents, add tool calls and results, data accessed, errors, model metadata and export events. Mask personal information by default.

Can anyone see audit logs?

They should not be open to everyone. Platforms gate access by permission; in a Discord server, for example, the View Audit Log permission is required. Apply least privilege to your own chatbot logs, and log who accessed them.

Does Copilot have audit logs?

Yes. Microsoft says Copilot and AI-app activities are logged automatically when auditing is enabled. Check what each record includes, because some do not contain the prompt text or full response.

How long should we keep chatbot audit logs?

There is no single answer. Set retention by event class, tested against your investigation needs and regulatory obligations, and keep personal information only as long as you have a reason to. Vendor defaults, such as 180 days for standard Purview audit, are starting points, not policy.

How do we test chatbot audit logging?

Write test cases covering completeness, policy decisions, masking, access control and retention, then run them before launch and after every major change. Keep the results as evidence of ongoing monitoring.