17 October 2026 · Conversational AI

Chatbot Load Testing for Developers: Find QPS Without Model Jitter

Test full chatbot journeys, mock the model to find sustainable QPS, then use OpenTelemetry to track TTFT, latency, throughput, and failures.

Effective chatbot load testing simulates complete multi-turn journeys on the same transport your production agents use, and separates infrastructure capacity tests (mocked model) from end-to-end model latency tests. Define the journeys first, pick a tool that matches your protocol, ramp to saturation, then record time to first token, percentile latencies, throughput and failures. Interpret those numbers against acceptance criteria you can rerun after every change.

TL;DR:

Give each critical journey its own scenario, including authentication, payments, handoffs, and error paths; vary utterances and pauses across conversations rather than testing single prompts.

Choose JMeter for WebSocket flows, Locust for Python streamed response parsing, or Azure Load Testing for managed JMeter runs; match production transport and refresh tokens.

Mock the model to find infrastructure capacity without token costs or model jitter, then run a smaller live test to measure perceived latency and cost.

Warm up, increase concurrency in documented steps, and stop when throughput plateaus or failures spike; check provider quotas first, since throttling can mimic capacity limits.

Track first token time, p50 and p95 latency, throughput, and failures; link traces to conversation turns to locate delays in retrieval, model, or backend calls.

Which conversational journeys should you test?
Which tool should you use, and when should you mock the model?
How do you ramp, warm up and find saturation?
Which metrics should you capture?
What do concrete test runs look like in JMeter, Locust and Azure Load Testing?
How do you read results and plan capacity?
How do you test voice, SMS, email and live chat together?
What security and compliance rules apply to load tests?
How should you load test private cloud and on-premises deployments?
How do you simulate failover and recovery under load?
What we think most teams get wrong
Skip the stitching: a platform built for enterprise chatbots
FAQ
Sources

Which conversational journeys should you test?

A bot that behaves in development can still fall over under real traffic. Microsoft’s guidance points out that concurrency exposes queues, provider quotas, downstream dependencies and long-running branches that functional tests rarely touch. That is why scenario design comes before tool choice.

Start from the flows the business cannot afford to see fail, and give each its own scenario or thread group so one slow journey cannot hide behind a healthy average.

Business-critical flows: authentication, account lookup, booking, payment and escalation each become a separate scenario.
Full multi-turn conversations: drive scripted or CSV-fed input through the whole dialogue, because later turns trigger retrieval, API calls and handoffs that single-prompt tests never reach.
Think time and variation: vary utterances and pause between turns the way a person would, rather than firing the next message instantly.
Error paths: include aborted conversations, invalid inputs and abandoned sessions, since real users produce them constantly.

Single-prompt benchmarks answer a narrow question: how fast does the model reply once? Your users ask a different one: does the whole conversation hold up when many others are talking too?

Which tool should you use, and when should you mock the model?

Tool choice follows protocol fidelity and scripting needs, not habit.

Apache JMeter: suits multi-thread tests across several flows, WebSocket or Direct Line connections and CSV-driven utterances.
Locust: suits teams who prefer Python user classes and need custom parsing of streaming responses.
Azure Load Testing: suits teams who want managed scale and the ability to run JMeter scripts across multiple engines.

Mocking is the other big decision. Databricks recommends mocking the LLM when you want to measure infrastructure capacity, then running a separate, smaller end-to-end test with the real model to capture user-visible latency and cost. The mocked run finds your throughput ceiling without paying for tokens or inheriting model jitter. The real-model run tells you what a person actually feels.

Whichever tool you pick, the client has to reproduce production authentication and token refresh, or tokens will expire mid-run and pollute your failure counts.

Pro Tip: Run the mocked infrastructure test first. If your backend saturates at a low user count with the model removed, no amount of model tuning will fix it.

How do you ramp, warm up and find saturation?

A single large burst tells you little. A stepped ramp tells you where sustainable capacity ends. The same Databricks guidance describes the pattern: warm up, raise users in levels, hold each level, and stop when successful throughput plateaus or failures and latency climb sharply.

Microsoft also notes that the load generator must match production transport and pacing. If production streams over WebSockets or server-sent events, a test that only sends synchronous HTTP changes the workload, and the results stop describing your system.

Build the ramp like this:

Warm up with a small, steady load so caches, connection pools and autoscalers settle before you measure anything.
Increase concurrent users in fixed steps, and document the spawn rate, step duration and maximum user count.
Hold each step long enough for throughput to stabilise before moving on.
Add healthchecks, and avoid automatic retries that inflate request counts and mask real failures.
Stop when queries per second flatten or the failure rate jumps, then record the user count at that point.

Token lifetime matters here. Short tests can often run on user credentials, but longer soak runs need machine-to-machine service principals so authentication does not expire halfway through. Check provider quotas and rate limits before you start, since hitting a quota looks like a capacity problem when it is a configuration one.

Databricks guidance: a stepped ramp identifies sustainable capacity more reliably than one large burst, because you see exactly where throughput stops climbing.

Rising test demand meets a throughput plateau

For headroom targets, capacity planning principles for small teams are a useful reference when deciding how close to the ceiling you are willing to run in production.

Which metrics should you capture?

HTTP timing alone cannot tell you why a conversation was slow. OpenTelemetry’s guidance explains that request duration folds together network time and model generation, so you need traces as well as metrics to see the sequence of retrieval, model and tool operations.

Collect these on every run:

Time to first token (TTFT): record it separately from total completion time, because streaming makes the first token the moment a user perceives a reply.
Latency percentiles: p50 and p95, never only the mean.
Throughput: queries per second at each ramp step.
Failures: error and rate-limit rates, plus retries and queueing.
Downstream behaviour: API and database latency, and spans for retrieval and tool calls.
Cost drivers: token usage and cost counters, which expose load growth that CPU dashboards cannot.
System health: CPU, memory and connection pool saturation alongside the application numbers.

Request duration includes network and model generation, so traces are needed to reveal the sequence of retrieval, model and tool operations.

With traces linked to each conversation turn, a slow reply stops being a mystery. You can see whether the model, the retrieval step or a backend call ate the time, and fix the right thing.

What do concrete test runs look like in JMeter, Locust and Azure Load Testing?

Each tool has a repeatable starting recipe. Treat these as skeletons to adapt to your own flows, not finished scripts.

JMeter. The Copilot Studio multi-thread sample shows the shape: one thread group per conversational flow, multi-turn utterances read from CSV, and WebSocket or Direct Line connections to match production. Add samplers for fetching a token, starting the conversation, sending an utterance and reading the bot response, so each step reports its own timing.

Locust. A Locust example for chat apps uses a ChatUser class that simulates a person asking and receiving answers. Extend it to parse streamed responses and timestamp the first token, then open the Locust web interface and set the user count and spawn rate.

Azure Load Testing. The LLMOps workshop notes that LLM app load tests can run on Azure Load Testing with JMeter scripts, scale across engines, support monitoring and fit into CI/CD.

A repeatable workflow looks like this:

Script one journey per flow and verify it passes with a single user.
Run the mocked-model ramp to saturation and export the dashboards.
Repeat at a smaller scale with the real model to capture TTFT and cost.
Check provider quotas and retry settings so throttling does not skew latency numbers.
Store results with the code revision so later runs have a baseline.

How do you read results and plan capacity?

Look for the saturation point, where throughput flattens even though you keep adding users. Record the user count and peak QPS at that point. That pair is your sustainable capacity boundary, and the number you plan production headroom against.

Then use the tail to decide what to fix:

p95 and p99 rising early: suspect downstream calls or queueing before blaming the model.
Failures climbing at a fixed user count: look at worker counts, connection limits and provider quotas.
Flat throughput with idle CPU: a lock, pool or rate limit is capping you, not hardware.

Our guide to configuring AI agent scalability settings covers the worker and configuration knobs that most often move that ceiling. If your results show a vertical limit instead, scaling VPS resources against metrics is a practical read on provisioning options.

Result	What it tells you	Next step
QPS plateaus as users rise	You have hit the capacity ceiling	Record users-at-peak and peak QPS
p95 climbs while p50 holds	Tail latency from a slow dependency	Inspect downstream spans
Failure rate jumps at one step	A quota, limit or crash threshold	Check limits, then rerun

Turn the findings into acceptance criteria for CI, for example a ceiling on p95 latency and an error rate budget, and rerun after every meaningful code or configuration change.

How do you test voice, SMS, email and live chat together?

Most enterprise deployments are not one channel. A customer might chat on the website, receive an SMS follow-up and take a callback, all drawing on the same CRM, model gateway and queueing layer. Testing each channel alone hides the contention between them.

Build a mixed workload that reflects your real channel split. Run the live chat scenarios on their production transport, add inbound SMS and email events at realistic rates, and include concurrent voice sessions where your platform handles them. Voice behaves differently from text because audio streams are continuous and latency-sensitive, a difference we explain in our piece on how voice and chat AI differ.

Watch the shared resources, not just each channel’s own numbers:

CRM and database connection pools used by every channel.
Model gateway quotas consumed by combined traffic.
Outbound queues for SMS and email that can back up silently.
Telephony or media capacity limits for voice.

Compare each channel’s latency in the mixed run against its solo baseline. If live chat p95 degrades only when outbound SMS campaigns run, you have found a shared-queue problem that no single-channel test would reveal. Vary the mix too: a campaign spike in SMS replies is a different stress from a surge of web chats.

What security and compliance rules apply to load tests?

Load tests create many realistic-looking conversations, and in private or sensitive environments that is itself a risk. The safest default is synthetic data: generated names, account numbers and messages that never map to a real person.

If a test must touch production-like systems, treat the test data with the same controls as live data. Restrict who can run tests and see the results, and keep logs and traces from leaking real personal information into dashboards.

Practical controls to put in place:

Synthetic identities: generate test customers and avoid copying production records.
Credential hygiene: use dedicated service principals with narrow permissions and rotate them after runs.
Log scrubbing: mask personal data in transcripts, traces and exported reports.
Isolation: run against a staging or dedicated environment so test traffic cannot reach real customers.
Outbound safeguards: disable real SMS, email and voice delivery, or route to sandbox endpoints, so a test never messages a live person.
Retention: delete transcripts and result artefacts once analysis is done.

For regulated sectors such as healthcare and finance, agree the test plan with your security and compliance owners before the first run. Our deployment thinking in the enterprise-scale AI deployment guide covers the production patterns these controls sit inside.

How should you load test private cloud and on-premises deployments?

When the platform runs in a private cloud or on your own hardware, the test has to respect where data lives. Keep load generators, the system under test and any captured results inside the approved environment, so test traffic and transcripts never cross a boundary your sovereignty requirements forbid.

For Australian data residency, that usually means running generators from within Australian infrastructure and storing results in Australian-hosted locations. Check that managed testing services you consider can run engines in the regions you need before committing, and confirm where their dashboards and exports are stored.

Other practices that matter on private deployments:

Size the generators honestly: a saturated load generator reports its own limits, not your system’s. Monitor generator CPU and network.
Test the real network path: firewalls, proxies and internal gateways add latency a public-cloud test would miss.
Mock external model calls when egress is restricted: this keeps infrastructure tests inside the boundary, then validate the real model path separately where policy allows.
Capture hardware metrics: with fixed capacity, CPU, memory, disk and network headroom are your scaling limits, so record them every step.

If you are weighing hosting options, our on-premises deployment page describes how we support private hosting for teams with strict residency needs.

How do you simulate failover and recovery under load?

A system that survives steady load can still fail badly when a component drops out. Resilience tests inject failures while traffic is flowing, because recovery under pressure behaves very differently from recovery at idle.

Hold the system at a safe, known load, such as a step below saturation, then break something deliberately and watch what users experience.

Failure scenarios worth running:

Worker or node loss: stop an application instance and check whether in-flight conversations recover or restart.
Dependency slowdown: add latency to a CRM or database call and confirm timeouts and fallbacks behave.
Model provider errors: return rate-limit and error responses to see whether retries amplify the problem.
Failover to a standby: trigger the switch and measure how long errors persist.

Record three things for each: the error rate during the failure, the time to recover, and whether any conversations lost state. Look for retry storms in particular, where clients retrying en masse turn a brief blip into a lasting outage.

Run these after you have established a clean baseline, so you can attribute every change to the fault you injected and not to underlying instability.

What we think most teams get wrong

Most teams test the model and ignore the conversation. They benchmark one prompt, publish a latency number and ship. That figure describes a playground, not a deployment, and it is why bots that looked fast in review stall in production.

The advice worth following is less glamorous. Run multi-turn journeys, keep transport faithful, mock the model for capacity and measure it separately for experience. Overrated: chasing a single headline throughput number. Underrated: traces, because without them a slow reply stays a guess.

Prioritise in this order: scenarios first, observability second, tooling third. Teams that choose the tool before they define the journey end up with beautiful dashboards for the wrong workload. And put the tests in CI, since capacity decays quietly with every dependency you add.

— Sowrabh

Skip the stitching: a platform built for enterprise chatbots

Not every team wants to assemble generators, mocks and tracing pipelines themselves. If you would rather run production agents on a platform with analytics built in, our Conversational AI platform gives you Voice, Live Chat and SMS agents hosted in Australia, with options for private cloud or on-premises deployment.

We keep reporting in the platform, so you spend less time wiring separate tools together, and our agents connect to your CRM and existing systems through standard integrations. Teams working around customer records can look at our Conversational AI CRM, and those with strict residency needs can discuss private hosting on our on-premises page. Bring your toughest conversation flow.

FAQ

What is load testing in QA for a chatbot?

Load testing checks how a system behaves under expected and peak numbers of concurrent users. For a chatbot, that means driving many simultaneous multi-turn conversations and measuring response times, throughput and failures, as the LLMOps workshop describes for LLM applications.

How do I test chatbot load without paying for model calls?

Mock the language model so requests return instantly, then measure how much traffic your own infrastructure can handle. Databricks recommends this approach, followed by a smaller end-to-end run with the real model to capture user-visible latency and cost.

What should I measure during chatbot load testing?

Measure time to first token separately from total completion time, along with p50, p95 and p99 latencies, throughput and failure rates. Add downstream latency and token cost counters, and use traces so you can tell model delays from backend ones.

Which tool is best for chatbot stress testing?

It depends on your protocol and scripting needs. Apache JMeter fits multi-flow WebSocket tests, Locust suits Python teams parsing streamed responses, and Azure Load Testing helps when you need managed scale for JMeter scripts.