Guides

Confidence threshold for a WhatsApp bot: when to hand off

TchaviBy Gael — Tchavi founder

· 15 min read

Back to blog

Table of contents

To decide which WhatsApp messages your bot answers alone, put a threshold on the classifier's confidence, then check it on hard messages and not only on easy ones. In my 11 November 2026 measurement, 0.90 was spotless on 40 ordinary messages and still let a wrong answer through on 12 hard ones. That wrong answer was a confident one: 0.96.

This is written for you if you are building a WhatsApp support inbox for a client's shop, in Nairobi, Kigali, Kampala or Lusaka. Your client wants the bot to answer the simple messages and a person to read the rest. The one real technical decision is the line between the two, and it comes down to a number. Here is how to choose it, what I measured, and what the number cannot do.

One thing up front: the messages are in French. I ran this on the French messages I used in October, 40 invented messages to a shop in Cotonou, plus 12 hard ones I wrote for this test. I did not write an English set to go with it. A set written quickly the same week would repeat the mistake lesson 2 warns about. The reasoning about thresholds does not depend on the language. The numbers do. If your client's customers write in English or Swahili, treat my table as a method and rebuild it on your own messages. I tested nothing in English or Swahili.

In short

  • Measured on 11 November 2026: 52 calls, 1.94 credits in total, zero errors.
  • Ordinary set, 40 invented messages from a shop's customers: 39 of 40 correct. Median latency 294 ms, p90 422 ms. Cost: 37.2 credits per 1,000 messages, at that day's catalogue rate.
  • Hard set, 12 messages (sarcasm, two requests in one message, genuine ambiguity): 9 of 12 correct. Median latency 338 ms.
  • A classifier that returns a confidence score is worth more than one that returns an answer: the human handoff is built on it.
  • A threshold tuned on easy examples betrays you on hard ones. Pushing it to 0.99 cleans the hard set but escalates 7 or 8 of 12 messages.
  • The miss to fear is the confident one: 0.96 on a misclassified message. At 0.80, 0.90 or 0.95, it reaches the customer.

The setup your client sees

The bot reads a message and classifies it into one of five intents (commande order, reclamation complaint, livraison delivery, paiement payment, autre other). Then a rule in your code picks a lane:

  • auto lane: the bot replies with the template for that intent;
  • human lane: the message goes to a person at the shop, with the proposed intent as a label.

What your client sees is a shorter human queue, and customers who do not get an off-target reply. Both depend on a single setting. If you first want to understand how a small model can answer before a stronger one steps in, read the cheap-model-first support bot. Here the subject is narrower: the line between "the bot answers" and "a human reads".

Lesson 1: a confidence score is worth more than an answer

A classifier that only says "payment" leaves you two options: trust it always, or never. One that says "payment, confidence 0.77" gives you a third, the one you need: trust it when it is sure, and hand off when it hesitates.

Here is the real response for message 3 of my set, a customer message I had labelled commande (order). Gloss: "Hi, I'll take the 3 wrappers + the pair of sandals. What do I do to confirm?"

{
  "choice": "paiement",
  "confidence": 0.77,
  "probabilities": {
    "commande": 0.18,
    "reclamation": 0,
    "livraison": 0,
    "paiement": 0.82,
    "autre": 0
  }
}

The answer is wrong: the customer is confirming an order, not asking how to pay. But it is wrong at 0.77, and that number saves the situation. With a threshold of 0.80, this message goes to a human instead of getting payment instructions for an order nobody has confirmed yet. The shop's customer notices nothing: they get a correct reply, a little later.

A note on reading the response: confidence is not the probability of the winning label. On this message probabilities.paiement is 0.82 and confidence is 0.77. Across my 52 calls, confidence is below the highest probability 9 times, by up to 0.10 (message 109: 0.51 against 0.61). I cannot explain it. So I apply the threshold to the confidence field, as returned, and the tables below use it.

This message is also the only miss in the ordinary set, and a stable one: it was already missed in all three October passes, at 0.78, 0.79 and 0.81. So it is not noise. It is a message this model misreads repeatably, and a well-placed threshold catches it every time.

Lesson 2: a threshold tuned on easy examples betrays you on hard ones

My ordinary set: 40 invented customer messages for a shop, written in French in a WhatsApp style, 8 per intent. The labels were written before any run and never changed. Result: 39 of 40.

On this set, any threshold between 0.80 and 0.99 gives the same verdict: zero wrong answers left in the auto lane. If you stop here, you pick 0.80 because it escalates least (1 message in 40), and you feel you have won.

So I wrote a second set of 12 hard messages, built from three traps: sarcasm, two requests in one message, and genuine ambiguity. Result: 9 of 12. Then for each threshold I counted how many messages go to a human and how many wrong answers stay in the auto lane.

ThresholdOrdinary: escalated to a humanOrdinary: wrong answers left in the auto laneHard: escalatedHard: wrong left in
0.801 of 4004 or 5 of 121
0.901 of 4006 of 121
0.952 of 4006 of 121
0.995 of 4007 or 8 of 120

The number itself moves between calls. I ran the hard set three times with identical text. The answers never changed, but the confidence did: message 109 came back at 0.51, then 0.50, then 0.59, and message 102 at 0.54, then 0.62, then 0.60. The largest spread is 0.09, and it sits on the middle-range messages. The very sure ones do not move: four messages returned 1.00 on every pass, and message 104 stayed at 0.96, 0.95, 0.95. The practical consequence: a threshold set at 0.62 is false precision. Set it where that wobble does not change the outcome. It is also why two rows of the table give a range rather than a number: at 0.99, message 110 comes back at 0.99 then 0.98; at 0.80, message 103 comes back at 0.77 then 0.80. Depending on the pass, they cross the line or they do not.

Look at the ordinary columns: they change nothing about the decision. 0.90 is spotless there. Now look at the hard columns: 0.90 lets a wrong answer through, and so do 0.80 and 0.95. Of the four thresholds tested, the only one that cleans the hard set is 0.99, and it escalates 7 or 8 messages of 12.

That is where the setting costs you. The bot exists so humans do not read everything. At 0.99, on messages like the hard set, humans read more than half of them. On the ordinary set, 0.99 costs only 5 of 40, or 12.5%. What your client will really pay therefore depends on the share of hard messages in their real inbox, and that is the number I do not have: 12 messages written to be hard are not a sample of a real inbox. Measure that share on your client's own messages before promising a figure.

Lesson 3: the miss to fear is the confident one

Here are the three errors of the hard set:

MessageTruePredictedConfidenceWhat a 0.80 threshold does with it
102paiementlivraison0.54escalated, correct
109livraisonreclamation0.51escalated, correct
104commandelivraison0.96answered alone, wrong

Message 109 reads (in French): "the parcel arrived but the courier wants 1,000 F extra, is that planned?" It is genuinely ambiguous: a delivery question or a complaint about the courier, two defensible readings. The model answered reclamation at 0.51. Low number, escalation, human: exactly what you want. The model knew it was guessing.

Message 104 reads: "I want to order, but first tell me whether you deliver to Abidjan." Two requests in one message. The model answered livraison at 0.96. At 0.80, 0.90 or 0.95, that answer goes to the customer: they get the delivery zones, and the order they wanted to place does not move forward. No disaster here, but a sale lost silently, and nobody at the shop will know, because no human saw the message.

A caveat:: I am the one who labelled this message commande. Another labeller might have picked livraison, since the delivery question comes second. I am not claiming the model is wrong. I am saying that its 0.96 gives no sign that the message holds two requests. The model knows when it is guessing (109), and it does not know when it is wrong (104).

Keep this: A threshold measures the model's hesitation. It does not measure its correctness. When the model is wrong without hesitating, none of the thresholds I tested helps you except 0.99, and at 0.99 you pay the bill from lesson 2. So you design around the confident error:

  • A first reply that states what it understood. For example: "You want to know if we deliver to Abidjan, is that right?" A customer whose request was misread sees it immediately and corrects it. I have not tested how this works in practice: it is a design choice, not a result.
  • An easy way to reach a person, at any point in the conversation, so the error does not become a dead end.
  • A sample of the auto lane read by a human, for example a few messages a day, to find the 104s your test set did not contain. Again I have no figure to give you on the right sample size.

What this changes in your code

The handoff rule fits in a few lines. The threshold is a constant your client can adjust, not a secret buried in the code:

// Confidence below which a human reads the message.
// 0.80 is the lowest tested here; read the table before choosing.
const THRESHOLD = 0.8;

function route(answer) {
  // answer = result.answers.intent
  if (!answer || typeof answer.confidence !== 'number') {
    return { lane: 'human', reason: 'no_answer' }; // when in doubt, a human
  }
  if (answer.confidence < THRESHOLD) {
    return { lane: 'human', reason: 'low_confidence', suggested: answer.choice };
  }
  return { lane: 'auto', intent: answer.choice, confidence: answer.confidence };
}

Two details matter more than the rest. First, a missing or malformed answer goes to the human, never to the auto lane. Second, store the confidence of every message in your database, including those in the auto lane: that is what lets you rebuild the table above on your client's real messages in a few weeks and move the threshold on facts.

Reproduce the measurement

You need a model that returns a confidence figure for each question: a plain chat model that answers "payment" does not give you one. I used Jev (jev-1.13) on Tchavi, through the /v1/decisions endpoint (quickstart, model page).

The setup is a single choice question named intent, with five labels in criteria, and the state Message WhatsApp d'un client d'une boutique : <text> (French for "WhatsApp message from a shop customer"). Here is the code I used, with the exact French labels, glossed in English in the comments:

const URL = process.env.DECISIONS_URL; // your provider's decisions endpoint
const KEY = process.env.API_KEY;

// Exact labels I ran, in French. Gloss in the comments.
const LABELS = {
  // the customer wants to order, change or cancel an order, or asks if an item is in stock
  commande:
    'le client veut commander, modifier ou annuler une commande, ou demande si un article est en stock',
  // the customer complains: damaged, wrong or missing item, unhappy with service or courier, refund
  reclamation:
    'le client se plaint : produit abîmé, erroné, manquant, service ou livreur mécontentant, remboursement',
  // delivery question: zones, lead times, parcel tracking, in-store pickup
  livraison: 'question sur la livraison : zones, délais, suivi du colis, retrait en boutique',
  // mobile money payment: number, confirmation, failure, payment methods and terms
  paiement:
    'paiement par mobile money : numéro, confirmation, échec, moyens et modalités de paiement',
  // everything else: opening hours, hiring, cold outreach, thanks, product questions
  autre:
    'tout le reste : horaires, recrutement, démarchage, remerciements, questions sur le produit',
};

async function classify(text) {
  const response = await fetch(URL, {
    method: 'POST',
    headers: { 'content-type': 'application/json', authorization: `Bearer ${KEY}` },
    body: JSON.stringify({
      model: 'jev-1.13',
      state: `Message WhatsApp d'un client d'une boutique : ${text}`,
      questions: {
        intent: {
          type: 'choice',
          // "What is the customer's main intent?"
          instructions: "Quelle est l'intention principale du client ?",
          criteria: LABELS,
        },
      },
    }),
  });
  const result = await response.json();
  return result.answers?.intent; // { choice, confidence, probabilities }
}

To rebuild the table: write your own messages and labels before running anything, pass them through classify, then for each threshold count the messages whose confidence is below it (escalated) and those whose confidence is at or above it and whose choice is not the right label (wrong answers left in the auto lane). My 52 messages are invented and I am not publishing them in bulk: write yours, they will fit your client better. Mine were in French, so I ran French definitions. I did not test English-language definitions, so if you write yours in English, run the whole table again before trusting any number from this article.

What these numbers do not say

  • 40 ordinary messages, 12 hard ones, invented, written and labelled by one person. A single labeller: someone else might have labelled differently (see message 104).
  • A one-message gap is not a difference. 39 of 40 against 40 of 40 proves nothing. On 12 messages, one error more or less moves the percentage by eight points.
  • One pass for both sets on 11 November. The three October passes on the ordinary set cover only that set, and they are where the stability of message 3 comes from.
  • The hard set is written to be hard. It shows that a threshold can betray you. It does not say how often, in a real inbox.
  • French only. Every message is in French. I tested nothing in another language: do not carry these numbers to another language without rerunning the test.
  • The human side's cost is not measured. I give the classification cost (37.2 credits per 1,000 messages). The time of a person reading an escalation is the real cost, and it depends on the shop.

FAQ

What confidence threshold should I use for a WhatsApp bot?

I cannot give you a universal threshold. On my 40 ordinary messages, 0.80, 0.90, 0.95 and 0.99 all leave zero wrong answers in the auto lane. On my 12 hard messages, only 0.99 cleans it, at the price of 7 or 8 escalations out of 12. Start at 0.80, measure on your client's real messages, and move the threshold with the numbers.

Why not just set everything to 0.99?

Because the bot's economics disappear. At 0.99, my ordinary set escalates 5 messages of 40 and my hard set 7 or 8 of 12. If your client's inbox looks like the hard set, a human reads almost everything and the bot is pointless.

Does a high threshold protect me from confident errors?

No, not at the thresholds I tested (0.80, 0.90, 0.95). Message 104 was misclassified at 0.96: any threshold up to 0.96 lets it through. A threshold measures hesitation, not correctness. Plan a first reply that states what was understood, and an easy way to reach a person.

Can I just use an LLM that answers "payment" with no confidence score?

You lose the handoff line: without a confidence you trust everything or nothing. An LLM can also serve as a second opinion on the messages where confidence is low, as I compared in Jev vs LLM.

Do these results hold for English or Swahili?

I do not know. All my messages are in French and I tested nothing else. The reasoning about thresholds does not depend on the language, but the numbers do: rebuild the table on your own messages.

How much does it cost to classify 1,000 messages?

In my 11 November 2026 measurement: 37.2 credits. That is the catalogue rate of that day and it can change; the model page shows the current price.

Share

Your first API call, paid with mobile money

No bank card. Top up with mobile money in your local currency; your first top-up adds 150 bonus credits, and credits never expire.