Jev vs LLM: when a decision model is enough

To classify customer messages, an LLM was slightly more accurate than Jev in my test, but Jev answers about 3.6 times faster and costs about 1.7 times less per classification. The accuracy gap is one message out of 40: not enough to call it. The speed and price gap repeats on every pass.

I measured this in October 2026, through Tchavi, with 40 invented WhatsApp messages for a shop in Cotonou. Here are the protocol, the numbers, and the setup I would use: Jev decides, an LLM writes. If you have not met Jev yet, read what Jev is first.

In short

• Tchavi, an OpenAI-compatible AI API gateway you pay for with mobile money, gives access to both: jev-1.13 (decision model, on /v1/decisions) and gpt-6-luna (chat model), with the same key.
• Over 40 messages and 3 passes: Jev 39/40 on every pass (97.5%), gpt-6-luna 40/40, 39/40, then 40/40 (99.2% on average).
• Median latency: about 326 ms for Jev, about 1,184 ms for gpt-6-luna.
• Cost of 1,000 classifications: about 43.6 credits (about KES 3.30) for Jev, about 74.4 credits (about KES 5.63) for gpt-6-luna, at October 2026 rates.
• To reply to the customer you need a chat model: Jev does not write. The right setup is Jev to decide, the LLM to write.

The protocol

I wanted a test anyone can redo. I wrote 40 customer messages for a shop, in WhatsApp style (light typos, abbreviations, capitals), all fictional. There are 5 classes, 8 messages per class. The messages and labels were in French, and I say this once: every test in this article was in French. English and Swahili messages were not tested. The class names stay as I used them, with a gloss:

• commande (order): order, change or cancel, ask about stock;
• reclamation (complaint): damaged or wrong product, incomplete parcel, unreachable service, double billing;
• livraison (delivery): zones, delays, tracking, shop pickup;
• paiement (payment): mobile money number, confirmation, failed payment, payment methods;
• autre (other): opening hours, hiring, cold sales pitches, thanks, a question about the fabric.

The labels were fixed before the first run and I never changed them. A few messages are ambiguous on purpose: "Je prends les 3 pagnes, comment je fais pour valider ?" ("I'll take the 3 wraps, how do I confirm?") can be an order or a payment question.

Both models got the same five class definitions.

Jev gets a single choice question. The code below is what I ran, with French labels and definitions as measured; the English comments are glosses:

{
  model: 'jev-1.13',
  // "WhatsApp message from a shop customer: <text>"
  state: `Message WhatsApp d'un client d'une boutique : ${text}`,
  questions: {
    intent: {
      type: 'choice',
      // "What is the customer's main intent?"
      instructions: "Quelle est l'intention principale du client ?",
      criteria: {
        // wants to order, change or cancel an order, or asks if an item is in stock
        commande: 'le client veut commander, modifier ou annuler une commande, ou demande si un article est en stock',
        // complains: damaged, wrong or missing product, unhappy with service or courier, refund
        reclamation: 'le client se plaint : produit abîmé, erroné, manquant, service ou livreur mécontentant, remboursement',
        // delivery question: zones, delays, parcel tracking, shop pickup
        livraison: 'question sur la livraison : zones, délais, suivi du colis, retrait en boutique',
        // mobile money payment: number, confirmation, failure, payment methods
        paiement: 'paiement par mobile money : numéro, confirmation, échec, moyens et modalités de paiement',
        // everything else: hours, hiring, cold pitches, thanks, product questions
        autre: 'tout le reste : horaires, recrutement, démarchage, remerciements, questions sur le produit',
      },
    },
  },
}

gpt-6-luna goes through /v1/chat/completions with a system instruction asking for a JSON object, the same definitions, and no special settings (shown here in English as a gloss; the run used the French text):

You classify WhatsApp messages from a shop's customers. Answer only with a JSON object {"intent": "<label>"} where <label> is exactly one of: - commande: … - reclamation: … (the same definitions)

Calls are sequential. If Tchavi answers 429 (the per-minute limit on my test key), the script waits for Retry-After and retries, and latency counts only the successful attempt. I ran three complete passes of each model.

The results

KES figures are at October 2026 rates: KES 1,000 buys about 13,204 credits, so one credit is about KES 0.0757.

Latencies are round trips from my machine, one request at a time: the ratio between the two is more reliable than the milliseconds themselves.

Errors, class by class

Jev made the same mistake on every pass: message no. 3 ("Je prends les 3 pagnes + la paire de sandales. Comment je fais pour valider ?", "I'll take the 3 wraps + the pair of sandals. How do I confirm?"), classed paiement instead of commande, with a confidence of 0.78 to 0.81. It is one of the ambiguous messages. The other 7 commande messages and the 32 messages of the other four classes were right.

gpt-6-luna was wrong only once over the three passes: "mon colis est où ? le livreur ne décroche pas" ("where is my parcel? the courier doesn't pick up"), classed reclamation instead of livraison, on the second pass only. The other two passes were perfect.

Two readings. The LLM is the more accurate, by a little (119 correct answers out of 120 against 117). And Jev's error is stable and comes from a truly ambiguous message, while the LLM's changes from one pass to the next.

Jev's confidence works as a filter

In every pass, 38 messages out of 40 had a confidence of 0.9 or higher, and those 38 were all correct. The ambiguous message was among the remaining 2. If you send everything under 0.9 to a human, they read 2 messages out of 40 and your code misses none on this set. My LLM returned only a label, with no signal of this kind.

A harder set

I also ran 12 difficult messages, in a single pass: sarcasm ("Super le livreur, 5 jours de retard et le sac est sale. Bravo hein.", "Great courier, 5 days late and the bag is dirty. Well done."), two intents in one message, thanks that hide a refund request. Result: 9 out of 12 for Jev, 9 out of 12 for gpt-6-luna. They are wrong together on two messages and each misses one that the other gets right. Twelve messages is too few to conclude, but it shows the LLM is not magically better once the message gets blurry.

What it really costs

Jev's price is 0.09 credits per thousand input tokens, output free. gpt-6-luna is 0.22 credits per thousand input and 1.1 output. A classification costs on average 0.0436 credits with Jev (about KES 0.0033) and about 0.074 with the LLM (about KES 0.0056).

Two billing details that explain the numbers:

• Jev expands your questions into the prompt, so it bills about 485 input tokens per message even when the message is one line. That weight is why the price gap with the LLM is only a factor of 1.7.
• gpt-6-luna reasons before answering, and those reasoning tokens are billed as output. On the first message of the second pass, I recorded 33 output tokens, 19 of them reasoning, for an answer that is a few words long.

The gap is about 30.8 credits (about KES 2.33) per 1,000 classifications: it matters from tens of thousands of messages, not for a small shop bot.

These rates are the October 2026 catalogue; each model's page (Jev 1.13, GPT-6 Luna) shows the current price. Top-ups start at KES 250.

Where the LLM wins

My test is a case that favours Jev: five fixed labels, short definitions. There are cases where I would take the LLM without hesitating, even though I did not measure them here:

• An open grid: "does this message follow our tone charter, with its 12 rules and their exceptions?" The LLM reads long, nuanced instructions; Jev answers typed questions with short criteria. I did not measure this case.
• When the answer is not a label: summarise, extract a name and an address, rephrase, translate.
• When you do not need speed: a nightly sort of 10,000 tickets does not care about 300 ms or 1,200 ms.

And in my test, the LLM was simply a bit more accurate. If a mistake costs you a lot and your volume is low, that weighs more than the price.

The combined setup: Jev decides, an LLM writes

Jev does not write, and a chat model is slow and expensive for mere sorting. So I put them one behind the other: Jev routes and scores, the LLM writes the reply in the right tone. I tested the setup on three messages (the torn dress, a question about Parakou, a customer who wrote three times without an answer). The code below is the same logic, with the prompts translated into English as an example; the measured run used French prompts:

// 1. Jev decides: intent, urgency, anger, in a single call
// same request as above, with three questions: intent (choice),
// urgency (score) and anger (noul)
const decision = await fetch('https://tchavi.com/api/v1/decisions', {
  /* method, headers and body as in the first example */
});
const { intent, anger } = (await decision.json()).answers;

// 2. The threshold picks the tone, then a chat model writes
const tone =
  anger.noul >= 0.7
    ? 'Start with a sincere apology and offer a concrete solution.'
    : 'Be warm and direct.';

const reply = await fetch('https://tchavi.com/api/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    Authorization: `Bearer ${process.env.TCHAVI_API_KEY}`,
  },
  body: JSON.stringify({
    model: 'gpt-6-luna',
    messages: [
      {
        role: 'system',
        content: `WhatsApp assistant for a shop in Cotonou. Message classed "${intent.choice}". 2 sentences maximum. ${tone} Do not invent any price, delay or policy.`,
      },
      { role: 'user', content: text },
    ],
  }),
});

Measured results:

Three things to take from it.

The decision costs between 0.042 and 0.043 credits, the writing between 0.130 and 0.255: sorting is 14% to 25% of the total cost of a handled message, but it is the only cost that messages needing no written reply also pay. The setup also lets you skip the big model when it is not worth it (an autre thank-you message can get a canned reply).

The anger threshold did pick the tone: the message at 0.97 got an apology ("I am sincerely sorry you had to write to us three times without an answer", translated from the French reply), the one at 0.03 a neutral answer.

But the LLM does not always obey the instruction. For the Parakou question, it replied "I'm checking whether delivery to Parakou is possible and how much it costs" (translated from the French), a promise your system is not ready to keep, despite "do not invent any policy". Give the LLM the real information (zones, rates) in the instruction, and read the first replies before letting the bot talk on its own.

What these numbers do not say

• 40 messages written by one person, in French: this is not a benchmark, and a one-message gap proves nothing.
• One request at a time from a single machine: under load, latencies may change.
• One LLM and one instruction: another model or another wording can move the accuracy.
• Nothing on English, Swahili, Fon, Wolof or mixed languages, and nothing on images: I only tested French text.

FAQ

Is Jev more accurate than an LLM for classifying?

Not in my test. gpt-6-luna scored 99.2% on average against 97.5% for Jev, one more message out of 40. That is too little to say one is better: the solid difference is speed and cost.

Why is Jev faster?

I do not know for certain. I observe that it does not produce text and that its answer is short. What I measured is the gap: about 326 ms against about 1,184 ms median, in October 2026.

When should I pick an LLM over Jev?

When the answer is text, when your grid is long and open, or when you have no volume: the price gap does not matter for a few hundred messages a day. You can also keep the LLM as a second opinion on the cases where Jev's confidence is low.

How do I test on my own messages?

Take 50 to 100 real messages, label them by hand, run them through both and count the mistakes, as in this article. The code is a simple fetch for Jev and a regular chat call for the LLM. For a first look at Jev calls, read the Jev guide and the decisions reference. Since I only tested French, this matters even more if your customers write in English or Swahili.

Do I need an SDK?

No. Both calls are plain HTTP requests, and I used fetch for everything (the published SDK has no decisions client).

How much do 1,000 classifications cost me?

In my test: about 43.6 credits (about KES 3.30) with Jev and about 74.4 credits (about KES 5.63) with gpt-6-luna, at October 2026 rates. These are catalogue prices and they can change.

create your account and top up KES 250

This article is part of the series "Building an AI product from Africa". Tchavi is an OpenAI-compatible AI API gateway: one key gives access to over 100 text, image, audio and video models, and you pay for it with mobile money, no bank card (countries and operators). More guides on the blog. Follow us: LinkedIn and X.