Cost vs quality: find the cheapest AI model that clears your bar

To choose a model, do not look at leaderboards: write 15 cases from your real task, set your pass bar before you run anything, run 3 or 4 models from the cheapest to the most expensive, and keep the cheapest one that clears the bar. I applied this method to extracting an order from a WhatsApp message. In October 2026, through Tchavi, all four models I tested clear my bar, and the cheapest per task costs 130 credits (about KES 9.8) per 1,000 tasks, against 2,766 credits (about KES 209) for the most expensive.

The model with the lowest price per token is not the one that costs least per task. The numbers are further down.

About the test data. The task is for a fictional shop in Cotonou, Benin. The 15 messages are in French, with West African cities (Cotonou, Abomey-Calavi, Porto-Novo, Parakou) and MTN MoMo or Moov Money payments. I quote them with an English gloss, and in the script below I translated the code and field names into English; the measured run used French names. I did not test English or Swahili messages, and I do not invent results for them. KES amounts are converted from credits at October 2026 rates (KES 1,000 buys about 13,204 credits, so 1 credit is about KES 0.076).

In short

• With Tchavi, an OpenAI-compatible AI API gateway you pay for with mobile money, you can run your 15 cases through several models with the same script, changing one string, and read the cost of each call in the x-credits-used header.
• My bar, set before the runs: at least 14 cases out of 15 correct, and 15 valid JSON out of 15.
• Result on 15 order messages: GPT-6 Luna 15 of 15 on both runs, GPT-5.4 Mini 15 then 14, Qwen 3.7 Flash 14 and 14, Claude Sonnet 5.5 15 (one run). All clear the bar.
• Cost for 1,000 tasks: Luna 130 credits (about KES 9.8), Qwen 3.7 Flash 256 credits (about KES 19), GPT-5.4 Mini 795 credits (about KES 60), Sonnet 5.5 2,766 credits (about KES 209).
• My set of 15 cases is too easy to separate the first three on quality. I explain below how to make it harder.

The method in 5 steps

1. Pick one real task. Not "my chatbot" in general: a task with a verifiable output. Extracting an order into JSON, sorting a ticket into categories.
2. Write 15 scored cases. Each case is a real message (or close to it) and the expected answer, written by hand. Put your shop's traps in: a customer correcting themselves, a quantity in words, a city that is not on your list, a message that is not an order.
3. Set the bar before you run. If you set it after seeing the scores, you tune it to the model you already prefer. Mine: 14 out of 15, JSON always valid.
4. Run 4 models, from cheapest to most expensive, with the same prompt and the same script. Note the successes and the credits.
5. Compute the cost for 1,000 tasks, and take the cheapest one that clears the bar.

My 15 cases

The task: an accessories shop in Cotonou receives orders on WhatsApp. The model must return JSON: the items and quantities, the city (Cotonou, Abomey-Calavi, Porto-Novo, other, or nothing), the payment method (MTN MoMo, Moov Money, cash, or nothing) and an "urgent" boolean.

A few of my 15 messages, as I wrote them (in French), with a gloss:

• "Bonjour je veux 2 chargeurs 65W et 1 coque pour Tecno Spark 10, livrez à Calavi svp, je paie MTN" (2 65W chargers and 1 case for a Tecno Spark 10, deliver to Calavi, paying with MTN)
• "Je prends 1 powerbank 20000mAh. Non attends, 2 powerbank. Cotonou, MoMo MTN" (a correction: the right quantity is 2)
• "une douzaine de coques iPhone 13 pour ma boutique à Calavi, paiement Moov" (a dozen iPhone 13 cases: a quantity in words, 12)
• "bonjour, 5 films de protection pour Samsung A14, Porto-Novo, pas pressé, je paierai par Moov" (5 screen protectors, "no rush": not urgent)
• "Annulez le casque, je veux plutôt 2 paires d'écouteurs. Cotonou." (cancel the headset, I want 2 pairs of earphones instead)
• "3 écouteurs sans fil + 1 chargeur voiture, livraison Parakou, espèces" (3 wireless earphones and a car charger, delivery to Parakou, cash; Parakou is not on the list: "other")
• "Combien coûte le chargeur 65W ?" (how much is the 65W charger? Not an order: empty item list)

A case passes only if everything is right: the items, the city, the payment and the urgency.

The scoring script

The cost of a call comes from the x-credits-used header. The script calls the model on each case, scores it, and adds up.

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.TCHAVI_API_KEY,
  baseURL: 'https://tchavi.com/api/v1',
});

// Product keywords (the measured run used the French words: chargeur, coque, ecouteur, ...)
const KEYWORDS = ['charger', 'case', 'earphone', 'headset', 'cable', 'powerbank', 'screen', 'film'];
const stripAccents = (s) => String(s).toLowerCase().normalize('NFD').replace(/[̀-ͯ]/g, '');

// expected: { items: { charger: 2 }, city: 'Cotonou', payment: null, urgent: false }
function score(answer, expected) {
  const errors = [];
  if (!answer || typeof answer !== 'object') return ['invalid_json'];

  const got = {};
  for (const item of answer.items ?? []) {
    const word = KEYWORDS.find((k) => stripAccents(item.product).includes(k));
    if (!word) {
      errors.push('unknown_product');
      continue;
    }
    got[word] = (got[word] ?? 0) + Number(item.quantity);
  }
  const sorted = (o) => JSON.stringify(Object.entries(o).sort());
  if (sorted(got) !== sorted(expected.items)) errors.push('items');
  if ((answer.city ?? null) !== expected.city) errors.push('city');
  if ((answer.payment ?? null) !== expected.payment) errors.push('payment');
  if (Boolean(answer.urgent) !== expected.urgent) errors.push('urgent');
  return errors;
}

export async function evaluate(model, cases, systemPrompt) {
  let passed = 0;
  let credits = 0;
  for (const c of cases) {
    const { data, response } = await client.chat.completions
      .create({
        model,
        messages: [
          { role: 'system', content: systemPrompt },
          { role: 'user', content: c.text },
        ],
        response_format: { type: 'json_object' },
      })
      .withResponse();
    credits += Number(response.headers.get('x-credits-used') ?? 0);

    let json = null;
    try {
      json = JSON.parse(data.choices[0].message.content ?? '');
    } catch {}
    const errors = score(json, c.exp);
    if (errors.length === 0) passed++;
    else console.log(model, 'case', c.id, errors.join(','));
  }
  const per1000 = (credits / cases.length) * 1000;
  return { model, passed, total: cases.length, credits, per1000 };
}

Two scoring choices to remember. Items are compared by keyword and total quantity, so "65W charger" is not penalised against "fast charger". And the comparison is strict on the rest: a wrong city fails the case.

To convert credits to KES, divide by about 13.2 (KES 1,000 buys about 13,204 credits at October 2026 rates).

The results

Four models, from the lowest price per token to the highest. At public prices, for a short prompt, in credits per 1,000 tokens, input then output: Qwen 3.7 Flash 0.066 and 0.29; GPT-6 Luna 0.22 and 1.1; GPT-5.4 Mini 1.65 and 10; Claude Sonnet 5.5 4.4 and 22.

I ran the 15 cases twice on the first three models, once on Sonnet 5.5:

The "per 1,000 tasks" figures are the average of the runs, scaled to 1,000: credits divided by 15, multiplied by 1,000.

The three failures:

• Qwen 3.7 Flash, case 8, on both runs: for "livraison Parakou" (delivery to Parakou) it wrote "city": "Parakou" instead of "other". It is a repeated error, so a prompt or model weakness, not chance.
• GPT-5.4 Mini, case 14, on the second run only: "Combien coûte le chargeur 65W ?" (how much is the 65W charger?) was read as an order for one charger. The first run was right: the error is not stable, which is why a single pass is not enough.
• Luna and Sonnet 5.5 missed no case.

What the bar decides

With my bar at 14 of 15, all four models pass, and the cheapest per task is GPT-6 Luna: about KES 9.8 per 1,000 extractions. Sonnet 5.5 also gets everything right, for about KES 209, which is 21 times more. In this test, paying for Sonnet would have brought me nothing.

If my bar had been "15 of 15 on both runs", only Luna would have passed among the three I reran. A bar is not a detail: it makes the decision. It depends on what an error costs you: a badly extracted order that a human rereads costs a few minutes; an order sent without review to the wrong place costs a customer.

The price per token lies

Qwen 3.7 Flash has the lowest price per token of the four: 0.066 input, 0.29 output, three times less than Luna on input and nearly four times less on output. Yet it costs twice as much per task.

The reason is in the usage. On my 15 cases, on the second pass, Luna produced 1,153 output tokens, 386 of them reasoning. Qwen 3.7 Flash produced 12,360, 11,738 of them reasoning tokens, about 780 thinking tokens per message for an extraction whose answer fits in a few dozen tokens. Reasoning tokens are billed as output. A cheap model that thinks a lot costs more than a slightly more expensive model that answers straight away.

That is the reason to measure on your task with the real cost from the header, not on the price grid. For a starting point on which four models to test, see the price comparison in F CFA (in French).

Limits of this test

• 15 cases is few. The first three models have at most one or two failures: that does not separate their quality. To make it harder: add messages with abbreviations or in your customers' language, ambiguous products, two orders in one message, and go up to 50 cases.
• I tested only four models. Another model, smaller or newer, might clear your bar at a better price: test it with the same script.
• The cases are mine. Use real messages from your shop, anonymised.
• The prompt matters. A more precise prompt (for example "a city outside the list is "other"", written more forcefully) might have fixed Qwen. That is another experiment: measure it too, because the prompt is free to improve and the model is not.
• Prices change. Redo the calculation when you change model or when prices move.

For a setup where a small model answers and a stronger one catches the hard cases, see the support bot with escalation.

FAQ

Why 15 cases and not 100?

Because writing 15 cases with their expected answer takes an hour, and that is enough to rule out clearly inadequate models. It is not enough to separate two good models: for that, go up to 50 or 100 cases drawn from your traffic.

Why run each model twice?

Because models vary from one pass to the next: GPT-5.4 Mini got 15 cases right, then 14. If a model only just clears your bar, a single run does not tell you whether it stays there.

How do I set the bar?

By what an error costs you. If a human rereads every output, 90% may be enough. If the result goes out without review, aim for 100% on your cases, and keep a second model for the doubtful ones.

Should I always take the smallest model that passes?

Not necessarily: take the cheapest per task that passes, not the one with the lowest price per token, as Qwen 3.7 Flash shows. And rerun the test when your messages change.

How do I read the cost in my code?

Every response carries the x-credits-used header. With the OpenAI SDK, .withResponse() gives you the HTTP response, hence the header, as in the script above. The credits and billing page explains the calculation.

create your account and top up KES 250

This article is part of the series "Building an AI product from Africa". Tchavi is an OpenAI-compatible AI API gateway: one key gives access to over 100 text, image, audio and video models, and you pay for it with mobile money, no bank card (countries and operators). More guides on the blog. Follow us: LinkedIn and X.