Does Your AI Assistant Understand Swahili? How to Test It

By Metridata Editorial Team  ·  Updated October 2026  ·  7 min read  ·  AI Testing & Evaluation

---

An AI assistant that works well in English can behave very differently once a customer writes to it in Swahili, or in the mix of Swahili, English and Sheng that many Kenyans use in a single message. If you sell to East Africa, or you are an overseas team about to, you should test for that before launch rather than learn about it from customers.

What the research says

AfroBench, a benchmark from researchers at McGill and other institutions, covers 64 African languages across 15 tasks and 22 datasets. Its authors report large gaps between English and African languages on most tasks, with performance varying by how much data exists for each language. They also found the gap is smaller for the closed commercial models they tested than for open ones. An earlier benchmark, IrokoBench, reported that the best open model it tested reached only 63% of the top proprietary model’s score on African languages.

Two cautions. These studies tested specific models at specific times, and models improve quickly, so do not read the numbers as a current ranking. And Swahili is one of the better-resourced African languages, so results for it will not match results for smaller languages. The sensible takeaway is modest: do not assume your assistant handles Swahili as well as English. Measure it.

What to test

Mixed-language messages. Real customers write things like a Swahili greeting, an English product name and a Sheng phrase in one line. Test whether the assistant understands the request and replies in a register the customer would actually use.

Local facts. Names of places and estates, the way people write shillings, dates in day-month order, and payment terms such as till numbers, paybill and M-Pesa confirmation messages. A model that gets these slightly wrong can send a customer to pay the wrong account.

Formal and informal register. A bank and a food delivery service should not sound the same. Check that the assistant matches the tone you intend and does not slip into stiff textbook Swahili or into slang that does not suit your brand.

Typos, short messages and voice notes transcribed to text. WhatsApp messages are short, unpunctuated and full of shortcuts. Test with the messages you actually receive, not clean sentences.

Safety behaviour in Swahili. Guardrails that hold in English do not always hold in other languages. Repeat your refusal and attack tests, including attempts to extract the system prompt or another user’s data, in Swahili and in mixed language, and compare the results with English.

Accuracy of facts and numbers. Run the same factual questions in both languages and check that the answers agree. Differences between the two versions are often where the errors hide.

How to run it

Use native speakers, not a translation tool. A translated English test set misses the way people really write. Ask two or three testers from different backgrounds to write the questions, since a Nairobi university student and a shopkeeper in Kisumu will phrase things differently.

For each answer, have the tester score three things: did it understand, was it correct, and did it sound right. Keep the scores by category so you can see where the weakness sits. A rough starting size is 50 to 100 questions per language category, which is our own rule of thumb rather than a standard. Then repeat the run after every model or prompt change, as we describe in our chatbot testing checklist.

What to do with the results

Failures tend to fall into a few groups, each with a different fix. If the assistant misunderstands, you may need a different model or clearer prompting. If it knows the language but gets your facts wrong, the problem is your knowledge base. If it is correct but sounds off, adjust the tone instructions and add examples. If its guardrails weaken in Swahili, limit what it can do rather than hoping a better prompt will hold.

Frequently Asked Questions