Nigeria told N-ATLAS builders to ship working tools. Local languages need local scorecards

THE SIGNAL IN ONE SENTENCE

A hackathon can make almost anything look useful for three minutes. The microphone behaves. The prepared question lands exactly where the model expects it. The Wi-Fi remembers its lines. Everyone applauds before somebody in the back asks what happens when a real person speaks quickly, changes languages halfway through a sentence and stands beside a generator that has opinions about audio quality. Nigeria's National AI Innovation Challenge has already removed one layer of theatre. The programme, run through the National Centre for Artificial Intelligence and Robotics and the Office for Nigerian Digital Innovation, calls itself build-only. Every valid submission must include a functioning technical artifact integrated with N-ATLAS, Nigeria's multilingual language model. It must also include evidence from actual users, real data or live benchmarks. Pitch decks, concept papers, mock-ups and research papers without a working build do not qualify. That is a good rule. The challenge opened on September 22 and reached its submission deadline on October 12 at 11:59 PM West Africa Time. It asks Nigerian universities, research groups, companies and independent developers to work in one of three areas: developer infrastructure, voice-first access or sector-specific fine-tuning. The official page describes N-ATLAS as an open-source model fine-tuned from Llama 3 8B. It says the system supports Yoruba, Hausa, Igbo and Nigerian-accented English, with automatic speech recognition for voice applications. Those are the programme's descriptions. They are not independent measurements of quality. The plain signal is that requiring a working tool is the right beginning, but a working tool is not yet proof that a language community is well served. For that, Nigeria needs local scorecards. A national average can hide a national collection of failures. Suppose a voice service performs well for one variety of Yoruba recorded in a quiet room, but struggles with another variety spoken outdoors. Suppose an Igbo education tool gives useful answers to formal prompts but loses the thread when students mix English into ordinary conversation. Suppose a Hausa public-service assistant is technically accurate but requires a smartphone, steady data and a careful reading of a small screen. One tidy accuracy number could call all three systems successful. The people using them would have a different report. The challenge already points builders toward real-world evidence. The validation requirement changes by problem statement. Developer-tool projects need at least two external beta testers. Voice-first projects need at least 50 documented real-user interactions. Sectoral fine-tuning projects need a benchmark of at least 500 examples against the base model. These thresholds are more useful than a promise to test later. They also need context. Fifty interactions can reveal bugs. They cannot, by themselves, represent a language spoken across different regions, generations, professions and social settings. Five hundred examples can show an improvement against a baseline. They can also reproduce the narrowness of whoever selected the examples. The first local scorecard should therefore break results down by language variety. Language labels are not uniform boxes. Pronunciation, vocabulary, rhythm and meaning vary by place and community. A model that performs well for the variety best represented in its data may quietly make everyone else do extra work. Builders should document where test speakers come from, which varieties are represented and where the sample is too small to support a claim. Results should be published separately where privacy allows, not blended into one reassuring score. The point is not to crown an official version of a language. It is to notice who the model treats as ordinary and who has to repeat themselves. The second scorecard should measure code-switching. People do not always stay inside the neat language lane a benchmark assigns them. A conversation may move between English and a Nigerian language because that is how the speaker thinks, works, jokes or names a technical idea. A useful voice tool should not fall apart when the language changes mid-sentence. It should identify the switch, preserve the meaning and avoid forcing a person to restart the interaction in the system's preferred mode. Testing should include natural conversations, not only sentences designed to make the model comfortable. It should record which switches caused transcription errors, which names or technical terms were lost and whether the system asked for clarification without blaming the speaker. The third scorecard should include the room. Automatic speech recognition often looks excellent when the audio comes from a close microphone in a quiet studio. Everyday services operate around traffic, markets, classrooms, offices, fans, rain, other conversations and inexpensive phone microphones. The challenge's voice-first track mentions WhatsApp voice notes, interactive voice response, low-bandwidth mobile applications and other accessible channels. Those delivery choices make environmental testing essential. Builders should compare results across devices, connection quality, background noise, speaking speed and distance from the microphone. They should report how often a person must repeat a request, how long the system takes to respond and whether it fails safely when confidence is low. An error rate that ignores the acoustic environment is a laboratory grade for a street-level service. The fourth scorecard should measure task completion, not merely model output. If the application gives agricultural information, did the user understand the answer and reach the intended next step? If it explains a public service, did the person find the correct form or office? If it supports learning, did the student improve after using it? If it provides basic health information, did it clearly state its limits and direct urgent cases to a qualified human? The answer can be linguistically fluent and practically useless. A task-level evaluation should record completion, time, retries, abandoned sessions, requests for human help and mistakes that changed the outcome. It should also record whether people trusted a wrong answer because it sounded natural. That last failure is especially important in high-stakes areas. The challenge lists health, agriculture, education, legal services and financial inclusion among the intended sectors. These applications can reduce barriers, but they can also create new ones if a confident model translates a right, dosage, price or deadline incorrectly. High-stakes tools need source links, uncertainty signals, constrained answers and a human escalation path. A model should not improvise where a person's health, money or legal position depends on precision. The fifth scorecard should be written with communities, not merely about them. Technical teams can measure word errors. Speakers can identify disrespect, stigma, awkward register, missing context and harms a generic benchmark never thought to ask about. Community reviewers should help choose test situations, define unacceptable failures and decide what counts as a respectful response. They should be paid for the work, credited where appropriate and given a say in whether recorded language data can be reused. This matters because voice data is not neutral raw material. A recording can reveal identity, location, health, family relationships and private circumstances. Local-language datasets may be difficult to replace, which makes them valuable and makes careless collection more dangerous. Every project should state what it records, why it needs the data, how long it keeps it, who can access it and how a participant can withdraw. Training data and evaluation data should not silently become a product asset simply because somebody clicked through a challenge form. The sixth scorecard should include access. NCAIR describes N-ATLAS as a digital public good and wants builders to create voice-first services that reach more Nigerians. Public value depends on the cost and conditions of use. Can the application run on a low-cost device? Does it need a continuous connection? How much data does one session consume? Is voice processing performed locally or sent to a server? Does the service work when bandwidth drops? Is there a text or human alternative? Can a person use it without surrendering contacts, location or unrelated phone data? A system can support a local language and still exclude the people who cannot afford its technical assumptions. The seventh scorecard should track the recovery path. Every system will misunderstand someone. The question is what happens next. Can the user see or hear what the system thought they said? Can they correct one word instead of starting over? Can they switch languages? Can they reach a person? Can they report a harmful answer? Does the team review recurring errors and publish what changed? The recovery experience is part of model quality. If a voice assistant fails three times and hangs up, it has not merely made a transcription error. It has denied service. This is why the challenge's insistence on working artifacts matters so much. A functioning tool exposes the entire path from speech to model to interface to outcome. It gives testers something real to interrupt, confuse, correct and improve. A framework on a slide cannot show how a person recovers after the fourth misunderstanding. The programme's three tracks could produce useful public infrastructure. Developer teams could publish documented software kits, evaluation tools and deployment recipes. Voice-first teams could make public information accessible through familiar channels. Research teams could build domain benchmarks and fine-tuned models for education, health, agriculture or legal information. The best projects could also produce shared test sets, error taxonomies and reporting templates that outlive the competition. That may be more valuable than one winning application. If every team measures performance differently, the programme receives a shelf of demos. If teams use a common local scorecard, Nigeria begins to build an evidence layer for language technology. The scorecard should not become another bureaucratic form nobody reads. It should be short enough to use and specific enough to reveal tradeoffs. Each project could publish the language varieties tested, speaker and recording conditions, code-switching results, task-completion rate, serious error examples, access requirements, data policy, human escalation route and known gaps. Results should include denominators. Ten errors out of one hundred sessions means something. Ten errors without the number of sessions means mood lighting. The programme page also sets out future screening, integration checks, shortlisting, mentorship and a November showcase. At the time this article was verified, it did not announce winners or independent performance results. Nobody should mistake the deadline for the outcome. The next useful evidence will come from the builds themselves: what they do, who tested them, where they fail and whether the people whose languages power the system recognize themselves in the evaluation. Nigeria is right to ask builders to ship something real. Now the scorecards need to be real too.