Final week, our lead software program engineer, Nelson Masuki and I offered on the MSRA Annual Convention to a room filled with good researchers, information scientists, and growth practitioners from throughout Kenya and Africa. We had been there to deal with a quietly rising dilemma in our area: the rise of artificial information and its implications for the way forward for analysis, significantly within the areas we serve.
Our presentation was anchored in findings from our whitepaper that in contrast outcomes from a conventional CATI survey information with artificial outputs generated utilizing a number of massive language fashions (LLMs). The session was a mixture of curiosity, concern, and significant pondering, particularly once we demonstrated how off-the-mark artificial information could be in locations the place cultural context, language, or floor realities are advanced and quickly altering.
We began the presentation by asking everybody to immediate their favorite AI app with some actual inquiries to mannequin survey outcomes. No two folks within the corridor acquired the identical solutions. Although the immediate was precisely the identical, and many individuals used the identical apps on the identical fashions, problem one.
The experiment
We then offered the findings from our experiments. Beginning with a CATI survey of over 1,000 respondents in Kenya, we carried out a 25-minute research on a number of areas: meals consumption, media and know-how use, data and attitudes towards AI, and views on humanitarian help. We then took the respondents’ demographic data (age, gender, rural-urban setting, training stage, and ADM1 location) and created artificial information respondents (SDRs) that precisely matched these respondents, and administered the identical questionnaire throughout a number of LLMs and fashions (even did repeat cycles with newer, extra superior fashions). The variations had been as various as they had been skewed – virtually all the time flawed. Artificial information failed the one true check of accuracy – the genuine voice of the folks.
Many within the room had confronted the identical pressure: world funding cuts, rising calls for for pace, and now, the attract of AI-generated insights that promise “simply pretty much as good” with out ever leaving a desk. However for these of us grounded within the realities of Africa, Asia, and Latin America, the concept of simulating the reality, of changing actual folks with probabilistic patterns, doesn’t sit proper.
This dialog, and others we had all through the convention, affirmed a rising fact – AI will undoubtedly form the way forward for analysis, however it should not exchange actual human enter. At the very least not but, and never within the components of the world the place fact on the bottom doesn’t dwell in neatly labeled datasets. We can not mannequin what we’ve by no means measured.
Why Artificial Information Can’t Exchange Actuality – But
Artificial information is precisely what it seems like: information that hasn’t been collected from actual folks, however generated algorithmically primarily based on what fashions suppose the solutions must be. Within the analysis world, this sometimes includes creating simulated survey responses primarily based on patterns recognized from historic information, statistical fashions, or massive language fashions (LLMs). Whereas artificial information can function a useful testing instrument, and we’re frequently testing its utility in managed experiments, it nonetheless falls brief in a number of important areas: it lacks floor fact, it missed nuance and context, and due to this fact it’s laborious to belief.
And that’s exactly the issue.
In our side-by-side comparability of actual survey responses and artificial responses generated by way of LLMs, the variations weren’t delicate – they had been foundational. The fashions guessed flawed on main indicators like unemployment ranges, digital platform utilization, and even easy family demographics.
I don’t imagine that is only a statistical problem. It’s a context problem. In areas equivalent to Africa, Asia, and Latin America, floor realities change quickly. Behaviors, opinions, and entry to companies are extremely native and deeply tied to tradition, infrastructure, and lived expertise. These are usually not issues a language mannequin skilled predominantly on Western web content material can intuit.
Artificial information can, certainly, be used
Artificial information isn’t inherently unhealthy. Lest you suppose we’re anti-tech (which we are able to by no means be accused of), at GeoPoll, we do use artificial information, simply not as a substitute of actual analysis. We use it to check survey logic and optimize scripts earlier than fieldwork, simulate potential outcomes and spot logical contradictions in surveys, and experiment with framing by working parallel simulations earlier than information assortment.
And sure, we may generate artificial datasets from scratch. With greater than 50 million accomplished surveys throughout rising markets, our dataset is arguably one of the consultant foundations for localized modeling.
Nonetheless, we’ve additionally examined its limits, and the findings are clear: artificial information can not exchange actual, human-sourced insights in low-data environments. We don’t imagine it’s moral or correct to interchange fieldwork with simulations, particularly when selections about coverage, funding, or support are at stake. Artificial information has its place. However in our view, it isn’t, and shouldn’t be, a shortcut for understanding actual folks in underrepresented areas. It’s a instrument to reinforce analysis, not a substitute for it.
Information Fairness Begins with Inclusion – GeoPoll AI Information Streams
There’s a major cause this issues. Whereas some are racing to construct the following massive language mannequin (LLM), few are asking: What information are these fashions skilled on? And who will get represented in these datasets?
GeoPoll is on this area, too. We now work with tech corporations and analysis establishments to offer high-quality, consented information from underrepresented languages and areas, information used to coach and fine-tune LLMs. GeoPoll AI Information Streams is designed to fill the gaps the place world datasets fall brief – to assist construct extra inclusive, consultant, and correct LLMs that perceive the contexts they search to serve.
As a result of if AI goes to be actually world, it must study from the whole globe, not simply guess. We should be certain that the voices of actual folks, particularly in rising markets, form each selections and the applied sciences of tomorrow.
Contact us to study extra about GeoPoll AI Information Streams and the way we use AI to energy analysis.
