Executive Summary

The artificial intelligence industry has become increasingly focused on achieving state-of-the-art results on frontier benchmarks. Metrics such as GPQA, ARC-AGI, and advanced reasoning evaluations now dominate research agendas and marketing narratives.

While these benchmarks measure impressive capabilities, they may not accurately represent how the majority of the world uses AI systems.

This study analyzes more than 50 million AI interactions from university students across India, offering one of the largest real-world datasets of emerging-market AI usage. The findings challenge the industry’s core assumption: that increasing model scale and benchmark performance necessarily improves user outcomes.

Our analysis reveals that 94% of real-world tasks involve mid-tier reasoning, including summarization, translation, coding assistance, document generation, and exam preparation. In these contexts, extremely large models provide only marginal improvements over well-optimized mid-sized models.

When factors such as latency, cost, and linguistic accuracy are considered together, smaller models frequently produce higher overall user satisfaction.

This suggests that the industry’s obsession with frontier benchmarks may be optimizing for the wrong objective. For billions of users, the most valuable AI systems are not the most powerful—they are the most accessible.

The Benchmark Obsession

Over the past several years, AI progress has increasingly been measured through benchmark performance.

Each new model announcement emphasizes improved scores across standardized tests designed to measure reasoning, mathematics, and problem-solving capability.

These benchmarks are useful for research comparison. However, they represent highly specialized tasks that differ significantly from everyday AI usage.

In practice, most people do not ask AI to solve advanced theoretical problems. They ask it to:

• Summarize documents

• Translate text

• Generate code scaffolding

• Explain academic concepts

• Draft professional communication

These tasks require competence, not perfection.

Real-World Usage Data

To understand how AI is actually used outside research environments, we analyzed 50 million queries submitted by university students across India over a twelve-month period.

The distribution of tasks was strikingly consistent:

Summarization and explanation: 41%

Translation and language assistance: 23%

Coding help and debugging: 18%

Document generation: 12%

Complex reasoning tasks: 6%

Only a small fraction of queries required capabilities associated with frontier-scale models.

The Latency-Cost Tradeoff

Large models introduce two practical disadvantages: latency and cost.

Frontier models typically require more compute resources and longer inference times. For users operating on mobile networks or low-bandwidth environments, this delay significantly degrades the user experience.

Smaller models respond faster and consume fewer resources. When tuned properly, they deliver responses that users perceive as equally useful for most tasks.

In user satisfaction surveys conducted alongside query analysis, participants consistently preferred systems that delivered fast and reliable answers, even when those answers were slightly less sophisticated.

A Different Paradigm: Good Enough AI

These findings suggest an alternative approach to AI development: optimize for sufficiency rather than perfection.

For billions of users, the most valuable AI system is one that is:

Fast
Affordable
Reliable
Multilingual

A model that scores slightly lower on a specialized benchmark but can serve millions of users efficiently may deliver far greater societal value than a frontier model accessible only to a small subset of the global population.

Conclusion

The pursuit of ever-higher benchmark performance has driven remarkable technological progress. However, benchmarks alone cannot define the success of AI systems.

If artificial intelligence is to reach billions of users worldwide, the industry must expand its definition of progress beyond frontier scores.

The next phase of AI development will not be defined solely by the most powerful models—but by the systems capable of delivering useful intelligence everywhere.