Why India Should Bet on Smaller, Specialized AI Models

I came across this sobering observation from Andrej Karpathy recently:

"Certainly, the moment money can buy dramatically better ChatGPT, things change. Large organizations get to concentrate their vast resources to buy more intelligence. And within the category of 'individual' too, the elite may once again split away from the rest of society. Their child will be tutored by GPT-8-pro-max-high, yours by GPT-6 mini."

This stark warning about the potential stratification of AI access led me to think about alternative approaches-particularly for countries like India that are working to establish their place in the global AI landscape.

In the midst of the global AI parameter race, where companies announce ever-larger models with hundreds of billions or even trillions of parameters, I find myself increasingly convinced that India's AI future lies in a different direction. Rather than joining this capital-intensive competition, India has a strategic opportunity to lead in the development of smaller, specialized AI models that could ultimately prove more impactful and sustainable.

The Parameter Race Has Peaked

Let's be honest: the parameter race is showing signs of diminishing returns. Yes, O-whatever (honestly find it hard to keep up, both the releases and the differences between them) and Claude 3.7 are impressive, Gemini 2.5 pro is the current GOAT, but the performance gains relative to their computational requirements don't always justify the increased scale. Meta'x latest underwhelming Llama release may well be case in point. Bigger, but not better. 

While some performance improvements have been achieved through architectural innovations (both software and hardware) rather than just raw scale, what remains undeniable is that the cost of entry to the frontier model race is extraordinarily high-often in the hundreds of millions of dollars for training and infrastructure.

This high barrier to entry effectively locks out most organizations and countries from competing at the frontier, regardless of their talent or innovative approaches. For India, attempting to compete in this arena means playing a game where the initial buy-in is prohibitively expensive. Our approach seeks to fundamentally change this dynamic by lowering the cost of entry and creating more distributed pathways to AI capabilities.

For India, attempting to compete in the frontier model arena means playing a game designed for those with vastly more capital. It's a competition rigged against newcomers, regardless of their talent or innovation.

The True Breakthrough of LLMs

The standout feature of LLMs, in my opinion, is not their ability to do things. It's their ability to understand things. This represents the true paradigm shift: computers that can comprehend context, nuance, and implicit meaning rather than merely executing explicit instructions.

This understanding capability is what makes LLMs revolutionary compared to previous technologies. And importantly, this capability isn't exclusive to massive models with trillions of parameters. Specialized models with careful training can develop deep understanding in specific domains while remaining relatively compact. A model trained exclusively on medical literature can develop sophisticated understanding of medical concepts without needing to also understand astrophysics, literature, and computer programming.

The question then becomes: do we need one massive model that understands everything, or a collection of specialized models that each understand their domain deeply? There is no evidence yet, but we must explore.

Rethinking Mixture of Experts

The Mixture of Experts (MoE) architecture has emerged as one approach to address the scaling challenges of large language models. Companies like Google (with Gemini) and Anthropic have turned to MoE designs where specialized "expert" neural networks handle different types of queries, activated by a router network that directs incoming requests to the appropriate expert.

While MoE is certainly more efficient than traditional dense models, I believe it still represents a half-measure that carries significant limitations:

1. Still resource-intensive: Though more efficient than fully dense models, state-of-the-art MoE systems still require massive computational resources to train and deploy. The router itself becomes a complex component that must understand enough about all domains to make intelligent routing decisions.
2. Architectural constraints: Forcing diverse expertise into a unified neural architecture imposes unnecessary constraints. Different tasks might benefit from entirely different model architectures, not just different parameters within the same architecture.
3. Limited specialization: The "experts" in modern MoE systems are specialized but still operate within the constraints of the broader model. Their specialization is limited compared to truly independent, purpose-built models.

The Specialized Model Alternative

Instead, consider a future built on constellations of smaller, specialized models:

• Why have one massive coding model when you could have 50 tailored to specific languages, frameworks, or domains?
• Why deploy energy-guzzling general-purpose models when task-specific models require a fraction of the resources?

SSDs are cheaper than GPUs anyway.

This isn't merely a consolation prize for those who can't afford frontier models-it may well be a better approach. Specialized models often outperform general models on specific tasks, run with lower latency, cost less to deploy, and can be updated more frequently.

Why India Is Perfectly Positioned

India has unique advantages that make this approach particularly attractive:

1. Software talent depth: India's vast pool of software engineers can be leveraged more effectively across multiple smaller projects than a single massive one.
2. 2Linguistic diversity: India's 22 official languages and hundreds of dialects demand specialized solutions rather than one-size-fits-all approaches.
3. 3Resource efficiency heritage: India has a rich tradition of creating frugal innovations that deliver maximum value with minimal resources.
4. Distributed expertise: From research institutions to startups, India has pockets of specialized knowledge that can each contribute domain-specific models.

There's even historical precedent for this approach. In the 1980s and 1990s, CDAC's work on Indian language computing showed how targeted solutions to specific problems could create an ecosystem that empowered millions, even without the resources of global tech giants.

Beyond MoE

What I'm proposing goes beyond Mixture of Experts to what we might call a "True Orchestration" approach. Rather than housing everything within a single model architecture, this approach:

• Uses a relatively small but capable orchestrator model to understand user intent
• Maintains a registry of completely independent specialized expert models
• Calls the relevant expert via API based on the determined intent
• Combines responses when needed
• Manages context and continuity across interactions

The key differences from traditional MoE approaches are profound:

1. Complete architectural independence: Each expert model can use whatever architecture is optimal for its specific domain-whether that's a transformer, CNN, GNN, or something entirely different.
2. Deployment flexibility: Expert models can be deployed wherever makes most sense-some locally, some in the cloud, some on specialized hardware.
3. Independent scaling: Each expert can be scaled according to its specific needs rather than being constrained by a one-size-fits-all approach.
4. Organizational collaboration: Different organizations can contribute expert models to the ecosystem without needing to integrate into a monolithic system.
5. Incremental improvement: The system can improve gradually as new experts are added or existing ones are enhanced, without requiring complete retraining or rearchitecting. 

If India pursues this, these domains are worth attention:

• Healthcare models trained on Indian medical data, understanding local disease patterns and treatment protocols
• Agricultural models optimized for India's diverse climates, crops, and farming practices
• Educational models designed for various curricula, languages, and pedagogical approaches
• Financial models tuned to India's unique economic landscape and inclusion challenges
• Governance models built to navigate India's administrative structures and public service delivery

Each of these represents an opportunity to create AI that is not just technically sound but contextually relevant to Indian realities.

True Orchestration vs. MoE

For India specifically, the True Orchestration approach offers several advantages over traditional MoE systems:

1. Economic efficiency: The training costs for a collection of smaller models plus an orchestrator are significantly lower than those for massive MoE systems. Initial estimates suggest 5-10x cost reduction for comparable capabilities.-
2. Infrastructure compatibility: Many parts of India still face infrastructure constraints. Smaller models can run on more modest hardware, making deployment feasible across a wider range of settings.
3. Distributed development: India's AI talent is spread across various institutions, companies, and regions. This approach allows different teams to contribute specialized models based on their unique expertise.
4. Data efficiency: Specialized models can achieve high performance with domain-specific datasets, which are often easier to collect and curate than the massive general datasets needed for large models.
5. Cultural and linguistic precision: India's remarkable diversity demands models with deep understanding of specific linguistic and cultural contexts-something that specialized models can provide more effectively than general ones.

Disrupting the New Rent Economy

There's another crucial dimension to this approach that goes beyond technical considerations: economic power dynamics. The current trend toward massive foundation models is actively creating new economic rents. These models are so resource-intensive to build and operate that they naturally create centralized control points where companies can extract ongoing payments for AI capabilities.

Our specialized model approach-especially if we can get these models running efficiently on CPUs of regular computers and laptops-returns power to the average user by short-circuiting that rent-seeking behavior of American firms. When specialized AI models can run locally on consumer hardware, users gain:

• Independence from usage-based billing models
• Freedom from constant connectivity requirements
• Control over their own data and privacy
• Protection from arbitrary API changes or price increases

This represents not just a technical alternative but an economic and philosophical one. It's about whether AI becomes another utility controlled by a handful of tech giants, or a capability that remains accessible to and controlled by ordinary people.

Returning to Karpathy's warning about AI stratification-where "the elite may once again split away from the rest of society" with premium AI access-our approach offers a compelling countermeasure. By creating specialized models that run on common hardware, we can help ensure that high-quality AI remains accessible to all segments of society, not just those who can afford premium subscriptions or dedicated infrastructure. This of course assumes that this will come to pass and there will be intense competition (and hence improvements) in this approach. 

A practical example illustrates the difference: A healthcare AI system for India might need to understand not just medical knowledge but regional disease patterns, local treatment protocols, and multiple languages. A True Orchestration approach would allow domain experts in each area to contribute specialized models rather than trying to force all this knowledge into a single architecture.

India's Strategic Path

For policymakers, researchers, entrepreneurs, and investors in India's AI ecosystem, this suggests a different approach to resource allocation:

• Invest in diverse smaller projects rather than a few massive ones
• Build and promote infrastructure and research that supports model orchestration and deployment
• Create standards for model interoperability and API communication
• Focus on solving specific, high-value problems rather than chasing general intelligence
• Develop expertise in orchestration technologies as a strategic capability

This approach also offers a viable path for public-private partnership. Government agencies could focus on building core infrastructure and standards, while private companies and research institutions develop specialized expert models in their domains of expertise.

A Different Kind of AI Leadership

The global AI landscape is rapidly evolving, and India has a chance to chart its own course-one that plays to its strengths rather than attempting to win a game where the deck is stacked against it. By embracing smaller, specialized models and sophisticated orchestration, India could develop an AI ecosystem that is not just globally competitive but uniquely valuable.

This approach might even represent the future of AI more broadly. As the limitations of monolithic models become more apparent, the industry may shift toward more modular, specialized approaches. By focusing on this direction now, India could find itself not just participating in the global AI ecosystem but helping to define its next evolution.

Importantly, this strategy aids not only India but also the rest of the not-so-rich world. The specialized model approach creates a viable path for countries with significant technical talent but limited capital to participate meaningfully in the AI revolution. It democratizes AI development, allowing nations to build on their unique strengths and address their particular needs without requiring the astronomical investments that frontier models demand. This could help prevent a future where advanced AI capabilities are concentrated exclusively among a handful of wealthy nations and corporations.

In the AI race, the smartest strategy isn't to run faster but to take a different path entirely. For India, that path leads through specialization, orchestration, and strategic collaboration-a distinctly Indian approach to artificial intelligence that builds on the country's unique strengths and addresses its particular needs. And by pioneering this path, India might just create a blueprint that empowers many other nations to develop their own AI capabilities tailored to their specific contexts.

Co- written with Claude.