For the first few years of enterprise generative AI, one of the first architecture questions was usually:
Which model should we use?
OpenAI? Claude? Gemini? Llama?
I increasingly think that is the wrong question.
The differences between today's models—in capability, specialization, latency, context size, deployment model, and cost—have become too significant to treat model selection as a one-time architecture decision.
The better question is:
Which model should handle this specific task, using this specific data, under this organization's policies, at an acceptable cost and latency, right now?
Once you ask the question that way, the architecture changes.
The model stops being the center of the system.
It becomes an execution resource.
And the layer that decides which intelligence to use, when, where, and under what conditions starts to become much more strategically important.
We are already moving toward a multi-model world
This is not just a theoretical architecture pattern.
The market is moving in this direction already.
OpenRouter exposes more than 400 models through a common API and provides routing controls around model selection, provider selection, cost, latency, availability, and data handling. Its Auto Router can classify requests by task type and select among eligible models rather than forcing the application to specify one model for every request.
AWS has introduced intelligent prompt routing in Amazon Bedrock. A single endpoint can dynamically choose between models based on predicted response quality and cost, currently within supported model families. AWS explicitly recommends monitoring both performance and cost as available models evolve.
Microsoft has gone even further conceptually with Model Router in Foundry. Microsoft describes it as a trained routing model that analyzes prompt complexity, reasoning requirements, task type, and other attributes, then dynamically selects an underlying model. Its routing modes explicitly trade among quality, cost, and latency.
Google's Model Garden similarly reflects a world in which enterprises choose across first-party, partner, and open models rather than commit to one foundation-model family. Google currently advertises more than 200 models in the catalog and makes the point directly: a bigger model is not always the better model for a business need.
These products are different, and they solve different problems.
But taken together, they point toward the same architectural conclusion:
Model choice is becoming dynamic.
Why would an enterprise want more than one model?
Because enterprise workloads are not uniform.
Consider a few requests that could all originate inside the same company.
Request 1: Classify 50,000 support tickets
The task is repetitive, relatively constrained, and easy to evaluate.
Using the most capable reasoning model available may add little business value while multiplying inference cost. A smaller model may be entirely adequate.
Request 2: Investigate suspicious AWS activity
Now the requirements are different.
The model may need to reason across CloudTrail events, IAM relationships, timing, behavioral context, and competing hypotheses.
Quality matters more than shaving fractions of a cent from the request. That may justify routing the work to a much stronger reasoning model.
Request 3: Summarize highly restricted acquisition documents
The most important requirement may not be capability at all.
It may be where the model executes and where the data is allowed to go. Perhaps policy requires that this workload stay inside an approved cloud boundary or run through locally hosted inference.
NVIDIA, for example, supports deploying NIM inference services in an organization's own cloud or data-center environment, giving enterprises another execution option beyond consuming a public hosted API.
Same enterprise. Same application platform. Three very different model decisions.
That is the core of the multi-model argument.
The five-dimensional routing problem
I think enterprise model selection is increasingly going to be governed by five dimensions.
Capability
Can the model reliably perform the task?
That includes more than general benchmark strength. Different workloads may emphasize:
- reasoning;
- coding;
- extraction;
- classification;
- multimodal understanding;
- long context;
- tool use;
- structured output.
A model that is excellent for one may be unnecessarily expensive—or simply weaker—for another.
Cost
What does the required quality cost?
This matters enormously once AI moves from experimentation into high-volume workflows. A prototype may generate a few hundred requests. A production customer-service workflow may eventually generate millions.
The economics of choosing a premium model for every transaction can change quickly.
I am deliberately not going deeply into this here because right-sizing intelligence deserves its own article in this series. But the principle is simple:
We learned not to run every cloud workload on the largest virtual machine available.
Why would we run every AI workload on the largest model available?
Security
What information is contained in the request?
A public marketing document and a confidential board document should not necessarily follow the same path. Security policy may determine:
- which providers are permitted;
- whether prompts may be retained;
- whether the model may train on supplied data;
- whether data can leave a geography;
- whether inference must remain inside a controlled environment.
OpenRouter itself illustrates how routing and data policy are becoming connected: its controls can restrict routing to zero-data-retention endpoints.
That is an important architectural clue.
Eventually, model routing becomes partly a data-routing problem.
Latency
How quickly does the answer need to arrive?
A customer-facing interaction may have very different latency requirements from an overnight research workflow. The most sophisticated model is not necessarily the best model if the user abandons the application waiting for it.
Microsoft's router explicitly incorporates responsiveness into its selection model, while OpenRouter supports provider selection based on throughput and latency.
Governance
Is the model allowed to perform this workload?
Suppose a developer discovers a new model that performs exceptionally well. Can they simply add it?
For a prototype, maybe. For an enterprise production system handling regulated information, probably not.
Microsoft lets administrators use Azure Policy to limit which models may participate in Foundry Model Router deployments and audit deployments that drift outside approved configurations. OpenRouter offers organizational guardrails for model and provider access, spending limits, and privacy policies.
That points toward an emerging enterprise requirement:
The model catalog must become governed infrastructure.
The application should not have to know
This is where the architectural shift becomes significant.
Today, many AI applications still contain model logic directly:
Application
|
+--> Call Model X
The developer decides the model. The model name is embedded in configuration or code. Changing providers may involve testing, rewriting integrations, modifying prompts, and redeploying applications.
That feels very similar to earlier generations of infrastructure architecture where applications knew too much about the hardware underneath them.
The more scalable model looks something like:
Application
|
v
AI Gateway / Control Plane
|
+--> Policy evaluation
+--> Task classification
+--> Model eligibility
+--> Cost / latency target
+--> Routing decision
|
+-------------------+-------------------+
| | |
v v v
Public Models Private Cloud Local Models
Now the application asks for a capability, not necessarily a specific model.
For example:
Task: summarization
Data classification: internal
Max latency: 2 seconds
Quality tier: balanced
Max cost: $0.02
The control layer determines how to satisfy that request.
That is a very different architecture.
The model becomes interchangeable infrastructure
This does not mean foundation models become commodities in the economic sense. The strongest models will continue to differentiate themselves.
But from the enterprise architecture perspective, the goal should increasingly be to reduce unnecessary coupling between application logic and model provider.
Cloud architecture provides a useful analogy.
We generally do not want an application to decide: “Run this specifically on server number 713.”
We describe requirements and allow infrastructure layers to determine where the workload executes.
AI appears to be moving toward a similar abstraction.
The enterprise application expresses intent. The control plane chooses intelligence.
That gives organizations options when:
- a model improves;
- pricing changes;
- a provider suffers an outage;
- policy changes;
- a workload becomes more sensitive;
- latency deteriorates;
- a better specialized model appears.
This is also why standardized integration layers are becoming important. The Model Context Protocol, for example, standardizes how AI applications connect to external data, tools, and workflows across model ecosystems.
Model abstraction and tool abstraction are evolving at the same time.
That is not accidental.
One application may span three AI worlds
I think enterprise AI infrastructure will increasingly span three execution domains.
Public AI
Hosted frontier models accessed through provider APIs or abstraction services.
Strengths include immediate access to leading models, no infrastructure management, rapid model evolution, and elasticity.
This is best suited to workloads where public hosted inference fits the security and governance requirements.
Private cloud AI
Models consumed inside cloud-provider governance boundaries through services such as Amazon Bedrock, Microsoft Foundry, or Google's managed model platforms.
These environments can simplify integration with enterprise identity, policy, logging, networking, and regional controls. AWS Bedrock, for example, provides managed access to foundation models, while Microsoft can enforce model-router choices through Azure Policy.
Local or self-hosted AI
Open-weight or commercially supported models running on infrastructure controlled by the organization.
NVIDIA NIM is one example of infrastructure designed to support this pattern across cloud and data-center deployments.
The important point is not that one of these will win.
It is almost the opposite.
The enterprise may need all three.
Multi-model does not mean model chaos
There is an obvious counterargument.
If enterprises struggled to govern three cloud providers, why would we voluntarily introduce dozens of AI models?
That concern is legitimate.
A poorly designed multi-model strategy can create:
- inconsistent behavior;
- difficult testing;
- cost surprises;
- provider sprawl;
- data-governance gaps;
- model-version drift;
- impossible audit trails.
The answer cannot be: let every development team pick whatever model it likes.
That is not multi-model architecture. That is AI sprawl.
The point of the abstraction layer is to allow diversity underneath while creating consistency above.
Developers might see:
/ai/generate
/ai/reason
/ai/embed
/ai/code
while the platform team governs:
- approved models and providers;
- data classifications;
- cost ceilings;
- routing policies;
- evaluation thresholds;
- fallbacks;
- logging;
- regional restrictions.
That separation is what turns model choice from application complexity into platform capability.
Routing without evaluation is dangerous
There is another problem.
How does the router know that the cheaper model is actually good enough?
Cost alone cannot answer that. Latency cannot answer it. Model reputation cannot answer it.
An enterprise needs evaluation.
For a support-ticket workflow, perhaps the organization maintains a test set of previously categorized tickets. For code generation, it may run automated tests. For financial analysis, it may compare outputs against validated cases. For cybersecurity, human analysts may score accuracy, evidence quality, and false-confidence behavior.
Now routing can become evidence-based.
The system can ask:
Among the models allowed for this data class, which ones meet our quality threshold for this task?
Then: which of those satisfies our latency requirement?
Then: which is most economical?
That is much more sophisticated than sending every prompt to whichever model won the latest public leaderboard.
And it creates an important architectural loop:
REQUEST
|
v
POLICY
|
v
ROUTING
|
v
MODEL
|
v
EVALUATION
|
+----> TELEMETRY ----> ROUTING POLICY
The router learns from outcomes. Not just benchmarks.
Reliability is another reason to abstract
Multi-model architecture also provides something enterprises already value deeply in infrastructure: resilience.
If an application depends directly on one model and that provider becomes unavailable, rate-limited, degraded, or unsuitable, the application may fail with it.
OpenRouter explicitly supports model fallbacks, and Microsoft Foundry Model Router includes automatic failover.
But failover in AI is more complicated than failing over a web server.
Two models may:
- interpret prompts differently;
- support different context lengths;
- behave differently with tools;
- return different structured outputs;
- have different safety behavior.
So an enterprise cannot simply declare every model interchangeable.
Fallback must be tested.
Again, evaluation becomes part of infrastructure.
The control plane becomes the strategic layer
This brings me back to the architectural idea I find most interesting.
If models continue multiplying, improving, specializing, and changing price, then enterprises probably should not build their long-term AI strategy around a single one.
The strategic layer becomes the system that knows:
- who is making the request;
- what they are allowed to access;
- what data is involved;
- what the task requires;
- which models are approved;
- how those models have performed;
- how much they cost;
- how quickly they respond;
- where they execute;
- what happened after the request.
That starts to look less like an API gateway.
And more like an AI control plane.
ENTERPRISE AI CONTROL PLANE
Identity Policy Model Registry
Routing Security Evaluation
Cost Telemetry Audit
Provider Health Execution Policy
The model remains enormously important.
But the control plane determines how the enterprise consumes intelligence.
That distinction will matter more as AI moves deeper into production workflows.
This is not the same as avoiding vendor commitment
There is a temptation to frame multi-model architecture purely as a way to avoid vendor lock-in.
That is part of the benefit. But I think it undersells the idea.
The goal is not simply: we can swap OpenAI for Anthropic tomorrow.
The goal is:
We can dynamically use different intelligence for different work without rebuilding the application each time.
That is much more powerful.
One provider may remain strategically preferred. A particular model may handle 80% of requests. Another may dominate coding. A small local model may handle sensitive classification. A premium reasoning model may be reserved for difficult cases.
The architecture does not require equal distribution.
It requires choice.
What enterprise architects should begin asking
The question I would put in front of CIOs, CTOs, platform leaders, and enterprise architects today is no longer simply:
Which LLM are we standardizing on?
I would ask:
What happens when the model we standardize on is no longer the best model for every workload six months from now?
And then:
Can our architecture adapt without rewriting every AI application?
That leads to a more useful set of questions:
- Is model selection embedded inside our applications or abstracted?
- Can policy prevent restricted data from reaching certain providers?
- Can different workloads use different quality tiers?
- Can we route based on task type?
- Can we enforce budgets and cost ceilings?
- Can we compare output quality across models?
- Do we know which model answered each request?
- Can we change providers without breaking the application?
- Can some requests run locally while others use public APIs?
- Can we audit the decision later?
If the answer to most of those is no, the organization may have an AI integration.
It probably does not yet have an AI platform.
My takeaway
For the first phase of generative AI adoption, choosing the right model was a reasonable architecture decision.
The market was smaller. The capability differences were easier to understand. Most enterprise AI projects were experiments.
That period is ending.
Today, model catalogs span hundreds of options, cloud platforms are introducing dynamic routers, private inference is increasingly accessible, policy is beginning to govern model eligibility, and standardized protocols are reducing coupling between models and enterprise tools.
The question therefore changes from:
Which model should we use?
to:
Which intelligence should we use for this workload, under these conditions?
That is the architectural shift.
The future of enterprise AI is unlikely to be one model everywhere.
It will be many models, many execution environments, and increasingly sophisticated decisions about where each request belongs.
And in that world:
The model becomes an execution resource. The control plane becomes the strategic layer.
Next in the series
Article 2 — The FinOps of AI: Right-Sizing Intelligence
If a small model can successfully complete a task for a fraction of the cost, why send every request to the premium model?
The next article will look at the economics of model routing, cost per successful outcome, inference budgets, and why AI needs the same right-sizing discipline cloud infrastructure eventually learned.
References
- OpenRouter: Models
- OpenRouter: Auto Router
- OpenRouter: Provider Routing
- OpenRouter: Zero Data Retention
- OpenRouter: Guardrails
- AWS: Understanding intelligent prompt routing in Amazon Bedrock
- Microsoft: Model router for Microsoft Foundry
- Microsoft: Govern model router deployments with Azure Policy
- Google Cloud: Model Garden
- NVIDIA: NIM documentation
- Model Context Protocol: What is MCP?