AzureFeeds

In AI applications, tokens are now cost — and token economics deserves architectural attention

For a long time, AI application design started with model capability: Can the model write code? Can it reason? Can it use tools? Can it handle long context? Those questions still matter, but in the age of agentic applications, they are no longer sufficient. The more important production question is this: How many tokens does the architecture burn to complete one useful task?

A classic chat application often maps one user turn to one model call. An agentic system is different. One user goal can trigger planning, retrieval, tool selection, tool execution, result interpretation, reflection, repair, and summarization. The user sees one instruction; the system may execute dozens of model calls behind the scenes. Tokens are no longer just a measure of text length. They become a measure of system design, runtime behavior, developer workflow, and business cost.

GitHub Copilot’s 2026 move to usage-based billing through GitHub AI Credits captures the industry shift clearly. Usage is now aligned with token consumption, including input, output, and cached tokens. That matters because Copilot has evolved from an in-editor assistant into an agentic platform that can handle long, multi-step coding sessions across repositories. In that world, a tiny prompt and a multi-hour autonomous coding workflow should not be treated as the same economic unit.

Token economics is therefore not about telling developers to “write shorter prompts.” It is about designing systems where:

• useful context is preserved, while noise is removed;


• repeated context is cached or deduplicated;


• simple tasks do not pay for frontier models;


• short-term state is managed structurally instead of copied repeatedly;


• every model call is metered, comparable, and governed.

In short: token economics is the practice of making agentic AI economically sustainable.

Scenario thinking: GitHub Copilot billing, Copilot SDK, GPT-5.5, Anthropic, and MAI-Code Model

The new GitHub Copilot billing model provides a useful framing for developers. Copilot is no longer only autocomplete. It is becoming a programmable agentic platform. It can use models, call tools, work across files, stream responses, and participate in long-running coding workflows. With the GitHub Copilot SDK, developers can embed that agentic runtime into their own applications, services, and developer tools.

That is powerful, but it also changes the cost model. Once an agent loop becomes programmable, token cost also needs to become programmable. If a system can plan, call tools, edit files, retry, repair, and summarize, it also needs to meter, route, cache, compress, and evaluate.

EvalAgentic gives this idea a concrete playground. The project groups models into cost and capability tiers:

This tiering lets us reason about real scenarios:

1. GPT-5.5-class models are valuable for hard reasoning and engineering workflows, but they should not be the default for every step. Using a frontier model for simple classification is like hiring a principal architect to label folders.


2. Anthropic high-capability models can be excellent for complex reasoning and coding, but they benefit from routing discipline. Requirements analysis, test interpretation, deployment explanation, and code generation may not need the same model tier.


3. MAI-Code Model-style coding models should be treated as specialized capability layers. Their value is not just “better code generation”; it is deciding when code-specialized intelligence should be invoked in a larger agent pipeline.

The real question is not “Which model is the best?” It is: Which model is the most economical and reliable for this step of this workflow?

Four engineering techniques for saving tokens

Context Compression: turn long text into executable structure

 

Implementation principle

Context Compression converts long natural-language context into the structured information an agent actually needs. Business documents are often verbose: resumes, contracts, product manuals, requirements, and support logs contain narrative text, boilerplate, repeated explanations, and low-value context. The next agent step may only need a few fields.

EvalAgentic demonstrates this with a long resume-like input that is compressed into a compact JSON object. Instead of injecting the full original text into every prompt, the system extracts key fields and dynamically injects only the data required by the current task.

A practical compression pipeline includes:

1. Redundancy detection — identify long-tail text, repeated descriptions, stale history, and low-value context.


2. Structured extraction — use Copilot or a mid-tier model to transform prose into JSON, tables, or typed schemas.


3. Dynamic injection — inject only the fields needed for the next step.


4. Recoverable references — preserve source pointers so compressed context remains auditable.

How to evaluate

• Prompt token reduction before and after compression.


• Answer quality and task success rate.


• Schema fidelity and missing-field rate.


• Latency improvement.


• Cost per successful task.

Compression is not summarization. Summaries are designed for humans. Structured compression is designed for agents.

Prompt Deduplication / Cache: stop paying twice for the same context

Implementation principle

Many agent systems waste tokens because they repeatedly send the same context. The same resume, contract, repository README, user profile, API documentation, or business rule can be copied across turns and agents.

Prompt Deduplication / Cache applies a simple principle: if context has already been processed, do not pay to process it again unless it has changed.

A concrete design includes:

• compute a hash or semantic key for source context;


• reuse extracted structured results when content is identical or equivalent;


• apply a TTL for repeated entities, such as the 24-hour cache pattern shown in EvalAgentic;


• organize stable prompt prefixes to benefit from provider-level prompt caching where available;


• store shared context in an artifact store or memory layer so multiple agents do not copy the same blob.

How to evaluate

• Cache hit rate.


• Cached token ratio.


• Duplicate prompt rate.


• Cost delta before and after caching.


• Correctness under cache, especially stale-cache failures.

Caching is not “save everything forever.” Good caching knows when to reuse and when to invalidate.

On-Demand Model Routing: let task complexity decide model tier

Implementation principle

On-Demand Model Routing routes each request to the cheapest model that can complete the task reliably. The entry point can use a rule tree, a lightweight classifier, or a hybrid complexity score.

EvalAgentic’s routing tree is intentionally easy to explain:

INCOMING REQUEST
└─ Prompt < 500 tokens? ── YES ─→ TINY: classify / extract
└─ NO ──→ multi-step reasoning?
├─ NO ─→ MID: dialogue / summary
└─ YES ─→ LARGE: agent / code

The engineering logic is straightforward:

• simple classification and keyword matching go to TINY;


• summarization and structured conversion go to MID;


• multi-step reasoning, coding, cross-file changes, and orchestration go to LARGE;


• code-specialized models such as MAI-Code Model can be placed in the coding phase rather than used across the whole pipeline.

How to evaluate

• Routing accuracy.


• Cost per route.


• Quality regression by tier.


• Escalation rate from small models to larger models.


• End-to-end success rate.

Routing does not mean “always use the smallest model.” It means frontier intelligence is reserved for the steps where it actually changes the outcome.

Short-term Memory: preserve state instead of replaying history

Implementation principle

Short-term Memory controls context growth across multi-turn and multi-agent workflows. Without it, agents often replay the full conversation history, full tool outputs, and full intermediate reasoning on every turn. The context grows; quality may not improve; the bill definitely does.

A better design stores state structurally:

• user goal;


• current plan;


• tool outputs and references;


• failure reasons;


• next actions;


• handoff artifacts between agents.

In a multi-agent coding pipeline, the Requirements Agent should hand off a structured spec. The Coding Agent should read that spec, not the entire prior conversation. The Testing Agent should consume testable artifacts, not every word produced by the Coding Agent.

How to evaluate

• Context growth curve across turns.


• Memory retrieval precision.


• Rework rate caused by missing state.


• Recovery quality after failed steps.


• Average input tokens per turn.

Short-term memory is not about remembering everything. It is about remembering the next useful thing.

EvalAgentic as a concrete evaluation example

EvalAgentic is effective as an evangelism project because it turns token economics into an observable before/after system.

The architecture has five layers:

1. Frontend — frontend/index.html provides Tabs A / B / C, live SSE logs, and before/after charts.


2. API — backend/server.py exposes FastAPI routes and Server-Sent Events streaming.


3. Orchestration — eval.py handles A/B evaluation; coding_agents.py handles the multi-agent coding scenario.


4. Core — compressor.py, router.py, gh_models.py, and token_meter.py implement compression, routing, Copilot SDK calls, and token metering.


5. Providers — GitHub Copilot SDK and Microsoft Agent Framework provide model access and agent orchestration.

Tab A: Compression comparison

Tab A compares long-form context before and after structured compression. The key message is that token saving does not come from writing a clever sentence. It comes from converting verbose context into a structured artifact that downstream agents can consume efficiently.

Tab B: On-demand model routing

Tab B demonstrates that cost is not only about raw token count. If a system routes simple tasks to cheaper tiers and reserves expensive models for complex reasoning, total cost can fall even if some token counts increase. This is a subtle but important point: token economics is not token starvation; it is model portfolio optimization.

Tab C: Coding scenario — multi-agent with Agent Framework

Tab C is the most persuasive demo. The same deliverable — a Taobao-like goods-list site with HTML + JavaScript frontend, Flask backend, and Docker deployment — is produced twice by a four-agent pipeline:

1. Requirements Agent;


2. Coding Agent;


3. Testing Agent;


4. Deployment Agent.

The before pipeline uses no compression and sends every agent to GPT-5.5 / LARGE.
The after pipeline injects a compressed JSON spec and uses on-demand routing: requirements can use MID, coding can use LARGE, testing can use MID, and deployment can use TINY.

This mirrors real enterprise development. Architecture and complex code generation may deserve frontier models. Test interpretation, deployment packaging, and simple validation often do not.

Summary and refinement based on the project diagrams

The EvalAgentic README describes three important visuals: the architecture flow, the routing tree, and the token-meter design. Together, they form a governance loop:

User Scenario
↓
Context Compression
↓
Prompt Deduplication / Cache
↓
On-Demand Model Routing
↓
Short-term Memory
↓
Token Metering & Budget Actions
↓
Before / After Evaluation

Optimize the path, not only the prompt

Many teams start token optimization by editing prompt wording. That helps, but the largest waste usually lives in the execution path: how many calls are made, how much context is repeated, how often tools retry, and whether every step uses the same expensive model. EvalAgentic makes the path visible through A/B comparisons.

Token Meter is the control plane of cost governance

EvalAgentic’s token_meter.py uses a non-invasive interceptor pattern:

INTERCEPTOR (@token_meter)
↓
COUNTER CORE: accounting / budget threshold / trigger
↓
ACTION HUB: throttle (>80% budget) / rollback (>budget)

This is the right architectural instinct. Production systems need thresholds, throttling, rollback, and traceability. Without those controls, one retry loop can quietly turn a small user request into a budget incident.

Cost metrics must be evaluated with quality metrics

A system that cuts cost by 80% but drops success rate by 50% is not optimized. It is broken more cheaply. The evaluation matrix should combine cost, quality, latency, and reliability:

Narrative

A simple three-line narrative works well for demos:

1. Token is no longer a technical detail. It is the bill of your architecture.


2. EvalAgentic shows the same scenario before and after cost-aware design.


3. The goal is not to make models cheaper; the goal is to make agent systems economically governable.

For a developer audience, the sharper version is:

A good agent does not use the biggest model everywhere. It uses the right intelligence at the right step, with the right context, under the right budget.

Practical recommendations for real projects

1. Establish a token baseline first. Measure input, output, retries, tool calls, and cost per scenario before optimizing.


2. Make compression a component, not a prompt habit. Define schemas, cache policies, and fallback behavior.


3. Introduce a model routing matrix. Route by task type, complexity, risk, latency, and cost.


4. Define handoff contracts between agents. Pass structured artifacts, not endless conversation history.


5. Evaluate every optimization with A/B tests. Compare cost, quality, latency, and stability.


6. Add budget actions. Throttle at a threshold, rollback on breach, and add circuit breakers for failed retries.

Closing: token economics is the second curve of agent engineering

The first phase of AI application development was about calling models. The second phase was about putting models into products. The next phase of agentic AI is about running those systems reliably, affordably, and governably.

EvalAgentic matters because it turns Context Compression, Prompt Deduplication / Cache, On-Demand Model Routing, and Short-term Memory into something developers can run, compare, and explain. It moves token economics from opinion to instrumentation.

Future AI applications will not only ask: How smart is this agent?
They will ask: How many tokens does it spend per completed task? Which model did it use? Did it hit cache? Did retries run away? Did the system reserve frontier intelligence for the steps that deserved it?

References

• kinfey/EvalAgentic


• GitHub Copilot is moving to usage-based billing


• Updates to GitHub Copilot billing and plans


• Copilot SDK – GitHub Docs