TECHPARTNER CONSULTING

Codester demo · Scripts & Code → Python

ModelRouter — Self-Hosted Model Router for Coding Agents

Self-hosted model router for coding agents. One local endpoint, three routing patterns, a daily budget cap.

$24 on CodesterPackage test suite: 50 passedRuns below: real, captured from the package

The config is the product

Which model answers which task is a file you edit, not a setting inside someone else's product. This is config.example.yaml from the package, exactly as shipped: a free local model through Ollama, a cheap API model, a strong one for escalation — plus the three routing tiers, effort levels, and a $5 daily budget that warns at 80% and stops at 100%. API keys are never written in the file; the router reads them from environment variables at request time.

# modelrouter-ai example config.
# Copy to modelrouter.yaml and edit. API keys are NEVER written here:
# key_env names an environment variable, and the router reads the key
# from the environment at request time.
#
# Prices are USD per 1M tokens. Local models are 0.0. Edit freely;
# `modelrouter costs` uses exactly these numbers.

models:
  # Local model via Ollama's OpenAI-compatible endpoint. No key needed.
  - id: qwen3-local
    endpoint: http://localhost:11434/v1
    key_env: ""
    input_price: 0.0
    output_price: 0.0
    context_window: 32768

  # Cheap API model. Set OPENAI_API_KEY (or your provider's key) in the
  # environment before starting the router.
  - id: gpt-cheap
    endpoint: https://api.openai.com/v1
    key_env: OPENAI_API_KEY
    input_price: 0.15
    output_price: 0.60
    context_window: 128000

  # Strong API model for escalation and critique.
  - id: gpt-strong
    endpoint: https://api.openai.com/v1
    key_env: OPENAI_API_KEY
    input_price: 2.00
    output_price: 8.00
    context_window: 200000

# Routing tiers. Roles name model ids; effort levels below may override
# individual roles without changing the pattern.
tiers:
  single:
    model: qwen3-local
  cascade:
    draft: qwen3-local
    strong: gpt-cheap
    min_length: 40
    # refusal_patterns:   # optional override of the built-in list
    #   - "i can't"
  critique:
    primary: gpt-cheap
    critic: gpt-strong
    threshold: 7
    max_rounds: 2

# Effort levels. Each level overrides tier roles; anything not listed
# keeps the tier default above. low = cheapest, high = strongest.
effort:
  default: medium
  low:
    single:
      model: qwen3-local
    cascade:
      draft: qwen3-local
      strong: qwen3-local
    critique:
      primary: qwen3-local
      critic: qwen3-local
  medium: {}
  high:
    single:
      model: gpt-strong
    cascade:
      draft: gpt-cheap
      strong: gpt-strong
    critique:
      primary: gpt-strong
      critic: gpt-strong

policies:
  default_tier: cascade
  request_timeout: 120
  budget:
    daily_usd: 5.0
    warn_fraction: 0.8
    stop_fraction: 1.0

Ask the router first — no backend is called

route --explain is a dry run: it prints which model would answer, in what order, and exactly what would make the router escalate — before a single token is spent. Two decisions from the example config above: the default cascade at medium effort, and the critique tier turned up to high.

$ python3 -m modelrouter --config config.example.yaml route --explain "write a function that adds two numbers" --tier cascade

prompt: 'write a function that adds two numbers'  (dry run — no backend was called)
tier:   cascade  (effort: medium)
flow:   qwen3-local -> gpt-cheap
why:    Draft on qwen3-local; escalate to gpt-cheap on upstream error, empty/short output (min 40 chars), refusal, or a missing code block when code was requested.
roles:  {"draft": "qwen3-local", "min_length": 40, "strong": "gpt-cheap"}

$ python3 -m modelrouter --config config.example.yaml route --explain "design a rate limiter for a public API" --tier critique --effort high

prompt: 'design a rate limiter for a public API'  (dry run — no backend was called)
tier:   critique  (effort: high)
flow:   gpt-strong -> gpt-strong -> revise on gpt-strong (up to 2 rounds)
why:    gpt-strong drafts, gpt-strong scores 0-10, revise while below threshold 7.
roles:  {"critic": "gpt-strong", "max_rounds": 2, "primary": "gpt-strong", "threshold": 7}

A live cascade: the cheap draft fails, the strong model answers

The real proxy, run against the package's own fake backends (tests/fakebackend.py), so the whole loop runs offline. One request, model "cascade", prompt "write a function that adds two numbers":

The local model's draft was "ok" — two characters, no code block.
The gate failed it and the router escalated. The agent received:

Here is the answer:
```python
def add(a, b):
    return a + b
```

The routing log entry for that request (routing.jsonl — the prompt is
stored as a hash, never as text):

{"attempts": [{"completion_tokens": 50, "cost_usd": 0.0, "model": "local-fast", "prompt_tokens": 100}, {"completion_tokens": 50, "cost_usd": 0.002, "gate_failures": [], "model": "api-strong", "prompt_tokens": 100}], "budget_warning": false, "completion_tokens": 100, "cost_usd": 0.002, "duration_ms": 4.3, "effort": "medium", "escalation_reason": "too_short+missing_code_block", "models_tried": ["local-fast", "api-strong"], "outcome": "ok", "project": null, "prompt_hash": "32e7e83618322931839f73500d03c422fc472e11fec5f41260a6aa73d5822f84", "prompt_tokens": 200, "request_id": "5b2c3d137ad54c7f", "tier": "cascade", "ts": "2026-11-11T18:04:34Z"}

What the day cost

Every request is priced from the config's own price table and written to a local ledger. modelrouter costs on the data from the run above — the local draft cost nothing; the escalation cost two-tenths of a cent. (An HTML version of the same report is included.)

# modelrouter costs

## Spend by day

| Day | Requests | Prompt tokens | Completion tokens | Cost (USD) |
| --- | ---: | ---: | ---: | ---: |
| 2026-11-11 | 2 | 200 | 100 | $0.0020 |

**Total (shown days): $0.0020**

## Spend by model

| Model | Requests | Cost (USD) |
| --- | ---: | ---: |
| api-strong | 1 | $0.0020 |
| local-fast | 1 | $0.0000 |

## Spend by project

| Project | Requests | Cost (USD) |
| --- | ---: | ---: |
| _no project-tagged requests yet_ | | |

From the package

The router also serves a small dashboard: requests, tokens, spend against the daily budget, and the model table. Screenshot shipped inside the Codester package.

The ModelRouter dashboard, from the package's own screenshots.

What it does