LLMRouter / notes

FIELD NOTES · AGENT INTEGRATION

From saving costs to knowing your work: model routing for everyday AI assistants

Connect WorkBuddy and Codex with LLMRouter. Choose a strategy. Train your own router.

Connect an open-source model router to familiar office and coding assistants—from the first request to a complete tool round trip.

Zijie Lei · Zhongjie Dai · Tao Feng · Kunlun Zhu · Fangxu Yu · Haozhen Zhang · Ge Liu · Jiaxuan You

·24 min read·Paper PDF ↗·GitHub ↗

Start with everyday work

The same assistant may rewrite an email, explain a function, and then investigate a bug across several files. We often select a model at the beginning and leave it in place for all three. Switching manually is another option, but it repeatedly puts the model-selection decision back on the person trying to get work done.

Connecting LLMRouter to an everyday assistant moves that decision into a separate routing layer. The assistant still manages the conversation, files, and tools. You still open WorkBuddy for office tasks or Codex for programming. The change happens after the assistant sends a model request.

This post follows those two integrations. We will look at the connection between assistant and router, trace an office request and a programming request, and then consider how to verify both the integration and its usefulness.

The goal is to bring routing into an existing workflow. Connectivity is the first step; choosing a suitable model and finishing the task are what we need to evaluate next.

Choose a router—and shape it around your work

You may want fast email rewrites, more reliable debugging, and a larger budget for important tasks. Someone else may make different trade-offs between speed, style, and cost. LLMRouter makes those choices something you can work on: select a routing method first, then adapt it with representative tasks and preference signals.

You can choose an existing strategy or train your own router.Training here means learning which model to select for a request; it generally does not require retraining every backend LLM. The table below separates several levels of customization.

What you wantMethods to exploreWhat you provideHow it adapts
Assign models according to your workflowRulesKeywords, candidates, and defaultsEncodes explicit task assignments. You update the rules; they do not learn automatically.
Choose flexibly from the task descriptionLLM routingA decision model, candidate descriptions, and routing-prompt changes if neededAn LLM judges candidate suitability. The standard configuration is not a complete user-preference system.
Fit model selection to your own tasksKNN, SVM, MLPRepresentative tasks, candidate evaluations, consistent embeddings, and trained artifactsLearns selection from your task examples. Task-specific customization is distinct from per-user personalization.
Make different choices for different usersPersonalizedRouterUser-conditioned routing evaluations, candidate metadata, a trained GNN, and inference-time user_idUses user identity in selection. The current assistant adapter does not pass user_id; its input mapping needs extending.
Explore user and session preferencesGMTRouterUser/session/query/LLM/response data in its required format, training configuration, and checkpointsSupports a preference-learning direction. Connecting complete session information to this service still requires dedicated adaptation and validation.
Consult choices made for similar tasksLLM routing + optional memoryEnable memory and collect query → model records; per-user retrieval also requires a user identifierAdds previous choices as prompt context. Those records are not satisfaction feedback and do not automatically update learned-router weights.

Getting to know your work takes feedback

Start with recurring tasks and record completion, elapsed time, cost, and which model outputs you prefer. Turn those signals into the supervision format required by the selected algorithm. For trainable methods, retrain periodically and compare the new policy with the old one on held-out tasks before updating the deployed artifact.

Real tasks → outcome and preference feedback → training data → train or update → held-out evaluation → deploy

This is a practical path toward a router that fits an individual or team. It is not an automatic online-learning feature of this integration. Repeated use of unchanged rules does not make them understand you better. Cost savings and a better experience also need measurement; limited feedback, changing candidates, or a shifting task distribution are reasons to evaluate again.

Next, we connect model requests to one routing layer, describe the available strategy choices, and walk through KNN preparation. PersonalizedRouter and GMTRouter are directions for extending personalization, rather than completed assistant-integration demos in this article.

Implementation references: PersonalizedRouter · personalized training config · GMTRouter config · routing memory & adapter.

Place the router behind the assistant

There are three roles in this interaction. The assistant receives the task, assembles context, and executes permitted tools. LLMRouter selects a backend according to its configured policy. The model produces an answer or a tool call, which travels back to the assistant.

Figure 1 · Requests move right; model results return to the assistant. Tool execution can lead to another model request.

One distinction matters here: a shared routing layer does not imply a shared API. The WorkBuddy integration uses Chat Completions, while the Codex integration uses Responses. A server exposing only the former is not sufficient for the latter.

ConnectionWorkBuddyCodex
Client setupCustom modelCustom provider
ProtocolChat CompletionsResponses
Key checkThe intended rule matchesText and tool calls complete

Keep two kinds of credentials separate: a token the assistant may need to access the router, and the upstream API keys the router uses to access model providers. Upstream keys stay on the router. A placeholder client key is appropriate only when the local router does not require endpoint authentication.

Beyond rules: prepare and switch routing strategies

Connecting an assistant and choosing a routing algorithm are separate configuration decisions. WorkBuddy and Codex connect to a server API. If the address, protocol, and routing entry model stay compatible, changing strategy usually means updating the server configuration and restarting it, rather than reconnecting the assistant. The later rules example is an observable starting point, not the limit of this integration.

Choose a strategy and prepare its inputs

StrategyWhat to prepareIntegration boundary
rulesKeywords, a default rule, candidates, and upstream credentials.No training; matches only configured keywords.
random / round_robinCandidates; optional weights for random.No training; useful for checking dispatch, without judging task requirements.
llmA decision-making LLM, its endpoint/key, and candidate descriptions.No local classifier training; routing can add a model call and latency.
llmrouter + KNN / SVM / MLPAlgorithm configuration, training data and embeddings, trained artifacts, matching candidate names, and dependencies.Loaded through the adapter; verify real model selections.
MF, Elo, Graph, other registered or custom methodsMethod-specific data, parameters, weights, and optional dependencies.Registry/plugin loading is available; registration alone does not establish assistant compatibility.
Multi-round methods, RouterR1, methods needing user or multimodal inputsSpecialized inputs and execution pipelines.Require separate adaptation; do not assume a name change provides full support.

The current adapter passes only {"query": text} to the algorithm and extracts a model name from model_name, predicted_llm, or predicted_llm_name. It does not automatically pass user profiles, raw images, or complete multi-turn state to a learned router. Some multi-round methods return a completed answer and have dedicated execution paths in the CLI. That is a different contract from selecting a backend and letting the assistant continue its workflow.

The shared foundation: one service configuration and callable models

Use Python 3.10 or newer in an isolated environment and install from source. These commands target the repository version linked below. Add any algorithm-specific dependencies required by its documentation.

Shell
git clone https://github.com/ulab-uiuc/LLMRouter.git
cd LLMRouter
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .

Prepare your own assistant-router.yaml. Keep its serve, api_keys, and llms sections, and replace the router section when changing strategies. You can adapt openclaw_router/config.yaml, replacing its demonstration providers, models, and credentials with working values. The snippets below omit the full candidate definitions and are not standalone service configurations.

Each llms entry needs a stable candidate name plus real provider, model, and base_url settings. Backends must support the tools and context lengths your assistant uses. Configure at least two eligible candidates when comparing strategies: the code selects the sole candidate directly if only one remains after filtering.

Path one: use an LLM to choose the backend

Merge these fields into the service YAML and set OPENAI_API_KEY in the server environment. Here, model identifies the decision-making model; the final answering backend still comes from llms. Write useful descriptions for the candidates, which the routing prompt includes. Other decision-model providers can be configured using the server’s supported fields.

YAML · merge into service configuration
router:
  strategy: llm
  provider: openai
  base_url: https://api.openai.com/v1
  model: "<routing-model-id>"

api_keys:
  openai: ${OPENAI_API_KEY}

Start with the service command below and inspect several different tasks in the logs. The earlier keyword assignments are not expected outcomes for this strategy: selection is now a model judgment.

Path two: prepare a KNN router

KNN makes the preparation steps concrete. Copy configs/model_config_train/knnrouter.yaml and configs/model_config_test/knnrouter.yaml into your own training and inference configurations, then check each path.

  1. Prepare candidates and supervision.Candidate metadata, query data, evaluated routing records, and query embeddings must describe a consistent model pool and data pipeline. The KNN implementation chooses the highest-performance model per query as its training label and uses embedding_id to find the corresponding vector. Labels are model names, not arbitrary difficulty tiers.
  2. Align names.The training model_name labels, candidate-metadata keys, and service llms keys should agree. If a training label is qwen2.5-7b-instruct, expose a candidate with that name; renaming it to lightweight does not establish a reliable mapping. The candidate’s model field contains the real upstream ID. Prefer exact matching over the adapter’s fuzzy matching.
  3. Prepare the embedding model.KNN inference currently uses allenai/longformer-base-4096, which may download on first use. Training vectors must use consistent encoding and preprocessing. Set LLMROUTER_EMBEDDING_DEVICE=cpu if needed. The repository’s example data is demonstration material, not evidence of suitability for your office or coding tasks.
  4. Train or load an existing artifact.The training save_model_path and inference load_model_path should refer to the same artifact. Prefer absolute paths in your own configurations to avoid differences in relative-path resolution across components. Router construction also reads data_path inputs; copying only a pkl file is insufficient.
Shell · train, then test selection only
llmrouter train --router knnrouter \
  --config /absolute/path/to/knn-train.yaml

llmrouter infer --router knnrouter \
  --config /absolute/path/to/knn-inference.yaml \
  --query "Explain this Python function" --route-only

If you already have a trusted artifact compatible with the model pool and features, skip training and run the route-only check. Confirm that its returned name exists among the service candidates, then replace the server’s router section:

YAML · merge into service configuration
router:
  strategy: llmrouter
  llmrouter_name: knnrouter
  llmrouter_config: "/absolute/path/to/knn-inference.yaml"
  llmrouter_model_path: "/absolute/path/to/knnrouter.pkl"

llmrouter_model_path overrides the algorithm’s model-loading path. The standalone CLI check above instead uses load_model_path in the inference YAML itself. Point both at the same artifact.

For SVM or MLP, follow the same sequence using the method’s own train/test YAMLs, training command, and artifact. Update llmrouter_name, llmrouter_config, and llmrouter_model_path together. A KNN pkl cannot simply be reused by MLP. MF, Graph, personalized, and other methods have different data requirements: follow their documentation and first check compatibility with this text-input, model-name-output contract.

Start the service and reconnect the existing workflow

Shell · run from the repository root
python -m openclaw_router \
  --config /absolute/path/to/assistant-router.yaml \
  --host 127.0.0.1 --port 8000

Specify host and port explicitly because this entry point overrides the YAML values with its CLI defaults. If the service address is unchanged, the WorkBuddy custom-model settings and Codex provider configuration can remain in place. Requests must use a routing entry model; specifying a candidate name can pin the backend. This local check assumes no endpoint authentication and uses auto to explicitly request routing:

Shell · local checks
curl -s http://127.0.0.1:8000/routers
curl -s http://127.0.0.1:8000/v1/models
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"auto","messages":[{"role":"user","content":"Explain this Python function"}]}'

/routers reports available strategies and the configured strategy; /v1/models reports candidates. Neither proves that an algorithm ran successfully. For a learned strategy, look for both Loaded LLMRouter: knnrouter and request-time Strategy=llmrouter(knnrouter) messages, and inspect nearby errors or fallback warnings. Successful initialization alone does not guarantee that an artifact will be readable on the first inference call.

After changing strategy, verify text responses and tool calls using the client checks below. Algorithm configurations, trained artifacts, and client versions must be compatible.

Implementation references (pinned revision): adapter & strategies · registry & loader · KNN config · KNN implementation · server entry point.

Office work: send the first routed request from WorkBuddy

Start with a task whose route is easy to inspect: translating an English sentence into Chinese. We want to see more than a translated answer. We want to confirm that the request reached LLMRouter and that the policy selected its backend. This example uses keyword rules to make this decision easy to follow.

Make the selection visible

The policy below sends translation and rewriting prompts to lightweight, code-related prompts to coding, and certain reasoning prompts to reasoning. These names are configured aliases, not guarantees of model capability. Rules run from top to bottom; the first match wins, and unmatched requests use the default.

YAML · policy excerpt
serve:
  host: "127.0.0.1"
  port: 8000
  show_model_prefix: false
  show_model_suffix: true

router:
  strategy: rules
  rules:
    - keywords: ["math", "prove that", "calculus", "probability"]
      model: reasoning
    - keywords: ["code", "debug", "python", "api", "deploy"]
      model: coding
    - keywords: ["translate this", "proofread", "rewrite this"]
      model: lightweight
    - default: coding

  fallback_models:
    - coding
    - lightweight

This is not a complete deployment configuration. Each candidate also needs a real model ID, upstream endpoint, and credentials. The deployment shown in the video reads OPENROUTER_API_KEY and starts with ./deploy/start-local-router.sh; use that script only if it exists in your migration checkout. Restart the demo server after changing a policy that it loads at startup.

Check the server before connecting the client. The server exposes the following endpoints; consult your deployment documentation if your version differs.

Shell
curl -s http://127.0.0.1:8000/health
curl -s http://127.0.0.1:8000/v1/models

Point the custom model at the router

In the WorkBuddy version shown in the video, open Settings → Model → Custom. Set the URL to http://127.0.0.1:8000/v1 and Model Name to LLMRouter. Turn Custom Protocol off so the client appends /chat/completions. Enable Tool Calling if that toggle is available; this text-only example does not require Vision/Image.

Set the API Key according to the router’s authentication requirements. Use a non-empty placeholder only for a local endpoint without authentication. The model name matters too: in the current server implementation, a candidate alias may pin the request to one backend, so use the routing entry point LLMRouter.

Follow one request through the system

Save and select the custom model, open a new conversation, and send the request below. Its English keyword intentionally matches the example policy. A Chinese phrase meaning “please translate” is not in this keyword list and should not be assumed to match.

Prompt
translate this into Chinese: The train leaves at nine.

Under this policy, translate this should select lightweight. Inspect the matched rule and real model ID in the router logs, then check whether the assistant completes the translation. With show_model_suffix enabled, the server adds [model: <upstream-model-id>] to the final text as an additional check.

Next, try debug this python function and explain the fix. and prove that the square root of 2 is irrational. to check the coding and reasoning paths. These prompts test routing behavior; they do not establish the relative capabilities of the selected models.

WorkBuddy · Integration walkthrough: client configuration and routed requests. MP4 ↓

Programming: connect Codex to the same routing layer

Programming adds another requirement. An assistant may read a file, call a model, run a tool, and explain the result. Receiving a sentence from a model does not establish compatibility with that whole loop. The Codex integration therefore checks both the Responses interface and a complete tool-call round trip.

Before starting, confirm that the router supports the Responses interface required by Codex and obtain a model profile compatible with your Codex client. That profile is distinct from an upstream candidate alias: the former must work with the client’s tool capabilities; the latter identifies a backend choice. Do not assume an example backend alias is a valid Codex model.

Configure a custom provider

Back up the user-level ~/.codex/config.toml, then merge the following settings into it without duplicating a TOML table. Provider configuration belongs in the user-level file; the official configuration reference notes that the corresponding provider settings are ignored in project-level files.

TOML · configuration example
model = "<Codex-supported-tool-capable-model-slug>"
model_provider = "llmrouter"

[model_providers.llmrouter]
name = "LLMRouter"
base_url = "http://127.0.0.1:8000/v1"
wire_api = "responses"
supports_websockets = false
requires_openai_auth = false

Replace the model placeholder with a real profile compatible with the client and deployed router. wire_api = "responses" specifies the interface. This example disables WebSockets and assumes the local endpoint needs no authentication. If it requires a token, configure env_key and its environment variable as directed by the operator. Upstream model credentials stay on the router.

Save and fully restart Codex. Confirm the provider and model configuration through the selection interface supported by your client, then start a new local conversation. The UI varies by version; a request arriving at the router is the decisive connection check.

Test text, then one complete tool round trip

Use a translation or short summary to check basic connectivity. Next, place a small example file in a test workspace and ask Codex to read it and explain a function. That task requires both model participation and a real tool operation by the assistant.

  1. Confirm in the logs that the request uses the Responses path, and record the selected backend.
  2. Check that the client executes the file-reading tool call and the model continues after receiving the result.
  3. Check that the assistant produces a final answer rather than stopping at the tool call.

If model attribution is enabled in the migration, the final text should identify only the model used for that answer. Do not expect attribution text in every tool-protocol message; extra text can break its structure. Repeated suffixes are a reason to inspect how old attribution labels are removed from conversation history.

Translation, summarization, and implementing Two Sum are useful sample workloads, but they do not guarantee assignments to tiers such as quick, moderate, or strong. The deployed Codex policy determines those choices. Verify them in the logs.

Codex · Integration walkthrough: client configuration and routed requests. MP4 ↓

After connection: does it help?

A successful answer establishes that one path works. Bringing routing into daily use requires two separate checks: whether requests follow the intended route, and whether the routed workflow completes tasks effectively.

Rule out integration failures first

SymptomCheck first
Connection failsRunning service, host, port, and /v1 prefix.
One backend for everythingAccidentally pinned aliases, default rules, and the actual rule matches in logs.
Policy edits have no effectWhether the demo server has reloaded its configuration or restarted.
401 / 403Router endpoint authentication versus upstream provider authentication.
Text works, tools failResponses compatibility, model profile, tool definitions, and result formats.

Then compare work outcomes

Choose tasks you actually do: a few email rewrites, translations, file explanations, and debugging tasks. Run the same set with a fixed model and with the router. Record completion, elapsed time, cost, and how often you have to correct the assistant. For tool-using tasks, measure the complete work session rather than just one model response.

Rules make a useful starting point because they are easy to explain and debug. They can also miss equivalent phrasing or react poorly to prompts containing several keywords. They do not inherently understand task difficulty. LLMRouter’s broader range of routing methods offers directions for later experiments; use your own tasks to decide whether a more complex policy is worthwhile.

This walkthrough does not include a controlled comparison, so it does not establish lower cost, faster execution, or better quality. It establishes an inspectable integration path: get requests to the right place, then connect routing decisions to actual task outcomes.

Make model selection part of the workflow

WorkBuddy and Codex expose different integration points, but the goal is the same: keep the assistant’s familiar interaction while making backend selection a separate decision that can be configured, observed, and improved.

A practical starting point is one assistant and one real task. Expand the model pool and routing policy after that works. When you can explain both why a model was selected and whether it finished the job, LLMRouter has begun to move from an open-source project into everyday work.

References & further reading

  1. LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers — arXiv:2608.06867 (PDF).
  2. LLMRouter — GitHub

    Project source and routing methods. Check integration support against your deployed version.

  3. Codex — Configuration reference

    Custom providers, the Responses protocol, and configuration scope.

  4. LLMRouter — Project documentation

Compatibility depends on client versions and backend capabilities. Follow the verification steps for your deployment.