The LLM Council: Six AI Models Review Everything I Build, via One OpenRouter API

The most useful tool in my workflow isn't a model. It's a committee.

Before I act on anything I've drafted (a game-design document, an architecture plan, a security review, an essay), I run one command. Twenty to sixty seconds later, six frontier models from six different AI labs have each independently critiqued it, anonymously, in a structured format that forbids praise. Then the coding agent I'm already working in reads all six critiques and tells me where they agree, where they disagree, and which findings are a single voice I can probably ignore.

I call it the council. It's a small Python script plus a Claude Code skill, it costs pennies per run, and the entire thing is possible because of one piece of infrastructure: OpenRouter.

What OpenRouter Is (and Why It's the Backbone Here)

OpenRouter is an API gateway for AI models: one endpoint, one API key, one bill, hundreds of models from OpenAI, Google, Anthropic, xAI, DeepSeek, Moonshot, and dozens of other providers. You send a standard chat-completion request with a model ID like openai/gpt-5.6-sol or moonshotai/kimi-k3, and OpenRouter routes it to the right provider.

For most developers the pitch is flexibility: swap models without rewriting code, avoid lock-in. For this project the pitch is something more specific: OpenRouter makes a multi-vendor workflow trivially cheap to build. Getting six models from six different labs to review the same document would otherwise mean six accounts, six API keys, six billing relationships, and six slightly different SDKs. With OpenRouter it's a for loop.

The Idea: Karpathy's llm-council

In late 2025, Andrej Karpathy published llm-council, a self-described Saturday hack he built to compare models while reading books. You ask a question and it runs a three-stage pipeline:

1. First opinions. The question goes to every model on the council; each answers independently.
2. Peer review. Each model reads the other answers, anonymized so there's no brand loyalty, and ranks them for accuracy and insight.
3. Chairman synthesis. A designated chairman model reads everything and writes the final answer.

Karpathy's repo runs on OpenRouter too; that's what makes a casual Saturday hack across four vendors possible at all. He shipped it with a note that he wouldn't support it and offered it "for other people's inspiration."

When I sat down one Saturday in May to build my own, I realized the version I wanted was smaller than the original.

My Version: Keep the Review, Cut the Rest

Karpathy's council answers questions. My problem was different: I already had the draft. What I kept getting burned by was acting on a plan that one model (usually the one that helped me write it) thought was great. A single reviewer, human or model, has blind spots, and the model that helped write a plan is in no position to grade it.

So I cut two of the three stages:

• Stage 1 (generation): gone. I already wrote the draft.
• Stage 2 (anonymized peer review): kept. This is the part that adds signal. Independent critics who can't see each other, can't see who else is on the panel, and are explicitly told not to praise.
• Stage 3 (chairman): replaced. Instead of a separate synthesis model, the coding agent already in my session does the clustering. It has the context of what I'm building; a chairman doesn't.

What's left is a simple tool: a single-file Python CLI, council-review, under 400 lines with zero dependencies beyond the standard library. It fans a file out to the council in parallel threads, collects the critiques, anonymizes them as Reviewer A through F, and writes one markdown file. That's it. The intelligence lives on either side of it: my draft going in, the clustering coming out.

Unzip it into your skills folder, add your OpenRouter key, and run it on any file:

# Install
unzip council-skill.zip -d ~/.claude/skills/

# One-time: add your OpenRouter key to ~/.zshrc
export OPENROUTER_API_KEY="sk-or-v1-..."

# Then, in any Claude Code session
/council plan.md

The zip holds three files: the CLI, its README, and the skill instructions that teach Claude the run-and-cluster workflow documented below.

How a Council Run Works

The invocation is one line, either directly or through the /council skill in Claude Code:

council-review INSPIRATION_PLAN.md --type plan \
  --focus "exploit metas, timescales, smallest shippable slice"

Every reviewer gets the same prompt scaffold, and it's engineered to produce critique rather than encouragement:

You are an anonymous peer reviewer. Critique the following plan rigorously.

Your job is to find what's wrong, weak, missing, or overconfident.
Be specific and direct. Do not praise. Do not summarize. Do not soften.

## Issues
- [severity: high|med|low] <issue> — <one sentence explanation>
## Missing
- <what isn't there but should be>
## Overconfidence
- <claims that need more evidence or are too strong>
## Verdict
<ship it / rework it, in a paragraph>

A few design choices that turned out to matter:

• Anonymity during, transparency after. Each model sees only the draft, not the other reviews or the roster. The output file reveals the identity key (Reviewer A → openai/gpt-5.6-sol, and so on) so I can weigh voices after the fact, but no reviewer can play to the room.
• Pinned models, checked every run. The council is six exact model IDs pinned in a config file, not floating aliases. Every invocation checks the pins against OpenRouter's live catalog and warns loudly if a model has gone missing or been deprecated. Model churn is real; silent substitution is worse.
• Failures don't sink the run. If one reviewer times out or errors, its section becomes an error block and the other five proceed.
• The synthesis is where the value is. The /council skill has Claude read the file and cluster every finding into three buckets: consensus (flagged by all or nearly all reviewers; act on these), contested (reviewers disagree, sometimes in opposite directions; my call), and outliers (a single voice; noted, rarely acted on). Six raw reviews are noise; the cross-reviewer pattern is the product.

A Sounding Board With Receipts

I've run the council on architecture plans, feature specs, essays, security postures, and one very strange game-design question. Three runs show the range.

The voice assistant teardown

This site has a voice agent. I handed the council a brief describing its voice-command navigation (an LLM-interpreted trigger manifest, a 500ms polling bridge, a confirm-before-navigate handshake) and asked what a modern rebuild would do differently. All four reviewers on that run converged on the same problem:

"Navigating to any internal page ends the voice call via full page reload — a catastrophic UX cliff that makes the voice assistant a dead end."

That consensus became a ranked roadmap, and the roadmap became shipped fixes. The last consensus item on the list was uptime alerting, and the code comment above it still reads "this is the last of the council's consensus items." The review doesn't end as opinions; it ends as a to-do list I work through.

The Dungeon Master question

For Propagate, my deterministic pixel farm sim, I asked the council something with no right answer: should an LLM "Dungeon Master" agent help run the game's battle set-pieces, and how much authority should it get? I gave them four escalating tiers to judge.

The council was unanimous on exactly one thing: the most ambitious tier, full LLM control of battle choreography, was rejected by every reviewer. Beyond that they earned their seats differently. The GPT reviewer produced the binding engineering catch: who returns and whom they target are game-world facts, so the nemesis must live in deterministic data with the LLM providing voice only. That's an authority boundary I'd been about to blur. A separate reviewer argued the brief's own ranking was backwards: the story payoff wasn't battle choreography at all, it was continuity, a named foe who comes back. That reversal became the build order. The feature shipped as "Krul the Howler returns — the third raid of his war," and it's one of the best things in the game.

I overrode the council once in that review: a reviewer wanted combat text cut entirely; I liked the text and fixed the pacing instead. That's the arrangement. The council advises, I decide.

The design brainstorm

Most recently, all six seats reviewed the spec for a feature where whispered suggestions leave "seeds" in a farmer's mind that may later germinate into action. The council caught an incentive inversion (the design rewarded refusing the player and gave nothing to a heeded-then-lapsed urge), a determinism hazard in float decay, and a self-blocking budget loop. Out of the contested pile came the design rule the feature now runs on: pressure damps germination. A nagged mind does not sprout. Optimal play became whisper-once-then-let-it-rest, which matches the fiction. I would not have gotten there alone, and no single model got there either; it came out of the disagreement.

The paper trail

The clearest evidence that this became a real workflow is that the git history of both projects now reads like council minutes. From Propagate's log:

1547256  Council brief: the counter-offensive (a town that strikes back)
9f5c92f  Phase 1 THE NEMESIS: the war gets a name
         (deterministic arc, LLM voices only)
e160322  Phase 3 ONE BEAT: the marquee duel gets its authored moment
         (council L2, starved as prescribed)
aa09ab7  Kimi-K3 review fixes (4 P0 + 2 P1) + battle HP bars

And from this site's repo:

67e3c58  Add uptime alerting for the voice stack
         (last council consensus item)

"LLM voices only" and "starved as prescribed" are council verdicts, executed. The pattern I've settled into: write the brief, convene the council, cluster, then work the consensus list top-down, committing against it item by item until a commit message can honestly say the last one shipped.

What Three Months of Councils Taught Me

Roughly three months of near-daily runs have settled into a few rules.

1. Capability diversity beats vendor diversity

One council run made this point for me: six API seats, six different labs, and every single one of them converged on the same critique of the same brief. Six vendors, one opinion. The only reviewer that produced anything new that day was a seventh seat I'd added as an experiment: a coding agent with access to the actual repository. It differed not by vendor but by capability: it could read the code the plan described and check the plan against reality.

So my default answer to "should we add model X to the council?" is now no; add a code-access seat instead. Frontier models trained on similar data reviewing the same brief will correlate. A reviewer with different evidence won't.

2. Six seats. Not seven.

When a new model earns a place (Kimi K3 did, after topping a frontend arena leaderboard), it swaps in for an existing seat rather than growing the panel. More seats blur the consensus signal: findings flagged by "most" reviewers stop meaning much, and synthesis cost grows faster than insight does.

3. Learn who wins which tiebreaks

After enough runs, patterns emerge in who's right about what. On my councils, the GPT seat has repeatedly delivered the decisive engineering catch (authority boundaries, latency realism, fallback parity), while the strongest story and product-feel judgment has come from elsewhere. So when consensus doesn't settle a question, I weight by track record per domain rather than treating six votes as equal. Your weights will differ; the point is to have them and to have earned them.

4. Popularity charts measure the wrong thing

It's tempting to staff a council from OpenRouter's most-used models list. Don't. That ranking is dominated by token volume from coding agents, which measures cheap-and-good-at-code, not good-at-critique. Reviewing is its own skill. Pick seats by probing candidates on a review task you know well.

5. Infrastructure fails quietly

My favorite bug in the project had nothing to do with AI. For a while, every reviewer silently returned a 401, because a five-character placeholder export OPENROUTER_API_KEY lower in my .zshrc was shadowing the real key. The CLI now refuses to run if the key doesn't look like a real OpenRouter key (sk-or-…), and fails loudly instead of politely. Multi-model workflows multiply your failure modes by N; make every failure loud.

Build Your Own in an Afternoon

You don't need a framework for this. The whole thing is five steps.

1. Get one OpenRouter API key

Sign up at openrouter.ai, buy a few dollars of credits, and put the key in your shell profile. This single key is every vendor on your council.

export OPENROUTER_API_KEY="sk-or-v1-..."

2. Pin a council

Around six models, different vendors, exact IDs. Mine, currently: openai/gpt-5.6-sol, google/gemini-3.1-pro-preview, x-ai/grok-4.5, deepseek/deepseek-v4-pro, moonshotai/kimi-k3, anthropic/claude-opus-5. Check your pins against OpenRouter's model catalog on every run so a deprecation never silently shrinks your panel.

3. Fan out with a structured critique prompt

The core of the whole system fits in a screenful of standard-library Python:

import json, os, urllib.request
from concurrent.futures import ThreadPoolExecutor

MODELS = ["openai/gpt-5.6-sol", "google/gemini-3.1-pro-preview",
          "x-ai/grok-4.5", "deepseek/deepseek-v4-pro",
          "moonshotai/kimi-k3", "anthropic/claude-opus-5"]

PROMPT = """You are an anonymous peer reviewer. Critique rigorously.
Find what's wrong, weak, missing, or overconfident. Do not praise.
Respond with: ## Issues (severity-tagged), ## Missing,
## Overconfidence, ## Verdict.

--- DRAFT ---
{draft}"""

def review(model, draft):
    req = urllib.request.Request(
        "https://openrouter.ai/api/v1/chat/completions",
        data=json.dumps({"model": model, "messages": [
            {"role": "user", "content": PROMPT.format(draft=draft)}
        ]}).encode(),
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}",
                 "Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        return json.load(r)["choices"][0]["message"]["content"]

draft = open("plan.md").read()
with ThreadPoolExecutor(len(MODELS)) as pool:
    reviews = list(pool.map(lambda m: review(m, draft), MODELS))

4. Anonymize and write one file

Label the critiques Reviewer A, B, C… and write them to a single markdown file, with the identity key at the top. Anonymity keeps you honest too: you read Reviewer D's argument on its merits before you know which lab it came from.

with open(".council-review.md", "w") as f:
    for i, (model, text) in enumerate(zip(MODELS, reviews)):
        f.write(f"## Reviewer {chr(65+i)}\n\n{text}\n\n---\n\n")

5. Cluster the findings

Hand the file to whatever agent you work in (mine is a Claude Code skill that does this automatically) with one instruction: don't summarize each reviewer in turn; the file already does that. Group every finding into consensus, contested, and outliers, order consensus by severity, then walk through them one at a time. The cross-reviewer pattern is the product.

What It Costs

Dimension	Value
Models per run	6 (pinned, six vendors)
Typical wall-clock	20–60 seconds, gated by the slowest reviewer
Typical cost per run	~$0.10–$0.40 via OpenRouter, by draft size
Code	One Python file, standard library only
API keys to manage	1
Output	One markdown file

That price is worth pausing on. A structured, adversarial, six-perspective review of a document you're about to spend days or weeks executing costs a few tens of cents and returns in under a minute. The council has caught plan-killing flaws (inverted incentives, authority leaks, unbuildable timelines) for forty cents a run. Nothing else in my toolchain returns that much for that little, and it exists because OpenRouter collapsed six vendor relationships into one API call.

FAQ

What is OpenRouter used for?

OpenRouter gives developers one API endpoint, one key, and one bill for hundreds of AI models across providers like OpenAI, Google, Anthropic, xAI, DeepSeek, and Moonshot. It's used to switch models without code changes, avoid vendor lock-in, and, as in this article, build multi-model workflows that would otherwise require an account with every lab.

What is an LLM council?

A group of AI models from different vendors that independently respond to the same input, with the outputs then compared, ranked, or synthesized. The pattern was popularized by Andrej Karpathy's llm-council project. The premise: models from different labs have different blind spots, and their agreement is a stronger signal than any single model's confidence.

How is this different from Karpathy's llm-council?

His pipeline generates answers, has the models rank each other's answers, and ends with a chairman model writing a synthesis. Mine drops generation (I wrote the draft), keeps the anonymized peer review, and replaces the chairman with the coding agent already in my session, which clusters findings into consensus, contested, and outliers.

Does the council actually disagree, or do the models all say the same thing?

Both, and each is informative. Full consensus on a flaw is the strongest "fix this" signal I have. But models trained on similar data do correlate (I've had six vendors return one opinion), which is why the highest-value seat I've added wasn't a seventh model but a reviewer with access to the actual repository.

Which models should sit on a council?

Around six, from different vendors, pinned to exact versions, chosen by probing candidates on review tasks you know well rather than by popularity rankings (which mostly measure coding-agent token volume). When a new model earns a seat, swap it in; don't grow the panel.