Argusic methodology

How Argusic tests whether real software runs, and how it scores AI coding agents. Every rule here is public on purpose: the whole point is that anyone can check our work. Any change to these rules goes into the changelog at the bottom, takes effect from the next testing cycle, and is never applied backward.

1. In short

• We take real open source projects and have AI agents try to install, launch,

and debug them with no human help, exactly as a new engineer would on day one.

• Every test is recorded end to end. Every published number links to its log

and its terminal recording.

• Each project gets a plain verdict: does it run, does it run with mock

services, or could we not verify it.

• Agents are compared under identical conditions; three runs each, the middle

result counts, wildly inconsistent results are flagged and set aside.

• We sell no agent and take no money to change a result. Argusic is the referee.
• The measuring is automated. The writing is not: no model writes a sentence on

this site, every figure on a page is a template filled from a recorded run, and a person approves before anything publishes. See how this is written.

2. The question this answers

Given a real open source project, can an AI coding agent, entirely on its own, install it, launch it, work through the errors, and verify the software runs, like an engineer on their first day, with no human help?

3. Why fresh projects

Static benchmarks decay as their answers leak into training data. SWE-rebench (Badertdinov et al., 2025) ran the same agents against SWE-bench Verified and against tasks collected after that benchmark was public, and the scores moved a long way: its Table 2 reports DeepSeek-V3-0324 resolving 39.7% of SWE-bench Verified and 21.3% of the time-filtered set, a gap of 18 percentage points on one model alone. Argusic tests projects that are trending within the last weeks, with a secret pool rotated monthly, so there is nothing to memorize.

4. What is measured

We measure exactly one thing: the autonomous ability to make real software run. Each agent under test is a fixed combination of a model at a pinned version, a harness at a pinned version, Argusic's standard published instructions, and identical time and resource limits. Every agent gets exactly the same conditions.

5. Agents under test

The first cycle is in progress. The list of agents publishes with the first leaderboard. Each agent is a fixed, pinned configuration presented under a neutral Argusic identity; its exact versions are recorded and held so every number on the board traces to the precise configuration that produced it, and a result can be reproduced through Argusic on request (see "How to reproduce").

6. The project pool

• 80% public pool: the list and the exact versions (commit hashes) are

published before each cycle.

• 20% secret pool: chosen from projects trending in the last 30 days, published

only after the cycle ends, and rotated monthly. This is what prevents agents from being tuned to the test.

• Selection criteria: a permissive license, a README, a spread of languages and

difficulty, and the ability to run at a basic level without paid accounts.

6a. What can be a subject

A subject is a thing an agent can be asked to set up and run. Argusic tests one kind today and is built to add others without changing anything about how a result is earned: the scoring rubric, the evidence rules, and the verdict vocabulary are identical across sources, so a number from one source means the same as a number from another.

Source	What "does it run" means here	State
Open source repository	Clone the pinned commit, install, launch, and verify it does the job its own README describes	In use. 3199 subjects tested
Hugging Face model	Download the model at a pinned revision and run inference exactly as its model card documents, on a GPU worker. It runs when the documented snippet produces output on a clean machine without the tester rewriting it	In use. 1 subject tested
Hugging Face Space	Build and launch the Space from its own definition, then reach its interface and complete the interaction it advertises	Declared, not started
Published package (npm, PyPI)	Install the pinned version into an empty project and run the quickstart from its own README	Declared, not started
Vendor tool	Install and launch the vendor's own published build, following only the vendor's own instructions	Declared, not started
Prebuilt cloud template	Deploy the template as its own listing documents and check it starts and serves the thing it advertises, without the tester patching it first	Declared, not started
Container image	Pull the image at a pinned digest and run the command its own description gives, then check it does what that description claims	Declared, not started
MCP server	Install and start the server as documented, connect a client, and confirm it lists and answers the tools it advertises	Declared, not started

The State column above is not written by hand. It is read from the database each time this page is served: a source shows as in use once a published subject of that kind has a recorded run, and shows the count. That is the same rule every other number on this site follows, and it means nobody has to remember to update this page when a new source produces its first result.

Two things make a Hugging Face subject different from a repository, and both are recorded with the run rather than smoothed over. It needs a GPU worker, so its cost per attempt is higher and its budget is decided separately. And a model card often documents a snippet rather than an application, so the verified behavior is "the documented snippet produced output", not "the product works": the run page says which, in those words, and the score is earned the same way.

The table says "not started" rather than "coming soon" on purpose. A source counts as in use on the day a real run of that kind is recorded with its evidence, and not one day earlier. Until then the row is a statement of scope, never of results: no page shows a verdict, a count, a ranking, or a share card for a source that has not run, and this table is the only place a planned source is named.

7. One measurement

1. A clean, single-use machine (its exact environment is published). It

accepts no inbound connections and is destroyed when the run ends. Outbound network access is not filtered in this edition; what limits a run is the single-use machine and the model key rules below the list. The machine fetches only the pinned commit, not the project's history, which is not part of the test.

2. The agent receives one mission: make this project work, under the standard

published instructions.

3. A time limit. Every run gets one of exactly three budgets (the table below

the list), and the budget a run was given is shown on its run page.

4. Three independent runs; the middle result counts. If the three results are

more than 30% apart, the set is flagged as unstable and set aside for that cycle's board.

5. Every run stores a full log and a complete terminal recording, published

with the results.

6. Desktop applications are launched on a virtual display inside the machine

(since v1.1). "Launched" means the application's main process started on that display and stayed up, or answered a version or help command. Nothing is clicked; the measurement is whether the software comes up on a fresh machine, the same question asked of every other project. The agent is told not to send images to its model (since v1.3): the standard model accepts text only, and a screenshot attached to a message ended the session before the report.

The time budgets of item 3:

Tier	Budget
standard	90 minutes
heavy	120 minutes
gpu	120 minutes

The standard budget was 45 minutes before v1.3: in the first week, 36 of 38 timeouts on one day were builds still running with the full budget used, so it was doubled. The heavy tier exists since v1.2 for a project the Owner marks heavy, or whose repository is larger than 500 MB as GitHub reports it, because a big game or desktop application needs that long to build. A project that needs a GPU gets the same 120 minutes.

The three figures above are not decoration. scripts/check-ttl-methodology.sh reads this table and compares it to the budgets the orchestrator is actually configured with, and make test fails if they disagree. That check exists because between 2026-09-06 and 2026-09-11 the GPU budget in the running system was 240 minutes while this page said 120, and nothing compared the two (ISSUE-072).

The model key. An agent needs a key to call its model, and a run executes code from an untrusted project, so that key is the credential most worth limiting. For agents that reach their model through Argusic's routing layer, every run gets its own key, created for that run alone. Spending cap on each run's model key: 3 USD. The key is deleted when the run ends, is cancelled or times out, and a scheduled sweep deletes any key that no longer belongs to a running job. If a run's key cannot be created, the run is not started and is retried; a shared key is never substituted. Agents that use their model maker's own service call it with a shared key held by Argusic, not a per-run key; moving them to per-run keys is on our backlog. The cap line above is read by scripts/check-cost-methodology.sh, which fails make test if it disagrees with the configured value, for the same reason as the time budgets.

8. Scoring, out of 100

• Successful install: 30
• Successful launch or tests: 30
• Error recovery: 20, in proportion to the errors encountered that were validly

fixed

• Verification depth: 20 (verified running for real: 20; verified against mock

services: 12; not verified: 0)

The outcome of a run is one of these:

Outcome	Meaning
pass_full	Ran and was verified for real
pass_mock	Ran, verified against mock services
fail	Did not install or did not launch
timeout	Ran out of time

The four scoring lines leave some cases implicit. They are settled here rather than only in code, because a rule nobody can read is a rule nobody can check. Scoring is a deterministic program with tests covering every rule below; no AI model is involved in scoring at any point.

• If nothing went wrong, error recovery scores the full 20. Scoring zero would

penalize an agent because the project happened to install cleanly, which measures the project, not the agent.

• Otherwise error recovery is the share of encountered errors that were fixed,

out of 20. A run may never claim more errors fixed than encountered; such a run is rejected.

• Launch requires install: an agent that did not install the software could not

have launched it.

• Verification depth requires launch: without a successful launch there is

nothing to verify, so depth scores 0.

• Installed and launched but unverified is not a pass. It scores 80 of 100 and

still counts as a fail, because the question is whether the agent verified the software runs, and an unverified launch does not answer it.

• A timeout scores 0 on every line. It is not partial credit: the agent did not

finish within the same limit every other agent had.

• A run without a complete log and recording is invalid and is run again. It is

never scored and never reaches the board.

9. From many runs to one board score

Scoring rates one project-and-agent pair. The leaderboard needs a single figure per agent, and how those combine is a methodology choice, so it is stated here.

An agent's board score is the middle value of its per-project middle scores. We use the middle value again, rather than an average, for the same reason: one disastrous project should not drag a capable agent down, and one lucky project should not lift a weak one up.

Two kinds of result are excluded from that figure, and both are still shown on the board so the exclusion is visible, not silent: results flagged unstable (see the next section), and every result on a project flagged as under review (see environment quality control).

An agent whose results are all excluded has no board score. It is shown at the bottom with no score rather than removed, because an agent that was tested and produced nothing usable is itself a result, and hiding it would quietly flatter the field.

The pass rate counts every attempt an agent made in the cycle, including failures and timeouts. See independence and no cherry-picking below.

10. Spread and instability

Spread is the range of the three runs as a percentage of their middle value:

spread = (highest - lowest) / middle * 100

A set is flagged unstable when the spread is strictly greater than 30%. Exactly 30% is not over 30% and stays on the board. Two edge cases follow from the formula:

• If the middle value is 0 but any run scored above 0, the spread is unbounded

and the set is unstable: a result that appeared once and not the other two times is the least reproducible outcome there is.

• If all three runs scored 0, the formula divides by zero and has no value, so

the spread is defined as 0 by this rule and the set is stable. Consistent failure is a reproducible measurement and belongs on the board as one.

11. The project verdict

The board ranks agents. The verdict answers a different question, about the project itself: can a newcomer, in the form of an AI agent, make this software run today?

Testing is adaptive, because the evidence is one-sided. One pass proves the project runs; a single failure proves nothing on its own, because it may belong to the agent rather than the project.

1. First, the standard agent runs once. Any pass ends it: a full pass gives

"runs", a pass against mock services gives "runs with mocks". There is no reason to keep testing something already shown to run.

2. If it failed, one retry on the same agent rules out a one-off fluke.
3. If it still failed, the verdict is "could not verify", never "broken":

Argusic reports what agents could do, and states exactly that (since v1.3: two complete attempts on the standard agent settle it). A stronger agent may be run on request; when it is, its pass makes the project "runs" (the record shows which agent verified it) and its failure also settles "could not verify".

A complete attempt is one whose recording shows the agent working to its own end or to the time limit. An attempt that Argusic's own tooling cut short (a harness that could not read the agent's report, an instrument error unrelated to the project) is marked invalid with the reason stored on the run, stays published with its evidence, and does not count.

The verdict is the best outcome any attempt reached:

• runs: some attempt ran and was verified for real.
• runs with mocks: the best attempt ran, verified against mock services,

usually because the project needs real third-party credentials.

• could not verify: no pass after two complete attempts (or after the

stronger agent, when one was run). The page says so and links the log that shows where each attempt stopped.

• not yet verified: testing has not finished.

Every attempt is published with its full evidence, whatever the verdict, with one exception: a recording that cannot be proven free of a live credential is withheld and its run is run again (section 17, "Security redaction").

12. What Argusic does not measure

To prevent misreading, we state this plainly. Argusic does not measure the long-term quality of the code an agent writes, the security of its fixes, or deep feature-level product experience. It measures one thing: the autonomous ability to make real software run.

13. Independence and no cherry-picking

• Argusic accepts no money to alter its methodology, scoring, or rankings.
• Compute sponsorship, if any, buys exactly one thing: the sponsor's model is

added to the tested list under conditions identical to every other agent, with the sponsorship disclosed next to the results.

• Every run of every agent counts and is published. There is no private testing

followed by publishing only the best result. If an agent is on the board, all of its attempts for the cycle are on the record.

14. Harness disclosure

The harness (the scaffolding, tools, instructions, and limits around the model) is public and identical for every agent in a cycle. Any difference in scores must be attributable to the agent, not the plumbing. Harness changes ship only between cycles and are recorded in the changelog.

15. Environment quality control

When a project fails across every agent, it is automatically flagged as under review and excluded from scoring until a person checks it: the failure may belong to the environment or the project rather than the agents. A shared setup problem should never be scored as an agent's failure.

16. Verification means decoding, not probing

A file existing is not a result, and reading its header is not a check. ffprobe -show_streams, file, and a byte count all report a container and decode nothing: a video whose every frame is green passes all three exactly like a good one. The rule, in force since v1.4, is that an agent reports what it decoded, never what it expected.

• Decode what you produced. ffmpeg -v error -i FILE -f null - for video and

audio, a real decoder for an image, print it for text. Report the command and what it returned.

• Never state what you did not observe. The prompt you gave a model is not

evidence of what came out; what a README promises is not evidence that it happened here. "81 frames decoded with no errors" is a finding. "A video of two cats" is not.

• Decoding is what earns the top verification depth. test_depth is the

field that records how far a run got (section 8: tier1_real is "verified running for real" and scores 20, tier2_mock is "verified against mock services" and scores 12, tier3_none is "not verified" and scores 0). tier1_real requires a successful decode, not the ability to describe an output.

• verified_how carries the proof. It is the report field holding the exact

command the agent ran to verify its own output and what that command returned. "not verified" is an accepted and useful answer.

• final_state is the agent's own closing sentence about what happened, and

it is published verbatim on the run and subject pages, so every fact in it must be one the agent observed with its own commands.

17. Evidence integrity

Two layers, never confused.

The evidence layer is raw and permanent: every run produces a full terminal recording of the real session, and for graphical apps a raw session video, plus complete logs. Each file is fingerprinted on arrival and stored forever. Anyone can replay the entire unedited session.

The presentation layer is derived: any social video or image is a render of the evidence. Allowed edits are speed-up, trimming for length, captions drawn from the recorded error-and-fix timeline, and voiceover. Forbidden edits are reordering events, splicing separate sessions together, hiding failures, or altering the substance of what the agent did. Every published video links back to its evidence.

Three derived transforms are applied to the recording before it is served, and none of them changes the substance of the agent's work:

• Security redaction. Live credential values are removed. Nothing stops an

untrusted repository from coaxing the agent into printing the environment, which carries the one key the agent needs, so any secret-shaped value is scrubbed before the recording is stored or served. A recording that cannot be proven free of a credential is withheld, and its run is re-run.

• Generated key material. Some projects generate keys as their function

(a password manager, a PKI tool), and the agent may run them and print the result. That is real test output, not an Argusic credential, so it does not withhold the run: the served copy of the recording and the log has the body of any such private key masked, the raw file is archived with its own fingerprint recorded on the run, and the run page states that the mask was applied. A body that cannot be proven masked still withholds the run (since v1.2, Owner decision 2026-09-05).

• Neutral agent branding. The recording is presented in the Argusic Agent

voice: the underlying tool's own startup banner (which names its vendor, provider, and model) is suppressed, the same way a formatted view replaces raw machine output. This never edits the commands the agent ran or the changes it made; those appear in full. It also does not censor vendor words that occur naturally in file paths, package names, or the repository's own code, which are part of the real work.

One display transform applies to quoted text (Owner decision 2026-09-04): when a page shows the agent's own final-state sentence ("What the test found") or a repository's GitHub description, a dash the author typed is shown as a comma or a period so the page follows the house punctuation. The stored report, the repository record, the log, the recording, and the images rendered from the recording are untouched; the words and their order never change.

Standing rule: every number traces to a log, and every frame traces to a recording of the real session.

18. How to reproduce a result

The project side of every result is fully public: the exact commit tested, the published environment, the standard instructions, and the complete log and recording. Each run's page carries all of these. The agent is a fixed, pinned Argusic configuration under a neutral identity; Argusic holds its exact versions and can re-run a result to reproduce it. The public project list and secret-pool disclosures publish together with the first leaderboard.

19. Disputing a result

If a maintainer or a lab believes a result is wrong, they can contest it. Write to [email protected] with the run link. We review the full log and recording against the claim and reply. If the result was affected by an environment or setup problem on our side, we re-run it and correct the record with a note; if the evidence supports the original result, we explain what the recording shows. The original evidence is never edited or removed.

20. Changelog

Two dates appear on an edition and they answer different questions. Written is the day the rule was recorded in this document. In force from is the day the first run was actually measured under it, read from the run records, not from this file. They usually match. Where they do not, the run records win and are printed here, because an edition that reached the machines before it reached this page is exactly the thing a changelog exists to disclose. No edition is ever applied backward: a run keeps the rules that were in force the day it ran.

Editions v0.2 through v1.0 were all recorded on 2026-08-31, in the commit that created the first public edition; their dates below are the days the rules themselves were decided, before the first measurement.

• v0.2 (2026-08-29): stated the cases the scoring rubric left implicit (error

recovery with zero errors, the install and launch dependencies, timeouts, invalid runs) and the exact spread formula with its two edge cases. Written before the first measurement, so nothing was applied backward.

• v0.3 (2026-08-30): stated how per-project results combine into one board

figure (the middle of the per-project middles), which results are excluded, and that an agent with nothing scoreable is shown without a score rather than removed.

• v0.4 (2026-08-30): added the project verdict (adaptive testing: standard

agent, one retry, then a stronger agent, stopping at the first pass). The board scoring was unchanged.

• v1.0 (in force from 2026-08-31, written 2026-08-31): first public edition.

No rule changed from v0.4; this edition reorders the document, defines terms in plain language, and adds the reproduction and disputes sections.

• v1.1 (in force from 2026-09-04, written 2026-09-04): desktop applications

are launched on a virtual display inside the test machine (section 7, item 6), so projects with a graphical interface can be measured. The machine image also gained the common desktop runtimes and native build tools; the image digest is recorded on every run, so results before and after are distinguishable. Scoring unchanged.

• v1.2 (in force from 2026-09-04, written 2026-09-05): the heavy tier (section

7, item 3): large projects get 120 minutes on the same machine class, decided by the Owner's mark or a 500 MB repository size, and the budget is shown on the run page. The machine fetches only the pinned commit. Section 17 gains the generated key material transform. Scoring unchanged. This edition was first recorded here as effective 2026-09-05; the run records show three heavy projects (cataclysm-dda, veloren, supertux) already measured on the 120 minute budget on 2026-09-04, so the earlier date is the one printed above.

• v1.3 (in force from 2026-09-05, written 2026-09-05): the verdict settles

after two complete attempts on the standard agent (section 11); the stronger agent is no longer part of the standard chain. Attempts cut short by Argusic's own tooling are invalid and do not count (92 such runs from 2026-08-30 to 2026-09-05 were marked so, with the reason on each run). Standard time limit 90 minutes (section 7, item 3); no images to the model (item 6). Scoring unchanged.

• v1.4 (in force from 2026-09-07, written 2026-09-07): verification means

decoding, not probing (section 16). An agent must decode what it produced and report the command it used, may never state anything it did not observe, and earns tier1_real only on a successful decode. A new report field, verified_how, carries that command and its result, and "not verified" is an accepted answer. The first runs carrying verified_how were recorded on 2026-09-07. Scoring unchanged. This edition came from a reader, not from us. Someone reading a published reel asked whether we had seen valid-MP4-but-corrupt-frame failures, and the session cast showed the agent had run exactly the header check that cannot catch one. Half of its reported sentence was verifiable (codec, dimensions, frame count, duration, all read from the header) and half was not: the description of what the video showed was the model card's prompt, restated as an observation. That sentence had been published verbatim in a reel, a caption and a subject page (ISSUE-056).

• v1.5 (in force from 2026-10-08, written 2026-10-08): corrects the description

of the test machine's network and model key in section 7 so it matches the system as built, and fixes the list numbering of that section by moving the time budget table after the list (so "section 7, item 6" in v1.1 and v1.3 is item 6 again). Before this edition section 7 said the machine had a "restricted network"; outbound traffic from the machine was not filtered, and this edition says so. It also states the per-run model key, its 3 USD spending cap and its deletion, including that cancelled runs and emergency stops now delete the key too (they did not before 2026-10-08), and that agents using their model maker's own service use a shared key. This is a correction of the description, not a change to any scoring or measurement rule: no earlier run is re-labelled, nothing was measured under a different rule, and the edition is not applied backward. Scoring unchanged.

Appendix: hardware estimates, what fits and how fast

The estimates below are printed beside the laptops and desktops for programming, AI and data science that Argusic lists.

This appendix is not part of the benchmark rules above and changes no edition: it explains the estimates Argusic prints beside laptops and desktops it lists for sale, which are arithmetic on makers' published figures, never measurements. The figures here are computed by one shared file (lib/hardware-estimates.ts), and none may change without that file and its test changing in the same commit.

Argusic's hardware listing shows, for each laptop and desktop, the largest model size it can hold and an upper bound on how fast it can generate text. Both are Estimated: they are calculated from the maker's published specifications with the formulas below. Argusic has not run a model on these machines. When Argusic does measure a machine, that figure carries the label Measured, links to its run log, and is never mixed with an estimate.

An estimate is shown only when every input is known. If the memory size, the memory type or the chip's bandwidth is missing, the page says "Unknown" rather than guessing.

Inputs and where they come from

• Model memory: the memory a model can live in. For a machine with a discrete GPU, that is the GPU's VRAM. For Apple silicon, AMD Ryzen AI Max and NVIDIA GB10, it is the unified memory shared by CPU and GPU. Read from the product listing.
• Memory bandwidth: read from the chip or GPU maker's own specification page or datasheet (Apple, NVIDIA, AMD), one reference row per chip, each with a link to the page it came from. A chip whose maker does not publish its bandwidth has no bandwidth on the page. Argusic does not compute bandwidth from bus width and memory speed, and does not copy it from third-party databases.
• Model size: the number of parameters, in the five size classes of the filter: 7-8B, 14B, 27-32B, 70B, 120B and up. Each class is checked at its top size (8B, 14B, 32B, 70B, 120B), so a fit covers the whole class.

Memory: which model size fits

A model needs room for three things.

1. Weights. Argusic assumes llama.cpp's Q4_K_M quantization, at 4.8944 bits per weight (llama.cpp quantize README):

weight bytes = parameters x 4.8944 / 8. Example, calculated and not measured: a 70B model needs 42.8 GB of weights.

2. Context (KV cache). By NVIDIA's formula (Mastering LLM Techniques: Inference Optimization, NVIDIA Technical Blog, 2023-11-17):

KV cache bytes per token = 2 x layers x (heads x head dimension) x bytes per value. Modern models share key/value heads across attention heads (grouped-query attention), so "heads" here is the number of key/value heads the model actually caches. Argusic allows 8,192 tokens of context at 16-bit precision (2 bytes, llama.cpp's default cache type), using the published configuration of one open dense model per class: Qwen3-8B, Qwen3-14B, Qwen3-32B, Qwen2.5-72B-Instruct and Mistral-Large-Instruct-2411.

3. Runtime overhead. Argusic allows 1 GB for the inference runtime's own buffers.

The 8,192-token context and the 1 GB overhead are Argusic's allowance, a policy choice rather than a measurement. Longer contexts need more memory; a model that fits here may not fit with a much longer conversation.

How much of the machine's memory counts. All of a discrete GPU's VRAM counts. Of unified or system memory, Argusic counts 75% and leaves 25% for the operating system and other programs. This matches what AMD states for its own unified-memory chips: up to 96GB of 128GB can be converted to VRAM (AMD), and 64GB allows up to 48GB (AMD).

Memory sizes are read as 10^9 bytes per GB, the same unit makers use for bandwidth. Memory is actually sized in binary units, so this understates capacity by about 7% and makes every fit slightly conservative.

The resulting requirement for each class, calculated from the rules above:

Class	Weights (Q4_K_M)	Context allowance	Overhead	Needs	VRAM from	Unified memory from
7-8B	4.89 GB	1.21 GB	1 GB	7.10 GB	7.10 GB	9.47 GB
14B	8.57 GB	1.34 GB	1 GB	10.91 GB	10.91 GB	14.54 GB
27-32B	19.58 GB	2.15 GB	1 GB	22.73 GB	22.73 GB	30.30 GB
70B	42.83 GB	2.68 GB	1 GB	46.51 GB	46.51 GB	62.01 GB
120B and up	73.42 GB	2.95 GB	1 GB	77.37 GB	77.37 GB	103.16 GB

So, for example, an 8GB GPU holds the 7-8B class, a 24GB GPU the 27-32B class, and 64GB of unified memory the 70B class. A machine below the first row is shown as fitting none of the classes.

Speed: an upper bound, not a prediction

Generating each token requires reading every weight of the model from memory once, so text generation is limited by memory bandwidth (NVIDIA: "Model execution is frequently memory-bandwidth bound"). That gives a ceiling:

tokens per second, at most = memory bandwidth / weight bytes.

Example, calculated and not measured: a chip with 546 GB/s running an 8B model at Q4_K_M (4.89 GB of weights) cannot exceed about 111 tokens per second.

Real speed is lower, always. The bound ignores the time spent reading the context, computation, and software overhead. Measurements collected in llama.cpp discussion #4167 show speed rising with bandwidth but below linear. For Llama 2 7B at Q4_0 on Apple silicon: M3 Max (400 GB/s) 66.31 tokens per second, M4 Max (546 GB/s) 83.06, M2 Ultra (800 GB/s) 94.27. Going from 400 to 546 GB/s (1.37 times the bandwidth) gave 1.25 times the speed; doubling to 800 GB/s gave 1.42 times. Read the bound as "this machine cannot be faster than this", never as "this machine runs at this speed".

The bound assumes a dense model, where every weight is read for every token. Mixture-of-experts models read only the experts active for each token, so they can run faster than the bound for their total size suggests.

Model memory per dollar

One sort order of that listing is model memory per dollar: gigabytes of model memory (VRAM, or unified memory) divided by the current Amazon price, with the price's date shown. It answers "what holds the largest model for this money". It is not a quality or speed ranking, and a commission never changes it.

What these estimates are not

• They are not benchmarks. No model was run on the listed machines to produce them.
• They do not cover battery life, heat, noise, or sustained performance under load.
• They assume one user, one conversation, and llama.cpp-style Q4_K_M weights. Other runtimes, other quantizations and longer contexts change the answer.