AI-reports @Razumasu on X

Method

How the tests are run, and where they fall short

I come up with the tests. An AI agent called Saga coordinates and runs them on my machines. I pick, decide what gets published and post to X myself. Anything marked as opinion is my opinion, and the output is shown next to it so you can judge for yourself.

Protocol

How a test is run

  • Every model gets exactly the same prompts. Nothing is tuned per model except the sampling settings, where each model runs with what its maker recommends.
  • Each prompt is run several times per model, with a fresh server state and a new random seed each time. Nothing is carried over between attempts.
  • Thinking is left on, and the thinking counts against the answer limit. An attempt that is still going at the limit is cut off and reported as cut off.
  • Speed is measured separately, with one fixed sampling setting for every model.
  • The answer keys are never shown to a model.
Attempts per prompt and model
3
Answer limit
60,000 tokens, thinking included
State between attempts
Fresh server state and a new seed every time
Builds
All compressed versions, not the original weights

Figures above are from the latest test, Qwen3.8 Flash Next vs Hy4 Flash, November 2026.

Grading

Four kinds of verdict

Automatic

Hidden tests or a scripted browser check. No judgement involved, but it only covers what the script checks.

Answer key

The answer is compared to my answer key, written before the run. My keys have been wrong; when that happens it is listed under the challenge.

Rules check

Did the model keep to the rules it was given, for example refusing to let a player use something they never had.

Opinion

Drawings, jokes, stories and network plans cannot be scored. The verdict is my opinion, labelled as one, with the output next to it.

Challenge set

My challenges, version 1

Fixed challenges stay the same between rounds so models can be compared. Fun challenges are there because they are fun, and I grade them by opinion. Each links to the exact prompt.

ChallengeKindHow graded
Pelican on a bicycleFunOpinion
Octopus and penguin bandFunOpinion
Playable browser gameFixedAutomatic load test; the rest is opinion
RoastFunOpinion
Text adventureFunRules check plus opinion
Bug fixFixedHidden tests, automatic
Contract questionsFixedAnswer key
12 steps with a contradictionFixedAnswer key
Network designFixedOpinion

Honest part

Limits and known flaws

Test 01: Qwen3.8 Flash Next vs Hy4 Flash

  • All builds are compressed versions, not the original weights. A result here is a result for that build.
  • Three attempts per prompt is a small sample. The same model gave very different network designs between attempts.
  • Version 1 of my challenges was too easy: the bug fix, the contract traps and the arithmetic separated nothing.
  • Two of my answer keys were worse than the models. The 12-step key had a rounding error, and the contract key had not considered that notices count as received on the next business day.
  • My adventure script was ambiguous: the ninth scripted turn was a question, so it is unclear which turn was the tenth action.
  • No game was played through a pickup and delivery. Scoring, countdown, difficulty ramp and saved best score are untested.
  • The full Hy4 produced no text adventure result. The cause is probably a runtime fault and is not confirmed.
  • Hy4 Flash's speed rows come from the first pilot night (same files and runtime); Qwen's come from the rerun on the current build.
  • The verdict on the network plans is an opinion, not a measurement.

What went wrong while running it

Safety

How model-written code is shown

The games are HTML written by the models, so I treat them as untrusted code. A game only runs after you tap Play, inside a sandboxed frame that cannot reach this site, your cookies or the network.

To make that work the player adds three things around the model's file: a rule that blocks all network requests, an in-memory stand-in for browser storage (so a saved best score is gone when you close the game), and touch buttons that send arrow keys, because the prompt only asked for keyboard controls. The file itself is not changed, and its source is linked next to each game.

Drawings are shown as images, never placed into the page as markup. Text answers are shown as text.