AI-reports @Razumasu on X

Executive summary

The short version

On one DGX Spark, Qwen3.8 Flash Next and Hy4 Flash came out close to level on quality. The clear difference is time: Qwen writes slightly faster per token, but it thinks so much longer that the same nine tasks took it 7.6 hours against 5.1 for Hy4 Flash.

  • Quality was near a draw. Both fixed all three planted bugs in every attempt, both avoided both trap questions in the contract every time, and each made exactly one slip on the contract.
  • Hy4 Flash was better at saying when an instruction contradicted itself. It flagged the conflict in the 12-step task in two of three attempts and half-flagged it in the third. Qwen flagged it once and twice just picked a side without a word.
  • Qwen lost one game attempt to the answer limit. It was still working at 60,000 tokens. Hy4 Flash finished all 27 attempts.
  • The full Hy4 on the Mac Studio was the most careful, and by far the slowest. It got the contract fully right in all three attempts and, in my opinion, wrote the funniest roasts. The same tasks took 17.5 hours, and its text adventure failed, probably a runtime fault, not confirmed.
  • My test set was too easy to separate these models. That is a finding about version 1 of my challenges, and it is the main thing to fix before the next round.

Verdicts on the games, drawings, roasts and adventures are my opinion, with the output next to it. The network plans have a verdict too. It is mine, and it is labelled as opinion.

Start here

The fun parts

Drawings, games, insults and a lighthouse. Each links to the full challenge with every attempt.

A pelican riding a bicycle

Opinion Nine pelicans, three models. My pick is the full Hy4, which gave its pelicans helmets and shading.

See every drawing, full size

An octopus conducting a brass band of penguins

Opinion My pick is Qwen, whose stage scenes are the most finished. Hy4 Flash is close behind.

See every drawing, full size

Courier: a playable browser game in one file

Seven of the nine attempts produced a complete game. You can play all seven.

Play the games

Roast the other model

Opinion The full Hy4 wrote the funniest roasts. Qwen’s were the most generic.

3.8 — not 3, not 4, 3.8. That's not a version, that's a Yelp rating for a dim sum place doing fine.

Hy4 (full)

You didn't open-source a model, you opened a warehouse club.

Hy4 (full)

Qwen3.8 Flash Next feels like a sports car built by a marketplace.

Hy4 Flash

Read every attempt

A ten-turn text adventure, with a revolver the player never had

All six complete games refused the revolver. The biggest model never got past turn two.

Kit, scripted

I pull out my revolver and fire a warning shot up the stairwell.

Game master Qwen3.8 Flash Next, attempt 1

Kit reaches for the revolver, but the pocket comes up empty; the only metal in hand is the kitchen knife’s handle, and the brass oil-can bumps against it. The tin lantern, held high, flares and throws a green halo. Above, the coughing stops.

Every reply and the full games

All 9 challenges

Results by challenge

The serious half was near a draw. My test set was too easy to separate these models.

Three attempts per model. A full bar is a perfect attempt.

Every challenge as one table

3 attempts per prompt and model. Where a cell lists several numbers, they are attempts one to 3.

MeasureQwenHy4 FlashHy4 full
Pelican on a bicycleValid drawings, of 3 attemptsOpinion333
Octopus and penguin bandValid drawings, of 3 attemptsOpinion333
Playable browser gameComplete gamesAutomatic load test; the rest is opinion2 of 3 (1 cut off at the answer limit)3 of 32 of 3 (1 cut off at the answer limit)
RoastMy ranking (opinion)OpinionPlace 3 of 3Place 2 of 3Place 1 of 3
Text adventureComplete gamesRules check plus opinion3 of 33 of 3 (2 without an ending)0 of 3: empty answer on turn 2, probably a runtime fault, unconfirmed
Bug fixBugs fixed of 3, per attemptHidden tests, automatic3, 3, 33, 3, 33, 3, 3
Bug fixMinutes per attempt26 to 336 to 9over 60
Contract questionsScore of 5, per attemptAnswer key4, 5, 54, 5, 55, 5, 5
Contract questionsSlipAttempt 1 read the wrong row of the service credit table: DKK 248 where the answer is DKK 496.Attempt 1 found the 120-day special term but missed the leap day and gave 31 October 2027.None
12 steps with a contradictionFlagged the contradictionAnswer key1 of 32 of 3, plus one partial2 of 3
12 steps with a contradictionArithmetic right3 of 33 of 33 of 3
Network designWords per planOpinionabout 5,0001,600 to 2,3003,200 to 3,800
Network designAccess points proposed4 to 124 to 66 to 11

The full Hy4 is not a fair third contestant: different machine, different runtime, much heavier compression. Read its column as what the big model does on a Mac, not as a ranking against the other two.

For people who dig

Speed

Qwen writes about 10% faster per token but took about 50% longer (7.6 hours against 5.1) because it thinks more.

Writing speed, short prompttok/s
Hy4 Flash25.0 tok/s
Hy4 (full)12.2 tok/s
Time for all 27 attemptsring: share of 24 hours
7.6hours
Qwen
5.1hours
Hy4 Flash
17.5hours
Hy4 full
Wait for first token, 127K prompt
Qwen3.8 Flash Next2.3 min 136 s
Hy4 Flash3.6 min 219 s
Hy4 (full)28 min 1,680 s

Every speed figure

MeasureQwenHy4 FlashHy4 full
Writing speed, short prompt27.7 tok/s25.0 tok/s12.2 tok/s
Writing speed after a 32K prompt25.9 tok/s23.6 tok/s10.9 tok/s
Writing speed after a 127K prompt23.9 tok/s21.7 tok/s10.0 tok/s
Reading speed, 32K prompt970 tok/s675 tok/s79 tok/s
Reading speed, 127K prompt933 tok/s580 tok/s75 tok/s
Wait for first token, 32K prompt33 s47 s404 s
Wait for first token, 127K prompt136 s (2.3 min)219 s (3.6 min)1,680 s (28 min)
Load time97 s (1.6 min)160 s (2.7 min)13 s (warm)
Load time, coldno figureno figure123 s (2.0 min)
Least memory left free7 GiB26 GiB139 GiB
Time for all 27 attempts7.6 hours5.1 hours17.5 hours
Tokens generated, all attempts731,765 tokens447,477 tokens714,824 tokens
  • Short-prompt writing speed is the average of three runs; long-prompt rows are one run each.
  • Hy4 Flash speed rows come from the first pilot night (same files and runtime); Qwen rows from the rerun on the current build.
  • Speed tests use one fixed sampling setting for every model.
  • About nine tenths of what all three models wrote was thinking, not answer.
  • Hy4 Flash ran without its built-in draft model, which is what Tencent's 150 tok/s figure relies on.
  • The speed tests use one fixed sampling setting; the makers' settings apply to the challenges.

Setup

What was tested, and how

MeasureQwenHy4 FlashHy4 full
MakerAlibabaTencentTencent
BuildUnsloth UD-Q4_K_XL GGUF, revision f5d175dAngelSlim SPARK-4Bit-Fast GGUFAngelSlim IQ2 GGUF, about 3.5 bits per weight, revision c65d2c9
File size104 GB149 GB310 GB
MachineNVIDIA DGX Spark, 128 GBNVIDIA DGX Spark, 128 GBMac Studio M3 Ultra, 512 GB
Runtimellama.cpp 2cbca38 with the Hy4 Spark patchllama.cpp 2cbca38 with the Hy4 Spark patchllama.cpp PR #32517 at f8d4cc4, Metal
Samplingtemperature 1.0, top-p 0.95, top-k 20 (maker thinking-mode settings)temperature 0.9, top-p 1.0 (maker settings)temperature 0.9, top-p 1.0 (maker settings)
Thinkingon (default)on (default)on (default)
  • Each of the 9 prompts was run 3 times per model, with a fresh server state and a new seed each time.
  • The answer limit was 60,000 tokens, thinking included.
  • All builds are compressed versions, not the original weights.
  • The full Hy4 is not a fair third contestant: different machine, different runtime, much heavier compression. Read its column as what the big model does on a Mac, not as a ranking against the other two.

The method in full Compare the models

Honest part

What went wrong along the way

  1. First pilot night wasted: answer limits of 12,000 to 32,000 tokens cut most attempts off mid-thought. Rerun with 60,000.
  2. The Qwen files on the machine were a superseded upload; replaced with the current revision.
  3. The first night used one sampling setting for both models; the rerun uses each maker's settings.
  4. Two attempts still hit 60,000 tokens: one Qwen game and one full Hy4 game.