MLX Model Test Report

Generated 2026-02-25 11:16 · AFM MLX Backend · v0.9.5

mlx-model-test.sh --prompts /Users/syl/bin/test-stop-sequences.txt

Test Runs
34
Passed
32
Failed
2
Best tok/s
17.0
Fastest
mlx-community/Qwen3.5-35B-A3B-4bit @ stop-code-fence

Performance Ranking (by tokens/sec)

Click a row to jump to its full response below.

# Model / Config Status Temp Load (s) Tokens Gen (s) Tokens/sec Prompt
1 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-code-fence stop-code-fence OK 0.0 3.0 122 7.19
17.0
Write a Python function that computes factorial, inside a co...
2 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-guided-json-no-match stop-guided-json-no-match
--guided-json '{"type":"object","properties":{"name":{"type":"string"},"age":{"type":"integer"}},"required":["name","age"]}'
OK 0.0 3.0 1150 74.31
15.5
Generate a person record for Bob aged 25.
3 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-guided-json-brace stop-guided-json-brace
--guided-json '{"type":"object","properties":{"color":{"type":"string"},"hex":{"type":"string"}},"required":["color","hex"]}'
OK 0.0 3.0 819 53.21
15.4
Describe the color blue with its hex code.
4 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-cli-multi stop-cli-multi
--stop "```,END"
OK 0.0 3.0 326 21.32
15.3
Write a Python hello world in a code block, then write END.
5 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-cli-only stop-cli-only
--stop "3."
OK 0.0 3.0 309 20.54
15.0
List 10 fruits, numbered 1 through 10, one per line.
6 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-api-word stop-api-word OK 0.0 3.0 90 6.3
14.3
Name 5 programming languages and briefly describe each one.
7 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-no-match stop-no-match OK 0.0 3.0 134 9.51
14.1
What is 2+2? Answer briefly.
8 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-system-pirate stop-system-pirate OK 0.0 3.0 374 28.76
13.0
Tell me about treasure hunting on the high seas.
9 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-system-numbered stop-system-numbered OK 0.0 3.0 138 10.66
12.9
What are the main benefits of exercise?
10 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-non-streaming stop-non-streaming
--no-streaming
OK 0.0 3.0 161 12.73
12.7
List 10 planets or celestial objects, numbered 1 through 10,...
11 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-streaming stop-streaming OK 0.0 3.0 161 12.75
12.6
List 10 planets or celestial objects, numbered 1 through 10,...
12 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-four-max stop-four-max OK 0.0 3.0 102 8.34
12.2
List 10 items, numbered 1 through 10. One per line.
13 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-multi-word stop-multi-word OK 0.0 3.0 111 9.32
11.9
Write a 5-step recipe for making tea. Label each step as "St...
14 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-cli-api-dedup stop-cli-api-dedup
--stop "3."
OK 0.0 3.0 87 7.38
11.8
List 10 cities, numbered 1 through 10, one per line.
15 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-guided-json-comma stop-guided-json-comma
--guided-json '{"type":"object","properties":{"name":{"type":"string"},"age":{"type":"integer"},"city":{"type":"string"}},"required":["name","age","city"]}'
OK 0.0 3.0 58 4.98
11.6
Generate a person profile for someone named Alice who is 30 ...
16 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-high-temp stop-high-temp OK 1.0 3.0 150 12.89
11.6
List 10 random words, numbered 1 through 10, one per line.
17 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-long-phrase stop-long-phrase OK 0.0 3.0 44 3.8
11.6
Write a 3-paragraph essay about renewable energy. Start the ...
18 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-cli-api-merge stop-cli-api-merge
--stop "5."
OK 0.0 4.0 82 7.08
11.6
List 10 countries, numbered 1 through 10, one per line.
19 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-api-single stop-api-single OK 0.0 3.0 90 8.08
11.1
List 10 colors, numbered 1 through 10, one per line.
20 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-guided-json-value stop-guided-json-value
--guided-json '{"type":"object","properties":{"cities":{"type":"array","items":{"type":"string"}}},"required":["cities"]}'
OK 0.0 3.0 38 3.42
11.1
List 5 major world cities as a JSON array. Include Tokyo.
21 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-top-p stop-top-p OK 0.0 4.0 111 10.03
11.1
List 10 rivers, numbered 1 through 10, one per line.
22 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-api-multi stop-api-multi OK 0.0 3.0 98 8.92
11.0
List 10 animals, numbered 1 through 10, one per line.
23 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-html-tag stop-html-tag OK 0.0 3.0 107 9.94
10.8
Write an HTML unordered list of 5 fruits using <ul> and <li>...
24 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-unicode stop-unicode OK 0.0 3.0 38 3.59
10.6
List 5 items about space using bullet points (•).
25 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-seed-run1 stop-seed-run1 OK 0.0 3.0 80 7.58
10.6
List 10 flowers, numbered 1 through 10, one per line.
26 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-seed-run2 stop-seed-run2 OK 0.0 3.0 80 7.68
10.4
List 10 flowers, numbered 1 through 10, one per line.
27 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-low-max-tokens stop-low-max-tokens OK 0.0 3.0 40 4.25
9.4
List 10 mountains, numbered 1 through 10, one per line.
28 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-api-newline stop-api-newline OK 0.0 4.0 6 0.86
7.0
What is the capital of France? Answer in one sentence.
29 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-immediate stop-immediate OK 0.0 3.0 8 1.19
6.7
What is the capital of Japan?
30 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-api-double-newline stop-api-double-newline OK 0.0 3.0 6 0.92
6.5
Write a short paragraph about the ocean. Then write a second...
31 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-api-period stop-api-period OK 0.0 3.0 6 1.0
6.0
Tell me about the sun in three sentences.
32 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-special-chars stop-special-chars OK 0.0 3.0 7 1.26
5.6
List 3 facts about the moon. Use **bold** markdown for empha...

Failed Runs

ModelErrorConfigLoad (s)
mlx-community/Qwen3.5-35B-A3B-4bit @ stop-json-object-key stop-json-object-key Error code: 400 - {'error': {'type': 'mlx_error', 'message': 'The operation couldn’t be completed. (Jinja.TemplateException error 1.)'}} t=0.0 3.0
mlx-community/Qwen3.5-35B-A3B-4bit @ stop-json-object-no-match stop-json-object-no-match Error code: 400 - {'error': {'type': 'mlx_error', 'message': 'The operation couldn’t be completed. (Jinja.TemplateException error 1.)'}} t=0.0 3.0

Full Responses

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-code-fence 122 tokens · 17.0 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-guided-json-no-match 1150 tokens · 15.5 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-guided-json-brace 819 tokens · 15.4 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-cli-multi 326 tokens · 15.3 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-cli-only 309 tokens · 15.0 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-api-word 90 tokens · 14.3 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-no-match 134 tokens · 14.1 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-system-pirate 374 tokens · 13.0 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-system-numbered 138 tokens · 12.9 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-non-streaming 161 tokens · 12.7 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-streaming 161 tokens · 12.6 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-four-max 102 tokens · 12.2 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-multi-word 111 tokens · 11.9 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-cli-api-dedup 87 tokens · 11.8 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-guided-json-comma 58 tokens · 11.6 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-high-temp 150 tokens · 11.6 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-long-phrase 44 tokens · 11.6 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-cli-api-merge 82 tokens · 11.6 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-api-single 90 tokens · 11.1 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-guided-json-value 38 tokens · 11.1 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-top-p 111 tokens · 11.1 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-api-multi 98 tokens · 11.0 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-html-tag 107 tokens · 10.8 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-unicode 38 tokens · 10.6 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-seed-run1 80 tokens · 10.6 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-seed-run2 80 tokens · 10.4 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-low-max-tokens 40 tokens · 9.4 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-api-newline 6 tokens · 7.0 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-immediate 8 tokens · 6.7 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-api-double-newline 6 tokens · 6.5 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-api-period 6 tokens · 6.0 tok/s

mlx-community/Qwen3.5-35B-A3B-4bit @ stop-special-chars 7 tokens · 5.6 tok/s