MLX Model Test Report

Generated 2026-02-25 09:55 · AFM MLX Backend · v0.9.5-0cfba17

mlx-model-test.sh --prompts /Users/syl/bin/test-Qwen3.5-35B-A3B-4bit.txt --smart claude,codex

Test Runs
132
Passed
129
Failed
3
Best tok/s
42.5
Fastest
mlx-community/Qwen3.5-35B-A3B-4bit @ greedy

Performance Ranking (by tokens/sec)

Click a row to jump to its full response below.

# Model / Config Status codexclaude Temp Load (s) Tokens Gen (s) Tokens/sec Prompt
1 mlx-community/Qwen3.5-35B-A3B-4bit @ greedy greedy OK 5— 0.0 1.0 1615 38.03
42.5
 Explain why the sky is blue in exactly 3 bullet points.
2 mlx-community/Qwen3.5-35B-A3B-4bit @ raw-mode raw-mode
--raw		 OK 3— 0.7 1.0 1573 37.11
42.4
 Think step by step: what is 17 * 23?
3 mlx-community/Qwen3.5-35B-A3B-4bit @ long-prompt long-prompt OK 5— 0.0 1.0 1910 45.05
42.4
 Repeat after me exactly: The quick brown fox jumps over the ...
4 mlx-community/Qwen3.5-35B-A3B-4bit @ response-format-schema response-format-schema OK 2— 0.0 1.0 1634 38.56
42.4
 Generate a fictional person profile.
5 mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-multi tool-call-multi OK 4— 0.0 1.0 4096 96.68
42.4
 What is the weather in London and what time is it in Tokyo?
6 mlx-community/Qwen3.5-35B-A3B-4bit @ no-streaming no-streaming
--no-streaming		 OK 5— 0.7 1.0 1959 46.31
42.3
 Write a short poem about the moon.
7 mlx-community/Qwen3.5-35B-A3B-4bit @ guided-json-simple guided-json-simple
--guided-json '{"type":"object","properties":{"name":{"type":"string"},"age":{"type":"integer"}},"required":["name","age"]}'		 OK 3— 0.0 1.0 987 23.36
42.2
 Generate a person record.
8 mlx-community/Qwen3.5-35B-A3B-4bit @ default default OK 5— 0.7 1.0 991 23.46
42.2
 Explain why the sky is blue in exactly 3 bullet points.
9 mlx-community/Qwen3.5-35B-A3B-4bit @ pirate pirate OK 4— 0.7 1.0 2780 65.87
42.2
 Tell me about quantum computing.
10 mlx-community/Qwen3.5-35B-A3B-4bit @ scientist scientist OK 2— 0.3 1.0 2084 49.38
42.2
 Tell me about quantum computing.
11 mlx-community/Qwen3.5-35B-A3B-4bit @ eli5 eli5 OK 5— 0.7 1.0 4096 97.03
42.2
 What is the capital of Japan? Answer in one sentence.
12 mlx-community/Qwen3.5-35B-A3B-4bit @ long-output long-output OK 5— 0.7 1.0 1999 47.36
42.2
 Write a detailed recipe for chocolate chip cookies with ingr...
13 mlx-community/Qwen3.5-35B-A3B-4bit @ scientist scientist OK 4— 0.3 1.0 4096 97.06
42.2
 What is the capital of Japan? Answer in one sentence.
14 mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-xml tool-call-xml
--tool-call-parser xmlFunction		 OK 4— 0.0 1.0 1186 28.1
42.2
 What is the weather in Paris in celsius?
15 mlx-community/Qwen3.5-35B-A3B-4bit @ json-output json-output OK 5— 0.0 2.0 1063 25.21
42.2
 Respond with a valid JSON object containing keys "name", "ag...
16 mlx-community/Qwen3.5-35B-A3B-4bit @ guided-json-nested guided-json-nested
--guided-json '{"type":"object","properties":{"city":{"type":"string"},"population":{"type":"integer"},"landmarks":{"type":"array","items":{"type":"string"}}},"required":["city","population","landmarks"]}'		 OK 3— 0.0 1.0 949 22.52
42.1
 Describe Tokyo as a structured record with at least 3 landma...
17 mlx-community/Qwen3.5-35B-A3B-4bit @ prefill-large-4096 prefill-large-4096
--prefill-step-size 4096		 OK 5— 0.0 2.0 4096 97.23
42.1
 Given this architecture, what are the top 3 performance bott...
18 mlx-community/Qwen3.5-35B-A3B-4bit @ eli5 eli5 OK 2— 0.7 1.0 1227 29.14
42.1
 Tell me about quantum computing.
19 mlx-community/Qwen3.5-35B-A3B-4bit @ response-format-json response-format-json OK 3— 0.0 1.0 616 14.64
42.1
 Return a JSON object with keys "language", "year_created", a...
20 mlx-community/Qwen3.5-35B-A3B-4bit @ max-completion-tokens max-completion-tokens OK 5— 0.0 2.0 4096 97.32
42.1
 max_completion_tokens: 100
21 mlx-community/Qwen3.5-35B-A3B-4bit @ non-streaming-seeded non-streaming-seeded
--no-streaming --seed 123		 OK 5— 0.7 2.0 4096 97.37
42.1
 Write a 4-line poem about the ocean.
22 mlx-community/Qwen3.5-35B-A3B-4bit @ prefill-small-256 prefill-small-256
--prefill-step-size 256		 OK 5— 0.0 1.0 4096 97.41
42.0
 Given this architecture, what are the top 3 performance bott...
23 mlx-community/Qwen3.5-35B-A3B-4bit @ streaming-seeded streaming-seeded
--seed 123		 OK 5— 0.7 2.0 4096 97.42
42.0
 Write a 4-line poem about the ocean.
24 mlx-community/Qwen3.5-35B-A3B-4bit @ small-kv small-kv
--max-kv-size 2048		 OK 5— 0.7 1.0 1115 26.53
42.0
 Summarize the key ideas of machine learning in a few paragra...
25 mlx-community/Qwen3.5-35B-A3B-4bit @ kv-quantized kv-quantized
--kv-bits 4		 OK 5— 0.7 2.0 1173 27.91
42.0
 Summarize the key ideas of machine learning in a few paragra...
26 mlx-community/Qwen3.5-35B-A3B-4bit @ very-verbose very-verbose
--very-verbose		 OK 5— 0.7 1.0 473 11.25
42.0
 Hello, how are you?
27 mlx-community/Qwen3.5-35B-A3B-4bit @ seed-42-run2 seed-42-run2
--seed 42		 OK 5— 0.7 1.0 4096 97.53
42.0
 Write a limerick about a cat.
28 mlx-community/Qwen3.5-35B-A3B-4bit @ seed-42-run1 seed-42-run1
--seed 42		 OK 5— 0.7 1.0 4096 97.59
42.0
 Write a limerick about a cat.
29 mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-none tool-call-none OK 5— 0.0 1.0 553 13.18
42.0
 What is the weather in Tokyo?
30 mlx-community/Qwen3.5-35B-A3B-4bit @ pirate pirate OK 3— 0.7 1.0 884 21.07
42.0
 What is the capital of Japan? Answer in one sentence.
31 mlx-community/Qwen3.5-35B-A3B-4bit @ long-form long-form OK 5— 0.7 1.0 4096 97.61
42.0
 Write a detailed technical blog post explaining how Mixture-...
32 mlx-community/Qwen3.5-35B-A3B-4bit @ prefill-default prefill-default OK 5— 0.0 1.0 4096 97.64
42.0
 Given this architecture, what are the top 3 performance bott...
33 mlx-community/Qwen3.5-35B-A3B-4bit @ think-raw think-raw
--raw		 OK 3— 0.0 1.0 904 21.55
41.9
 Solve step by step: If a train travels at 60 mph for 2.5 hou...
34 mlx-community/Qwen3.5-35B-A3B-4bit @ multilingual multilingual OK 5— 0.3 1.0 1052 25.08
41.9
 Translate "Hello, how are you?" into French, Spanish, Japane...
35 mlx-community/Qwen3.5-35B-A3B-4bit @ math math OK 5— 0.0 1.0 1559 37.18
41.9
 A store sells notebooks for $3.50 each. If you buy 5 or more...
36 mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-auto tool-call-auto OK 4— 0.0 1.0 553 13.19
41.9
 What is the weather in Tokyo?
37 mlx-community/Qwen3.5-35B-A3B-4bit @ numbered-list numbered-list OK 5— 0.0 1.0 640 15.29
41.9
 List exactly 5 animals, one per line, numbered 1-5. No other...
38 mlx-community/Qwen3.5-35B-A3B-4bit @ no-penalty no-penalty
--presence-penalty 0.0		 OK 5— 0.8 1.0 4096 97.89
41.8
 Write a long essay about the history of bread making across ...
39 mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-complex tool-call-complex OK 4— 0.0 1.0 508 12.14
41.8
 Search recursively for all Swift files under Sources/ with a...
40 mlx-community/Qwen3.5-35B-A3B-4bit @ code-swift code-swift OK 2— 0.0 2.0 4096 97.9
41.8
 Write a Swift async function that fetches JSON from a URL us...
41 mlx-community/Qwen3.5-35B-A3B-4bit @ think-normal think-normal OK 5— 0.0 1.0 904 21.62
41.8
 Solve step by step: If a train travels at 60 mph for 2.5 hou...
42 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-single stop-single OK 5— 0.0 1.0 394 9.43
41.8
 List 10 fruits, numbered 1-10, one per line.
43 mlx-community/Qwen3.5-35B-A3B-4bit @ combined-samplers combined-samplers
--top-k 50 --min-p 0.03 --top-p 0.95		 OK 5— 0.8 1.0 1654 39.66
41.7
 Explain why the sky is blue in exactly 3 bullet points.
44 mlx-community/Qwen3.5-35B-A3B-4bit @ developer-role developer-role OK 5— 0.0 2.0 4096 98.19
41.7
 developer: You are a helpful coding assistant. Only respond ...
45 mlx-community/Qwen3.5-35B-A3B-4bit @ code-swift code-swift OK 5— 0.0 2.0 4096 98.34
41.6
 What is the capital of Japan? Answer in one sentence.
46 mlx-community/Qwen3.5-35B-A3B-4bit @ max-completion-tokens max-completion-tokens OK 2— 0.0 2.0 2937 70.54
41.6
 Describe the entire history of Rome.
47 mlx-community/Qwen3.5-35B-A3B-4bit @ developer-role developer-role OK 2— 0.0 2.0 530 12.73
41.6
 Write a Python function that reverses a string.
48 mlx-community/Qwen3.5-35B-A3B-4bit @ response-format-text response-format-text OK 5— 0.0 1.0 338 8.12
41.6
 Tell me the language Python, what year it was created, and w...
49 mlx-community/Qwen3.5-35B-A3B-4bit @ code-python code-python OK 5— 0.0 1.0 4096 98.75
41.5
 What is the capital of Japan? Answer in one sentence.
50 mlx-community/Qwen3.5-35B-A3B-4bit @ strict-format strict-format OK 5— 0.0 1.0 427 10.34
41.3
 Output EXACTLY 3 lines. Each line must contain exactly one w...
51 mlx-community/Qwen3.5-35B-A3B-4bit @ verbose verbose
--verbose		 OK 5— 0.7 1.0 316 7.65
41.3
 Hello, how are you?
52 mlx-community/Qwen3.5-35B-A3B-4bit @ minimal-prompt minimal-prompt OK 5— 0.7 1.0 236 5.72
41.3
 Hi
53 mlx-community/Qwen3.5-35B-A3B-4bit @ top-k top-k
--top-k 30		 OK 5— 0.8 1.0 595 14.42
41.3
 Explain why the sky is blue in exactly 3 bullet points.
54 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-multi stop-multi OK 5— 0.0 1.0 254 6.15
41.3
 Write a Python hello world program in a code block, then wri...
55 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-newline stop-newline OK 5— 0.0 1.0 238 5.77
41.2
 What is the capital of France? Answer in one sentence.
56 mlx-community/Qwen3.5-35B-A3B-4bit @ long-prompt long-prompt OK 5— 0.0 1.0 237 5.75
41.2
 What is the capital of Japan? Answer in one sentence.
57 mlx-community/Qwen3.5-35B-A3B-4bit @ greedy greedy OK 3— 0.0 1.0 237 5.75
41.2
 What is the capital of Japan? Answer in one sentence.
58 mlx-community/Qwen3.5-35B-A3B-4bit @ minimal-prompt minimal-prompt OK 4— 0.7 1.0 292 7.1
41.1
 What is the capital of Japan? Answer in one sentence.
59 mlx-community/Qwen3.5-35B-A3B-4bit @ high-temp high-temp OK 5— 1.5 1.0 589 14.33
41.1
 Explain why the sky is blue in exactly 3 bullet points.
60 mlx-community/Qwen3.5-35B-A3B-4bit @ response-format-json response-format-json OK 5— 0.0 1.0 237 5.77
41.1
 What is the capital of Japan? Answer in one sentence.
61 mlx-community/Qwen3.5-35B-A3B-4bit @ response-format-schema response-format-schema OK 5— 0.0 1.0 237 5.77
41.1
 What is the capital of Japan? Answer in one sentence.
62 mlx-community/Qwen3.5-35B-A3B-4bit @ special-chars special-chars OK 5— 0.0 1.0 237 5.78
41.0
 What is the capital of Japan? Answer in one sentence.
63 mlx-community/Qwen3.5-35B-A3B-4bit @ think-raw think-raw
--raw		 OK 5— 0.0 1.0 237 5.78
41.0
 What is the capital of Japan? Answer in one sentence.
64 mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-auto tool-call-auto OK 5— 0.0 1.0 237 5.78
41.0
 What is the capital of Japan? Answer in one sentence.
65 mlx-community/Qwen3.5-35B-A3B-4bit @ special-chars special-chars OK 5— 0.0 1.0 4028 98.2
41.0
 Repeat these characters exactly: <tag> "quotes" 'apostrophes...
66 mlx-community/Qwen3.5-35B-A3B-4bit @ guided-json-nested guided-json-nested
--guided-json '{"type":"object","properties":{"city":{"type":"string"},"population":{"type":"integer"},"landmarks":{"type":"array","items":{"type":"string"}}},"required":["city","population","landmarks"]}'		 OK 2— 0.0 1.0 237 5.78
41.0
 What is the capital of Japan? Answer in one sentence.
67 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-multi stop-multi OK 5— 0.0 1.0 237 5.78
41.0
 What is the capital of Japan? Answer in one sentence.
68 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-single stop-single OK 3— 0.0 1.0 237 5.78
41.0
 What is the capital of Japan? Answer in one sentence.
69 mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-complex tool-call-complex OK 2— 0.0 1.0 237 5.79
41.0
 What is the capital of Japan? Answer in one sentence.
70 mlx-community/Qwen3.5-35B-A3B-4bit @ very-verbose very-verbose
--very-verbose		 OK 5— 0.7 1.0 239 5.84
40.9
 What is the capital of Japan? Answer in one sentence.
71 mlx-community/Qwen3.5-35B-A3B-4bit @ min-p min-p
--min-p 0.05		 OK 5— 0.8 2.0 690 16.86
40.9
 Explain why the sky is blue in exactly 3 bullet points.
72 mlx-community/Qwen3.5-35B-A3B-4bit @ stop-newline stop-newline OK 5— 0.0 1.0 237 5.79
40.9
 What is the capital of Japan? Answer in one sentence.
73 mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-multi tool-call-multi OK 4— 0.0 1.0 237 5.79
40.9
 What is the capital of Japan? Answer in one sentence.
74 mlx-community/Qwen3.5-35B-A3B-4bit @ default default OK 5— 0.7 1.0 205 5.01
40.9
 What is the capital of Japan? Answer in one sentence.
75 mlx-community/Qwen3.5-35B-A3B-4bit @ logprobs logprobs
--max-logprobs 5		 OK 5— 0.0 1.0 186 4.55
40.9
 What is 1+1?
76 mlx-community/Qwen3.5-35B-A3B-4bit @ guided-json-simple guided-json-simple
--guided-json '{"type":"object","properties":{"name":{"type":"string"},"age":{"type":"integer"}},"required":["name","age"]}'		 OK 5— 0.0 1.0 237 5.8
40.9
 What is the capital of Japan? Answer in one sentence.
77 mlx-community/Qwen3.5-35B-A3B-4bit @ response-format-text response-format-text OK 2— 0.0 1.0 237 5.8
40.9
 What is the capital of Japan? Answer in one sentence.
78 mlx-community/Qwen3.5-35B-A3B-4bit @ small-kv small-kv
--max-kv-size 2048		 OK 5— 0.7 1.0 241 5.9
40.8
 What is the capital of Japan? Answer in one sentence.
79 mlx-community/Qwen3.5-35B-A3B-4bit @ logprobs logprobs
--max-logprobs 5		 OK 5— 0.0 1.0 237 5.81
40.8
 What is the capital of Japan? Answer in one sentence.
80 mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-none tool-call-none OK 3— 0.0 1.0 237 5.81
40.8
 What is the capital of Japan? Answer in one sentence.
81 mlx-community/Qwen3.5-35B-A3B-4bit @ numbered-list numbered-list OK 5— 0.0 1.0 237 5.81
40.8
 What is the capital of Japan? Answer in one sentence.
82 mlx-community/Qwen3.5-35B-A3B-4bit @ long-output long-output OK 2— 0.7 1.0 216 5.3
40.8
 What is the capital of Japan? Answer in one sentence.
83 mlx-community/Qwen3.5-35B-A3B-4bit @ raw-mode raw-mode
--raw		 OK 5— 0.7 1.0 224 5.49
40.8
 What is the capital of Japan? Answer in one sentence.
84 mlx-community/Qwen3.5-35B-A3B-4bit @ strict-format strict-format OK 4— 0.0 1.0 237 5.82
40.7
 What is the capital of Japan? Answer in one sentence.
85 mlx-community/Qwen3.5-35B-A3B-4bit @ multilingual multilingual OK 2— 0.3 1.0 232 5.7
40.7
 What is the capital of Japan? Answer in one sentence.
86 mlx-community/Qwen3.5-35B-A3B-4bit @ math math OK 2— 0.0 1.0 237 5.82
40.7
 What is the capital of Japan? Answer in one sentence.
87 mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-xml tool-call-xml
--tool-call-parser xmlFunction		 OK 4— 0.0 1.0 237 5.83
40.7
 What is the capital of Japan? Answer in one sentence.
88 mlx-community/Qwen3.5-35B-A3B-4bit @ top-k top-k
--top-k 30		 OK 5— 0.8 1.0 261 6.42
40.6
 What is the capital of Japan? Answer in one sentence.
89 mlx-community/Qwen3.5-35B-A3B-4bit @ think-normal think-normal OK 5— 0.0 1.0 237 5.84
40.6
 What is the capital of Japan? Answer in one sentence.
90 mlx-community/Qwen3.5-35B-A3B-4bit @ no-penalty no-penalty
--presence-penalty 0.0		 OK 2— 0.8 1.0 218 5.38
40.5
 What is the capital of Japan? Answer in one sentence.
91 mlx-community/Qwen3.5-35B-A3B-4bit @ seed-42-run1 seed-42-run1
--seed 42		 OK 5— 0.7 1.0 197 4.87
40.4
 What is the capital of Japan? Answer in one sentence.
92 mlx-community/Qwen3.5-35B-A3B-4bit @ no-streaming no-streaming
--no-streaming		 OK 3— 0.7 1.0 201 4.97
40.4
 What is the capital of Japan? Answer in one sentence.
93 mlx-community/Qwen3.5-35B-A3B-4bit @ high-temp high-temp OK 5— 1.5 1.0 207 5.13
40.4
 What is the capital of Japan? Answer in one sentence.
94 mlx-community/Qwen3.5-35B-A3B-4bit @ top-p top-p
--top-p 0.9		 OK 5— 0.8 1.0 466 11.54
40.4
 Explain why the sky is blue in exactly 3 bullet points.
95 mlx-community/Qwen3.5-35B-A3B-4bit @ streaming-seeded streaming-seeded
--seed 123		 OK 3— 0.7 2.0 206 5.11
40.4
 What is the capital of Japan? Answer in one sentence.
96 mlx-community/Qwen3.5-35B-A3B-4bit @ long-form long-form OK 5— 0.7 1.0 199 4.94
40.3
 What is the capital of Japan? Answer in one sentence.
97 mlx-community/Qwen3.5-35B-A3B-4bit @ code-python code-python OK 2— 0.0 1.0 267 6.63
40.3
 Write a function that finds all prime numbers up to n using ...
98 mlx-community/Qwen3.5-35B-A3B-4bit @ top-p top-p
--top-p 0.9		 OK 5— 0.8 1.0 227 5.64
40.3
 What is the capital of Japan? Answer in one sentence.
99 mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-complex tool-call-complex OK 5— 0.0 1.0 299 7.43
40.2
 tools: [{"type":"function","function":{"name":"search_files"...
100 mlx-community/Qwen3.5-35B-A3B-4bit @ seed-42-run2 seed-42-run2
--seed 42		 OK 2— 0.7 1.0 197 4.9
40.2
 What is the capital of Japan? Answer in one sentence.
101 mlx-community/Qwen3.5-35B-A3B-4bit @ prefill-default prefill-default OK 5— 0.0 1.0 454 11.3
40.2
 What is the capital of Japan? Answer in one sentence.
102 mlx-community/Qwen3.5-35B-A3B-4bit @ prefill-small-256 prefill-small-256
--prefill-step-size 256		 OK 3— 0.0 1.0 454 11.32
40.1
 What is the capital of Japan? Answer in one sentence.
103 mlx-community/Qwen3.5-35B-A3B-4bit @ json-output json-output OK 5— 0.0 2.0 237 5.92
40.0
 What is the capital of Japan? Answer in one sentence.
104 mlx-community/Qwen3.5-35B-A3B-4bit @ verbose verbose
--verbose		 OK 4— 0.7 1.0 222 5.55
40.0
 What is the capital of Japan? Answer in one sentence.
105 mlx-community/Qwen3.5-35B-A3B-4bit @ non-streaming-seeded non-streaming-seeded
--no-streaming --seed 123		 OK 2— 0.7 2.0 206 5.15
40.0
 What is the capital of Japan? Answer in one sentence.
106 mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-xml tool-call-xml
--tool-call-parser xmlFunction		 OK 5— 0.0 1.0 270 6.76
39.9
 tools: [{"type":"function","function":{"name":"get_weather",...
107 mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-multi tool-call-multi OK 5— 0.0 1.0 355 8.9
39.9
 tools: [{"type":"function","function":{"name":"get_weather",...
108 mlx-community/Qwen3.5-35B-A3B-4bit @ combined-samplers combined-samplers
--top-k 50 --min-p 0.03 --top-p 0.95		 OK 5— 0.8 1.0 178 4.47
39.8
 What is the capital of Japan? Answer in one sentence.
109 mlx-community/Qwen3.5-35B-A3B-4bit @ max-completion-tokens max-completion-tokens OK 2— 0.0 2.0 237 5.96
39.7
 What is the capital of Japan? Answer in one sentence.
110 mlx-community/Qwen3.5-35B-A3B-4bit @ developer-role developer-role OK 5— 0.0 2.0 237 5.98
39.6
 What is the capital of Japan? Answer in one sentence.
111 mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-auto tool-call-auto OK 5— 0.0 1.0 270 6.81
39.6
 tools: [{"type":"function","function":{"name":"get_weather",...
112 mlx-community/Qwen3.5-35B-A3B-4bit @ prefill-large-4096 prefill-large-4096
--prefill-step-size 4096		 OK 3— 0.0 2.0 454 11.46
39.6
 What is the capital of Japan? Answer in one sentence.
113 mlx-community/Qwen3.5-35B-A3B-4bit @ repetition-penalty repetition-penalty
--repetition-penalty 1.2		 OK 5— 0.8 2.0 4096 104.0
39.4
 Write a long essay about the history of bread making across ...
114 mlx-community/Qwen3.5-35B-A3B-4bit @ kv-quantized kv-quantized
--kv-bits 4		 OK 5— 0.7 2.0 232 5.91
39.2
 What is the capital of Japan? Answer in one sentence.
115 mlx-community/Qwen3.5-35B-A3B-4bit @ with-penalty with-penalty
--presence-penalty 1.5		 OK 5— 0.8 1.0 4096 104.53
39.2
 Write a long essay about the history of bread making across ...
116 mlx-community/Qwen3.5-35B-A3B-4bit @ min-p min-p
--min-p 0.05		 OK 5— 0.8 2.0 245 6.31
38.8
 What is the capital of Japan? Answer in one sentence.
117 mlx-community/Qwen3.5-35B-A3B-4bit @ with-penalty with-penalty
--presence-penalty 1.5		 OK 4— 0.8 1.0 221 5.89
37.5
 What is the capital of Japan? Answer in one sentence.
118 mlx-community/Qwen3.5-35B-A3B-4bit @ short-output short-output OK 2— 0.7 1.0 50 1.34
37.3
 Describe the entire history of Rome.
119 mlx-community/Qwen3.5-35B-A3B-4bit @ agent-no-cache-turn3 agent-no-cache-turn3 OK 5— 0.0 1.0 203 5.49
37.0
 Write a unit test for the timeout feature you just described...
120 mlx-community/Qwen3.5-35B-A3B-4bit @ repetition-penalty repetition-penalty
--repetition-penalty 1.2		 OK 4— 0.8 2.0 204 5.56
36.7
 What is the capital of Japan? Answer in one sentence.
121 mlx-community/Qwen3.5-35B-A3B-4bit @ short-output short-output OK 1— 0.7 1.0 50 1.41
35.4
 What is the capital of Japan? Answer in one sentence.
122 mlx-community/Qwen3.5-35B-A3B-4bit @ agent-no-cache-turn2 agent-no-cache-turn2 OK 5— 0.0 1.0 105 3.21
32.7
 Now add a --timeout flag to the CLI that sets a request time...
123 mlx-community/Qwen3.5-35B-A3B-4bit @ agent-no-cache-turn1 agent-no-cache-turn1 OK 5— 0.0 1.0 66 2.3
28.7
 Read the file Sources/MacLocalAPI/main.swift and explain wha...
124 mlx-community/Qwen3.5-35B-A3B-4bit @ agent-cached-turn1 agent-cached-turn1
--enable-prefix-caching		 OK 2— 0.0 1.0 64 2.29
28.0
 What is the capital of Japan? Answer in one sentence.
125 mlx-community/Qwen3.5-35B-A3B-4bit @ agent-no-cache-turn3 agent-no-cache-turn3 OK 3— 0.0 1.0 64 2.29
28.0
 What is the capital of Japan? Answer in one sentence.
126 mlx-community/Qwen3.5-35B-A3B-4bit @ agent-cached-turn2 agent-cached-turn2
--enable-prefix-caching		 OK 1— 0.0 1.0 64 2.29
27.9
 What is the capital of Japan? Answer in one sentence.
127 mlx-community/Qwen3.5-35B-A3B-4bit @ agent-no-cache-turn2 agent-no-cache-turn2 OK 2— 0.0 1.0 64 2.3
27.9
 What is the capital of Japan? Answer in one sentence.
128 mlx-community/Qwen3.5-35B-A3B-4bit @ agent-cached-turn3 agent-cached-turn3
--enable-prefix-caching		 OK 1— 0.0 1.0 64 2.3
27.9
 What is the capital of Japan? Answer in one sentence.
129 mlx-community/Qwen3.5-35B-A3B-4bit @ agent-no-cache-turn1 agent-no-cache-turn1 OK 2— 0.0 1.0 64 2.54
25.2
 What is the capital of Japan? Answer in one sentence.

Failed Runs

ModelErrorcodexclaudeConfigLoad (s)
mlx-community/Qwen3.5-35B-A3B-4bit @ agent-cached-turn1 agent-cached-turn1 Connection error. 5— t=0.0 --enable-prefix-caching 1.0
mlx-community/Qwen3.5-35B-A3B-4bit @ agent-cached-turn2 agent-cached-turn2 Connection error. 5— t=0.0 --enable-prefix-caching 1.0
mlx-community/Qwen3.5-35B-A3B-4bit @ agent-cached-turn3 agent-cached-turn3 Connection error. 5— t=0.0 --enable-prefix-caching 1.0

AI Analysis (--smart)

▶ codex Analysis · avg score: 4.1/5

▶ claude Analysis

Full Responses

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ greedy 1615 tokens · 42.5 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ raw-mode 1573 tokens · 42.4 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ long-prompt 1910 tokens · 42.4 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ response-format-schema 1634 tokens · 42.4 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-multi 4096 tokens · 42.4 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ no-streaming 1959 tokens · 42.3 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ guided-json-simple 987 tokens · 42.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ default 991 tokens · 42.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ pirate 2780 tokens · 42.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ scientist 2084 tokens · 42.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ eli5 4096 tokens · 42.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ long-output 1999 tokens · 42.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ scientist 4096 tokens · 42.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-xml 1186 tokens · 42.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ json-output 1063 tokens · 42.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ guided-json-nested 949 tokens · 42.1 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ prefill-large-4096 4096 tokens · 42.1 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ eli5 1227 tokens · 42.1 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ response-format-json 616 tokens · 42.1 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ max-completion-tokens 4096 tokens · 42.1 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ non-streaming-seeded 4096 tokens · 42.1 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ prefill-small-256 4096 tokens · 42.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ streaming-seeded 4096 tokens · 42.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ small-kv 1115 tokens · 42.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ kv-quantized 1173 tokens · 42.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ very-verbose 473 tokens · 42.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ seed-42-run2 4096 tokens · 42.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ seed-42-run1 4096 tokens · 42.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-none 553 tokens · 42.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ pirate 884 tokens · 42.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ long-form 4096 tokens · 42.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ prefill-default 4096 tokens · 42.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ think-raw 904 tokens · 41.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ multilingual 1052 tokens · 41.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ math 1559 tokens · 41.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-auto 553 tokens · 41.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ numbered-list 640 tokens · 41.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ no-penalty 4096 tokens · 41.8 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-complex 508 tokens · 41.8 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ code-swift 4096 tokens · 41.8 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ think-normal 904 tokens · 41.8 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ stop-single 394 tokens · 41.8 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ combined-samplers 1654 tokens · 41.7 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ developer-role 4096 tokens · 41.7 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ code-swift 4096 tokens · 41.6 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ max-completion-tokens 2937 tokens · 41.6 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ developer-role 530 tokens · 41.6 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ response-format-text 338 tokens · 41.6 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ code-python 4096 tokens · 41.5 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ strict-format 427 tokens · 41.3 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ verbose 316 tokens · 41.3 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ minimal-prompt 236 tokens · 41.3 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ top-k 595 tokens · 41.3 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ stop-multi 254 tokens · 41.3 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ stop-newline 238 tokens · 41.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ long-prompt 237 tokens · 41.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ greedy 237 tokens · 41.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ minimal-prompt 292 tokens · 41.1 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ high-temp 589 tokens · 41.1 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ response-format-json 237 tokens · 41.1 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ response-format-schema 237 tokens · 41.1 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ special-chars 237 tokens · 41.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ think-raw 237 tokens · 41.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-auto 237 tokens · 41.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ special-chars 4028 tokens · 41.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ guided-json-nested 237 tokens · 41.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ stop-multi 237 tokens · 41.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ stop-single 237 tokens · 41.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-complex 237 tokens · 41.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ very-verbose 239 tokens · 40.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ min-p 690 tokens · 40.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ stop-newline 237 tokens · 40.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-multi 237 tokens · 40.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ default 205 tokens · 40.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ logprobs 186 tokens · 40.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ guided-json-simple 237 tokens · 40.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ response-format-text 237 tokens · 40.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ small-kv 241 tokens · 40.8 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ logprobs 237 tokens · 40.8 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-none 237 tokens · 40.8 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ numbered-list 237 tokens · 40.8 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ long-output 216 tokens · 40.8 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ raw-mode 224 tokens · 40.8 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ strict-format 237 tokens · 40.7 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ multilingual 232 tokens · 40.7 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ math 237 tokens · 40.7 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-xml 237 tokens · 40.7 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ top-k 261 tokens · 40.6 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ think-normal 237 tokens · 40.6 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ no-penalty 218 tokens · 40.5 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ seed-42-run1 197 tokens · 40.4 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ no-streaming 201 tokens · 40.4 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ high-temp 207 tokens · 40.4 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ top-p 466 tokens · 40.4 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ streaming-seeded 206 tokens · 40.4 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ long-form 199 tokens · 40.3 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ code-python 267 tokens · 40.3 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ top-p 227 tokens · 40.3 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-complex 299 tokens · 40.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ seed-42-run2 197 tokens · 40.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ prefill-default 454 tokens · 40.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ prefill-small-256 454 tokens · 40.1 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ json-output 237 tokens · 40.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ verbose 222 tokens · 40.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ non-streaming-seeded 206 tokens · 40.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-xml 270 tokens · 39.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-multi 355 tokens · 39.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ combined-samplers 178 tokens · 39.8 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ max-completion-tokens 237 tokens · 39.7 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ developer-role 237 tokens · 39.6 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ tool-call-auto 270 tokens · 39.6 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ prefill-large-4096 454 tokens · 39.6 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ repetition-penalty 4096 tokens · 39.4 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ kv-quantized 232 tokens · 39.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ with-penalty 4096 tokens · 39.2 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ min-p 245 tokens · 38.8 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ with-penalty 221 tokens · 37.5 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ short-output 50 tokens · 37.3 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ agent-no-cache-turn3 203 tokens · 37.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ repetition-penalty 204 tokens · 36.7 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ short-output 50 tokens · 35.4 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ agent-no-cache-turn2 105 tokens · 32.7 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ agent-no-cache-turn1 66 tokens · 28.7 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ agent-cached-turn1 64 tokens · 28.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ agent-no-cache-turn3 64 tokens · 28.0 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ agent-cached-turn2 64 tokens · 27.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ agent-no-cache-turn2 64 tokens · 27.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ agent-cached-turn3 64 tokens · 27.9 tok/s

▶ mlx-community/Qwen3.5-35B-A3B-4bit @ agent-no-cache-turn1 64 tokens · 25.2 tok/s