Structured Outputs Test Report

Generated 2026-02-23 17:45 · afm v0.9.4 · Apple M-series · macOS 26

ALL TESTS PASSED

Total Tests

88

Passed

88

Failed

0

Skipped

0

Models

7

Total Time

227.1s

Test Results by Backend / Model

▶ AFM 14/14 passed · 9.5s ALL PASS

Status	Test / Prompt	Time	Detail / Output
✓ PASS	1.1 Simple object (strict, non-streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is the capital of France? Answer with the country name and capital." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "name": { "type": "string" }, "capital": { "type": "string" } }, "required": [ "name", "capital" ], "additionalProperties": false } } } }`	1.9s	`{ "name": "France", "capital": "Paris" }`
✓ PASS	1.2 Simple object (strict, streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is the capital of Japan? Answer with the country name and capital." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "name": { "type": "string" }, "capital": { "type": "string" } }, "required": [ "name", "capital" ], "additionalProperties": false } } } }`	0.5s	`{ "name": "Japan", "capital": "Tokyo" }`
✓ PASS	1.3 Array of objects (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 3 colors with their hex codes." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "hex": { "type": "string" } }, "required": [ "name", "hex" ] } } }, "required": [ "colors" ], "additionalProperties": false } } } }	1.0s	`{ "colors": [ { "hex": "#FF5733", "name": "FireBrick" }, { "name": "SkyBlue", "hex": "#00A9FF" }, { "name": "Yellow", "hex": "#FFFF00" } ] }`
✓ PASS	1.4 Array of objects (strict, streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 2 colors with their hex codes." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "hex": { "type": "string" } }, "required": [ "name", "hex" ] } } }, "required": [ "colors" ], "additionalProperties": false } } } }	0.8s	`{ "colors": [ { "name": "Red", "hex": "#FF5733" }, { "hex": "#007BFF", "name": "Blue" } ] }`
✓ PASS	1.5 Mixed types (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Tell me about the movie Inception. Provide title, year, rating out of 10, and whether it is available on streaming." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "title": { "type": "string" }, "year": { "type": "integer" }, "rating": { "type": "number" }, "available": { "type": "boolean" } }, "required": [ "title", "year", "rating", "available" ], "additionalProperties": false } } } }	0.5s	`{ "year": 2010, "available": true, "title": "Inception", "rating": 8.8 }`
✓ PASS	1.6 Nested objects (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Give me info about Albert Einstein including his city and country." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "person": { "type": "object", "properties": { "name": { "type": "string" }, "address": { "type": "object", "properties": { "city": { "type": "string" }, "country": { "type": "string" } }, "required": [ "city", "country" ] } }, "required": [ "name", "address" ] } }, "required": [ "person" ], "additionalProperties": false } } } }	0.6s	`{ "person": { "address": { "country": "Germany", "city": "Munich" }, "name": "Albert Einstein" } }`
✓ PASS	1.7 String enum (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Analyze the sentiment of: I love this product, it is amazing!" } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "sentiment": { "type": "string", "enum": [ "positive", "negative", "neutral" ] }, "confidence": { "type": "number" } }, "required": [ "sentiment", "confidence" ], "additionalProperties": false } } } }	0.4s	`{ "sentiment": "positive", "confidence": 1 }`
✓ PASS	1.8 Bounded array (strict, non-streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "List some popular programming languages." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "items": { "type": "array", "items": { "type": "string" }, "minItems": 2, "maxItems": 4 } }, "required": [ "items" ], "additionalProperties": false } } } }`	0.4s	`{ "items": [ "Python", "JavaScript", "Java", "C++" ] }`
✓ PASS	1.9 Optional properties (strict, non-streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "Tell me about someone named Alice." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "name": { "type": "string" }, "nickname": { "type": "string" }, "age": { "type": "integer" } }, "required": [ "name" ], "additionalProperties": false } } } }`	0.3s	`{ "name": "Alice" }`
✓ PASS	1.10 Refs and defs (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 2 students with their names and letter grades." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "students": { "type": "array", "items": { "$ref": "#/$defs/Student" } } }, "required": [ "students" ], "$defs": { "Student": { "type": "object", "properties": { "name": { "type": "string" }, "grade": { "type": "string" } }, "required": [ "name", "grade" ] } }, "additionalProperties": false } } } }	0.6s	`{ "students": [ { "name": "Alice", "grade": "A" }, { "name": "Bob", "grade": "B+" } ] }`
✓ PASS	1.11 Non-strict json_schema (fallback to text) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "List 2 colors with hex codes." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false }`	0.7s	Got text response (81 chars)
✓ PASS	1.12 json_object mode (plain text expected) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "List 2 animals as JSON with name and type fields." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false }`	0.8s	Got text response (143 chars)
✓ PASS	1.13 No response_format (plain text regression) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is 2+2?" } ], "max_tokens": 2000, "temperature": 0.3, "stream": false }`	0.3s	Got text response (15 chars)
✓ PASS	1.14 Mixed types (strict, streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Tell me about the movie The Matrix. Provide title, year, rating out of 10, and whether it is available on streaming." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "title": { "type": "string" }, "year": { "type": "integer" }, "rating": { "type": "number" }, "available": { "type": "boolean" } }, "required": [ "title", "year", "rating", "available" ], "additionalProperties": false } } } }	0.7s	`{ "title": "The Matrix", "rating": 8.8, "available": true, "year": 1999 }`
Section total		9.5s

▶ MLX:Qwen3-30B-A3B-4bit 14/14 passed · 44.7s ALL PASS

Status	Test / Prompt	Time	Detail / Output
✓ PASS	2.1 Simple object (strict, non-streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is the capital of France? Answer with the country name and capital." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "name": { "type": "string" }, "capital": { "type": "string" } }, "required": [ "name", "capital" ], "additionalProperties": false } } } }`	1.5s	`{ "name": "France", "capital": "Paris" }`
✓ PASS	2.2 Simple object (strict, streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is the capital of Germany? Answer with the country name and capital." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "name": { "type": "string" }, "capital": { "type": "string" } }, "required": [ "name", "capital" ], "additionalProperties": false } } } }`	2.2s	`{ "name": "Germany", "capital": "Berlin" }`
✓ PASS	2.3 Array of objects (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 3 colors with their hex codes." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "hex": { "type": "string" } }, "required": [ "name", "hex" ] } } }, "required": [ "colors" ], "additionalProperties": false } } } }	4.8s	`{ "colors": [ { "name": "Red", "hex": "#FF0000" }, { "name": "Blue", "hex": "#0000FF" }, { "name": "Green", "hex": "#008000" } ] }`
✓ PASS	2.4 Array of objects (strict, streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 2 colors with their hex codes." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "hex": { "type": "string" } }, "required": [ "name", "hex" ] } } }, "required": [ "colors" ], "additionalProperties": false } } } }	3.7s	`{ "colors": [ { "name": "Black", "hex": "#000000" }, { "name": "White", "hex": "#FFFFFF" } ] }`
✓ PASS	2.5 Mixed types (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Tell me about the movie Inception. Provide title, year, rating out of 10, and whether it is available on streaming." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "title": { "type": "string" }, "year": { "type": "integer" }, "rating": { "type": "number" }, "available": { "type": "boolean" } }, "required": [ "title", "year", "rating", "available" ], "additionalProperties": false } } } }	5.0s	`{ "title": "Inception", "year": 2010, "rating": 8.8, "available": true }`
✓ PASS	2.6 Nested objects (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Give me info about Albert Einstein including his city and country." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "person": { "type": "object", "properties": { "name": { "type": "string" }, "address": { "type": "object", "properties": { "city": { "type": "string" }, "country": { "type": "string" } }, "required": [ "city", "country" ] } }, "required": [ "name", "address" ] } }, "required": [ "person" ], "additionalProperties": false } } } }	2.3s	`{ "person": { "name": "Albert Einstein", "address": { "city": "Ulm", "country": "Germany" } } }`
✓ PASS	2.7 String enum (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Analyze the sentiment of: I love this product, it is amazing! Return sentiment and confidence." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "sentiment": { "type": "string", "enum": [ "positive", "negative", "neutral" ] }, "confidence": { "type": "number" } }, "required": [ "sentiment", "confidence" ], "additionalProperties": false } } } }	1.8s	`{ "sentiment": "positive", "confidence": 0.95 }`
✓ PASS	2.8 Bounded array (strict, non-streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "List some popular programming languages." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "items": { "type": "array", "items": { "type": "string" }, "minItems": 2, "maxItems": 4 } }, "required": [ "items" ], "additionalProperties": false } } } }`	1.8s	`{ "items": [ "JavaScript", "Python", "Java", "C++" ] }`
✓ PASS	2.9 Refs and defs (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 2 students with their names and letter grades." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "students": { "type": "array", "items": { "$ref": "#/$defs/Student" } } }, "required": [ "students" ], "$defs": { "Student": { "type": "object", "properties": { "name": { "type": "string" }, "grade": { "type": "string" } }, "required": [ "name", "grade" ] } }, "additionalProperties": false } } } }	2.6s	`{ "students": [ { "name": "Alice Johnson", "grade": "A" }, { "name": "Bob Smith", "grade": "B" } ] }`
✓ PASS	2.10 Non-strict json_schema ▶ Show request payload { "messages": [ { "role": "user", "content": "List 2 colors with hex codes as JSON." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": false, "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "hex": { "type": "string" } }, "required": [ "name", "hex" ] } } }, "required": [ "colors" ], "additionalProperties": false } } } }	4.7s	`{ "colors": [ { "name": "Red", "hex": "#FF0000" }, { "name": "Blue", "hex": "#0000FF" } ] }`
✓ PASS	2.11 json_object mode ▶ Show request payload `{ "messages": [ { "role": "user", "content": "Return a JSON object with an 'animals' key containing 2 animals, each with name and type fields." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_object" } }`	3.0s	`{ "animals": [ { "name": "Buddy", "type": "Dog" }, { "name": "Whiskers", "type": "Cat" } ] }`
✓ PASS	2.12 No response_format (plain text regression) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is 2+2?" } ], "max_tokens": 2000, "temperature": 0.3, "stream": false }`	1.6s	Got text response (156 chars)
✓ PASS	2.13 Nested objects (strict, streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Give me info about Marie Curie including her city and country." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "person": { "type": "object", "properties": { "name": { "type": "string" }, "address": { "type": "object", "properties": { "city": { "type": "string" }, "country": { "type": "string" } }, "required": [ "city", "country" ] } }, "required": [ "name", "address" ] } }, "required": [ "person" ], "additionalProperties": false } } } }	5.1s	`{ "person": { "name": "Marie Curie", "address": { "city": "Warsaw", "country": "Poland" } } }`
✓ PASS	2.14 Mixed types (strict, streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Tell me about the movie The Matrix. Provide title, year, rating out of 10, and whether it is available." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "title": { "type": "string" }, "year": { "type": "integer" }, "rating": { "type": "number" }, "available": { "type": "boolean" } }, "required": [ "title", "year", "rating", "available" ], "additionalProperties": false } } } }	4.6s	`{ "title": "The Matrix", "year": 1999, "rating": 8.7, "available": true }`
Section total		44.7s

▶ MLX:gpt-oss-20b-MXFP4-Q4 14/14 passed · 44.3s ALL PASS

Status	Test / Prompt	Time	Detail / Output
✓ PASS	2.1 Simple object (strict, non-streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is the capital of France? Answer with the country name and capital." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "name": { "type": "string" }, "capital": { "type": "string" } }, "required": [ "name", "capital" ], "additionalProperties": false } } } }`	1.9s	`{ "name": "France", "capital": "Paris" }`
✓ PASS	2.2 Simple object (strict, streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is the capital of Germany? Answer with the country name and capital." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "name": { "type": "string" }, "capital": { "type": "string" } }, "required": [ "name", "capital" ], "additionalProperties": false } } } }`	5.0s	`{ "name": "Germany", "capital": "Berlin" }`
✓ PASS	2.3 Array of objects (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 3 colors with their hex codes." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "hex": { "type": "string" } }, "required": [ "name", "hex" ] } } }, "required": [ "colors" ], "additionalProperties": false } } } }	3.4s	`{ "colors": [ { "name": "Red", "hex": "FF0000" }, { "name": "Blue", "hex": "0000FF" }, { "name": "Green", "hex": "008000" } ] }`
✓ PASS	2.4 Array of objects (strict, streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 2 colors with their hex codes." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "hex": { "type": "string" } }, "required": [ "name", "hex" ] } } }, "required": [ "colors" ], "additionalProperties": false } } } }	3.9s	`{ "colors": [ { "name": "Crimson", "hex": "#DC143C" }, { "name": "SkyBlue", "hex": "#87CEEB" } ] }`
✓ PASS	2.5 Mixed types (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Tell me about the movie Inception. Provide title, year, rating out of 10, and whether it is available on streaming." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "title": { "type": "string" }, "year": { "type": "integer" }, "rating": { "type": "number" }, "available": { "type": "boolean" } }, "required": [ "title", "year", "rating", "available" ], "additionalProperties": false } } } }	3.5s	`{ "title": "Inception", "year": 2010, "rating": 8.8, "available": true }`
✓ PASS	2.6 Nested objects (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Give me info about Albert Einstein including his city and country." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "person": { "type": "object", "properties": { "name": { "type": "string" }, "address": { "type": "object", "properties": { "city": { "type": "string" }, "country": { "type": "string" } }, "required": [ "city", "country" ] } }, "required": [ "name", "address" ] } }, "required": [ "person" ], "additionalProperties": false } } } }	2.4s	`{ "person": { "name": "Albert Einstein", "address": { "city": "Ulm", "country": "Germany" } } }`
✓ PASS	2.7 String enum (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Analyze the sentiment of: I love this product, it is amazing! Return sentiment and confidence." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "sentiment": { "type": "string", "enum": [ "positive", "negative", "neutral" ] }, "confidence": { "type": "number" } }, "required": [ "sentiment", "confidence" ], "additionalProperties": false } } } }	1.9s	`{ "sentiment": "positive", "confidence": 0.95 }`
✓ PASS	2.8 Bounded array (strict, non-streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "List some popular programming languages." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "items": { "type": "array", "items": { "type": "string" }, "minItems": 2, "maxItems": 4 } }, "required": [ "items" ], "additionalProperties": false } } } }`	2.9s	`{ "items": [ "Python", "JavaScript", "Java", "C++" ] }`
✓ PASS	2.9 Refs and defs (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 2 students with their names and letter grades." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "students": { "type": "array", "items": { "$ref": "#/$defs/Student" } } }, "required": [ "students" ], "$defs": { "Student": { "type": "object", "properties": { "name": { "type": "string" }, "grade": { "type": "string" } }, "required": [ "name", "grade" ] } }, "additionalProperties": false } } } }	2.5s	`{ "students": [ { "name": "Alice Johnson", "grade": "A" }, { "name": "Bob Smith", "grade": "B" } ] }`
✓ PASS	2.10 Non-strict json_schema ▶ Show request payload { "messages": [ { "role": "user", "content": "List 2 colors with hex codes as JSON." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": false, "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "hex": { "type": "string" } }, "required": [ "name", "hex" ] } } }, "required": [ "colors" ], "additionalProperties": false } } } }	4.0s	`{ "colors": [ { "name": "Red", "hex": "FF0000" }, { "name": "Blue", "hex": "0000FF" } ] }`
✓ PASS	2.11 json_object mode ▶ Show request payload `{ "messages": [ { "role": "user", "content": "Return a JSON object with an 'animals' key containing 2 animals, each with name and type fields." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_object" } }`	2.0s	`{ "animals": [ { "name": "Buddy", "type": "Dog" }, { "name": "Whiskers", "type": "Cat" } ] }`
✓ PASS	2.12 No response_format (plain text regression) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is 2+2?" } ], "max_tokens": 2000, "temperature": 0.3, "stream": false }`	2.5s	Got text response (154 chars)
✓ PASS	2.13 Nested objects (strict, streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Give me info about Marie Curie including her city and country." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "person": { "type": "object", "properties": { "name": { "type": "string" }, "address": { "type": "object", "properties": { "city": { "type": "string" }, "country": { "type": "string" } }, "required": [ "city", "country" ] } }, "required": [ "name", "address" ] } }, "required": [ "person" ], "additionalProperties": false } } } }	4.6s	`{ "person": { "name": "Marie Curie", "address": { "city": "Warsaw", "country": "Poland" } } }`
✓ PASS	2.14 Mixed types (strict, streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Tell me about the movie The Matrix. Provide title, year, rating out of 10, and whether it is available." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "title": { "type": "string" }, "year": { "type": "integer" }, "rating": { "type": "number" }, "available": { "type": "boolean" } }, "required": [ "title", "year", "rating", "available" ], "additionalProperties": false } } } }	3.8s	`{ "title": "The Matrix", "year": 1999, "rating": 8.7, "available": true }`
Section total		44.3s

▶ MLX:gemma-3-4b-it-8bit 14/14 passed · 46.5s ALL PASS

Status	Test / Prompt	Time	Detail / Output
✓ PASS	2.1 Simple object (strict, non-streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is the capital of France? Answer with the country name and capital." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "name": { "type": "string" }, "capital": { "type": "string" } }, "required": [ "name", "capital" ], "additionalProperties": false } } } }`	3.3s	`{ "name": "France", "capital": "Paris" }`
✓ PASS	2.2 Simple object (strict, streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is the capital of Germany? Answer with the country name and capital." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "name": { "type": "string" }, "capital": { "type": "string" } }, "required": [ "name", "capital" ], "additionalProperties": false } } } }`	2.0s	`{ "name": "Germany", "capital": "Berlin" }`
✓ PASS	2.3 Array of objects (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 3 colors with their hex codes." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "hex": { "type": "string" } }, "required": [ "name", "hex" ] } } }, "required": [ "colors" ], "additionalProperties": false } } } }	2.5s	`{ "colors": [ { "name": "Red", "hex": "FF0000" }, { "name": "Blue", "hex": "0000FF" }, { "name": "Green", "hex": "008000" } ] }`
✓ PASS	2.4 Array of objects (strict, streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 2 colors with their hex codes." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "hex": { "type": "string" } }, "required": [ "name", "hex" ] } } }, "required": [ "colors" ], "additionalProperties": false } } } }	4.6s	`{ "colors": [ { "name": "Black", "hex": "#000000" }, { "name": "White", "hex": "#FFFFFF" } ] }`
✓ PASS	2.5 Mixed types (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Tell me about the movie Inception. Provide title, year, rating out of 10, and whether it is available on streaming." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "title": { "type": "string" }, "year": { "type": "integer" }, "rating": { "type": "number" }, "available": { "type": "boolean" } }, "required": [ "title", "year", "rating", "available" ], "additionalProperties": false } } } }	2.7s	`{ "title": "Inception", "year": 2010, "rating": 8.8, "available": true }`
✓ PASS	2.6 Nested objects (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Give me info about Albert Einstein including his city and country." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "person": { "type": "object", "properties": { "name": { "type": "string" }, "address": { "type": "object", "properties": { "city": { "type": "string" }, "country": { "type": "string" } }, "required": [ "city", "country" ] } }, "required": [ "name", "address" ] } }, "required": [ "person" ], "additionalProperties": false } } } }	2.6s	`{ "person": { "name": "Albert Einstein", "address": { "city": "Ulm", "country": "Germany" } } }`
✓ PASS	2.7 String enum (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Analyze the sentiment of: I love this product, it is amazing! Return sentiment and confidence." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "sentiment": { "type": "string", "enum": [ "positive", "negative", "neutral" ] }, "confidence": { "type": "number" } }, "required": [ "sentiment", "confidence" ], "additionalProperties": false } } } }	2.8s	`{ "sentiment": "positive", "confidence": 0.95 }`
✓ PASS	2.8 Bounded array (strict, non-streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "List some popular programming languages." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "items": { "type": "array", "items": { "type": "string" }, "minItems": 2, "maxItems": 4 } }, "required": [ "items" ], "additionalProperties": false } } } }`	2.8s	`{ "items": [ "Python", "JavaScript", "Java", "C++" ] }`
✓ PASS	2.9 Refs and defs (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 2 students with their names and letter grades." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "students": { "type": "array", "items": { "$ref": "#/$defs/Student" } } }, "required": [ "students" ], "$defs": { "Student": { "type": "object", "properties": { "name": { "type": "string" }, "grade": { "type": "string" } }, "required": [ "name", "grade" ] } }, "additionalProperties": false } } } }	2.7s	`{ "students": [ { "name": "Alice Johnson", "grade": "A" }, { "name": "Bob Smith", "grade": "B" } ] }`
✓ PASS	2.10 Non-strict json_schema ▶ Show request payload { "messages": [ { "role": "user", "content": "List 2 colors with hex codes as JSON." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": false, "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "hex": { "type": "string" } }, "required": [ "name", "hex" ] } } }, "required": [ "colors" ], "additionalProperties": false } } } }	3.9s	`{ "colors": [ { "name": "Red", "hex": "#FF0000" }, { "name": "Blue", "hex": "#0000FF" } ] }`
✓ PASS	2.11 json_object mode ▶ Show request payload `{ "messages": [ { "role": "user", "content": "Return a JSON object with an 'animals' key containing 2 animals, each with name and type fields." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_object" } }`	2.2s	`{ "animals": [ { "name": "Buddy", "type": "Dog" }, { "name": "Whiskers", "type": "Cat" } ] }`
✓ PASS	2.12 No response_format (plain text regression) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is 2+2?" } ], "max_tokens": 2000, "temperature": 0.3, "stream": false }`	4.1s	Got text response (325 chars)
✓ PASS	2.13 Nested objects (strict, streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Give me info about Marie Curie including her city and country." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "person": { "type": "object", "properties": { "name": { "type": "string" }, "address": { "type": "object", "properties": { "city": { "type": "string" }, "country": { "type": "string" } }, "required": [ "city", "country" ] } }, "required": [ "name", "address" ] } }, "required": [ "person" ], "additionalProperties": false } } } }	5.7s	`{ "person": { "name": "Marie Curie", "address": { "city": "Warsaw", "country": "Poland" } } }`
✓ PASS	2.14 Mixed types (strict, streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Tell me about the movie The Matrix. Provide title, year, rating out of 10, and whether it is available." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "title": { "type": "string" }, "year": { "type": "integer" }, "rating": { "type": "number" }, "available": { "type": "boolean" } }, "required": [ "title", "year", "rating", "available" ], "additionalProperties": false } } } }	4.6s	`{ "title": "The Matrix", "year": 1999, "rating": 8.7, "available": false }`
Section total		46.5s

▶ MLX:Qwen3-Coder-Next-4bit 14/14 passed · 45.9s ALL PASS

Status	Test / Prompt	Time	Detail / Output
✓ PASS	2.1 Simple object (strict, non-streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is the capital of France? Answer with the country name and capital." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "name": { "type": "string" }, "capital": { "type": "string" } }, "required": [ "name", "capital" ], "additionalProperties": false } } } }`	1.6s	`{ "name": "France", "capital": "Paris" }`
✓ PASS	2.2 Simple object (strict, streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is the capital of Germany? Answer with the country name and capital." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "name": { "type": "string" }, "capital": { "type": "string" } }, "required": [ "name", "capital" ], "additionalProperties": false } } } }`	2.1s	`{ "name": "Germany", "capital": "Berlin" }`
✓ PASS	2.3 Array of objects (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 3 colors with their hex codes." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "hex": { "type": "string" } }, "required": [ "name", "hex" ] } } }, "required": [ "colors" ], "additionalProperties": false } } } }	3.7s	`{ "colors": [ { "name": "Red", "hex": "#FF0000" }, { "name": "Blue", "hex": "#0000FF" }, { "name": "Green", "hex": "#008000" } ] }`
✓ PASS	2.4 Array of objects (strict, streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 2 colors with their hex codes." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "hex": { "type": "string" } }, "required": [ "name", "hex" ] } } }, "required": [ "colors" ], "additionalProperties": false } } } }	3.8s	`{ "colors": [ { "name": "Black", "hex": "#000000" }, { "name": "White", "hex": "#FFFFFF" } ] }`
✓ PASS	2.5 Mixed types (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Tell me about the movie Inception. Provide title, year, rating out of 10, and whether it is available on streaming." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "title": { "type": "string" }, "year": { "type": "integer" }, "rating": { "type": "number" }, "available": { "type": "boolean" } }, "required": [ "title", "year", "rating", "available" ], "additionalProperties": false } } } }	4.3s	`{ "title": "Inception", "year": 2010, "rating": 8.8, "available": true }`
✓ PASS	2.6 Nested objects (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Give me info about Albert Einstein including his city and country." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "person": { "type": "object", "properties": { "name": { "type": "string" }, "address": { "type": "object", "properties": { "city": { "type": "string" }, "country": { "type": "string" } }, "required": [ "city", "country" ] } }, "required": [ "name", "address" ] } }, "required": [ "person" ], "additionalProperties": false } } } }	2.6s	`{ "person": { "name": "Albert Einstein", "address": { "city": "Ulm", "country": "Germany" } } }`
✓ PASS	2.7 String enum (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Analyze the sentiment of: I love this product, it is amazing! Return sentiment and confidence." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "sentiment": { "type": "string", "enum": [ "positive", "negative", "neutral" ] }, "confidence": { "type": "number" } }, "required": [ "sentiment", "confidence" ], "additionalProperties": false } } } }	2.6s	`{ "sentiment": "positive", "confidence": 0.95 }`
✓ PASS	2.8 Bounded array (strict, non-streaming) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "List some popular programming languages." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "items": { "type": "array", "items": { "type": "string" }, "minItems": 2, "maxItems": 4 } }, "required": [ "items" ], "additionalProperties": false } } } }`	1.9s	`{ "items": [ "Python", "JavaScript", "Java", "C++" ] }`
✓ PASS	2.9 Refs and defs (strict, non-streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "List 2 students with their names and letter grades." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "students": { "type": "array", "items": { "$ref": "#/$defs/Student" } } }, "required": [ "students" ], "$defs": { "Student": { "type": "object", "properties": { "name": { "type": "string" }, "grade": { "type": "string" } }, "required": [ "name", "grade" ] } }, "additionalProperties": false } } } }	3.7s	`{ "students": [ { "name": "Alice", "grade": "A" }, { "name": "Bob", "grade": "B" } ] }`
✓ PASS	2.10 Non-strict json_schema ▶ Show request payload { "messages": [ { "role": "user", "content": "List 2 colors with hex codes as JSON." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": false, "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "hex": { "type": "string" } }, "required": [ "name", "hex" ] } } }, "required": [ "colors" ], "additionalProperties": false } } } }	4.7s	`{ "colors": [ { "name": "red", "hex": "#FF0000" }, { "name": "blue", "hex": "#0000FF" } ] }`
✓ PASS	2.11 json_object mode ▶ Show request payload `{ "messages": [ { "role": "user", "content": "Return a JSON object with an 'animals' key containing 2 animals, each with name and type fields." } ], "max_tokens": 2000, "temperature": 0.3, "stream": false, "response_format": { "type": "json_object" } }`	2.6s	`{ "animals": [ { "name": "Buddy", "type": "Dog" }, { "name": "Whiskers", "type": "Cat" } ] }`
✓ PASS	2.12 No response_format (plain text regression) ▶ Show request payload `{ "messages": [ { "role": "user", "content": "What is 2+2?" } ], "max_tokens": 2000, "temperature": 0.3, "stream": false }`	1.9s	Got text response (156 chars)
✓ PASS	2.13 Nested objects (strict, streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Give me info about Marie Curie including her city and country." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "person": { "type": "object", "properties": { "name": { "type": "string" }, "address": { "type": "object", "properties": { "city": { "type": "string" }, "country": { "type": "string" } }, "required": [ "city", "country" ] } }, "required": [ "name", "address" ] } }, "required": [ "person" ], "additionalProperties": false } } } }	4.1s	`{ "person": { "name": "Marie Curie", "address": { "city": "Warsaw", "country": "Poland" } } }`
✓ PASS	2.14 Mixed types (strict, streaming) ▶ Show request payload { "messages": [ { "role": "user", "content": "Tell me about the movie The Matrix. Provide title, year, rating out of 10, and whether it is available." } ], "max_tokens": 2000, "temperature": 0.3, "stream": true, "response_format": { "type": "json_schema", "json_schema": { "name": "test_schema", "strict": true, "schema": { "type": "object", "properties": { "title": { "type": "string" }, "year": { "type": "integer" }, "rating": { "type": "number" }, "available": { "type": "boolean" } }, "required": [ "title", "year", "rating", "available" ], "additionalProperties": false } } } }	6.3s	`{ "title": "The Matrix", "year": 1999, "rating": 8.7, "available": true }`
Section total		45.9s

▶ CLI:AFM 9/9 passed · 4.7s ALL PASS

Status	Test / Prompt	Time	Detail / Output
✓ PASS	3.1 --guided-json simple object ▶ Show request payload `{ "command": ".build/release/afm -s What is the capital of France? Answer with the country name and capital. --guided-json {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\"},\n \"capital\": {\"type\": \"string\"}\n },\n \"required\": [\"name\", \"capital\"],\n \"additionalProperties\": false\n}" }`	1.2s	`{ "name": "France", "capital": "Paris" }`
✓ PASS	3.2 --guided-json array of objects ▶ Show request payload `{ "command": ".build/release/afm -s List 2 colors with their hex codes. --guided-json {\n \"type\": \"object\",\n \"properties\": {\n \"colors\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\"},\n \"hex\": {\"type\": \"string\"}\n },\n \"required\": [\"name\", \"hex\"]\n }\n }\n },\n \"required\": [\"colors\"],\n \"additionalProperties\": false\n}" }`	1.0s	`{ "colors": [ { "name": "Crimson", "hex": "#FF5733" }, { "name": "SkyBlue", "hex": "#1A90FF" } ] }`
✓ PASS	3.3 --guided-json mixed types ▶ Show request payload `{ "command": ".build/release/afm -s Tell me about the movie Inception. Provide title, year, rating out of 10, and whether it is available on streaming. --guided-json {\n \"type\": \"object\",\n \"properties\": {\n \"title\": {\"type\": \"string\"},\n \"year\": {\"type\": \"integer\"},\n \"rating\": {\"type\": \"number\"},\n \"available\": {\"type\": \"boolean\"}\n },\n \"required\": [\"title\", \"year\", \"rating\", \"available\"],\n \"additionalProperties\": false\n}" }`	0.6s	`{ "available": true, "rating": 8.8, "year": 2010, "title": "Inception" }`
✓ PASS	3.4 --guided-json nested objects ▶ Show request payload { "command": ".build/release/afm -s Give me info about Albert Einstein including his city and country. --guided-json {\n \"type\": \"object\",\n \"properties\": {\n \"person\": {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\"},\n \"address\": {\n \"type\": \"object\",\n \"properties\": {\n \"city\": {\"type\": \"string\"},\n \"country\": {\"type\": \"string\"}\n },\n \"required\": [\"city\", \"country\"]\n }\n },\n \"required\": [\"name\", \"address\"]\n }\n },\n \"required\": [\"person\"],\n \"additionalProperties\": false\n}" }	0.5s	`{ "address": { "city": "Munich", "country": "Germany" }, "name": "Albert Einstein" }`
✓ PASS	3.5 --guided-json string enum ▶ Show request payload `{ "command": ".build/release/afm -s Analyze the sentiment of: I love this product, it is amazing! --guided-json {\n \"type\": \"object\",\n \"properties\": {\n \"sentiment\": {\n \"type\": \"string\",\n \"enum\": [\"positive\", \"negative\", \"neutral\"]\n },\n \"confidence\": {\"type\": \"number\"}\n },\n \"required\": [\"sentiment\", \"confidence\"],\n \"additionalProperties\": false\n}" }`	0.4s	`{ "confidence": 1, "sentiment": "positive" }`
✓ PASS	3.6 --guided-json refs and defs ▶ Show request payload { "command": ".build/release/afm -s List 2 students with their names and letter grades. --guided-json {\n \"type\": \"object\",\n \"properties\": {\n \"students\": {\n \"type\": \"array\",\n \"items\": {\"$ref\": \"#/$defs/Student\"}\n }\n },\n \"required\": [\"students\"],\n \"$defs\": {\n \"Student\": {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\"},\n \"grade\": {\"type\": \"string\"}\n },\n \"required\": [\"name\", \"grade\"]\n }\n },\n \"additionalProperties\": false\n}" }	0.7s	`{ "students": [ { "grade": "A+", "name": "John Doe" }, { "name": "Jane Smith", "grade": "B-" } ] }`
✓ PASS	3.7 Plain text (no --guided-json, regression) ▶ Show request payload `{ "command": ".build/release/afm -s What is 2+2?" }`	0.3s	Got text response (10 chars)
✓ PASS	3.8 Invalid --guided-json (error) ▶ Show request payload `{ "command": ".build/release/afm -s Hello --guided-json not valid json" }`	0.0s	Exit code 1: Error: The operation couldn’t be completed. (ArgumentParser.ValidationError error 1.)
✓ PASS	3.9 --guided-json with array schema (error) ▶ Show request payload `{ "command": ".build/release/afm -s Hello --guided-json [1,2,3]" }`	0.0s	Exit code 1: Error: The operation couldn’t be completed. (ArgumentParser.ValidationError error 1.)
Section total		4.7s

▶ CLI:MLX:Qwen3-30B-A3B-4bit 9/9 passed · 31.5s ALL PASS

Status	Test / Prompt	Time	Detail / Output
✓ PASS	4.1 --guided-json simple object ▶ Show request payload `{ "command": ".build/release/afm mlx -m mlx-community/Qwen3-30B-A3B-4bit -s What is the capital of France? Answer with the country name and capital. --guided-json {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\"},\n \"capital\": {\"type\": \"string\"}\n },\n \"required\": [\"name\", \"capital\"],\n \"additionalProperties\": false\n}" }`	3.5s	`{ "name": "France", "capital": "Paris" }`
✓ PASS	4.2 --guided-json array of objects ▶ Show request payload `{ "command": ".build/release/afm mlx -m mlx-community/Qwen3-30B-A3B-4bit -s List 2 colors with their hex codes. --guided-json {\n \"type\": \"object\",\n \"properties\": {\n \"colors\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\"},\n \"hex\": {\"type\": \"string\"}\n },\n \"required\": [\"name\", \"hex\"]\n }\n }\n },\n \"required\": [\"colors\"],\n \"additionalProperties\": false\n}" }`	4.9s	`{ "colors": [ { "name": "Red", "hex": "#FF0000" }, { "name": "Blue", "hex": "#0000FF" } ] }`
✓ PASS	4.3 --guided-json mixed types ▶ Show request payload { "command": ".build/release/afm mlx -m mlx-community/Qwen3-30B-A3B-4bit -s Tell me about the movie Inception. Provide title, year, rating out of 10, and whether it is available on streaming. --guided-json {\n \"type\": \"object\",\n \"properties\": {\n \"title\": {\"type\": \"string\"},\n \"year\": {\"type\": \"integer\"},\n \"rating\": {\"type\": \"number\"},\n \"available\": {\"type\": \"boolean\"}\n },\n \"required\": [\"title\", \"year\", \"rating\", \"available\"],\n \"additionalProperties\": false\n}" }	4.4s	`{ "title": "Inception", "year": 2010, "rating": 8.8, "available": true }`
✓ PASS	4.4 --guided-json nested objects ▶ Show request payload { "command": ".build/release/afm mlx -m mlx-community/Qwen3-30B-A3B-4bit -s Give me info about Albert Einstein including his city and country. --guided-json {\n \"type\": \"object\",\n \"properties\": {\n \"person\": {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\"},\n \"address\": {\n \"type\": \"object\",\n \"properties\": {\n \"city\": {\"type\": \"string\"},\n \"country\": {\"type\": \"string\"}\n },\n \"required\": [\"city\", \"country\"]\n }\n },\n \"required\": [\"name\", \"address\"]\n }\n },\n \"required\": [\"person\"],\n \"additionalProperties\": false\n}" }	3.7s	`{ "name": "Albert Einstein", "address": { "city": "Ulm", "country": "Germany" } }`
✓ PASS	4.5 --guided-json string enum ▶ Show request payload `{ "command": ".build/release/afm mlx -m mlx-community/Qwen3-30B-A3B-4bit -s Analyze the sentiment of: I love this product, it is amazing! Return sentiment and confidence. --guided-json {\n \"type\": \"object\",\n \"properties\": {\n \"sentiment\": {\n \"type\": \"string\",\n \"enum\": [\"positive\", \"negative\", \"neutral\"]\n },\n \"confidence\": {\"type\": \"number\"}\n },\n \"required\": [\"sentiment\", \"confidence\"],\n \"additionalProperties\": false\n}" }`	3.4s	`{ "sentiment": "positive", "confidence": 0.9 }`
✓ PASS	4.6 --guided-json refs and defs ▶ Show request payload { "command": ".build/release/afm mlx -m mlx-community/Qwen3-30B-A3B-4bit -s List 2 students with their names and letter grades. --guided-json {\n \"type\": \"object\",\n \"properties\": {\n \"students\": {\n \"type\": \"array\",\n \"items\": {\"$ref\": \"#/$defs/Student\"}\n }\n },\n \"required\": [\"students\"],\n \"$defs\": {\n \"Student\": {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\"},\n \"grade\": {\"type\": \"string\"}\n },\n \"required\": [\"name\", \"grade\"]\n }\n },\n \"additionalProperties\": false\n}" }	4.1s	`{ "students": [ { "name": "Alice", "grade": "A" }, { "name": "Bob", "grade": "B" } ] }`
✓ PASS	4.7 Plain text (no --guided-json, regression) ▶ Show request payload `{ "command": ".build/release/afm mlx -m mlx-community/Qwen3-30B-A3B-4bit -s What is 2+2?" }`	5.1s	Got text response (2369 chars)
✓ PASS	4.8 Invalid --guided-json (error) ▶ Show request payload `{ "command": ".build/release/afm mlx -m mlx-community/Qwen3-30B-A3B-4bit -s Hello --guided-json not valid json" }`	1.2s	Exit code 1: MLX model: mlx-community/Qwen3-30B-A3B-4bit Loading MLX model: mlx-community/Qwen3-30B-A3B-4bit [/
✓ PASS	4.9 --guided-json with array schema (error) ▶ Show request payload `{ "command": ".build/release/afm mlx -m mlx-community/Qwen3-30B-A3B-4bit -s Hello --guided-json [1,2,3]" }`	1.2s	Exit code 1: MLX model: mlx-community/Qwen3-30B-A3B-4bit Loading MLX model: mlx-community/Qwen3-30B-A3B-4bit [/
Section total		31.5s

Source data: structured-outputs-20260223_174042.jsonl