| # | Test | Group | Coverage | Status | Duration | Details |
|---|---|---|---|---|---|---|
| 1 | swift test execution | UnitTest | unit smoke standard full | FAIL | 2.7s | Expected: test output\nActual: FAIL: no parseable output |
| 2 | Binary exists at .build/release/afm | Preflight | unit smoke standard full | PASS | 0.0s | file exists |
| 3 | Server reachable at http://127.0.0.1:9998 | Preflight | unit smoke standard full | PASS | 0.0s | 200 OK |
| 4 | /v1/models contains model ID | Lifecycle | unit smoke standard full | PASS | 0.0s | model in response |
| 5 | Basic completion returns content | Lifecycle | unit smoke standard full | PASS | 1.6s | non-empty content or reasoning |
| 6 | Stop string '5' absent from output | Stop | unit smoke standard full | PASS | 0.3s | no '5' in content |
| 7 | finish_reason is 'stop' with stop sequence | Stop | unit smoke standard full | PASS | 0.0s | stop |
| 8 | Multi-word stop 'and' truncates correctly | Stop | unit smoke standard full | PASS | 0.4s | no 'and' in output |
| 9 | Stop on newline produces single line | Stop | unit smoke standard full | PASS | 0.4s | single line |
| 10 | Multiple stop sequences [7, 12] | Stop | unit smoke standard full | PASS | 0.8s | neither found |
| 11 | Empty stop array is no-op | Stop | unit smoke standard full | PASS | 0.3s | valid output |
| 12 | Streaming: stop string '5' absent | Stop | unit smoke standard full | FAIL | 0.4s | Expected: no '5' in stream\nActual: FAIL: found '5' |
| 13 | Stop sequence '3.' truncates list | Stop | standard full | PASS | 0.4s | no '3.' in output |
| 14 | Stop 'stopped' doesn't fire on 'stopping' | Stop | standard full | PASS | 0.1s | output produced |
| 15 | Stop 'llo' fires mid-word in 'hello' | Stop | standard full | PASS | 0.1s | no 'llo' |
| 16 | ChoiceLogprobs JSON schema valid | Logprobs | smoke standard full | PASS | 0.2s | valid schema |
| 17 | top_logprobs count <= requested (5) | Logprobs | smoke standard full | PASS | 0.2s | count valid |
| 18 | logprobs=false returns null | Logprobs | smoke standard full | PASS | 0.2s | null logprobs |
| 19 | top_logprobs=99 returns 400 | Logprobs | smoke standard full | PASS | 0.0s | 400 |
| 20 | Streaming logprobs present and valid | Logprobs | standard full | PASS | 0.2s | valid |
| 21 | top_logprobs=0 returns empty arrays | Logprobs | standard full | PASS | 0.2s | empty top_logprobs |
| 22 | Think extraction (model lacks <think> support) | Think | smoke standard full | SKIP | skip |
|
| 23 | Basic tool call: finish_reason=tool_calls, valid args | Tools | smoke standard full | PASS | 0.9s | valid |
| 24 | tool_choice=none suppresses tool calls | Tools | smoke standard full | PASS | 2.0s | no tool calls |
| 25 | Tool arguments are valid JSON dict | Tools | standard full | PASS | 0.7s | valid JSON |
| 26 | Streaming: tool calls with finish_reason | Tools | standard full | PASS | 0.7s | valid |
| 27 | Multi-tool: at least 1 tool call with 2 tools | Tools | standard full | PASS | 1.2s | >=1 calls |
| 28 | Array param: todos is JSON array (not string) | Tools | standard full | PASS | 0.9s | array |
| 29 | Nullable param: anyOf [string, null] does not crash | Tools | standard full | PASS | 0.7s | no crash |
| 30 | Multi-turn: tool result produces valid follow-up | Tools | standard full | PASS | 1.6s | text response |
| 31 | Multi-turn nullable: full round-trip (anyOf params) | Tools | standard full | PASS | 2.1s | valid follow-up |
| 32 | No tools: normal text response | Tools | standard full | PASS | 0.3s | text response |
| 33 | First unique request: uncached suffix remains | Cache | standard full | PASS | 0.5s | cached_tokens < prompt_tokens |
| 34 | Shared-prefix request: cached_tokens>0 | Cache | standard full | PASS | 0.2s | >0 |
| 35 | Exact replay: cached response matches cold response | Cache | standard full | PASS | 0.5s | same content |
| 36 | Different unique prompt: uncached suffix remains | Cache | standard full | PASS | 0.5s | cached_tokens < prompt_tokens |
| 37 | Streaming shared-prefix: cached_tokens>0 in usage chunk | Cache | standard full | PASS | 0.8s | >0 |
| 38 | Streaming replay: content matches non-streaming warmup | Cache | standard full | PASS | 1.0s | same content |
| 39 | Concurrent x8 shared-prefix: uncached suffix remains on every branch | Cache | standard full | FAIL | 5.4s | Expected: 0 < cached_tokens < prompt_tokens for all 8\nActual: FAIL: 1:0/55, 3:0/55, 4:0/55, 5:0/55, 6:0/55, 7:0/55, 8:0/55 |
| 40 | Concurrent x8 shared-prefix: divergent suffix responses stay isolated | Cache | standard full | FAIL | 5.4s | Expected: each of 8 responses keeps only its own marker\nActual: FAIL: 1:error=Expecting value: line 1 column 1 (char 0); 2:error=Expecting value: line 1 column 1 (char 0); 3:error=Expecting value: line 1 column 1 (char 0); 4:error=Expecting value: line 1 column 1 (char 0); 5:error=Expecting value: line 1 column 1 (char 0); 6:error=Expecting value: line 1 column 1 (char 0); 7:error=Expecting value: line 1 column 1 (char 0); 8:error=Expecting value: line 1 column 1 (char 0) |
| 41 | Two simultaneous requests: both 200 | Concurrent | standard full | PASS | 0.2s | 200+200 |
| 42 | Three simultaneous requests: all 200 | Concurrent | standard full | PASS | 0.3s | all 200 |
| 43 | Empty messages → 400 | Error | smoke standard full | PASS | 0.0s | 400 |
| 44 | Malformed JSON → 400 | Error | smoke standard full | PASS | 0.0s | 400 |
| 45 | Missing messages field → 400 | Error | smoke standard full | PASS | 0.0s | 400 |
| 46 | response_format json_object returns valid JSON | Error | smoke standard full | PASS | 0.2s | valid JSON |
| 47 | max_tokens=5 is respected | Error | smoke standard full | PASS | 0.2s | <=10 tokens |
| 48 | OPTIONS /v1/chat/completions → 200 (CORS) | Error | smoke standard full | PASS | 0.0s | 200 |
| 49 | developer role accepted (mapped to system) | Error | smoke standard full | PASS | 3.3s | valid response |
| 50 | chat_template_kwargs (model lacks thinking) | Kwargs | standard full | SKIP | skip |
|
| 51 | Prefix cache reuse across requests | Cache | standard full | PASS | 0.5s | valid response on cache hit |
| 52 | Issue #32: sequential guided-json → nullable tool (no crash) | Cache | standard full | PASS | 2.0s | both succeed |
| 53 | Sequential: 3 nullable tool calls (cache reuse) | Cache | standard full | PASS | 2.6s | 3/3 succeed |
| 54 | Multi-turn: 5-msg tool conversation with prefix cache | Cache | standard full | PASS | 1.8s | valid response |
| 55 | Long system prompt: 3 sequential requests (prefix reuse) | Cache | standard full | PASS | 10.2s | 3/3 succeed |
| 56 | Sequential stress: 5 mixed requests (text/tool/json/stream/nullable) | Cache | standard full | PASS | 2.7s | 5/5 succeed |
| 57 | Multi-turn: 5-turn growing conversation (prefix reuse) | Cache | standard full | PASS | 3.9s | 5/5 turns |
| 58 | Streaming + non-streaming sequential (shared prefix) | Cache | standard full | PASS | 0.6s | 3/3 succeed |
| 59 | Issue #32 comprehensive: nullable tool + multi-turn + prefix | Cache | standard full | PASS | 2.0s | valid response |
| 60 | json_schema produces valid schema-matching JSON | Structured | standard full | PASS | 0.4s | valid JSON |
| 61 | Function name correctly extracted | XMLTools | standard full | PASS | 1.0s | get_weather |
| 62 | Parameter values are correct string types | XMLTools | standard full | PASS | 1.0s | strings |
| 63 | Mixed-type params (string+bool+int) parse correctly | XMLTools | standard full | PASS | 1.3s | valid types |
| 64 | Nested object param survives XML parsing | XMLTools | standard full | PASS | 1.3s | valid dict |
| 65 | tool_choice=required forces tool call | XMLTools | standard full | PASS | 1.0s | tool_calls |
| 66 | tool_choice={function: get_time} calls correct function | XMLTools | standard full | PASS | 0.8s | get_time |
| 67 | Tool call IDs are unique | XMLTools | standard full | PASS | 1.3s | unique IDs |
| 68 | Streaming: XML tool call assembles valid JSON args | XMLTools | standard full | PASS | 0.8s | valid |
| 69 | Streaming: array param is JSON array (not string) | XMLTools | standard full | PASS | 1.0s | array |
| 70 | Tool call matches OpenAI schema (id, type, function.name, function.arguments) | XMLTools | standard full | PASS | 1.0s | valid schema |
| 71 | Normal XML tool call works with afm_adaptive_xml | AdaptiveXML | standard full | PASS | 1.0s | valid tool call |
| 72 | Streaming tool call emits valid deltas | AdaptiveXML | standard full | PASS | 1.0s | name + JSON args |
| 73 | Multi-turn: model responds after tool result | AdaptiveXML | standard full | PASS | 0.7s | non-empty content |
| 74 | tool_choice=none suppresses tool calls | AdaptiveXML | standard full | PASS | 2.1s | no tool_calls |
| 75 | Multiple tools: correct function selected | AdaptiveXML | standard full | PASS | 0.8s | get_time |
| 76 | Argument types coerced (string, bool, int) | AdaptiveXML | standard full | FAIL | 1.2s | Expected: correct types\nActual: FAIL: type coercion: ['case_sensitive=str(True)', 'max_results=str(5)'] |
| 77 | Tool call valid (with or without grammar constraints) | AdaptiveXML | standard full | PASS | 1.0s | valid |
| 78 | Array of objects coercion (question tool pattern) | AdaptiveXML | standard full | PASS | 2.0s | array of dicts |
| 79 | Number (float) and boolean coercion | AdaptiveXML | standard full | PASS | 1.0s | correct types |
| 80 | Nested object with typed fields coercion | AdaptiveXML | standard full | PASS | 1.3s | correct types |
| 81 | Streaming array coercion (incremental path) | AdaptiveXML | standard full | PASS | 1.3s | array in stream |
| 82 | XML entity decoding in tool call values | AdaptiveXML | standard full | PASS | 2.1s | decoded |
| 83 | EBNF grammar enforces all required params present | AdaptiveXML | standard full | PASS | 1.2s | command + description |
| 84 | EBNF structured params: array gets json_array constraint | AdaptiveXML | standard full | PASS | 1.2s | tags is array |
| 85 | Calculator tool call (non-streaming) | Grammar | standard full | PASS | 1.1s | calculate(expression) |
| 86 | Calculator tool call (streaming) | Grammar | standard full | PASS | 0.8s | calculate(expression) via SSE |
| 87 | Two tools: grammar allows correct selection | Grammar | standard full | PASS | 0.9s | get_weather selected |
| 88 | Two tools: grammar selects calculate | Grammar | standard full | PASS | 0.7s | calculate selected |
| 89 | Grammar enforces 3 required params (send_email) | Grammar | standard full | PASS | 1.4s | to + subject + body |
| 90 | Grammar constrains array param at generation time | Grammar | standard full | PASS | 1.3s | tags is array |
| 91 | Grammar array param via streaming | Grammar | standard full | PASS | 1.1s | tags is array (SSE) |
| 92 | Complex schema: string + int + array + object | Grammar | standard full | PASS | 2.1s | correct types |
| 93 | Header absent when grammar enabled (tool strict:true) | StrictWiring | smoke standard full | PASS | 0.8s | no header |
| 94 | Header absent when grammar enabled (schema strict:true) | StrictWiring | smoke standard full | PASS | 0.4s | no header |
| 95 | No header when strict absent | StrictWiring | smoke standard full | PASS | 0.6s | no header |
| 96 | Streaming json_schema strict:true returns valid JSON | StrictWiring | smoke standard full | PASS | 0.4s | valid JSON with name+age |
| 97 | Streaming tool strict:true returns valid tool call | StrictWiring | smoke standard full | PASS | 0.8s | get_weather(location) |
| 98 | strict:false does not error (best-effort) | StrictWiring | smoke standard full | PASS | 0.7s | valid response |
| 99 | POST /v1/files upload returns file ID | Batch | standard full | PASS | 0.0s | file-* |
| 100 | GET /v1/files/:id returns file metadata | Batch | standard full | PASS | 0.0s | file object |
| 101 | POST /v1/batches creates batch | Batch | standard full | PASS | 0.0s | in_progress/completed |
| 102 | GET /v1/batches/:id polls to completed | Batch | standard full | PASS | 2.1s | completed |
| 103 | Output JSONL contains both results (2/2) | Batch | standard full | PASS | 0.0s | assert-1 + assert-2 |
| 104 | GET /v1/batches lists completed batch | Batch | standard full | PASS | 0.0s | list with ≥1 batch |
| 105 | DELETE /v1/files/:id removes uploaded file | Batch | standard full | PASS | 0.0s | deleted=true |
| 106 | SSE multiplex: 2 non-streaming requests tagged with custom_id | Batch | standard full | PASS | 0.7s | sse-a + sse-b + [DONE] |
| 107 | SSE multiplex: streaming interleaved with finish_reason | Batch | standard full | PASS | 0.6s | str-x + str-y chunks + stop + [DONE] |
| 108 | SSE multiplex rejects duplicate custom_ids | Batch | standard full | PASS | 0.0s | 400 |
| 109 | SSE multiplex rejects empty requests | Batch | standard full | PASS | 0.0s | 400 |
| 110 | batch + top_k + streaming | PairwiseSmoke | standard full | PASS | 0.2s | content present |
| 111 | batch + presence_penalty + non-streaming | PairwiseSmoke | standard full | PASS | 0.2s | finish_reason present |
| 112 | batch + repetition_penalty + logprobs | PairwiseSmoke | standard full | PASS | 0.2s | response valid |
| 113 | batch + all sampling params combined | PairwiseSmoke | standard full | PASS | 0.2s | response valid |
| 114 | streaming parity (same seed → same output) | PairwiseSmoke | standard full | PASS | 0.3s | match |
| 115 | cache idempotency (same seed → same output) | PairwiseSmoke | standard full | PASS | 0.3s | match |