Questionnaire Flag Bench

The count is an outcome. It should be an input.

When one call evaluates the whole questionnaire and the response is built from the returned array, the model decides how many results exist. Long arrays lose items: truncation when output tokens run short, merging of near-identical questions, one malformed character sending the parse into an except branch that returns an empty list. None of it raises, because nothing is holding a list of what it was owed.

Reconciliation flips that. A slot is created for every question id before any call is made. Calls fan out one per question, each retried on its own under a concurrency limit, each filling exactly one slot. Any slot still empty at the end becomes an explicit Unknown carrying the reason it failed. The count now comes from the input, so 28 results for 30 questions isn't a bug to catch. It can't be expressed.

What it guarantees

The count, absolutely. 30 questions give 30 flags under rate limits, timeouts, refusals and total outage. Turn the failure rate to 30% and watch it hold.

Not verdicts. Batched inference is non-deterministic even at fixed sampling settings, so a borderline question can flip Yes/Unknown between runs. That's a separate problem, handled with pinned retrieval order, schema-constrained generation and cached context, and measured separately as a flip rate. Treating the two as one metric is why this class of bug stays open.

Retrieval is the other half

A pipeline can be deterministic at the model layer and still flip verdicts because the evidence moved. ANN indexes don't promise identical neighbours across queries, so retrieval needs a fixed top_k and a total ordering. Sort by (-score, chunk_id) so ties never resolve by whichever candidate the index happened to emit first.

# one slot per question, before any call
slots = {q.id: None for q in questions}

# fan out, bounded, independent retries
async def run(q):
    async with sem:
        slots[q.id] = await evaluate_one(q)

# without return_exceptions the first failure
# cancels its siblings and shortens the set
await asyncio.gather(
    *(run(q) for q in questions),
    return_exceptions=True,
)

# every empty slot becomes an auditable Unknown
results = []
for q in questions:
    r = slots[q.id]
    if r is None:
        r = EvalResult(
            question_id=q.id,
            verdict=Verdict.UNKNOWN,
            status="floored",
            error=last_error,
        )
    results.append(r)

# the validator refuses to build a report
# whose count disagrees with the input
assert len(results) == len(questions)