Getting reliable JSON and structured output from LLMs

Define a tight schema

Constrain values wherever possible.

{
  "type": "object",
  "required": ["category", "priority", "summary"],
  "properties": {
    "category": {"enum": ["billing", "bug", "how_to", "other"]},
    "priority": {"enum": ["low", "medium", "high"]},
    "summary": {"type": "string", "maxLength": 200}
  },
  "additionalProperties": false
}

Use provider features

Many APIs support JSON mode or schema-constrained output, which greatly reduces malformed responses. Still validate — constrained output can be well-formed but wrong.

Validate, retry, fall back

Treat model output as untrusted input.

1. Parse and validate against the schema (for example with a schema validation library).
2. On failure, retry once, including the validation error in the prompt.
3. If it still fails, log and route to a fallback path or human review.

Check semantics, not just shape

A valid JSON object can still contain the wrong category. Include structured-output cases in your evaluation set and check field-level accuracy.

Structured-output mistakes

These cause fragile pipelines.

1. Describing the JSON format only in prose instead of a schema.
2. Allowing free-text fields where an enum would work.
3. Crashing the whole batch on one malformed response.
4. Checking that JSON parses but never checking the values.

Worked example: invoice field extraction

A team extracts vendor name, invoice number, date, and total from invoices. They define a schema with a date format and numeric total, validate each response, and cross-check the extracted total against line items where available.

Responses that fail validation get one retry with the error message; persistent failures go to a review queue. Measuring field-level accuracy on 100 labelled invoices shows dates are the weakest field, so they add examples of tricky date formats to the prompt.