Structured Outputs
Structured Outputs refers to techniques that force an LLM to return data in a specific, machine-readable format (like valid JSON conforming to a JSON Schema) in...
Definition
Structured Outputs refers to techniques that force an LLM to return data in a specific, machine-readable format (like valid JSON conforming to a JSON Schema) instead of unstructured natural language text.
Why It Exists
Integrating LLMs into production pipelines requires feeding their output into other software systems. If a model returns natural language (e.g., "The status is successful"), your program cannot parse it reliably. Forcing the model to return structured data (e.g., {"status": "success"}) allows you to parse it directly into database models.
How It Works
STRUCTURED OUTPUT PIPELINE
┌──────────────┐
│ Prompt + │ ──► [ LLM Generation ] ──► [ Output Validation ]
│ JSON Schema │ - Verify matches Schema
└──────────────┘ - Catch syntax issues
│
├─► [ Success ] ──► Parsed Pydantic Model
│
└─► [ Failure ] ──► Retry / Repair Loop
1. **Schema Injection**: The application passes a JSON schema to the model API.
2. **Model Constraint**: The model adjusts its token generation probabilities to only allow tokens that conform to the target schema.
3. **Pydantic Validation**: The application parses the returned JSON string into a Pydantic model, throwing a structured error if validation fails.Real-World Example
In Cursor, when you select edit options, the backend requests structured JSON outputs defining the file paths, line numbers, and specific diff changes, allowing the editor to apply the code edits automatically.
Python Example
Here is a production-style pattern demonstrating how to define a validation schema using Pydantic, validate mock JSON outputs, and implement a repair fallback if validation fails:
from pydantic import BaseModel, Field, ValidationError
from typing import Dict, Any, Optional
# Define the target structure using Pydantic
class TicketDetails(BaseModel):
priority: str = Field(..., description="High, Medium, or Low")
category: str = Field(..., description="Billing, Technical, or General")
assigned_team: str = Field(..., description="Target queue queue")
def parse_model_output(raw_text: str) -> Optional[TicketDetails]:
"""Parses and validates raw text output against Pydantic schema model."""
try:
# Validate and instantiate
return TicketDetails.model_validate_json(raw_text)
except ValidationError as e:
print(f"Validation failed: {e}")
return None
# Simulation
if __name__ == "__main__":
# Case 1: Valid structured response from LLM
llm_output_valid = '{"priority": "High", "category": "Billing", "assigned_team": "finance_team"}'
details = parse_model_output(llm_output_valid)
if details:
print("Success! Assigned queue queue:", details.assigned_team)
# Case 2: Invalid structured response (missing assigned_team)
llm_output_invalid = '{"priority": "Low", "category": "Technical"}'
print("\nAttempting to parse invalid output:")
parse_model_output(llm_output_invalid)Interview Questions
- Beginner: What format is most commonly used for structured LLM outputs?
- Intermediate: What is Function Calling (or Tool Use) in LLMs and how does it relate to structured outputs?
- Advanced: How do modern LLM APIs (like Gemini or OpenAI) guarantee that their generated text conforms perfectly to a Pydantic schema?
Interview Answers
- Beginner: JSON is the most common format, as it is native to web services and easily parsed by modern programming languages.
- Intermediate: Function Calling is a technique where the model is provided with a list of tools (functions) defined by their name, description, and JSON schema parameters. Instead of answering a query directly, the model outputs a structured JSON object containing the function name and the arguments to call it.
- Advanced: They use Constrained Decoding (Grammar-guided decoding). During token generation, the API gateway uses a compiler-style parser to inspect the output against the JSON schema. It dynamically adjusts the model's output probabilities, setting the probability of invalid tokens (like letters when a number is required) to zero, guaranteeing that the model only generates schema-compliant tokens.
Common Mistakes
- Relying solely on system prompts: Trying to force structured outputs using only prompt instructions (e.g.,
"Output only JSON"). Models regular slip up and include markdown wrappers or conversational text. Always combine your prompts with API-level schema constraints or Pydantic validation layers.
Assignment
Define a Pydantic model class representing an Ingredient (name: str, quantity: float, unit: str). Write a validation wrapper that parses a mock JSON list of ingredients safely.