Production checklist for shipping an AI feature

Quality

Know what “good enough” means before launch.

1. Evaluation set covering common, edge, and adversarial cases.
2. Agreed thresholds for accuracy, refusals, and latency.
3. Regression runs wired into CI.

Safety and privacy

Review risks with security and privacy owners.

1. Prompt-injection and misuse testing.
2. Personal data minimised and handled per policy and law.
3. Output filtering where needed; clear disclosure that AI is involved.

Operations

Treat the model as an unreliable dependency.

1. Timeouts, retries, and fallbacks when the model is slow or down.
2. Logs and dashboards for errors, latency, cost, and feedback.
3. Spend limits and alerts.
4. Feature flag for quick rollback.

Rollout

Release to internal users, then a small percentage, reviewing feedback and logs at each step before expanding.

Launch mistakes teams regret

These are common post-launch surprises.

1. No fallback when the model provider has an outage.
2. No way for users to report bad outputs.
3. Logs that store personal data unnecessarily.
4. Launching to everyone at once with no feature flag.

Worked example: a staged launch

A team launches an AI summary feature to internal staff for two weeks, collecting feedback via a thumbs-up/down control with optional comments. They fix two recurring issues, then release to a small share of customers with monitoring on latency, cost, and feedback.

After another fortnight with stable metrics, they expand gradually. A provider outage during the rollout triggers the fallback — showing the original content without a summary — and users barely notice.