12. Operations and Observability
12. Operations and Observability
Operational goals
Operations should answer four questions quickly: Is the storefront available? Are orders moving? Are payments and courier updates reconciled? Is the system exposing data or producing unsafe answers? Observability must support those questions without logging sensitive data.
Health checks
Expose a lightweight public health check that confirms the Worker is serving. Keep deeper dependency checks authenticated or internal so health probing cannot expose database state. Track D1, media, payment sandbox/production adapter, courier adapter, AI, Vectorize, analytics, and notification health independently.
Logs and metrics
| Signal | Required dimensions | PII rule |
|---|---|---|
| Request latency | route, status, environment | No request body by default |
| Error count | route, error code, provider | No tokens or raw provider payloads |
| Checkout failures | stage, code, retryable | Hash/order reference only |
| Payment reconciliation | order, attempt, provider status | Redact customer fields |
| Chat quality | channel, intent, retrieval confidence, fallback | No raw chat by default |
| Inventory | adjustment type, actor scope, result | No customer address |
| Security | auth failure, rate limit, policy block | Avoid raw credentials/PII |
Cloudflare Workers includes observability features, and AI Gateway can add request/token/cost analytics and logging for AI traffic.1 2 Use sampling and retention limits appropriate to traffic and privacy policy.
Backups and recovery
D1 Time Travel can support point-in-time recovery within its documented window.3 Recovery is not a substitute for a written runbook. Export critical business data on an approved schedule, store it securely, test restoration in a separate environment, and record the last successful restore test.
Media requires a separate recovery plan: list important buckets/paths, retention, versioning where applicable, and restore verification. Vectorize can be rebuilt from versioned D1 knowledge documents, so D1 source content and sync status are more important than treating the index as the only copy.
Payment and order reconciliation
Run reconciliation for pending, unknown, confirmed, cancelled, refunded, and provider-disputed attempts. Compare internal order totals, attempt references, provider status, and settlement evidence. Unknown states require manual review or a safe retry path; never silently convert unknown to paid.
Support workflow
Support agents should be able to search by order code, verify identity, see the safe order timeline, add a private note, escalate to the owner, and link a knowledge document when the chatbot lacks an answer. Public chat escalation should route to the approved support channel without exposing internal notes.
Incident response
- Identify the affected route, environment, data class, and time window.
- Stop or disable the smallest affected mutation path if required.
- Preserve logs and evidence without copying secrets or raw PII.
- Check payment/order consistency before making corrections.
- Deploy a reviewed mitigation or rollback.
- Verify with smoke tests and safe fixtures.
- Document root cause, impact, remediation, and follow-up test.
AI quality review
Review low-confidence retrieval, repeated fallback answers, negative chat feedback, invented-number blocks, policy refusals, and staff/customer boundary violations. Convert recurring missing knowledge into reviewed CMS documents rather than adding more model instructions alone.