Open incidents from your systems

Make a server secret

A project's admins make keys on Settings → Keys (services.my-beast.com/<workspace>/<project>/settings/keys). Beside Incidents, choose New key, keep Server secret, give it a label that says where it is used, and choose Make key.

Copy the secret straight away: only its fingerprint is kept, so it is shown once. Put it on the server that calls, beside Incidents' address:

MYBEAST_INCIDENTS_URL=https://services.my-beast.com/incidents
MYBEAST_INCIDENTS_KEY=sk_…

If the project has Secrets, id has put both lines there already. A server secret opens Incidents in its own project only. To change one, make a new one, move the caller to it, then revoke the old one.

Open an incident

Send what is wrong and how bad it is. The rest is optional, but a dedupe key is worth sending:

curl -X POST "$MYBEAST_INCIDENTS_URL/api/v1/incidents" \
  -H "Authorization: Bearer $MYBEAST_INCIDENTS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Checkout fails for card payments",
    "severity": "sev1",
    "summary": "Card payments answer 502 since 09:07. PayPal still works.",
    "source": {
      "product": "monitoring",
      "ref": "checkout-cards",
      "url": "https://grafana.example.com/d/checkout"
    },
    "dedupe_key": "monitoring:checkout-cards"
  }'

{
  "id": "019a2f7c-5e3b-7d21-9c4f-2b8e6a1d3f50",
  "reference": "INC-12",
  "url": "https://services.my-beast.com/acme/shop/incidents/incidents/12"
}

The incident opens at Investigating with no lead, and is told as “An incident declared”. Keep the id or the reference to post updates later; url is its page. Every field is in the API reference.

Open one incident per outage

A monitor that fails again and again shouldn't open an incident each time. Give each thing that can go wrong a dedupe_key of your own, such as monitoring:checkout-cards. While an incident with that key is open, a call with it opens nothing new. Instead:

• the answer is 200 OK with the open incident's id, reference and url;
• its timeline gets a line, “called again with the same dedupe key”, with the call's summary;
• a worse severity raises it, and everyone is told; its title stays as it was;
• the call isn't counted as another incident in the month.

Once the incident is resolved or declined, the key is free again: the next call with it opens a new incident. With a dedupe key, retrying a call that timed out is safe too.

Post updates

Post to the incident by its id or its reference. An update can move the status and change the severity at the same time:

curl -X POST "$MYBEAST_INCIDENTS_URL/api/v1/incidents/INC-12/updates" \
  -H "Authorization: Bearer $MYBEAST_INCIDENTS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "We found the cause: the card provider'\''s new certificate. A fix is going out.",
    "status": "identified"
  }'

With a new status, the message goes out with that change's notice. Without one, it is an update on the timeline and sends no notice. An update from the API is never internal: it can reach your status page.

Resolve it

When your monitoring sees it working again, resolve the open incident by its dedupe key:

curl -X POST "$MYBEAST_INCIDENTS_URL/api/v1/incidents/resolve" \
  -H "Authorization: Bearer $MYBEAST_INCIDENTS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "dedupe_key": "monitoring:checkout-cards",
    "summary": "Card payments work again since 09:48."
  }'

The summary goes out with the notice that it is resolved. If a person resolved it first, or no incident with that key is open, the answer is 404: there is nothing left to do.

How the timeline names a caller

Each call is a line on the timeline, marked API, in the caller's name. The name is the first of:

1. source.product in the call, such as monitoring;
2. the one that opened the incident, for later calls about it;
3. the server secret's label;
4. api.

With source.url, the incident's page links back to what you sent it from, and shows source.ref and the dedupe key beside it.

What can go wrong

• 401: the secret is missing, wrong, revoked or made for another product.
• 422: a field is missing or wrong, or the project hasn't added Incidents. The answer says which.
• 429: more than 120 calls in a minute with one secret. Wait for Retry-After.
• 402: the project's subscription has ended. Nothing new is taken in until it is renewed.

If Incidents doesn't answer, tell people another way: a failed call opens nothing. Every error is in the API reference.