Sentry

Sentry collects errors from our servers, web apps and mobile apps, and groups them into issues. This page covers how to set it up so an issue points at the exact build that caused it, and how to triage what it shows.

How it works

Every build gets its own release name. A release is a label Sentry puts on every event, so you can see which build an error first appeared in. We make the release name the same as the Docker image tag, so a Sentry issue maps straight to an image and a commit.

How to build it

Environments

Set the Sentry environment to development, staging or production. Then one Sentry project can hold all three, and you filter by environment.

Local development sends nothing. The SDK init is skipped when the app runs on a laptop, so a stray DSN in someone's .env cannot fill the team's quota with local errors.

import * as Sentry from "@sentry/node";

if (process.env.NODE_ENV !== "development-local" && process.env.SENTRY_DSN) {
  Sentry.init({
    dsn: process.env.SENTRY_DSN,
    environment: process.env.SENTRY_ENVIRONMENT,
    release: process.env.SENTRY_RELEASE,
    sendDefaultPii: false,
    tracesSampleRate: 0.1,
  });
}

sendDefaultPii: false keeps IP addresses, cookies and request bodies out of Sentry. A trace sample rate of 0.1 keeps one request in ten for performance data, which is plenty and keeps cost down.

Release per build, baked into the task definition

In buildspec.yml, derive the name next to the image tag, then substitute it into each task definition file:

pre_build:
  commands:
    - IMAGE_TAG=$(echo "$CODEBUILD_RESOLVED_SOURCE_VERSION" | cut -c1-7)
    - SENTRY_RELEASE_NAME="api@${IMAGE_TAG}"
post_build:
  commands:
    - |
      SRC="deploy/${TARGET_ENV}/taskdef.json"
      if ! grep -q '<SENTRY_RELEASE>' "$SRC"; then
        echo "ERROR: $SRC has no <SENTRY_RELEASE> placeholder"; exit 1
      fi
      sed "s|<SENTRY_RELEASE>|${SENTRY_RELEASE_NAME}|g" "$SRC" > taskdef.json

And in taskdef.json, as a plain environment value, not a secret:

"environment": [
  { "name": "SENTRY_RELEASE", "value": "<SENTRY_RELEASE>" }
]

The grep guard fails the build if someone deletes the placeholder. Without it, a new environment's task definition silently goes back to one release for every deploy.

Any other task definition the build registers itself (a scheduled job, a nightly purge) needs the same sed.

Source maps

Our server code is compiled and bundled, so a raw stack trace points at the bundled JavaScript, not your source. Upload source maps for each release so Sentry can show the original TypeScript line:

export SENTRY_ORG=example-org SENTRY_PROJECT=api SENTRY_AUTH_TOKEN=...   # token from SSM
npx sentry-cli sourcemaps inject ./dist
npx sentry-cli sourcemaps upload --release="$SENTRY_RELEASE_NAME" ./dist

Run this in the build, after the bundle step. The auth token is a secret: read it from SSM in the build, never commit it.

For mobile apps, upload debug symbols after each release build, or traces from obfuscated builds are unreadable. For Flutter that is dart run sentry_dart_plugin.

Alert rules

Start with these four. Send them to the team channel, not to one person's email.

Alert	Condition	Why
New issue in production	A new issue is created, environment = production	Something new broke
Regression	An issue changes from resolved to unresolved	A fix did not hold, or a release reverted it
Spike	An issue is seen more than N times in 1 hour	A known error suddenly got worse
New issue in latest release	First seen in the newest release	The deploy caused it, so rollback is likely the fix

Triage flow

Every issue ends in one state: fixed, ticketed, or ignored with a written reason. An issue list nobody clears is an issue list nobody reads.

Rules

Rule

The Sentry release is set per build and baked into the task definition. Never put it in SSM. A static value sends every deploy under one release, so "first seen in release" means nothing. And SSM is read when a task starts, not at deploy. A shared value is re-read by every later task launch, so a newer build's value silently relabels an older running image.

Rule

Do not drop whole classes of errors in beforeSend. Filter by tag inside Sentry instead. A filter that hides "all errors under 500" also hides every validation bug users hit.

Rule

Never send tokens, passwords, emails or message content to Sentry. Log ids, codes, field paths and lengths instead. Anything you log at warning level or above can end up in an event.

Mistakes we made

• Every deploy was the same release. The release was a static SSM value, bootstrap, in every environment for months. A comment claimed the build overwrote it. No such code existed. Regression-to-deploy links were useless.
• Validation errors were invisible. beforeSend dropped every event under status 500 and every message shaped like a translation key. GraphQL returns HTTP 200 for errors, and the request body is not logged. So a user stuck on "please check the fields" left no trace anywhere. The fix: log client errors at warning level with the failing field paths, and tag them client_error: true in Sentry.
• Then every rejected request became an event. After that fix, one attacker sending junk created about 600 Sentry events in a week. A scripted client can use up the monthly quota and bury real errors. Keep the log line and the metric, but sample or rate-limit client-error events in Sentry.
• Traces pointed at bundled JavaScript. The release was correct, but source maps were never uploaded, so stack traces were unreadable. Add the upload step on day one.
• A few web error reports were dropped. The web apps forward Sentry events through their own server. Node tried IPv6 after 250 ms and the network had no IPv6 route. Set NODE_OPTIONS="--dns-result-order=ipv4first --no-network-family-autoselection" when your network is IPv4 only.

Checklist

• environment set per env, local dev sends nothing.
• Release name equals the image tag, substituted into every task definition the build produces.
• Build fails if a task definition lost the <SENTRY_RELEASE> placeholder.
• Source maps or debug symbols uploaded per release.
• sendDefaultPii is false. No tokens or personal data in messages.
• The four alert rules exist and post to the team channel.
• Client errors are visible but sampled.