Skip to main content

Mistakes we made

Every item here really happened while we built uploads, serving and moderation. Each one has: what happened, why, and what to do instead. Read this before you start, and add to it when something new bites.

The pattern behind most of them​

Signing needs no permission, checks no object, and knows nothing about CORS. So most upload and serving bugs are silent on the server. The fix is always the same: test the real flow in a real browser, in every environment, and make the client report the real error.

Uploads​

  • The SDK's default checksum broke uploads, twice. What: presigned uploads returned 403, first for simple PUTs, later for every multipart chunk. Why: AWS SDK v3 by default adds an x-amz-checksum-crc32 to a presigned URL, computed from an empty body. S3 compares the real body to it and refuses. Instead: build every presigning S3 client with requestChecksumCalculation: "WHEN_REQUIRED", and leave a comment saying why. See Presigned PUT.

  • Content-Type was believed to be signed. It was not. What: code comments said the signature locked the content type. Why: the presigner does not sign Content-Type unless you list it in signableHeaders. A URL check showed X-Amz-SignedHeaders=host. Instead: pass signableHeaders, then print a URL and read the signed headers.

  • Size limits were a client promise. What: early presigned PUTs did not sign the size. A caller could declare 1 KB and upload much more. Why: the size was checked before signing, but never bound to the request. Instead: sign Content-Length, or use a POST policy with content-length-range. See Presigned POST.

  • A seven-file form lost a submission. What: in production, six files uploaded, one did not, and the submit never ran. The person saw "check your connection". Six orphan files stayed in the bucket. Why: each file did its own bot check and presign, serially. One failure ended the flow, and nothing cleaned up. Instead: one presign call for all files, one bot check, an upload ticket bound to the keys, bounded parallel uploads, retries on the same URL, and a lifecycle rule for the scratch prefix.

  • The rate limit was charged per file. What: 30 presigns an hour per IP meant about 4 registrations an hour, shared by everyone behind one office or carrier NAT. Why: the limiter ran inside the per-file loop. Instead: charge the limit once per call.

  • Every failure showed the same message. What: ten different causes all became "check your connection". Why: the client collapsed every error into one code. Instead: pass the server's real error key to the UI and to logs.

CORS, IAM and config drift​

  • The production bucket had no CORS rules. What: every browser upload in production failed at the preflight. Why: dev had CORS in Terraform, staging had it added by hand, prod had none. Server tests and curl passed, because CORS only applies in browsers. Instead: CORS in Terraform for every environment. Test with an OPTIONS preflight after every apply. See Serving files.

  • CORS lists went stale after a hostname rename. What: uploads failed from the new web hostname. One staging list even had the production origin in it. Why: the rename did not include the bucket CORS lists, and someone fixed the wrong bucket. Instead: list every place a hostname appears (CORS, CloudFront, OAuth, CSP) and change them together.

  • Bucket rules lived only in the live bucket. What: a CORS rule and a lifecycle rule existed in production but not in Terraform. Why: they were added by hand and never copied into Terraform. Instead: codify every hand fix the same day. An apply from stale config would have deleted both and broken every upload.

  • Read grants listed prefixes one by one. What: production showed text but blank images. Why: the role could read 6 of 13 prefixes. Presigning still worked, so the API returned 200 and the image URLs 403'd. Nothing was logged. Instead: read on the whole bucket, writes scoped by prefix.

  • Moderation permissions existed in dev only. What: the moderation code was ready, but the role permission it needed was applied to dev alone. Why: IAM lives in the infrastructure repo and is applied per environment, by hand. Code promotion does not carry it. Instead: list the IAM changes in the feature's promotion checklist, and apply them in an environment before the code reaches it.

Serving​

  • "CDN or presigned" was decided by a bucket-name match. What: production served every asset as a presigned S3 GET. The CDN was never used, so its CORS policy did nothing. Why: the code checked whether the bucket name contained a marker string. The marker matched no bucket in production. Instead: make public or private an explicit setting per asset type.

  • The CDN 403'd every key. What: the CloudFront distribution worked for no file. Why: it had Origin Access Control, but no bucket policy for CloudFront and no KMS decrypt grant for the bucket's key. Instead: OAC needs all three: the OAC, a bucket policy with AWS:SourceArn, and KMS decrypt if the bucket uses a customer key.

  • Rows pointed at missing objects. What: some catalog images 404'd. Why: the database rows and the bucket had drifted apart. Presigning does not check that an object exists. Instead: copy objects with the data, and check every key with HeadObject.

  • An overwritten file stayed stale at the edge. What: a re-uploaded bundle kept serving the old copy. Why: it was cached for a year at a fixed name. Instead: a new file name per version, or a CloudFront invalidation.

Video​

  • Phone videos were unreadable for moderation. What: iPhone clips were rejected as unreadable. Why: iPhones record HEVC by default, and the video moderation service only reads H.264. The original file was served and moderated as uploaded. Instead: transcode every upload on the server to H.264 and AAC, then moderate the clean copy. See Video.

  • A note claimed the client converted video. What: the API docs said HEVC was converted on the client. Why: it was a plan, never built. The client sent clientTranscoded: false. Instead: check the client code before you trust a note about it.

  • A key pattern blocked a new file type. What: staff could not play any video. Every URL came back "not found". Why: the staff URL signer's key pattern allowed only image and PDF extensions. Instead: when a feature gains a file type, search for every key pattern that guards it.

Moderation​

  • A fail-soft moderator caught nothing. What: photo moderation was on, but nothing was ever flagged. Why: the role had no permission to call the service. Every call failed and the code moved on. Instead: grant permission, switch on, then test with a known-bad sample in that environment. See Moderation.

  • The review console could not close a case. What: every case stayed open, and every author showed zero strikes. Why: remove and ban ran, but nothing marked the case actioned. Strikes count actioned cases. Instead: closing the case is part of the action. Test the whole loop.

  • No undo, and no notices. What: there was no unban anywhere. A "notify by email" box in the console did nothing. Why: both were assumed to exist. Neither was ever built. Instead: every action has an undo. Check the back end behind every UI control.

  • One threshold table for two providers. What: distress posts passed the cheap first check. Why: one provider gave calibrated probabilities, the other gave round numbers. The same text scored 0.08 and 0.9. Instead: a threshold table per provider, tuned on real samples in every supported language.

  • A permanent error was treated as temporary. What: a check with no permission, and another with a daily quota of zero, were retried on every request, adding seconds per post. Why: the errors looked like throttling. Instead: permission and daily-quota errors open the circuit breaker at once.

  • A store rejection blamed reporting, but login was broken. What: the store said there was no way to report content. Why: the reviewer could not log in, so never saw the controls. Instead: test the exact build the reviewer gets, and send a screen recording of report and block.

Checklist before you ship a file feature​

  • Tested the real upload in a real browser, in every environment.
  • Printed a presigned URL and checked its signed headers and that it has no checksum parameter.
  • Bucket CORS and lifecycle rules are in Terraform for every environment.
  • IAM for this prefix is applied in every environment the code will reach.
  • Opened a served file as a real viewer, not only checked the API response.
  • The client shows the real error.
  • Moderation tested with a known-bad sample, and the review loop closes a case.
Last updated on