Moderation

Any app where users can see each other's photos, videos or text needs moderation. The app stores reject apps without it. This page covers automatic checks for images, video and text, the human review queue, and the special, legal rules for child sexual abuse material.

How it works

Images: check before publish

Video: async jobs

Text: a ladder of checks on the write path

The three decisions

Decision	Meaning	Use for
allow	Passed. Visible.	Most content
block	Refused. Delete the file. Tell the user.	Clear violations: explicit nudity, sexual content, abuse
review	A human must look	Borderline content, and anything where a block would harm a real user

review comes in two kinds. Pick per surface and write it down.

Kind	What the user sees	Use when
Quarantine (hidden until approved)	Their own content is visible to them only	Profile photos, dating or matching profiles, anything a stranger sees first
Publish and flag	Visible to all, a case is filed	Text where blocking hurts, such as a person in distress asking for help

Rule

Self-harm or distress is never a block rule. Blocking a person asking for help silences them. Publish, flag it for a human, and consider showing support resources.

Images with Rekognition

import { DetectModerationLabelsCommand, RekognitionClient } from "@aws-sdk/client-rekognition";

const rk = new RekognitionClient({});
const BLOCK = ["Explicit", "Explicit Nudity", "Explicit Sexual Activity"];
const REVIEW = ["Non-Explicit Nudity of Intimate parts and Kissing", "Violence", "Visually Disturbing"];

export async function moderateImage(bucket: string, key: string) {
  const res = await rk.send(
    new DetectModerationLabelsCommand({
      Image: { S3Object: { Bucket: bucket, Name: key } }, // no bytes through the API
      MinConfidence: 60,
    }),
    { abortSignal: AbortSignal.timeout(5000) },
  );
  const labels = res.ModerationLabels ?? [];
  // Labels are a tree. Match the label AND its parent.
  const hit = (names: string[]) =>
    labels.some((l) => names.includes(l.Name ?? "") || names.includes(l.ParentName ?? ""));
  const decision = hit(BLOCK) ? "block" : hit(REVIEW) ? "review" : "allow";
  return { decision, labels, modelVersion: res.ModerationModelVersion };
}

Facts checked against the AWS docs in October 2026:

• Formats. Rekognition Image reads JPEG and PNG only. An image from S3 can be at most 15 MB, and at least 80 pixels on each side. If you accept WebP or HEIC, convert first, or Rekognition will error on it.
• Labels are a 3-level tree. In the current model (version 7), Explicit is a top-level label and Explicit Nudity sits under it. Older label names (for example Suggestive) are not in the version 7 list. A block list with an old name matches nothing, silently.
• MinConfidence. If you leave it out, the default is 50. Below 50 means more false positives. We used 60.
• Rekognition does not detect CSAM. AWS says the moderation APIs "don't detect whether an image includes illegal content, such as CSAM". See the CSAE section below.

tip

Store ModerationModelVersion with each verdict. When AWS ships a new model, you can see which decisions came from which model, and re-check your label lists.

Positive checks

For profile photos, "nothing bad found" is not enough. We also wanted exactly one clear face. So the check is positive: pass only if the photo has what it must have.

• Profile photo: one face, above a confidence level, facing the camera, not too small or too big in the frame. No face, or two faces, is a block with a reason the app can show.
• Profile video: a good face visible in a set share of the seconds. Cars, places and objects fail because they have no face. Never build a list of banned objects. It can never be complete.

Put every threshold in config and tune it on a test set of real samples.

Video with Rekognition

• StartContentModeration is async. You get a JobId and read results later with GetContentModeration, paging with NextToken.
• You can pass an SNS topic for a "job done" message, or just poll from a background job. Polling every minute is simple and fine for short clips.
• Use a new ClientRequestToken per attempt. Rekognition returns the old (failed) job if you reuse a token.
• Stored video must be H.264 in MP4 or MOV. Transcode first. See Video.
• There is a per-account limit on concurrent video jobs (20 in the AWS docs). Start jobs from a queue, in batches.
• After a few failed attempts, send the video to human review. Never approve it.

Text with AI

• Check on the write path. The API calls the moderator inside the "create post" handler, before the insert. A check in the client can simply be skipped. App store reviewers will not accept that for user content.
• Use a ladder. A cheap, fast moderation API first. A second opinion from an LLM for unclear cases. Each rung has a circuit breaker and the whole check has a time budget.
• Cache verdicts by text plus a fingerprint of the policy, so a policy change forces a re-check.
• Decide what happens when every rung is down. Our text check published the post and counted it. The unjudged post must still reach a re-check queue later.

The human review queue

• One case per target (a message, a photo, a post), not one per report. Count distinct reporters.
• Auto-hide after a set number of distinct reporters. Show "This may be inappropriate. Tap to view." until a human decides.
• The API decides visibility. Every read returns a moderation state. Clients only render it.
• Strikes and bans. Strikes count actioned cases. A ban after a set number of strikes. Every ban needs an unban.
• The AI files cases too. An AI review verdict creates a case from a system reporter shown as "AI moderator", so staff work one queue.
• Tell people. The author should hear their content was removed or restored, and the reporter should hear the outcome.
• Service level. Commit to a response time (we promised the stores 24 hours) and make the queue sort by priority and age.

Audit trail

• Append-only audit rows for every decision: who, what, when, which target, which rule or label.
• Audit staff views of private media too, not only decisions.
• Keep content out of the audit payload. Store ids and outcomes. The content stays where it lives, behind its own access rules.
• A shared moderation queue that spans several data stores should hold ids only. Staff read the content through the owning service.

App store rules for user content

As of October 2026. Re-check before every submission.

• Apple, App Review Guideline 1.2 (guidelines): apps with user content need a way to filter objectionable content, a way to report it with timely responses, a way to block abusive users, and published contact details.
• Google Play, Child Safety Standards policy (policy), for social and dating apps: published standards that forbid child sexual abuse and exploitation, an in-app way to report concerns, action on CSAM once you know of it, and a process to report it to the authorities.

Child sexual abuse material (CSAE and CSAM)

CSAE means child sexual abuse and exploitation. CSAM means child sexual abuse material. The rules here are legal duties, not product choices. Get your lawyer to sign off the runbook.

• Detection. General moderation models do not detect CSAM. Known material is found by hash matching against industry hash lists, using a specialist service. Plan this before launch if users can upload images.
• Reporting is a legal duty. In the US, providers must report apparent CSAM to the NCMEC CyberTipline (18 U.S.C. 2258A, CyberTipline). Since the REPORT Act (May 2024), the provider must preserve the report contents for one year. Other countries have their own laws and portals. Confirm yours with legal. As of October 2026.
• Never let staff download it. No download button, no "open original", no forwarding, no screenshots into tickets. Keep it in place, restrict access to the fewest people, and log every access.
• Never dismiss a child-safety report without the escalation owner's review.
• Legal hold beats erasure. The normal "delete my account" job must skip data under a hold.
• Name a monitored contact and a backup. An unmonitored inbox fails the store policies.

Rules

Rule

Moderate on the server, on the write path. Never trust a client-side check.

Rule

Before switching moderation on in an environment, check the role can call the service there. A missing permission makes a fail-soft moderator approve everything while looking like it works.

Rule

Thresholds and label lists live in config, are tuned on real samples, and are re-checked when the provider's model version changes.

Rule

Every enforcement action has an undo: remove has restore, ban has unban.

Rule

Suspected CSAM is preserved in place, never downloaded, reported by the escalation owner, and kept out of normal erasure.

Mistakes we made

• A moderator that caught nothing. Photo moderation was coded fail-soft: on an error it logged and moved on. The role had no Rekognition permission, so every call failed with AccessDenied and nothing was ever flagged. It looked like a working feature. Grant the permission first, then switch on, then test with a known-bad sample.
• The review console could never close a case. Remove and ban ran and were audited, but nothing marked the case actioned. Every case stayed open, and every author showed zero strikes, because strikes count actioned cases. Closing the case must be part of the action.
• There was no unban anywhere. Not in the console, not in the API. A ban was also narrower than it sounded: it stopped new posts and replies, but did not hide old content or stop reading. Write down what each action really does.
• The notices box did nothing. The console showed a "notify member by email" box, but no code path sent any notice. Nothing was ever removed. It was never built. Check the back end before trusting a UI control.
• A store rejection for "no way to report content" was not about reporting. The reviewer could not get past a broken login, so never saw the report controls. Fix the login, then send a short screen recording that shows report and block.
• "Block" had been replaced by "mute". The store asks for a way to block abusive users. Present the strongest control you have and name it plainly in the review notes.
• Scores from two providers were treated as one scale. One API gave a calibrated probability. An LLM gave round numbers. The same sentence scored 0.08 on one and 0.9 on the other. One threshold table across both let distress posts through. Use a threshold table per provider.
• No single threshold could catch distress posts. On one API, real distress posts scored below some clean posts. Adding two words changed a score from a miss to a hit. We now send every post to a second opinion for the self-harm categories. It costs about 2.7 seconds per post, so the post path has a 5-second budget.
• Posts in other languages scored as clean. Distress posts in several Indian languages scored as low as clean English posts. Test every language you support.
• A dead rung slowed every post. A permission error was retried every minute, costing seconds per post. A permission error must open the circuit breaker at once.
• A quota of zero looked like throttling. In one account the LLM's daily token quota was 0. The error said "Too many tokens per day", not "access denied", so the code treated it as temporary. Treat a daily-quota error as permanent, and check quotas in every account.
• The model refused the worst content. The LLM provider's own safety filter refused to classify an explicit death threat. That rung fails on exactly the text it should catch, so the ladder needs another rung that does not.

Checklist

• Each surface has a written choice: quarantine or publish-and-flag, and what happens on an outage.
• Moderation runs on the server, before the content goes live.
• The role can call the moderation service in this environment. Tested with a known-bad sample.
• Only JPEG or PNG reach Rekognition Image. Video is H.264.
• Label lists match the current model version. Matching uses name and parent name.
• Thresholds are in config and tuned on real samples, in every supported language.
• Video jobs use a new token per attempt, and failures go to human review.
• One case per target, distinct reporters, auto-hide, strikes, ban and unban.
• Actions close the case. Authors and reporters get notices.
• Audit rows for every decision and every staff view of private media.
• CSAE runbook signed off by legal, contact named, hash matching planned.
• No download path for suspected CSAM. Erasure skips legal holds.