Retention and deletion

Every piece of personal data needs an end date and a job that enforces it. "Until deletion" is an end date only if deletion actually reaches every copy. Read this before you add a table, a bucket, a log line, a cache key or an analytics property.

How it works

Retention is how long we keep data. Under GDPR it is the "storage limitation" principle (Article 5(1)(e)): keep personal data no longer than the purpose needs. DPDP has the same idea: erase when the purpose is served or consent is withdrawn, unless a law needs it kept.

Retention is a policy decision first and a cron job second. The DPO approves the numbers; engineers build one job per rule.

How to build it

1. A starting retention schedule

These are example values from a real review, not legal advice. Your DPO sets the real ones.

Data	Example period	Clock starts	Action
Account and profile	Life of account	Account deletion	Erase within the DSR deadline
Dormant account	24 months	Last sign-in	Warn twice, then erase
AI chat transcripts	12 months rolling	Message created	Hard delete
Voice audio sent to an AI	Not stored	n/a	Verify the provider side too
AI onboarding transcripts	90 days	Created	Hard delete
Devices and push tokens	90 days	Last seen	Hard delete
Refresh tokens, IP, user agent	90 days after expiry	Expiry	Hard delete
One-time codes, rate-limit counters	Minutes to hours	Created	Cache TTL
Notification feed	30 days	Created	Hard delete
Moderation reports	1 year after resolution	Resolution	Delete; reporter anonymised on their own deletion
Security audit logs	12 months to 3 years	Created	Delete; actor anonymised on erasure
Consent records	Account life plus a fixed evidence period	Account closed	Delete
Anonymised account shell	Indefinite	n/a	No longer personal data
Hashed identifier tombstone (anti-abuse)	Short, for example 180 days	Deletion	Delete; must be disclosed in the policy
Application logs	14 to 30 days	Written	Log group retention
Database and function logs	30 days	Written	Log group retention
Build logs	90 days	Written	Log group retention
Automated database backups	14 days	Taken	Native expiry
Manual snapshots	30 days maximum	Taken	Scheduled delete
Old object versions	30 days	Became noncurrent	Bucket lifecycle rule
Public form leads (no account)	30 days	Submitted	Purge job plus bucket rule
Data export archives	7 days	Created	Own lifecycle rule
Payment and tax records	Set by tax law, often 6 to 8 years	Transaction	Ask your accountant

2. The purge job

// One job per retention rule. Idempotent. Dry-run first. Counts always logged.
export async function purgeNotifications({ dryRun }: { dryRun: boolean }) {
  const cutoff = new Date(Date.now() - 30 * 24 * 3600 * 1000);
  const count = await db.notifications.countOlderThan(cutoff);
  if (count > SAFETY_LIMIT) throw new Error(`refusing to delete ${count} rows`); // data-loss guard
  if (!dryRun) await db.notifications.deleteOlderThan(cutoff);
  log.info({ job: 'purge_notifications', cutoff, count, dryRun });
}

• Schedule it in every environment, through infrastructure code.
• Alarm when it has not run, not only when it fails.
• Register it as the enforcement_ref of the rule in the data map.

3. Object storage

Versioned buckets keep "deleted" files. A plain delete only writes a delete marker; the bytes stay.

resource "aws_s3_bucket_lifecycle_configuration" "assets" {
  bucket = aws_s3_bucket.assets.id
  rule {
    id     = "expire-noncurrent"
    status = "Enabled"
    filter {}
    noncurrent_version_expiration { noncurrent_days = 30 }
    abort_incomplete_multipart_upload { days_after_initiation = 7 }
  }
}

When you erase one user's files, delete every version of each key, not just the current one.

4. Backups

You cannot erase one user from a snapshot. The accepted position is:

1. Backups expire on a short, fixed cycle, and the privacy policy says so.
2. Any restore is followed by re-applying every erasure done since the backup was taken. So keep a list of erased subject ids (hashed) for longer than the backup window.
3. Manual snapshots get a delete date when they are taken.

5. Logs

Logs are the most common place for personal data and secrets to leak.

• Set a retention period on every log group. No retention means forever.
• Redact at the logger, not at each call site: authorization, cookies, API keys, internal keys, tokens including tokens in the URL query string, passwords, email, phone.
• Never put personal data in cache key names, metric dimensions or URLs.

const logger = pino({
  redact: {
    paths: [
      'req.headers.authorization', 'req.headers.cookie', 'req.headers["x-internal-key"]',
      'req.query.token', 'req.url',  // a ?token= in the URL is still a token; or strip the query in a serializer
      '*.password', '*.email', '*.phone',
    ],
    censor: '[redacted]',
  },
});

6. Analytics

• Send no raw personal data to analytics. If a property must identify a person, hash it before it leaves the device, and mark it in the event schema (pii: true, type: hash).
• A hash is pseudonymous, not anonymous. An unsalted SHA-256 of an email or user id can be matched by anyone who has the same email. Treat hashed data as personal data: it still needs consent, retention and deletion.
• Prefer safe proxies (char_count, result_count) over hashing free text at all.
• Set the vendor's own retention to the shortest value that meets the need.

7. Vendors

Deleting our rows does not delete the vendor's copy. For each processor, record either its deletion API (call it from the erasure job) or its retention setting (state it in the policy). Typical list: auth provider user record, analytics, attribution, crash reporting, error tracking, email provider suppression, AI provider logs.

Rules

Rule

No new table, bucket, log group or cache key without a retention rule and the job that enforces it. Data with no end date is kept forever by default, and "forever" is never a lawful default.

Rule

Never log secrets or personal data. Redact at the logger. A token in a URL query string is still a token, and logs are read by more people than the database.

Rule

Enable lifecycle expiry of noncurrent versions on every versioned bucket that holds user files. Otherwise deleted photos and audio stay retrievable.

Mistakes we made

• Purge jobs ran in dev only. The task definition existed in every environment, but only dev had a schedule. Production kept the notification feed forever. Deploy schedules through infrastructure code and check that they exist in production.
• Log groups with no retention. Database log groups, some functions and build logs had none, so they kept everything forever. Database error lines can carry the values that failed.
• Tokens in logs. User tokens passed as ?token= on websocket and streaming routes, and a service-to-service key in a header, were written to application logs on every call. The redaction list missed the URL and that header. The key had to be rotated.
• Manual snapshots never expired. Several "before the migration" snapshots of the full production database were kept indefinitely. Erased users lived on inside them, which contradicted our own "14 days" backup statement.
• Personal data in cache key names. One-time-code cache keys contained the raw email or phone in the key itself. Key names show up in monitoring and dumps. Hash them.
• A one-time-code stand-in posted codes to a chat channel. During setup, recipient phone/email and the live code were posted to an internal chat. Time-box any stand-in like this and remove it before real users arrive.
• Unsalted hashes called "anonymous". An AI service stored an unsalted SHA-256 of the user id and described it as anonymous. It is pseudonymous and still in scope.

Checklist

• Retention rule agreed with the DPO and written in the data map.
• Purge job built, idempotent, dry-run tested, scheduled in every environment.
• Alarm when the job does not run.
• Bucket has noncurrent-version expiry; erasure deletes all versions.
• Log groups have retention; logger redacts tokens, keys and personal data.
• No personal data in URLs, cache key names or metric dimensions.
• Analytics properties are PII-free or hashed and marked in the schema.
• Vendor retention or deletion API recorded.
• Backup window stated in the privacy policy; restore runbook re-applies erasures.