DevOps PeachUsers upload photos and view feeds of other people's photos. It is very read-heavy and the objects are large — so images and metadata take different paths.
flowchart LR APP["App"] -->|"1. ask to upload"| API["API"] API -->|"2. pre-signed URL"| APP APP -->|"3. PUT bytes"| S3["Object storage: originals"] S3 -->|"4. event"| Q["Queue"] Q --> W["Workers: resize, strip EXIF, scan"] W --> TH["Object storage: thumbnails"] W --> DB["Metadata DB"] TH --> CDN["CDN"] CDN --> V["Viewers"] %% step APP: The client asks the API for permission to upload. %% step API: The API returns a short-lived pre-signed URL; it never touches the image bytes. %% step S3: The client uploads directly to object storage — scales without web servers. %% step Q: The upload event goes to a queue so processing is asynchronous and retryable. %% step W: Workers create sizes, remove location data and check content. %% step DB: Metadata (owner, size, caption, keys) goes to the database. %% step CDN: Viewers get thumbnails from the CDN, never from the origin directly.
1–2. Requirements and estimates
Upload (≤ 20 MB), view photo, feed of followed users, delete. 10 M daily users, 1 M uploads/day ≈ 12/s (peaks 100/s); 50 views per user per day ≈ 6,000 reads/s. Storage: 1 M × (4 MB original + ~0.5 MB thumbnails) ≈ 4.5 TB/day → object storage with lifecycle tiers, not disks.
3. API
POST /v1/uploads → {"upload_url": "<pre-signed PUT>", "photo_id": "p-981"}
GET /v1/photos/{id} → metadata + CDN URLs
GET /v1/feed?cursor=... → photos of followed users
DELETE /v1/photos/{id}
4. Data model
photos(photo_id, owner_id, created_at, status, original_key, sizes, caption)— statusuploading → processing → ready.follows(follower_id, followee_id).- Feeds: fan-out on write (push photo IDs into followers' feed lists in Redis) for normal users; fan-out on read for celebrities with millions of followers.
5. Key choices
| Choice | Why |
|---|---|
| Pre-signed URLs | Uploads bypass the web tier; the URL expires in minutes and allows one key only |
| Event → queue → workers | Processing spikes do not affect uploads; failed jobs retry; DLQ |
| Several sizes (thumb, medium, full) | Phones should not download 4 MB for a 200 px thumbnail |
| CDN with long TTL, versioned keys | ~95% of image traffic served from the edge |
| Private originals bucket | Only the CDN (origin access control) reads it |
6. Bottlenecks and extras
Upload spikes → the queue absorbs them, workers autoscale on depth. Large uploads → multipart upload. Privacy → strip EXIF GPS data. Abuse → content scanning before ready. Cost → move originals to an infrequent-access tier after 30 days. Deletion → remove from storage, CDN cache and feeds.
Traps: streaming uploads through the API servers; generating thumbnails synchronously in the upload request; a public bucket; serving full-size images to mobile feeds.