Technical Report
Distributed Systems  ·  Privacy Engineering

Hybrid P2P-Cloud Anonymous Transfer Engine

A privacy-first file transfer platform that races a direct WebRTC peer-to-peer connection against a cloud relay — and then self-destructs the evidence. Hope for the best, plan for the worst, at every layer.

5 s
P2P race window
16 KB
DataChannel chunk
3-tier
Fallback hierarchy
≤ 24 h
Cloud file lifespan
FastAPI + Uvicorn  ·  React 18 / Vite 5  ·  Socket.IO signaling  ·  AWS S3 + Lambda  ·  Redis  ·  Kubernetes / Coturn
01

Executive Summary

Every mainstream file-transfer service routes your file through a corporate server and stores it on infrastructure you do not control. This engine challenges that model: files stream directly between browsers over a WebRTC DataChannel whenever the network permits, so zero servers touch the data and zero copies exist anywhere.

Because peer-to-peer connections fail more often than they succeed in the real world — symmetric NATs, carrier-grade NATs, corporate firewalls — the system treats connectivity as a probability, not a guarantee. A 5-second race decides between direct P2P and a cloud fallback that streams the file through AWS S3 and then destroys it with a serverless "Nuke Protocol," backed by a 24-hour lifecycle failsafe.

Best case
Direct P2PLAN-like speeds, zero server contact, nothing stored
Worst case
Cloud relayS3 multipart stream + 1-hour presigned URL, auto-destruct after download
Memory
FlatIndexedDB chunk buffer keeps RAM constant for any file size
Deploy
ProductionK8s manifests, non-root multi-stage containers, Prometheus, Render / Railway
02

Why This Project

The guiding question: what if file transfers could happen directly between browsers — no middleman, no storage, no trace? Pursuing it meant confronting WebRTC DataChannels, NAT traversal, STUN/TURN relays, and the reality that the internet is designed to prevent direct connections between devices.

The result is a system built around network connectivity as a probability:

  • Best case (P2P succeeds) — files stream directly at LAN-like speeds; no server sees the data.
  • Worst case (P2P fails) — the system silently pivots to S3 with a 1-hour presigned URL, then destroys the file via a Lambda nuke and a 24-hour lifecycle failsafe.
Philosophy

The most important code is not the happy path — it is the fallback logic. Every component has a failure mode, and the system's quality is defined by how gracefully it handles each one.

03

Microservices Architecture

Three independently deployable services sit behind a unified Nginx Ingress: a React SPA, a FastAPI backend, and a Node.js Socket.IO signaling relay. External dependencies are Redis (ephemeral presence), AWS S3 + Lambda (cloud fallback), and a self-hosted Coturn STUN/TURN server.

Browser · React SPA useSignaling Socket.IO client 30 s heartbeat useHybridTransfer 5 s race controller p2p ⇄ cloud useWebRTC RTCPeerConnection DataChannel IndexedDB zero-RAM 16 KB chunks Peer Browser receives DataChannel direct P2P · 16 KB WebSocket ICE candidates fallback after 5 s Signaling Node.js Socket.IO relay · SDP offer/answer + ICE only NAT Traversal Coturn STUN/TURN · host networking · ports 3478 / 5349 Cloud Infrastructure · Kubernetes FastAPI Backend auth · heartbeat · upload port 8000 Redis presence store TTL 60 s AWS S3 5 MB multipart stream 24 h lifecycle AWS Lambda Nuke Protocol delete after download Prometheus scrape /metrics 15 s
Figure 1 — Service topology. The accent path is the privacy-preserving default: signaling and TURN never see file bytes, and the DataChannel carries the file directly. The dashed edge is the 5-second fallback into the S3 relay.
04

The 5-Second Race Algorithm

The heart of the system is the useHybridTransfer hook. Instead of waiting indefinitely for WebRTC to negotiate, it runs an aggressive race between the P2P connection and a 5-second countdown.

startTransfer(file)
  │
  ├─ state: p2p_pending  → connectAndOffer()  → start 5 s countdown
  │
  ├─ P2P connected before 5 s ─────┐        ┌──── timer expires (5 s)
  │                                ▼        ▼
  │                          state: p2p_active   state: cloud_active
  │                                │                  │
  │                    sendFile() via DataChannel   POST /api/upload
  │                       (16 KB chunks)            (5 MB S3 multipart)
  │                                │                  │
  │                                └────► state: completed ◄────┘
Why 5 seconds?

WebRTC ICE negotiation completes in 1–3 seconds on open networks. A 5-second window gives STUN enough time to probe, but does not leave the user staring at a spinner when they are behind a corporate firewall that will never allow P2P. The user sees only a badge change: P2P Direct → Cloud Server Relay.

05

Zero-RAM Chunk Buffering with IndexedDB

Traditional receivers accumulate incoming chunks in a JavaScript array — a 500 MB file then consumes 500 MB of tab RAM and crashes on mobile. This engine writes every incoming 16 KB DataChannel chunk directly to IndexedDB (the TransferCache object store), bypassing RAM entirely.

The browser's IndexedDB implementation persists to disk (typically LevelDB), so memory usage stays flat regardless of file size. Only at the final step does the receiver read all chunks back, assemble a Blob, and trigger the download. This is the difference between "transfers files up to 50 MB" and "transfers files of arbitrary size."

Backpressure
Sender pauses when bufferedAmount exceeds the 64 KB bufferedAmountLowThreshold, resuming via onbufferedamountlow — natural flow control that adapts to network speed.
Chunk size
16 KB sits well below the SCTP ~64 KB fragmentation boundary while minimising per-chunk overhead — the reliability/speed sweet spot.
06

Burn-After-Reading Data Destruction

Cloud-relayed files are protected by a three-layer destruction cascade. Even if the user never triggers the Nuke Protocol, the file cannot exist beyond 24 hours — AWS enforces this at the infrastructure level.

LayerMechanismTriggerLatency
1abort_multipart_uploadUpload failure mid-streamImmediate
2AWS Lambda "Nuke Protocol" (head_object → delete_object)Post-download deletion requestSeconds
3S3 lifecycle policy24-hour automatic expiration≤ 24 hours

Graceful degradation at every layer

07

Three-Tier Transfer Hierarchy

A cascading fallback strategy that prioritises privacy at every step.

PriorityRouteLatencyPrivacyWhen used
1WebRTC Direct P2PLowestMaximum — zero server contactBoth peers on open / cone NAT
2TURN Relay via CoturnMediumHigh — encrypted relay, no storageOne or both peers behind restricted NAT
3AWS S3 Cloud UploadHighestModerate — temporary storage, auto-destructWebRTC fully fails (symmetric NAT, firewall)
08

End-to-End Transfer Flow

From anonymous session creation through lobby discovery, SDP exchange, and either a direct DataChannel transfer or the cloud fallback.

1  Alice → POST /api/auth/join {"anonymous_name": "Hacker_4821"}
2  Auth  → JWT token (UUID + name + 2 h expiry)
3  Alice → POST /api/heartbeat  (every 30 s)  → SETEX user_online:{uuid} 60
4  Bob   → GET  /api/users/online             → [{user_id, name}]   (Alice visible)
   ── Bob selects Alice from the lobby ──
5  Alice, Bob → emit('register', {uuid})      → Socket.IO userSocketMap
6  Alice → emit('offer',  {targetUuid, sdp})  → relayed to Bob
7  Bob   → emit('answer', {targetUuid, sdp})  → relayed to Alice
8  both  → emit('ice-candidate', …)           → trickled both ways

   ┌─ P2P succeeds (< 5 s) ───────────────────────────────────────┐
   │  DataChannel: JSON metadata {filename, size, type}           │
   │  DataChannel: binary chunks (16 KB)  → Bob writes IndexedDB  │
   │  Bob assembles Blob → triggers download                      │
   └─────────────────────────────────────────────────────────────┘
   ┌─ P2P fails (5 s timeout) ───────────────────────────────────┐
   │  Alice → POST /api/upload  (5 MB multipart stream to S3)    │
   │  S3    → 1-hour presigned URL → shared with Bob             │
   │  Bob   → downloads → triggers Lambda Nuke → S3 delete_object │
   └────────────────────────────────────────────────────────────┘
Signaling is a pure relay

Once ICE completes and the DataChannel opens, the signaling server is no longer involved. It never inspects or stores file data — only SDP offers/answers and ICE candidates.

09

Data Storage Schema

Redis presence store

Key pattern:   user_online:{uuid}
Value:         "{anonymous_name}"        e.g. "Hacker_4821"
TTL:           60 seconds
Heartbeat:     every 30 seconds (POST /api/heartbeat)
Scan method:   SCAN (non-blocking) + batched MGET

S3 object layout

Path:          uploads/{user_id}/{uuid4}.{extension}
Metadata:      {"sender_uuid": "{user_id}"}
Presigned URL: GET, expires in 3600 seconds (1 hour)
Lifecycle:     Expiration after 1 day
Multipart:     AbortIncompleteMultipartUpload after 1 day

IndexedDB (browser-side)

Database:      TransferCache
Object store:  chunks
Data:          raw binary ArrayBuffer (16 KB per entry)
Lifecycle:     cleared after Blob assembly and download trigger

Pydantic models

class JoinRequest(BaseModel):
    anonymous_name: str          # e.g. "Hacker_4821"

class TokenResponse(BaseModel):
    access_token: str            # HS256 JWT
    token_type: str              # "bearer"
10

API & Signaling Protocol

REST API — FastAPI, port 8000

MethodEndpointAuthPurpose
GET/ping—Health check
GET/metrics—Auto-instrumented Prometheus metrics
POST/api/auth/join—Anonymous session creation → HS256 JWT
POST/api/heartbeat✅Refresh Redis presence TTL to 60 s
GET/api/users/online—List active peers
POST/api/upload✅S3 cloud fallback upload → presigned URL

WebSocket signaling events — Socket.IO, port 3001

EventDirectionPayloadPurpose
registerClient → Server{uuid}Map UUID to socket ID
offerBidirectional{targetUuid, sdp}Relay WebRTC SDP offer
answerBidirectional{targetUuid, sdp}Relay WebRTC SDP answer
ice-candidateBidirectional{targetUuid, candidate}Trickle ICE candidates
disconnectAuto—Clean up UUID mapping
11

Security Architecture

Authentication flow

Client                                  Server
  │  POST /api/auth/join {"anonymous_name": "X"}
  │ ─────────────────────────────────────►  generate UUID4
  │                                         sign JWT (HS256): sub=uuid, name=X, exp=+2h
  │  ◄─────────────────────────────────────  {"access_token": "eyJ..."}
  │  POST /api/heartbeat  Authorization: Bearer eyJ...
  │ ─────────────────────────────────────►  decode + verify HS256 + check exp
  │                                         SETEX user_online:{uuid} 60
  │  ◄─────────────────────────────────────  200 OK

Container security posture

12

Key Design Decisions

16 KB DataChannel chunks
SCTP fragments messages crossing the ~64 KB boundary, destroying throughput. 16 KB stays well below it while minimising per-chunk overhead.
Redis ephemeral keys, not a DB
Presence is temporal. SETEX with a 60 s TTL means a crashed client's key simply evaporates — no cleanup jobs, no phantom users. 30 s heartbeat gives a 2× margin.
SCAN, not KEYS *
Redis is single-threaded; KEYS * blocks the whole keyspace. SCAN returns cursor-based pages and stays responsive at thousands of keys.
In-memory Redis fallback
InMemoryStore mirrors the async Redis API with asyncio.Lock and manual TTL, so python start.py runs with zero external dependencies.
aioboto3, not boto3
Synchronous boto3 blocks the event loop during uploads, serialising every transfer. aioboto3 wraps operations in async context managers for true concurrency.
Split Ingress resources
Nginx regex rewrite annotations apply to every path in an Ingress. One Ingress for /api rewrites, one without for / and /socket.io, prevents breaking static-asset paths.
Host networking for Coturn
TURN relay needs dynamic UDP ports across 49152–65535. Docker bridge NAT cannot handle that scale efficiently; host networking gives Coturn the raw stack.
IndexedDB, not in-memory arrays
A JS array of 500 MB of ArrayBuffers pushes the tab past 500 MB RAM. IndexedDB writes to disk, keeping RAM flat for arbitrary file sizes.
13

Challenges Faced & Solutions

#ProblemSolution
1Symmetric NAT — P2P worked on home networks but failed 100% on mobile data and corporate networksSelf-hosted Coturn TURN with host networking to relay encrypted media, plus the 5-second race so users never wait on a connection that will never form
2Railway dynamic $PORT — hardcoded --port 8000 made the container unreachable behind the proxyentrypoint.sh reads $PORT at runtime: exec uvicorn app.main:app --host 0.0.0.0 --port "${PORT:-8000}"
3Redis blocking the event loop — eager pool init at import time hung the whole FastAPI startup if Redis was slowLazy Redis init wrapped in the InMemoryStore fallback — connection attempted on first use, transparent takeover on failure
4Browser RAM exhaustion — a 200 MB file in a JS array killed the tab on mobileIndexedDB chunk buffering — each 16 KB chunk written straight to TransferCache; RAM stays constant
5DataChannel buffer overflow — sending as fast as FileReader reads silently dropped chunksBackpressure via bufferedAmountLowThreshold (64 KB) with pause/resume on onbufferedamountlow
6Nginx regex vs SPA routing — a single Ingress rewrote static assets to 404Two separate Ingress resources — one for /api rewrites, one for / and /socket.io
7Orphaned S3 multipart uploads — partial uploads persisted, accruing costThree-layer cleanup: immediate abort_multipart_upload, a 1-day lifecycle rule, and an on-demand Lambda
14

Deployment & Tech Stack

Backend
FastAPI, Uvicorn, redis.asyncio, aioboto3 / botocore, PyJWT (HS256), python-multipart, Pydantic, prometheus-fastapi-instrumentator
Frontend
React 18, Vite 5, Tailwind CSS 3, Socket.IO client, Axios, WebRTC APIs, IndexedDB
Infrastructure
Docker (multi-stage, non-root), Kubernetes, Nginx Ingress, Coturn, AWS S3 + Lambda, Prometheus (ServiceMonitor), Render / Railway, Let's Encrypt via cert-manager

One-command local development

python start.py
# boots FastAPI :8000 (hot-reload), Socket.IO :3001, Vite :5173
# waits 4 s, opens http://localhost:5173, monitors all 3 processes

Kubernetes deployment

kubectl apply -f infrastructure/k8s/app-secrets.yaml
kubectl apply -f infrastructure/k8s/api-deployment.yaml
kubectl apply -f infrastructure/k8s/frontend-deployment.yaml
kubectl apply -f infrastructure/k8s/socket-deployment.yaml
kubectl apply -f infrastructure/k8s/ingress.yaml
kubectl apply -f infrastructure/k8s/api-servicemonitor.yaml

Environment variables

VariableRequiredDefault
SECRET_KEY✅—
REDIS_URL—None → InMemoryStore
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY✅—
AWS_REGION✅eu-north-1
S3_BUCKET_NAME✅—
PORT—8000 / 3001

Hybrid P2P-Cloud Anonymous Transfer Engine · Technical Report · Built with FastAPI · React · WebRTC · Socket.IO · AWS · Redis · Kubernetes