Building global web apps
with multi-region hosting

Jordi Boggiano

Building the internet for over 10 years

Composer lead and other OSS

Dev & Ops at

Why host across
multiple regions?

Users in multiple locations

Latency gets really bad

Redundancy against
regional host failure

We use CDN for web assets,
why not the rest?

Why *not* host across
multiple regions?

Database without multi-master setup

Cloud-first DBs and NoSQL help here

Google Spanner, AWS Aurora, DynamoDB, mongoDB

Cloud providers don't help that much

No cross-region Redis replicas on AWS

No cross-region VPCs until late 2017


Devs and stakeholders do
not suffer from latency

Case Studies

Goals for repository

High reliability




Custom-built CDN

Mirrors feed from primary server when it is up

Invalidation almost instant

DNS-based Route 53 routing

Health-checks for failover

Easy to set up new nodes,
standalone and dumb


Only repository (metadata)

Proxy fallback to handle mirror races

Private Packagist

Goals for repository




High reliability

Custom-built CDN

Mirrors have a read replica of the DB

Local "caches" for redis / S3 data

API between replicas and primary


No redis replication

No way to talk to primary DB from replica regions (fixed late 2017)


Only repository (metadata + zip)

Updates & co handled in a single region, but customer installs safe against regional outages

Goals for the web app


Low latency

High reliability for data access

Low maintenance

First iteration

Terraform + AWS based setup

Primary region has site, db and workers

Replica regions have site and db

Redis sessions handled locally

Reads handled in every region

Writes talk to primary DB via VPC peering


Redis/SQL RTT of ~100ms is deadly

ProxySQL connection pools helped a bit

Second iteration

Proxy write requests to primary region

Session forwarding in HTTP headers

Client IP forwarding using proxy headers

HTTPS + VPC peering for security

Signed request/response headers for extra security

Failover to replica processing if proxying failed


HTTPS overhead for proxied requests vs clients hitting primary region

nginx local proxy for connection pooling helped

Network jitter fails random requests

AWS ELB times out random requests

Session size is limited in headers


Read-only in case primary
region is down

Higher complexity



Audience location

Responsiveness requirements

Team size

Tech stack

Choose cloud DBs early if necessary

Thank you