Building global web apps
with multi-region hosting

Jordi Boggiano
@seldaek

Building the internet for over 10 years
  → seld.be

Composer lead and other OSS
  → github.com/Seldaek

Dev & Ops at Teamup.com

Dev & Ops at Private Packagist

Why host across
multiple regions?

Users in multiple locations

Latency gets really bad

Redundancy against
regional host failure

We use CDN for web assets,
why not the rest?

Why *not* host across
multiple regions?

Database without multi-master setup

Cloud-first DBs and NoSQL help here

Google Spanner, AWS Aurora, DynamoDB, mongoDB

Cloud providers don't help that much

No cross-region Redis replicas on AWS

No cross-region VPCs until late 2017

Awareness

Devs and stakeholders do
not suffer from latency

Case Studies















Packagist.org

Goals for repository

High reliability

Simplicity

Global

Low-cost

Custom-built CDN

Mirrors feed from primary server when it is up

Invalidation almost instant

DNS-based Route 53 routing

Health-checks for failover

Easy to set up new nodes,
standalone and dumb

Downsides

Only repository (metadata)

Proxy fallback to handle mirror races

teamup.com

Goals for the web app

Global

Low latency

High reliability for data access

Low maintenance

First iteration

Terraform + AWS based setup

Primary region has site, db and workers

Replica regions have site and db

Redis sessions handled locally

Reads handled in every region

Writes talk to primary DB via VPC peering

Challenges

Redis/SQL RTT of ~100ms is deadly

ProxySQL connection pools helped a bit

Second iteration

Proxy write requests to primary region

Session forwarding in HTTP headers

Client IP forwarding using proxy headers

HTTPS + VPC peering for security

Signed request/response headers for extra security

Failover to replica processing if proxying failed

Challenges

HTTPS overhead for proxied requests vs clients hitting primary region

Challenges

nginx local proxy for connection pooling

                  upstream primary_server_backend {
                      server 10.0.123.123:443 max_fails=1 fail_timeout=30s;
                      # ...
                      keepalive 8;
                  }
                  server {
                      listen 127.0.0.1:1234;
                      location / {
                          proxy_pass https://primary_server_backend;
                          proxy_http_version 1.1;
                          proxy_ssl_session_reuse on;
                          proxy_set_header Connection "";
                          # ...
                      }
                  }
                

Challenges

Network jitter fails random requests

AWS ELB times out random requests

Session size is limited in headers

Downsides

Read-only in case primary
region is down

Workers in single region (but multi AZ)

Higher complexity

Summary

Case-by-case

Audience location

Responsiveness requirements

Team size

Tech stack

Choose cloud DBs early if necessary

Thank you

Questions?

@seldaek

slides.seld.be


Feedback

joind.in/talk/2a002















Private Packagist

packagist.com

Goals for repository

Security

Global

Speed

High reliability

Custom-built CDN

Mirrors have a read replica of the DB

Local "caches" for redis / S3 data

API between replicas and primary

Challenges

No redis replication

No way to talk to primary DB from replica regions (fixed late 2017)

Downsides

Only repository (metadata + zip)

Updates & co handled in a single region, but customer installs safe against regional outages