How B12 Hosts Websites at Scale

Goals for our new infrastructure

In building our new hosting infrastructure, we had several goals in mind:

• Infrastructure Reliability. While the availability requirements for all of our services are ambitious, the expectations for our customers’ websites is that they only go down when significant portions of the web are affected. To accomplish this, our hosting infrastructure relies on two relatively bulletproof pieces of internet architecture: Amazon’s Simple Storage Service (S3) and Fastly’s Content Delivery Network (CDN).

• Globally Distributed Low Latency. Our customers’ customers come from all over the world. We use Fastly’s CDN to cache our customers’ websites in several geographically distributed locations, dramatically decreasing page load times.

• Low Update Latency. There are two updates to customer websites that should appear near-instantaneously. First, if a customer or designer updates a website, those changes should appear the next time they view their website. Second, if a customer or designer adds a new domain to their website, we should route requests from that domain to the appropriate website as quickly as possible. A fine-grained use of Fastly’s cache invalidation mechanisms allows us to make these updates visible to customers with sub-second latency.

• SSL Support. Being a good citizen of the web means supporting SSL certificates on all customer websites.

• Scale. While you can never predict where your scalability bottlenecks will be across orders of magnitude, we’re pretty confident our infrastructure will scale to hundreds of thousands of customer websites without any significant re-architecture.

Architecture Overview

Configuration API on S3

The S3 bucket for the configuration API has the following layout to store configurations for our customers. We have sub-buckets for our different server environments (such as dev or production) to allow for normal development on the same infrastructure as production. Each server environment sub-bucket also holds different website environments (e.g., our customers’ staging and production environments) since our customers and designers iterate on their designs before publishing them.

config-bucket/
    dev/
        [...]
    production/
        staging/
            [...]
        production/
             joshblum.b12sites.com/
                 <shared-secret> # shh
                     b12-hosting-path: <joshblum-uid>
                     b12-force-ssl: 1
                     b12-redirect-from-0: about.html
                     b12-redirect-to-0: index.html    
                     b12-redirect-from-n: team-page.html
                     b12-redirect-to-n: team.html        
                     [...]
             [...]

In the above example we show the configuration for the domain joshblum.b12sites.com. When a request comes to our Fastly service from joshblum.b12sites.com, we programmatically build the URL to get the configuration API metadata. <shared-secret> is a secret shared between the API and our Fastly service. The data we store isn’t sensitive and is read-only, but we like that the obscure secret makes it harder to explore our domain configuration data ;).

A Fastly service can only read HTTP headers or request metadata during processing (it can’t read the request content), so we encode our configuration data as custom HTTP headers. The b12-hosting-path specifies the path to the website content for the particular customer in our hosting service (described below). b12-force-ssl specifies whether SSL is available for the given domain. We also store a set of redirects (e.g., b12-redirect-from-0, b12-redirect-to-0) if a customer wishes to set up aliases for their pages.

Clothes

Configuration Updates

Fastly Service Configuration

Index Rewriting:

• At the root of a customer website (/), we automatically redirect the request to index.html.

set req.url = regsub(req.url, "^/$", "/index.html");

Customer-Specific Redirects

• Customers can specify redirects for their website to internally redirect from one page to another. (Note: We use the Django Template language to write our VCL config for convenience, so the syntax isn’t 100% VCL here. More on that later!)

  {% raw %}
  {% for REDIRECT_FROM_i, REDIRECT_TO_i in B12_REDIRECTS %}
  if (req.http.{{REDIRECT_FROM_i}} && req.http.{{REDIRECT_TO_i}} &&
    req.url == req.http.{{ REDIRECT_FROM_i }}) {
    set req.url = req.http.{{ REDIRECT_TO_i }};
    error 802 "Force Redirect";
  }
  {% endfor %}
  {% endraw %}

Force HTTPS

• For all domains that have a SSL certificate (currently just *.b12sites.com), we force every request to HTTPs.

if (!req.http.Fastly-SSL && req.http.{{ B12_FORCE_SSL }} == "1") { 
  error 801 "Force SSL";  
}

Page Rewriting

• URLs that do not contain a dot in them are rewritten with a .html at the end, so if someone visited the page /about we would automatically rewrite the url to /about.html.

set req.url = regsub(req.url, "^([^\.]+?)([\/]?)$", "\1\.html");

404 Redirect

• If S3 returns a 404 or 403 for a nonexistent page, we redirect the browser to a customer branded 404.html page.

# If we get a 404 or 403 from the backend, trigger an error so we
# can redirect to a custom 404 page.  
if (beresp.status == 404 || beresp.status == 403) {     
  error 900 "Not Found";  
}

Website Storage & Hosting on S3

S3 Storage

Our customer websites are hosted in a single S3 bucket which contains sub-buckets for our server and website environments. Below is an example of the bucket layout:

hosting-bucket/
    dev/
        [...]
    production/
        staging/
            [...]
        production/
             <joshblum-uid>/
                index.html
                about.html
                img/[...]
             [...]

Content Updates

Whenever new content is pushed to S3, we purge all potentially stale content in Fastly. Similar to configuration changes, each website has a Surrogate Key with which its content is tagged and purged.

Lessons Learned

Building our new hosting infrastructure has been a wild ride. Along the way, we’ve picked up a few takeaways that apply more broadly.

Similar Architecture Across Environments

From the start, we designed our architecture to allow for our normal development workflows, namely working in a dev environment, testing large changes in a staging environment and ultimately releasing new features onto production. It was important that the infrastructure have as few differences as possible between the environments to allow for testing and to minimize production related bugs.

Automated Testing

During the development of our Fastly service, we realized it was key to have automated tests to verify the features we wanted to support. We have a testing service in Fastly that mirrors our production service and allows us to test new features as we develop them without downtime, regressions, or misconfigurations errors for our customers.

Templating Fastly Configs

  {% raw %}
    {% for REDIRECT_FROM_i, REDIRECT_TO_i in B12_REDIRECTS %}      
    if (resp.http.{{REDIRECT_FROM_i}} && resp.http.{{REDIRECT_TO_i}}) {        
      set req.http.{{REDIRECT_FROM_i}} = resp.http.{{REDIRECT_FROM_i}};        
      set req.http.{{REDIRECT_TO_i}} = resp.http.{{REDIRECT_TO_i}};      }    
    {% endfor %}
  {% endraw %}

Configuration Service Scalability & Resiliency

Amazon Web Services Gotchas

Next Steps

Conclusion

Fast and reliable web hosting is a small but crucial piece of the customer experience at B12. We hope that with our new infrastructure, all of our customers and their customers can enjoy websites at blazing fast speeds wherever they are in the world!