Analysis of a CDN Origin Server Traffic Spike

Last week, we experienced a sudden surge in traffic to our CDN origin server. The root cause was that a batch of files on the CDN had expired, triggering a massive number of origin fetches.

CDN Origin Fetch Process

1. First Fetch: If the origin server returns a Last-Modified header, the CDN records it.
2. Subsequent Fetches (After Expiry): Once the cached file expires, if the origin previously returned a Last-Modified header, the CDN will access the origin using an If-Modified-Since header (containing the value from the original Last-Modified). If the file has not changed, the origin returns a 304 status. However, if the origin did not return a Last-Modified header initially, the CDN will download the entire file, and the origin returns a 200 status.

The traffic spike occurred because the origin server was not returning the Last-Modified header. The solution is to ensure the origin server returns this header.

However, there are several important considerations. Let’s examine the Nginx configuration preceding the origin server:

location ~ /(?<name>.+?)/.* {
    expires 100d;

    rewrite ^/.+?/(.*)$ /$1 break;

    proxy_cache m_wdjcdn_com;
    proxy_cache_key "$request_uri";
    proxy_cache_valid 200 30d;

    proxy_set_header Host $name.cdn.xxxx.com;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_pass http://new-source-nodes;
}

1. Local Static Files: If the origin files are served directly from the local disk (using the root directive) rather than being proxied to a backend via proxy_pass, this issue does not arise. This is because Nginx fully supports Last-Modified and If-Modified-Since semantics for static files.
2. Proxy Cache Impact: Without the proxy_cache directive, even if the origin returns a Last-Modified header, a 304 response will not be generated. This is because Nginx does not save the headers unless they are cached locally. When files are cached to the local disk, Nginx retains them, allowing it to return a 304 response.
3. if_modified_since Directive: Nginx has a directive called if_modified_since, which defaults to exact. This means the value of the If-Modified-Since header must exactly match the previously obtained Last-Modified value; otherwise, a 200 response is sent. You can change if_modified_since to before, which allows the If-Modified-Since value to be greater than or equal to the Last-Modified value (i.e., indicating the resource has been updated).

http://nginx.org/en/docs/http/ngx_http_core_module.html