HTTP specification (RFC 7230) requires that an HTTP request must begin with an HTTP header. The size of the HTTP header however is not restricted in the RFC, allowing each web server implementation to define its own limit, as well as its own behavior when the limit is exceeded.
On the NetScaler load balancer, the HTTP header is limited to 24KB by default. If the load balancer receives an oversized HTTP header, it immediately stops processing all the layer 7 policies on that TCP connection and begins treating the traffic as TCP payload (CTX120674). This behavior is not quite straightforward when combined with content switching policies and may cause unexpected consequence and a bad user experience if overlooked. In this article, we are going to use an example to explain the behavior, followed by recommendations on how to customize this behavior.
Assume there is an HTTP content switching virtual server (CSV) bound to multiple layer 7 content switching policies:
- Policy A: if request = http://www.foo.com/url-a, then send traffic to application-a.
- Policy B: if request = http://www.foo.com/url-b, then send traffic to application-b.
- Policy C: if request = http://www.foo.com/url-c, then send traffic to application-c.
- Default Policy: if request = http://www.foo.com/any-other-url, then send traffic to application-x (default application).
A client establishes a TCP connection with the CSV and conducts multiple HTTP transactions sequentially by reusing this TCP connection. The header size is less than 24KB for all the requests, except request 3 which header size is greater than 24KB.
- Request 1 = http://www.foo.com/url-a (Header < 24 B)
- Request 2 = http://www.foo.com/url-b (Header < 24 B)
- Request 3 = http://www.foo.com/url-c (Header > 24KB)
- Request 4 = http://www.foo.com/url-a (Header < 24 B)
- Request 5 = http://www.foo.com/url-b (Header < 24 B)
- Request 6 = http://www.foo.com/url-c (Header < 24 B)
Some engineers may expect request 3 to be rejected and all the other requests to be sent to the desired application, but this is not true. The actual outcome is illustrated as below:
- Request 1 is sent to application-a by Policy A, as expected.
- Request 2 is sent to application-b by Policy B, as expected.
- Request 3 has header size > 24KB, so load balancer stops processing layer 7 policies on this TCP connection. Request 3 is considered as TCP payload and sent to application-x (instead of application-c) because application-x is the default "catch all" application.
- Furthermore, all the subsequent requests on the same TCP connection, such as request 4, 5, and 6, are also considered as TCP payload and sent to the default application (application-x), regardless of whether their header size is over the limit.
In summary, an oversized request can poison all the subsequent legitimate requests, resulting in bad user experience.
The header size limit and over-limit behavior need to be managed properly to avoid the unexpected outcomes. Our recommendations are:
1. Manage the header size limit.
- Define a standard header size limit for all the applications. For reference, legacy applications usually have a lower limit such as 8KB or 16KB, while modern platforms often have a high limit such as 32KB or more.
- On load balancer, set the header size limit to a number that is equal or higher than the standard limit on the applications, so that the legitimate traffic will not trigger the bypass layer 7 behavior.
2. Manage the over-limit behavior.
- On the default application, prepare to return a proper landing page instead of the HTTP 400 error code for unknown requests or requests with oversized headers.
- On load balancer, update the HTTP profile to reset the TCP connection when an oversized header is received, so that an invalid request would not poison the subsequent legitimate requests.