Optimization Study on Processing Order of NetScaler Load Balancer Layer 7 Policies

NetScaler is one of the most popular load balancer devices in modern cloud computing data centers. It supports four types of layer 7 policies:

• Responder: Generates HTTP response upon certain HTTP request.
• Rewrite: Modifies the HTTP request or response.
• Content Switching: Routes request to downstream applications.
• Filter: Accept, Deny or Drop certain requests.

A virtual server can have multiple types of layer 7 policies applied simultaneously. In this case, the final output would depend on the order that the policies are processed. For instance, assuming a virtual server, http://www.site1.com, has the following layer 7 policies: 

• Responder: IF request URL = http://www.site1.com/ THEN HTTP 301 redirect to https://www.site1.com/
• Rewrite: IF request = https://www.site1.com/a/ THEN rewrite it to https://www.site1.com/b/
• Content Switching: IF request URL Path = /a/ THEN route to application_A ELSEIF request URL Path = /b/ THEN route to application_B
• Filter: IF source IP = 1.2.3.4 THEN reset TCP connection

When clients access http://www.site1.com/a/:

• If Responder is processed first, clients are redirected to https://www.site1.com/a/.
• If Content Switching is processed first, the request is routed to application_A.
• If Rewrite is processed first, the request is modified to /b/ and then routed to application_B.
• If Filter is processed first, the client connection is reset.

The example above shows that the processing order is critical, as the output would be totally different if the processing order varies. However many documents and online resources contain conflicting information of the processing order. For instance: 

• This document indicates the processing order is Responder -> Content Switching -> Filter -> Rewrite.
• Another document indicates the processing order is Content Switching -> Responder -> Rewrite -> Filter.

So we decided to perform several tests to clarify what the processing order really is. Below are the test cases, followed by our conclusion. 

Test Cases

Preparation 

1. Configure a CSVserver with two Content Switching policies: IF request URL Path = /a/ THEN route to LBVserver-A ELSEIF request URL Path = /b/ THEN route to LBVserver-B.
2. Configure a Rewrite policy on the same CSVserver, to change the request URL Path from /a/ to /b/.
3. Configure a Filter policy on the same CSVserver to RESET the TCP connection if source IP = 1.2.3.4.
4. Configure a Responder policy on the same CSVserver, to HTTP 301 redirect the request URL from HTTP to HTTPS.

Test 1: All the policies above enabled

$ curl -I http://IP-address-of-CSV/

HTTP/1.1 301 Moved Permanently

Location: https://IP-address-of-CSV/

Output matches the action defined in the Responder policy. This indicates that Responder is the first place in the processing order.

Test 2: Disable Responder, keep all the other policies enabled

$ curl -I http://IP-address-of-CSV/

curl: (56) Recv failure: Connection reset by peer 

Output matches the action defined in the Filter policy. This indicates that Filter policy is the second place in the processing order.

Test 3: Disable both Responder and Filter, keep all the other policies enabled

$ curl -I http://IP-address-of-CSV/a/

Check host in LBVserver_A, the log shows

10.20.30.40 - - [11/Jun/2018:01:02:03 +0000] "HEAD /b/ HTTP/1.1" 200 0 "-" "curl/7.35.0" "-"

Output shows the request is routed to application_A. This indicates that Content Switching policy is the third place in the processing order, and accordingly, Request_Rewrite is the forth place in the processing order.

Conclusion

Based on the test results our conclusion is that on NetScaler CSVserver, the layer 7 policies are processed in the order of Responder -> Filter -> Content Switching -> Rewrite.

All the tests are executed on NetScaler MPX v11.5.

Knowing the order helps eBay manage the load balancer policies precisely and efficiently, avoid misconfigurations, and ensure the availability, security, and performance of our site.