Practical Approaches for Distributing HTTP Traffic

When it comes to handling lots of visitors, high-volume sites like Yahoo!, Netscape, and Microsoft have learned that the actual quality of service a Web server provides to end users typically depends on two parameters -- network-transfer speed and server-response time. Network-transfer speed is mainly a matter of your Internet-link bandwidth, while server-response time depends upon resources: fast CPU(especially for CGI programs), lots of RAM (especially for parallel-running HTTP daemon processes), and good I/O performance (especially for disk and network traffic).

What do you do when these resources are exhausted and your Web server is struggling against heavy traffic? You could install more RAM on existing machines, or perhaps replace the CPU with a faster one. You could also use faster or dedicated SCSI controllers and disks with shorter access times (perhaps a RAID system with a huge cache). Software could be tuned as well; you could adjust operating-system parameters and Web-server software to achieve better performance.

Or you can address the problem with an alternative approach: Improve performance by increasing the number of Web servers. This involves an attempt to distribute the traffic onto a cluster of back-end Web servers. Aside from the technical hurdles, this is an interesting approach, because the back-end servers don't need to be large-scale machines -- medium-scale hardware works just fine.

Assume there are N back-end servers available, named wwwX.foo.dom (where X is between 1 and N), and you want to use the cluster approach to solve the resource problem. The goal then is to balance the traffic (addressed to www.foo.dom) onto these available servers so that the technical distribution is totally transparent to the end user. Your Web-site visitors can still use canonical URLs of the http://www.foo.dom/bar/quux/ form to reach the Web cluster, and are not directly confronted with the fact that their requests are being served by more than one machine. They never see the underlying distribution. This is important both for backward compatibility and to avoid problems (for instance, bookmarking pages, or a back-end server crash). The new Web cluster should behave identically to the old single-machine approach.

The DNS Approach

The first solution is based on the Domain Name System (DNS); see Figure 1.

Here we exploit the fact that the first step a browser has to take to retrieve the URL www.foo.dom/bar/quux/ is to resolve the corresponding IP address for www.foo.dom. This is accomplished with a passive resolver library that calls a nearby DNS server, which then actively iterates over the distributed DNS server hierarchy on the Internet until it reaches your DNS server, which finally gives the IP address. Instead of giving a static address for www.foo.dom, the DNS server gives the address of one of the back-end Web servers. Which one depends on the scheme you want to use for balancing the traffic and the technical possibilities available for this decision.

The state of the art in DNS-server implementation is still the Berkeley Internet Name Daemon (BIND), which is currently developed and maintained by the Internet Software Consortium (ISC).

The key to this solution is the fact that BIND provides a nifty, but widely unknown, feature called "round robin." Round robin lets you select and provide a particular IP address from a pool of addresses when a DNS request arrives. In the meantime, the selection pointer to this pool is increased and when it reaches the last element, it starts again with the first one. It is configured making www.foo.dom an alias that is mapped to wwwX.foo.dom by using multiple CNAME (canonical name) resource records as shown below.


www.foo.dom. IN CNAME www1.foo.dom
IN CNAME www2.foo.dom
IN CNAME www3.foo.dom
IN CNAME www4.foo.dom
IN CNAME www5.foo.dom
IN CNAME www6.foo.dom


This sounds perfect in theory, because the traffic is distributed equally onto the Web cluster. But in practice, we have to fight with the fact that DNS servers cache the resolved data at any point in the DNS hierarchy both to decrease the resolver traffic and to speed up resolving. This caching is controlled by a time-to-live (TTL) value that is appended to each piece of information by our DNS server. It resides in the "start of authority" (SOA) resource record of the BIND zone file where the CNAME resource records reside.

Now we have a dilemma: When we set this TTL value too high, we decrease the DNS traffic on our Internet link, but we let the other DNS servers cache our information too long, which leads to bad HTTP traffic distribution over our Web cluster. On the other hand, when we set this TTL value too low, we increase our DNS traffic and the request time for the visitor dramatically, because the other DNS servers expire our information faster, so they have to resolve it more often. But we then have a better balancing of HTTP traffic. The decision for the best TTL value is thus dependent on the level of balancing that we want and how many intermittent delays we think the visitor will accept before deciding that the Web-cluster approach has reduced the quality of service. In practice, a TTL of one hour has been shown to be adequate.

One problem remains: When we change the SOA resource record in the zone file for foo.dom in order to achieve the effect for www.foo.dom, we also change the TTL of all other entries in this zone file. For instance, ftp.foo.dom is also assigned the decreased TTL, which increases the DNS traffic unnecessarily. To overcome this problem, we have to use another trick: We move the round-robin entry for www.foo.dom into a separate zone file that only gets the decreased TTL.
For the configuration, we use a round-robin subdomain, rr.foo.dom, to achieve the effect.


1 ;;
2 ;; named.boot -- BIND daemon boot configuration
3 ;;
4
5 :
6 :
7 ;type domain source-file
8 primary foo.dom db.foo
9 primary rr.foo.dom db.foo.rr
10 :
11 :



1 ;;
2 ;; db.foo -- BIND DNS zone file for the foo.dom domain
3 ;;
4
5 @ IN SOA world.foo.dom. root.world.foo.dom. (
6 1998021502 ; SERIAL
7 604800 ; REFRESH: Secondaries refresh after 1 week
8 3600 ; RETRY: Secondaries retry after 1 hour
9 604800 ; EXPIRE: Maximum TTL of Data is 1 week
10 86400 ; MINTTL: Minimum TTL of Data is 1 day
11 )
12
13 IN NS world.foo.dom.
14
15 ;;
16 ;; the resource record for www.foo.dom which
17 ;; maps to the Round-Robin domain
18 ;;
19 www IN CNAME www.rr.foo.dom.



1 ;;
2 ;; db.foo.rr -- BIND DNS zone file for the rr.foo.dom domain
3 ;;
4
5 ;;
6 ;; the start of authority (SOA) resource record which
7 ;; forces a minimal time-to-live (TTL) for this zone file
8 ;;
9 @ IN SOA world.foo.dom. root.world.foo.dom. (
10 1998021501 ; SERIAL
11 3600 ; REFRESH: Secondaries refresh after 1 hour
12 600 ; RETRY: Secondaries retry after 10 minutes
13 3600 ; EXPIRE: Maximum TTL of Data is 1 hour
14 1800 ; MINTTL: Minimum TTL of Data is 30 minutes
15 )
16
17 IN NS world.foo.dom.
18
19 ;;
20 ;; the multiple canonical name (CNAME) resource record
21 ;; which implies BIND's Round-Robin (RR) feature
22 ;;
23 www IN CNAME www1.rr.foo.dom.
24 IN CNAME www2.rr.foo.dom.
25 IN CNAME www3.rr.foo.dom.
26 IN CNAME www4.rr.foo.dom.
27 IN CNAME www5.rr.foo.dom.
28 IN CNAME www6.rr.foo.dom.
29
30 ;;
31 ;; the address (A) resource records for the
32 ;; final NAME -> IP mapping
33 ;;
34 www1 IN A 192.168.1.1
35 www2 IN A 192.168.1.2
36 www3 IN A 192.168.1.3
37 www4 IN A 192.168.1.4
38 www5 IN A 192.168.1.5
39 www6 IN A 192.168.1.6


The Reverse Proxy Approach

The DNS-based approach is simple and elegant, but has some drawbacks. The caching of DNS data and the simple round-robin decision scheme of BIND restricts its usefulness. For instance, when one of the back-end servers crashes, www.foo.dom is not available to all visitors of this back-end server, at least for the TTL we used. Even hitting the Reload button in the browser won't work, because once a particular back-end server is resolved, it remains the contact point for that particular visitor until the address information expires. The round-robin scheme also treats all back-end servers equally. For example, back-end servers can't be selected dependent on the requesting URL. Perhaps we only want to run CPU-intensive jobs (CGI programs, for
instance) on a subset of the back-end servers to avoid slow static-data serving.

The solution is a "reverse proxy," an HTTP proxy server that operates in the opposite direction of the commonly known one, hence the name. Usually, an HTTP proxy server is used near or in front of the browsers to bundle requests (when using a firewall) and to reduce bandwidth waste by performing data caching. Browsers call their proxy with the absolute URL http://www.foo.dom/bar/quux/ and the proxy either forwards this request to parent proxies or requests the relative URL /bar/quux/ from www.foo.dom. In other words, the proxy either forwards absolute URLs or translates them to relative URLs. In contrast to this, a reverse proxy masquerades as the final www.foo.dom server, and translates the relative URL back to an absolute URL addressed to one of its back-end servers.

Figure 2 shows how a reverse proxy resides side-by-side with the back-end servers and visually (for the browser or the other proxies) acts as the final Web server. But instead of serving the request itself, it determines a proper back-end server on-the-fly, turns the request over to it, and then forwards the response. No DNS tricks are needed here; www.foo.dom now actually resolves to the IP address of the reverse proxy in the DNS. For security and/or speed considerations, the back-end servers can even be placed on a separate subnet that stays behind the reverse proxy

Figure 3 This way, you separate the communication traffic between the reverse proxy and its back-end servers and even avoid N officially assigned IP addresses and DNS entries for the back-end servers. Additionally, you can place the back-end servers behind your company's firewall. A reverse proxy is a very elegant solution that provides maximum flexibility for your network topology.

Once this network/machine topology has been established, numerous benefits emerge. First, we have a single point of access -- the reverse proxy. This leads to simplified traffic logging and monitoring of the Web site, although we are now using a Web cluster instead of a single server. Secondly, we now have complete control over the back-end delegation scheme, because it's done locally in the reverse proxy for each request and not cached somewhere on the Internet. Because the delegation scheme is performed locally, changes are activated immediately. For instance, when one of the back-end servers crashes, we just change the delegation configuration of the reverse proxy, so the crashed back-end no longer leads to errors for the visitors. After it
is repaired we can reactivate it as simply as we deactivated it.

One problem remains: Which hardware, software, and configuration can be used to implement the reverse proxy? The choices are many. There are some dedicated proxy-software packages (for example, Squid Internet Object Cache, Netscape's or Microsoft's Proxy Server, and Sun's Netra Proxy Cache Server) and hardware-based solutions (such as Cisco Systems' LocalDirector and Coyote Point Systems' Equalizer) that can be used as reverse proxies. As part of the Apache team, I've designed a pragmatic and cheap, but nevertheless high-performance and flexible, all-in-one solution.

Because we have to rewrite a mass of relative URL requests to absolute URL requests to the back-end servers, we need a scalable server with a powerful URL-rewriting engine and an HTTP-proxy engine. Apache already provides these with its preforking process model and its mod_rewrite and mod_proxy modules. So, the idea is to strip down the full-featured Apache modules and configure them according to our required functionality.

Functionality was the problem here because mod_rewrite lacked the ability to perform random selection. After thinking about the reverse-proxy functionality, we also noticed that mod_proxy lacked the ability to divert HTTP responses back to itself. Because high performance is a major requirement for a reverse proxy, the only alternative would be Squid, the most popular dedicated proxy. But it's not easier to use this program as a reverse proxy. So we decided to stay on the Apache track and enhance it to create a full-featured reverse proxy. At the time of this writing, the patches are being considered for inclusion in the official Apache sources for version 1.3b6, but currently only Apache 1.3b5 is available. So we first had to create the Apache binary out of the original source code and the patches.


1 #!/bin/sh
2 ##
3 ## apache-rproxy.mk -- Build the apache-rproxy binary
4 ##
5
6 V=1.3b5
7
8 echo "Unpacking Apache $V distribution tarball" 1>&2
9 gunzip 10 cd apache_$V
11 echo "Patching sources for Reverse Proxy support" 1>&2
12 patch -p1 <../apache_$V.patch-rproxy >/dev/null 2>&1
13 cd src
14 echo "Configuring sources for Reverse Proxy usage" 1>&2
15 cat Configuration.tmpl |\
16 sed -e 's/^AddModule/# AddModule/g' |\
17 sed -e 's/^[# ]*\(AddModule.*mod_rewrite.o\)/\1/g' |\
18 sed -e 's/^[# ]*\(AddModule.*libproxy.a\)/\1/g' |\
19 sed -e 's/^[# ]*\(AddModule.*mod_mime.o\)/\1/g' |\
20 sed -e 's/^[# ]*\(AddModule.*mod_status.o\)/\1/g' |\
21 sed -e 's/^[# ]*\(AddModule.*mod_log_config.o\)/\1/g' |\
22 sed -e 's;^EXTRA_CFLAGS=.*;EXTRA_CFLAGS=DSERVER_SUBVERSION=
\\"rproxy\/1.0\\" -DBUFFERED_LOGS -DDYNAMIC_MODULE_LIMIT=0;g' |\
23 cat >Configuration
24 ./Configure >/dev/null 2>&1
25 echo "Building runtime binary" 1>&2
26 make >/dev/null 2>&1
27 strip httpd
28 cp httpd ../../apache-rproxy
29 echo "Cleaning up" 1>&2
30 cd ../..
31 rm -rf apache_$V


Above code shows a script that automatically builds the binary.
Online for corresponding patches and sample configuration files.

Online


BIND
www.isc.org/bind.html

Apache
www.apache.org

Apache Reverse Proxy Patch
www.engelschall.com/pw/wt/loadbalance/

Alternative products

Software-based solutions:

Squid Internet Object Cache
squid.nlanr.net/Squid/

Netscape Proxy Server 
www.netscape.com/proxcy/v3.5

Microsoft Proxy Server
www.microsoft.com/proxy/

Sun Netra Proxy Cache Server
www.sun.com/netra

Hardware-based solutions:

Cisco Systems LocalDirector
www.cisco.com/

Coyote Point Systems Equalizer
www.coyotepoint.com/equalizer.htm


After running this script, we receive a binary named apache-rproxy, which is a heavily stripped-down Apache plus the missing functionality added. So now we can start configuring it as our reverse proxy. We assume that we have a pool of exactly six back-end Web servers, named www1.foo.dom through www6.foo.dom. www5.foo.dom and www6.foo.dom should be dedicated to running CPU-intensive jobs, while www1.foo.dom through www4.foo.dom should mainly serve the static data. Inside these two subsets of back-end servers, the traffic should be balanced.

Let's start with the configuration of our available back-end servers. We create a file named apache-rproxy.conf-servers as below

1 ##
2 ## apache-rproxy.conf-servers -- Apache/mod_rewrite selection table
3 ##
4
5 # list of back-end servers which serve static
6 # pages (HTML files and Images, etc.)
7 static www1.foo.dom|www2.foo.dom|www3.foo.dom|www4.foo.dom
8
9 # list of back-end servers which serve dynamically
10 # generated page (CGI programs or mod_perl scripts)
11 dynamic www5.foo.dom|www6.foo.dom


Under the key static, we list the servers that serve the static data, and under the key dynamic, we list the ones dedicated to dynamic data.

Additionally, we need the actual Apache configuration file that controls the apache-rproxy binary

1 ##
2 ## apache-rproxy.conf -- Apache configuration for Reverse Proxy Usage
3 ##
4
5 # server type
6 ServerType standalone
7 Port 80
8 MinSpareServers
9 StartServers
10 MaxSpareServers
11 MaxClients
12 MaxRequestsPerChild 10000
13
14 # server operation parameters
15 KeepAlive on
16 MaxKeepAliveRequests 100
17 KeepAliveTimeout 15
18 Timeout 400
19 IdentityCheck off
20 HostnameLookups off
21
22 # paths to runtime files
23 PidFile /path/to/apache-rproxy.pid
24 LockFile /path/to/apache-rproxy.lock
25 ErrorLog /path/to/apache-rproxy.elog
26 CustomLog /path/to/apache-rproxy.dlog "%{%v/%T}t %h -> %{SERVER}e
URL: %U"
27
28 # unused paths
29 ServerRoot /tmp
30 DocumentRoot /tmp
31 CacheRoot /tmp
32 RewriteLog /dev/null
33 TransferLog /dev/null
34 TypesConfig /dev/null
35 AccessConfig /dev/null
36 ResourceConfig /dev/null
37
38 # speed up and secure processing
39
40 Options -FollowSymLinks -SymLinksIfOwnerMatch
41 AllowOverride None
42
43
44 # the status page for monitoring the reverse proxy
45
46 SetHandler server-status
47
48
49 # enable the URL rewriting engine
50 RewriteEngine on
51 RewriteLogLevel 0
52
53 # define a rewriting map with value-lists where
54 # mod_rewrite randomly chooses a particular value
55 RewriteMap server rnd:/path/to/apache-rproxy.conf-servers
56
57 # make sure the status page is handled locally
58 # and make sure no one uses our proxy except ourself
59 RewriteRule ^/rproxy-status.* - [L]
60 RewriteRule ^(http|ftp)://.* - [F]
61
62 # now choose the possible servers for particular URL types
63 RewriteRule ^/(.*\.(cgi|shtml))$ to://${server:dynamic}/$1
[S=1]
64 RewriteRule ^/(.*)$ to://${server:static}/$1
65
66 # and delegate the generated URL by passing it
67 # through the proxy module
68 RewriteRule ^to://([^/]+)/(.*) http://$1/$2 [E=SERVER:$1,P,L]
69
70 # and make really sure all other stuff is forbidden
71 # when it should survive the above rules...
72 RewriteRule .* - [F]
73
74 # enable the Proxy module without caching
75 ProxyRequests on
76 NoCache *
77
78 # setup URL reverse mapping for redirect reponses
79 ProxyPassReverse / http://www1.foo.dom/
80 ProxyPassReverse / http://www2.foo.dom/
81 ProxyPassReverse / http://www3.foo.dom/
82 ProxyPassReverse / http://www4.foo.dom/
83 ProxyPassReverse / http://www5.foo.dom/
84 ProxyPassReverse / http://www6.foo.dom/


Apache-rproxy first sets up the runtime parameters, then configures the auxiliary files that Apache uses with a custom log file that shows only the request delegation. Next, we add more directives to make Apache quiet on startup and to avoid runtime side effects. Then we activate the online status monitor for our proxy through URL www.foo.dom/rproxy-status. The actual reverse-proxy configuration follows. First we turn on the URL rewriting engine without logging. Then we activate the apache-rproxy.conf-servers file by defining a rewriting map called servers that has a random subvalue post-processing enabled (the rnd-feature, which was added by our patch). We then have to make sure that the status monitor is handled locally, rather than by the back-end servers, and that no one on the Internet can exploit us by using our reverse proxy as a standard proxy.

The delegation scheme is now ready to be implemented. First, we delegate all URLs to CGI programs and SSI pages to the servers under the key dynamic, which is either the server www5.foo.dom or www6.foo.dom. The one used is randomly chosen by mod_rewrite. All other URLs are then delegated to the servers under the key static. The delegation is activated by passing the URL through the Apache proxy module mod_proxy, while setting the environment variable SERVER

to provide the logging module with complete information to write the delegation log file. Then, we make sure no URLs survive. The program activates mod_proxy as a plain proxy without caching. We then configure mod_proxy as a reverse proxy by using the second feature we've patched into our Apache program. We force our reverse proxy to divert back to itself all URLs in http Location headers that the back-end servers send on HTTP redirects to again use the reverse proxy. Either the back ends are not directly accessible, or we want to let all traffic flow over our reverse proxy and avoid bypassed traffic.

Server Processes

Finally, we must calculate the number of servers (NOS) and the amount of RAM in MB (MBR) that we need for our reverse proxy. To calculate these values, we need three input parameters: the maximum number of HTTP requests per minute (RPM) we expect, the average number of seconds an HTTP request needs to be completely served (SPR = seconds per request), and the maximum number of MB an apache-rproxy process needs to operate under the operating system (SPS = server process size). The formulas in


;NOS = ceil(RPM * SPR * (1/60) * (100/80))
MBR = ceil((SPS * NOS * (100/70)) / 16) * 16


assume that because of lingering socket closes, 20 percent of the servers are not always available. These formulas also assume that we conservatively want to use only 70 percent of the available memory for our reverse proxy, and that we have only 16-MB chunks of RAM available.

For instance, when we run our reverse proxy under FreeBSD.

Performance Tuning Apache Under FreeBSD

When you are running Apache as your Web server on top of FreeBSD, you have a lot of parameters to tune in order to achieve maximum performance.

As with most operating systems, the TCP/IP listen queue is often the first limit encountered. It restricts the pending TCP requests. The second important parameter is the number of mbuf clusters, which should be increased. Additionally, you can increase the maximum number of allowed child processes and open file descriptors. So, for a heavily loaded machine you may want to increase these values in your kernel config as depicted as below

maxusers 256
options SOMAXCONN=256
options NMBCLUSTERS=4096
options CHILD_MAX=512
options OPEN_MAX=512


Additionally, you can try to use maximum optimization when building the kernel itself by using the GCC compiler flags
-mpentium, -O2, -fexpensive-optimizations, and
-fomit-frame-pointer, or even try to compile the kernel with the latest EGCS-based Pentium-GCC variant. But please be careful: Always keep a working kernel at hand when doing such optimization tests.

After tuning your operating system you can try to enhance the performance of Apache. In addition to setting the above kernel parameters you should first increase the corresponding Apache parameters when building, as depicted below in Example (a), and then tune the Apache configuration file as depicted in Example (b).


(a)
-DHARD_SERVER_LIMIT=256
-DDYNAMIC_MODULE_LIMIT=0
-DBUFFERED_LOGS

(b)
MinSpareServers 256
StartServers 256
MaxSpareServers 256
MaxClients 256

MaxRequestsPerChild 1000
KeepAlive on
KeepAliveTimeout 15
MaxKeepAliveRequests 64
Timeout 400
IdentityCheck off
HostnameLookups off

HostnameLookups on


Options FollowSymLinks
AllowOverride None



we see with the commands ps or top that each server process requires between 700 KB and 900 KB of RAM. So we have SPS = 0.9 MB. Because we have approximately 1000 requests per minute we use RPM = 1000. With HTTP benchmarks or just by appreciating, the processing time is between 0.5 and 4 seconds per request, so we use SPR = 2 seconds. Through the above formulas, we see that we need
NOS = ceil(1000 * 2 * (1/60) * (100/80)) = 42 servers to start, and that we have to make sure that our machine has at least
MBR = ceil((0.9 * 42 * (100/70)) / 16) * 16 = 64 MB of total RAM installed.

In Summary

We've looked at two methods for distributing the load across a number of low-cost servers. The DNS method is fairly standard, but does have its problems, including caching of DNS data and the potential that an individual machine in the cluster may fail. The reverse-proxy method improves reliability and control by allowing the administrator to designate subclusters for specific tasks, but we could go even further. For instance, we could write a script on the reverse proxy that periodically polls the back-end machines through rsh or ssh, and then adjusts the apache-rproxy.conf-servers table accordingly, restarting the reverse proxy through a kill-USR1 to reread the configuration file and replace the server processes with new ones. And finally, wouldn't it be nice if we could do some sort of server balancing based not on load, but on the actual bandwidth? This may soon be possible, as I am currently working on Apache::SiteSwitch, a mod_perl-based module that will allow for a distributed set of servers (each one has its own particular Internet link) to
negotiate for the optimum world-wide path between client and server. This will be especially interesting for Web sites with international mirrors.

Consulting Contact

If you require any similar consulting for your own enterprise, feel free to contact us