When running web services or applications within / behind web servers by utilizing HTTP or HTTPs you might get into trouble if network traffic is reaching the maximum of the available bandwidth.

This article will give you an idea what to do and how to optimize the utilization of network resources. Example configurations will be provided for Apache 2 web servers.

Introduction

As mentioned above, high-traffic web-sites or web services providing / consuming large amounts of data may run into trouble if they reach the maximum bandwidth of attached network connections. This may result in an unpredictable behaviour of your application from the view of external systems.

The following list contains some examples for unpredicatable behaviours. Remote systems might…

• get an unpredictive responsiveness regarding the processing time of requests.
• receive error messages regarding unanswered requests (timeout, bad gatway etc.).
• receive responses from your web service that might be fragmented / incomplete.
• retrieve error messages sporadically.
• stop working if they are relying on page refreshes in web browsers!

The Approach

De-/Compression of Data Streams

Our approach requires you to enable the dynamic compression and decompression of data streams within your web server. This will lead to an additional functionality of your webserver from the view of interfacing remote applications.

If remote applications do support the new way to talk with your server, the data stream to / from your web service might be compressed up to 95 percent! Real compression rates are dependant on data stream contents and the compression level configured. For the case, that remote applications do not support compressed data streams nothing will change for them and they will be able to continue working as they did before.

Backend applications and applications running in your web server are not affected by the new configuration, because the web server is performing the de-/compression transparently.

Drawback

The presented approach might lead to a higher number of processed requests. But the new communication feature of you web server as a drawback, that should be mentioned. Compressing and decomressing data streams on-the-fly will result in a higher workload of your server. This might be a problem, if it is already running with a high CPU utilization.

However, there are a number of options to optimize data transfer and the CPU utilization. Here are some examples:

• restrict comression of data streams to a dedicated number of contents types or a subset of provided applications
• static contents (css, JavaScript, images, …) might be stored precompressed
• play around with cache settings and modification timestamps – this might prevent the redundant delivery of data
• if there is a database running at your server then try to move it to a separate machine
• try to find a more redundant design for your application, so that requests might be answered by one out of a number of web services (build a cluster, cloud, …)

Example Configuration

To illustrate what to do at the server side to enable on-the-fly data de-/compression we will configure an Apache 2 server. Other web servers and application servers with HTTP interface do provide similar features.

Apache 2 relies on the module mod_deflate the perform de-/compression of data streams. The following configuration examples provide an overview how to customize your Apache server. However, modifications will be required to make teh examples work in your specific environment.

Please utilize the package manager of your present distribution to install all required software packages. If you have to compile and/or install all software packages by hand, you have to download the source codes of the desired Apache server (mod_deflate is included). mod_deflate requires the system library libz to be present in your server system.

Global Configurations

There are a number of available configuration switches to customize the behaviour of the module mod_deflate. The following configuration snippet is intended to be placed in the global part of your Apache configuration file (this means outside of VirtualHost nodes).

123456789101112131415161718

### compression of HTTP  data streamsLoadFile /usr/lib64/libz.so.1LoadModule deflate_module modules/mod_deflate.so<IfModule mod_deflate.c>  # set the level of compression  DeflateCompressionLevel 9   # Compress just a few content types  AddOutputFilterByType DEFLATE text/html text/plain text/xml   # enable extended logging  DeflateFilterNote Input input_info  DeflateFilterNote Output output_info  DeflateFilterNote Ratio ratio_info  LogFormat '"%r" %{output_info}n/%{input_info}n (%{ratio_info}n%%)' deflate  CustomLog /app/rishttp/ext/apache-2.0.63-inst1/logs/deflate_log deflateIfModule> 

Application Specific Configuration

The following configuration has to be added to the desired VirtualHost nodes or, if you do not utilize VirtualHost nodes, you have to integrate one or both of the following filter statements into the main configuration file of your Apache server.

12345678

<Location />  ### Insert Filters  # enable decompression of input streams  SetInputFilter DEFLATE  # enable compression of output streams  SetOutputFilter DEFLATELocation> 

Now you have to instruct your Apache server to reload the server configuration or just restart the web server. If the module ist installed properly and the added configuration statement are correct, the server is now running with the new and desired communication feature.

Verify Server Settings

Now, the server is running with the customized configuration and we have to test the server responses whether they do contain a comressed data stream. Therefore we have to request the server to return a compressed HTTP answer. The following screenshot contains an extract of a HTTP session.

Line 1: The command line HTTP client curl is requesting data by accepting compressed data stream.
Line 9: curl informs the server what kind of compression will be accepted.
Line 11: Starting with this line the server is sending its response.
Line 19: The server informs the client to be prepared to receive a compressed data stream.
Line 22: Starting at line 22 curl prints out the decompressed data stream received from the server.

There are a number of additional things to say and that are worth to be optimized. But at this point the configured feature has been verified and we are finished.

We want to conclude this article with an example of a HTTP session where a non-compressed response is retrieved even if the client was requesting a compressed data stream. This will help you to identify a non-working configuration option:

# request a compressed data stream - but the answer in not compressed$ curl --compressed -v http://10.182.164.82/rishttp/* About to connect() to 10.182.164.82 port 80*   Trying 10.182.164.82... connected* Connected to 10.182.164.82 (10.182.164.82) port 80> GET /rishttp/ HTTP/1.1> User-Agent: curl/7.15.1 (x86_64-suse-linux) libcurl/7.15.1 [...]> Host: 10.182.164.82> Accept: */*> Accept-Encoding: deflate, gzip>< HTTP/1.1 200 OK< Date: Thu, 03 Feb 2011 13:40:54 GMT< Server: Apache< ETag: W/"331-1296036761000"< Last-Modified: Wed, 26 Jan 2011 10:12:41 GMT< Content-Length: 331< Content-Type: text/html; charset=ISO-8859-1<html><head> <title>RIS Infoplattformtitle>[...]

Incoming search terms:

• http compress data stream
• linux compression filter data stream