mod_qos

In computer networking, the term quality of services (QoS) describes resource management rather than the quality of a service. Quality of services implements control mechanism to provide different priority to different users, applications, and data connections. It is used to guarantee a certain level of performance to data resources. The term quality of service is often used in the field of wide area network protocols (e.g. ATM) and telephony (e.g. VoIP) but rarely in conjunction with web applications. mod_qos is a quality of service module for the Apache web server implementing control mechanisms that can provide different priority to different HTTP requests.

But why do you need quality of service for a web application? Well, web servers require threads and processes to serve HTTP requests. Each TCP connection to the web server occupies one of these threads respectively processes. Sometimes a server gets too busy to serve every request due the lack of free processes or threads. Another parameter requiring control by mod_qos is the available bandwidth: all clients communicate to the server over a network link with a limited bandwidth. Overfilling the link, results in network congestion and poor performance.

Example situations where web applications require QoS:

• More resources are consumed if request processing takes a long time by an application, e.g. when request processing includes time consuming database queries.
• Oversubscription of link capabilities due too many concurrent clients uploading or downloading data.
• Penetration of the web server by attackers (DDoS).

mod_qos may be used to determine which requests should be served and which shouldn't in order to avoid resource oversubscription. The module collects different attributes such as the request URL, HTTP request and response headers, the IP source address, the HTTP response code, history data (user session and on per source IP address basis), the number of concurrent requests to the server (total or requests having similar attributes), the number of concurrent TCP connections (total or from a single source IP), and so forth.

Counteractive measures to enforce the defined rules are: request blocking, dynamic timeout adjustment, request delay, response throttling, and dropping of TCP connections.

The current release of the mod_qos module implements control mechanisms to manage:

• The maximum number of concurrent requests to a location/resouce (URL) or virtual host.
• Limitation of the bandwidth such as the maximum allowed number of requests per second to an URL or the maximum/minimum of downloaded kbytes per second.
• Limits the number of request events per second (special request conditions).
• It can also "detect" very important persons (VIP) which may access the web server without or with fewer restrictions.
• Generic request line and header filter to deny unauthorized operations.
• Request body data limitation and filtering (requires mod_parp).
• Limitations on the TCP connection level, e.g. the maximum of allowed connections from a single IP source address or dynamic keep-alive control.
• Prefers known IP addresses when server runs out of free TCP connections.

mod_qos is available at SourceForge.net under the GNU General Public License.

More information about mod_qos:

• Build
• Source
• Changes
• Configuration
• Request Level Control
• Privileged Users
• Variables
• Conditional Rules
• Events
• Request Level, Generic Filter
• Connection Level Control
• Client Level Control
• Log Messages
• Error Log
• Access Log
• Request Statistics
• Status Viewer
• Utilities
• Use Cases

Build

mod_qos requires OpenSSL, PCRE (don't use the version which comes with the Apache distribution), threading and shared memory support. mod_qos works with Apache version 2.2 (tested in mpm worker and prefork mode).
Just copy the module into the modules directory of the Apache server's source code and compile it using the following commands (all examples are using Apache 2.2.8 and mod_qos 9.25):

tar xfz httpd-2.2.8.tar.gz tar xfz mod_qos-9.25-src.tar.gz ln -s httpd-2.2.8 httpd cd httpd mkdir modules/qos cp ../mod_qos-9.25/apache2/* modules/qos ./buildconf ./configure --enable-so --enable-qos=shared --enable-ssl make cd ..

This creates a DSO module, which can be loaded into the Apache server using the following directive:

LoadModule qos_module <path to module>/mod_qos.so

You can also compile the module using apxs alternatively. Your httpd must support dynamically loaded objects (DSO). Verify this by checking the availability of mod_so: The command httpd -l must list the mod_so.c module. The following command compiles the module and installs the mod_qos into the server's modules directory.

cd mod_qos-9.25/apache2 apxs -i -c mod_qos.c cd ../..

The support tools may be build (at least on some Linux platforms) using the GNU autotools. Some of these utilities requires third-party libraries such as apr, apr-util, pcre, libpng, and OpenSSL.

cd mod_qos-9.25/tools ./configure make

Source

mod_qos is available for Apache version 2.2.

Configuration

Configuration is made on a per server basis (except the generic request filter). Commands within a virtual host are merged with the settings in the global configuration.

The QS_SrvMinDataRate, QS_SrvRequestRate, QS_RequestHeaderFilterRule, and all QS_Client* directives may be used outside of virtual host configurations only.

Request Level Control

• QS_LocRequestLimitMatch <regex> <number>
  Defines the number of concurrent requests for the specified request pattern (applied to the unparsed URL). The rule with the lowest number of allowed concurrent connection has the highest priority if multiple expressions match the request. By default, no limitations are active.
• QS_LocRequestPerSecLimitMatch <regex> <number>
  Defines the allowed number of requests per second to the URL (path and query) pattern. Requests are limited by adding a delay to each request (linear). By default, no limitation is active. This directive should be used in conjunction with QS_LocRequestLimitMatch only (you must use the very same regex pattern with the QS_LocRequestPerSecLimitMatch and QS_LocRequestLimitMatch directive).
• QS_LocKBytesPerSecLimitMatch <regex> <number>
  Defines the allowed download bandwidth to the location matching the defined URL (path and query) pattern. Responses are slowed down by adding a delay to each response (non-linear, bigger files get longer delay than smaller ones). By default, no limitation is active. This directive should be used in conjunction with QS_LocRequestLimitMatch only (you must use the very same regex pattern with the QS_LocKBytesPerSecLimitMatch and QS_LocRequestLimitMatch directive).
• QS_LocRequestLimit <location> <number>
  Defines the number of concurrent requests for the specified location (applied to the parsed path). By default, no limitations are active for locations. Has lower priority than QS_LocRequestLimitMatch directives.
• QS_LocRequestLimitDefault <number>
  Defines the default limitation for the maximum of concurrent per locations for those locations not defined by any QS_LocRequestLimit directive. It could also be used to limit the number of concurrent requests to a virtual host.
• QS_LocRequestPerSecLimit <location> <number>
  Defines the allowed number of requests per second to a location. The maximum number of requests is limited by adding a delay to each request (linear, each request gets the same delay). By default, no limitation is active. This directive should be used in conjunction with QS_LocRequestLimit only (you must use the same location for both directives). Has lower priority than QS_LocRequestPerSecLimitMatch.
• QS_LocKBytesPerSecLimit <location> <number>
  Throttles the download bandwidth to the defined kbytes per second. Responses are slowed by adding a delay to each response (non-linear, bigger files get longer delay than smaller ones). By default, no limitation is active. This directive should be used in conjunction with QS_LocRequestLimit only (you must use the same location for both directives). Has lower priority than QS_LocKBytesPerSecLimitMatch.
• QS_ErrorPage <URL>
  Defines an error page to be returned when a request is denied. The defined URL must be a (S)HTML document accessible by the client. You may enable server-side includes in order to present detailed error messages based on the error codes provided by mod_qos.
• QS_ErrorResponseCode <code>
  Defines the HTTP response code which is used when a request is denied. Requests denied at connection level usually get a HTTP 500er response code (ignoring the settings of the QS_ErrorResponseCode and QS_ErrorPage directives).
  Default codes are:
   400: if a request has no valid URL.
   403: for requests denied by a QS_Deny*, QS_Permit*, or QS_RequestHeaderFilter directive.
   413: when limiting the max. body data length by the QS_LimitRequestBody directive.
   500: for requests denied by any other directive.
  

Privileged Users

• Session: VIP identification is stored using a HTTP session cookie. mod_qos starts a new session when detecting a HTTP response header (the header name is defined by the QS_VipHeaderName directive). Alternatively, a new session is started when detecting an authenticated user, see QS_VipUser. The QS_Session* directives are used to set session attributes.
• Request: The QS_VipRequest process environment may be evaluated by mod_qos rules. This variable is set automatically when receiving a valid mod_qos session cookie. The QS_VipRequest variable may also be set by configuration using a QS_SetEnvIf* or SetEnvIf directive. VIP status last for the particular request only.
• Client IP address: VIP identification may be stored at the server side on a pre client IP address basis. The QS_VipIPHeaderName, QS_VipHeaderName, QS_VipIPUser, and QS_VipUser directives are used to define when a per IP address should be marked as a VIP user.

• QS_VipHeaderName <header name>[=<regex>] [drop]
  Defines a HTTP response header which marks a user as a VIP. mod_qos creates a session for this user by setting a cookie, e.g. after successful user authentication. Tests Optionally its value against the provided regular expression. Specify the action 'drop' if you want mod_qos to remove this control header from the HTTP response.
• QS_VipIPHeaderName <header name>[=<regex>] [drop]
  Defines a HTTP response header which marks a client source IP address as a VIP. Tests Optionally its value against the provided regular expression. Specify the action 'drop' if you want mod_qos to remove this control header from the HTTP response.
• QS_VipUser
  Creates a VIP session for users which has been authenticated by the Apache server, e.g. by the standard mod_auth* modules. It works similar to the QS_VipHeaderName directive.
• QS_VipIPUser
  Marks a source IP address as a VIP if the user has been authenticated by the Apache server, e.g. by the standard mod_auth* modules. It works similar to the QS_VipIPHeaderName directive.
• QS_SessionTimeout <seconds>
  Defines the session life time for a VIP. It is only used for session based (cookie) VIP identification (not for IP based). Default is 3600 seconds.
• QS_SessionCookieName <name>
  A cookie is used to identify requests coming from a user which has been identified as a VIP. This directive defines a custom cookie name for the mod_qos session cookie. Default is MODQOS.
• QS_SessionCookiePath <path>
  Defines a the cookie path. Default is "/".
• QS_SessionKey <string>
  Secret key used for cookie encryption. Used when using the same session cookie for multiple web servers (load balancing) or sessions should survive a server restart. By default, a random key is used which changes every server restart.

QS_ErrorPage                  /error-docs/qs_error.html

# restricts max concurrent requests for any location which has no
# individual rule:
QS_LocRequestLimitDefault                              200

# limits access to *.gif files to 100 concurrent requests:
QS_LocRequestLimitMatch       "^.*\.gif$"              100

# limits concurrent requests to the locations /images and /app/a:
QS_LocRequestLimit            /images                  100
QS_LocRequestLimit            /app/a                   300
# limits download bandwidth to 5Mbit/sec:
QS_LocKBytesPerSecLimit       /app/a                   640

# two locations (/app/b and /app/c) representing a single application:
QS_LocRequestLimitMatch       "^(/app/b/|/app/c/).*$"  300


# allows the application to nominate VIP users by sending a
# "mod-qos-vip" HTTP response header:
QS_VipHeaderName              mod-qos-vip
QS_SessionKey                 na&5san-sB.F4_0a=%D200ahLK1

Variables

• QS_VipRequest=yes
  Disables the pre location restrictions for this request. Requires the definition of a VIP header using the QS_VipHeaderName directive (this activates VIP verification). However, such an event does not create a VIP session. The user has the VIP status only for a single request.
  The variable is set by mod_qos when receiving a valid VIP session cookie.
• QS_KeepAliveTimeout=<seconds>
  Applies dynamic connection keep-alive settings overriding the Apache KeepAliveTimeout directive settings.
• QS_ErrorPage=<URL>
  Defines the error page overriding the setting made by the QS_ErrorPagex directive.
• QS_Delay=<milliseconds>
  Defines a number of milliseconds to deleay the request processing.
• QS_Event
  The variale processed by the QS_ClientEventPerSecLimit directive.
• QS_Block
  Variable processed by the QS_ClientEventBlockCount directve.
• QS_Cond
  Variable prcoessed by the QS_CondLocRequestLimitMatch directive.
• QS_EventRequest
  Variable processed by the QS_ClientEventRequestLimit directive.

• QS_SrvConn
  Number of concurrent connections. Value is set when using the QS_SrvMaxConn directive.

# privileged access for curl clients:
BrowserMatch             "curl"                   QS_VipRequest=yes

# allows privileged access to a single resource:
SetEnvIf     Request_URI /app/start.html          QS_VipRequest=yes

# allows privileged access from a specified source address
# or source address range:
SetEnvIf     Remote_Addr 172.18.3.32              QS_VipRequest=yes
SetEnvIf     Remote_Addr 192.168.10.              QS_VipRequest=yes

# set keep-alive timeout for MSIE version 5.x browser to 65 seconds:
BrowserMatch             "(MSIE 5\.)"             QS_KeepAliveTimeout=65

# dynamic error page URL (per host error page):
SetEnvIf     Host        (.*)                     QS_ErrorPage=/error-docs/$1.html

Conditional Rules

• QS_CondLocRequestLimitMatch <regex> <number> <condition>
  Rule works similar to QS_LocRequestLimitMatch but it is only enforced for requests whose QS_Cond variable matches the specified condition (regular expression). Every request matching the defined pattern is counted but the defined limitation is only enforced for those requests matching the specified condition.
  Only one QS_CondLocRequestLimitMatch rule is evaluated per request

# set the conditional variable to spider if detecting a
# "slurp" or "googlebot" search engine:
BrowserMatch             "slurp"                  QS_Cond=spider
BrowserMatch             "googlebot"              QS_Cond=spider

# limits the number of concurrent requests to two applications
# (/app/b and /app/c) to 300 but does not allow access by a "spider"
# if the number of concurrent requests exceeds the limit of 10:
QS_LocRequestLimitMatch       "^(/app/b/|/app/c/).*$"  300
QS_CondLocRequestLimitMatch   "^(/app/b/|/app/c/).*$"  10   spider

Events

• QS_EventRequestLimit <env-variable>[=<regex>] <number>
  Defines the number of concurrent events. Directive works similar to QS_LocRequestLimit but counting the requests having the same environment variable (and optionally matching its value too) set rather than having the same URL pattern.
• QS_EventPerSecLimit [!]<env-variable> <number>
  Defines how often requests may have the defined environment variable (literal string) set. It measures the occurrences of the defined environment variable on a request per seconds level and tries to limit this occurrence to the defined number. It works similar as QS_LocRequestPerSecLimit does but counts only the requests having the specified variable set (or not set if the variable name is prefixed by a "!"). If a request matches multiple events, the rule with the lowest bandwidth is applied. Events are limited by adding a delay to each request causing an event.
• QS_EventKBytesPerSecLimit [!]<env-variable> <number>
  Throttles the download bandwidth of all requests having the defined variable set to the defined kbytes per second. Responses are slowed by adding a delay to each response (non-linear, bigger files get longer delay than smaller ones). By default, no limitation is active. This directive should be used in conjunction with QS_EventRequestLimit only (you must use the same variable name for both directives).
• QS_SetEnvIf [!]<env-variable1> [!]<env-variable2> <variable=value>
  Sets the "variable=value" (literal string) if variable1 (literal string) AND variable2 (literal string) are set in the request environment variable list (not case sensitive). This is used to combine multiple variables to a new event type.
• QS_SetEnv <env-variable> <value>
  Sets the defined variable with the value where the value string may contain other environment variables surrounded by "${" and "}". The variable is only set if all defined variables within the value have been resolved.
• QS_SetEnvIfQuery <regex> [!]<env-variable>[=<value>]
  Directive works quite similar to the SetEnvIf directive of the Apache module mod_setenvif but the specified regex is applied against the query string portion of the request line. The directive recognizes the occurrences of $1..$9 within value and replaces them by the subexpressions of the defined regex pattern.
• QS_SetEnvIfParp <regex> [!]<env-variable>[=<value>]
  Directive parsing the request payload using the Apache module mod_parp. It matches the request URL query and the HTTP request message body data as well (application/x-www-form-urlencoded, multipart/form-data, and multipart/mixed) and sets the defined process variable (quite similar to the QS_SetEnvIfQuery directive). The directive recognizes the occurrences of $1..$9 within value and replaces them by the subexpressions of the defined regex pattern. This directive activates mod_parp for every request to the virtual host. You may deactivate mod_parp for selected requests using the SetEnvIf directive: unset the variable "parp" to do so. Important: request message body processing requires that the server loads the whole request into its memory (at least twice the length of the message). You should limit the allowed size of HTTP requests message body using the QS_LimitRequestBody directive when using QS_SetEnvIfParp!
• QS_SetEnvIfBody <regex> [!]<env-variable>[=<value>]
  Directive parsing the request body using the Apache module mod_parp. Specify the content types to process using the mod_parp directive PARP_BodyData and ensure that mod_parp is enabled using the SetEnvIf directive of the Apache module mod_setenvif. You should limit the allowed size of HTTP requests message body using the QS_LimitRequestBody directive when using mod_parp. The directive recognizes the occurrence of $1 within the variable value and replaces it by the subexpressions of the defined regex pattern.
• QS_SetEnvIfStatus <code> <env-variable>
  Sets the defined variable in the request environment if the HTTP response status code matches the defined value. This may be used in conjunction with the QS_ClientEventBlockCount directive.
• QS_SetEnvIfResBody <string> <env-variable>
  Adds the definedenvironment variable (e.g. QS_Block) if the response body contains the defined literal string. Used on a per Location level. Only one directive may be defined per location (one search string per response).
• QS_SetEnvResHeader <header name> [drop]
  Sets the defined HTTP response header to the request environment variables. Deletes the specified header if the action 'drop' has been specified.
• QS_SetEnvResHeaderMatch <header name> <regex>
  Sets the defined HTTP response header to the request environment variables if the specified regular expression (pcre not case sensitive) matches the header value.
• QS_SetReqHeader <header name> <env-variable>
  Sets the defined HTTP request header to the request if the specified environment variable is set.

# marks clients coming from the internal network:
SetEnvIf    Remote_Addr      ^192\.168\.            QS_Intra

# marks clients neither coming from the internal network
# nor are VIP clients as low priority clients:
QS_SetEnvIf !QS_VipRequest   !QS_Intra              QS_LowPrio

# limits the request rate for low priority (neither VIP nor internal)
# clients (and no more than 400 concurrent requests for them):
QS_EventPerSecLimit          QS_LowPrio             100
QS_EventRequestLimit         QS_LowPrio             400

# detects the variable "file" within the query portion of the URL:
QS_SetEnvIfQuery             file=([a-zA-Z]*)       QS_LowPrio=$1

# combine variables and propagate them to the application via HTTP header:
SetEnvIf    Content-Length   ([0-9]*)               QS_Length=$1
QS_SetEnv   QS_Type          "length=${QS_Length}; file=${QS_LowPrio}"
QS_SetReqHeader              X-File                 QS_Type

# limit the max. body size since mod_parp loads the whole message into
# the memory servers's:
QS_LimitRequestBody          131072

# body pattern detection, examples limits the maximum of concurrent
# the requests posting "id=1234" to ten:
QS_SetEnvIfParp  id=([0-9]*) PARP_PATTERN=$1
QS_EventRequestLimit         PARP_PATTERN=1234      10
# but ignore requests to the location /main/ (any any sub-locations):
SetEnvIf    Request_URI      /main/.*               !parp

Request Level, Generic Filter

• QS_DenyRequestLine '+'|'-'<id> 'log'|'deny' <pcre>
  Generic request line (method, path, query, and protocol) filter used to deny access for requests matching the defined expression (pcre). The action taken for matching rules is either 'log' (access is granted but the rule match is logged) or 'deny' (access is denied).
• QS_DenyPath '+'|'-'<id> 'log'|'deny' <pcre>
  Generic abs_path (see RFC 2616 section 3.2.2) filter used to deny access for requests matching the defined expression (pcre). The action taken for matching rules is either 'log' (access is granted but the rule match is logged) or 'deny' (access is denied).
• QS_DenyQuery '+'|'-'<id> 'log'|'deny' <pcre>
  Generic query (see RFC 2616 section 3.2.2) filter used to deny access for requests matching the defined expression (pcre). The action taken for matching rules is either 'log' (access is granted but the rule match is logged) or 'deny' (access is denied).
• QS_InvalidUrlEncoding 'log'|'deny'|'off'
  Enforces correct URL decoding in conjunction with the QS_DenyRequestLine, QS_DenyPath, and QS_DenyQuery directives. Default is "off" which means that incorrect encodings are ignored.
• QS_Decoding 'uni'
  Enableds additional string decoding functions which are applied before matching QS_Deny* and QS_Permit* directives. Default is URL decoding (%xx, \\xHH, '+').
  Available additional decodings:
  • uni: unicode decoding for MS IIS (%uXXXX and \uXXXX) encoded characters.
• QS_DenyEvent '+'|'-'<id> 'log'|'deny' [!]<env-variable>
  Rule matching requests having the defined process environment variable set (or NOT set if prefixed by a '!'). The action taken for matching rules is either 'log' (access is granted but the rule match is logged) or 'deny' (access is denied).
• QS_PermitUri '+'|'-'<id> 'log'|'deny' <pcre>
  Generic URL (path and query) filter implementing a request pattern whitelist. Only requests matching at least one QS_PermitUri pattern are allowed. If a QS_PermitUri pattern has been defined and the request does not match any rule, the request is denied albeit of any server resource availability. All rules must define the same action. pcre is case sensitive. You may use the qsfilter2 utility to generate rules based on access log files.
• QS_DenyInheritanceOff
  Disables inheritance of QS_Deny* and QS_Permit* directives (pattern definitions) to a location.
• QS_RequestHeaderFilter 'on'|'off'|'size'
  Filters request headers using validation rules provided by mod_qos. Suspicious headers (not matching the pattern or those which are too long) are normally dropped (removed from the request). Abnormal content-* headers cause request blocking. Only the defined headers are allowed. Custom rules (additional headers or different pattern/size definitions) may be added using the QS_RequestHeaderFilterRule directive. Filter is activated ('on') or deactivated ('off') on a per Location basis. The mode 'size' does not verify the pattern but limits the maximum length of request header values (similar to the Apache directive LimitRequestFieldsize but with an individual rule for each header field).
• QS_RequestHeaderFilterRule <header name> 'drop'|'deny' <pcre> <size>
  Used to add custom request header filter rules, e.g. to override the internal rules (different pcre or size) or to add additional headers which should be allowed. Definitions are made globally (outside VirtualHost). A list of all rules is shown at server startup when using LogLevel debug. pcre is case sensitive. The size parameter defines the maximum length of a header value. The action 'drop' removes a header not matching the pcre, the action 'deny' rejects a request including such a header not matching the pcre.
• QS_ResponseHeaderFilter 'on'|'off'
  Filters response headers using validation rules provided by mod_qos. Suspicious headers (not matching the pattern or those which are too long) are removed from the response. Only the defined headers are allowed. Filter is activated ('on') or deactivated ('off') on a per Location basis.
• QS_ResponseHeaderFilterRule <header name> <pcre> <size>
  Used to add custom response header filter rules, e.g. to override the internal rules (different pcre or size) or to add additional headers which should be allowed. Definitions are made globally (outside VirtualHost). A list of all rules is shown at server startup when using LogLevel debug. pcre is case sensitive. The size parameter defines the maximum length of a header value.

QS_ErrorPage                     /error-docs/qs_error.html
QS_RequestHeaderFilterRule       UA-CPU drop "^[a-zA-Z0-9]+$" 20

<Location />
   # enable header validation:
   QS_RequestHeaderFilter        on

   # don't allow access to the path /app/admin.jsp:
   QS_DenyPath        +admin     deny "^/app/admin.jsp$"

   # allow printable characters only within the request line:
   QS_DenyRequestLine +printable deny ".*[\x00-\x19].*"
</Location>

• QS_DenyQueryBody 'on|'off'
  Enables request body data filtering for the QS_DenyQuery directive.
• QS_PermitUriBody 'on|'off'
  Enables request body data filtering for the QS_PermitUri directive.
• QS_LimitRequestBody <bytes>
  Limits the allowed size of an HTTP request message body. This directive may be placed anywhere in the configuration. Alternatively, the limitation may be set as an environment variable using mod_setenvif (overriding the directive settings).

# configure the audit log writing the request body data to a file
# (use this log to generate a whitelist rules using qsfilter2
# when QS_PermitUriBody has been enabled)
CustomLog             logs/qsaudit_log  "%{qos-path}n%{qos-query}n"

# limit the max. body size since mod_parp loads the whole message into the
# memory servers's:
SetEnvIfNoCase Content-Type application/x-www-form-urlencoded QS_LimitRequestBody=131072
SetEnvIfNoCase Content-Type multipart/form-data               QS_LimitRequestBody=131072
SetEnvIfNoCase Content-Type multipart/mixed                   QS_LimitRequestBody=131072

# enable mod_deflate input filter for compressed request body data:
SetEnvIfNoCase Content-Encoding gzip|compress|deflate QS_DeflateReqBody

<Location /app>
   # don't allow certain string pattern within the request query or
   # the request message body data:
   QS_DenyQueryBody              on
   QS_DenyQuery       +s01       deny "(EXEC|SELECT|INSERT|UPDATE|DELETE)"
</Location>

Milestones: you may define a number of resources (request line patterns) as milestones. A client must access these resources in the correct order as they are defined within the server configuration. A client is not allowed to skip these milestones (but may access any other resource not covered by a milestone in between requests to milestones).

• QS_MileStone 'log'|'deny' <pattern>
  Defines request line patterns a client must access in the defined order as they are defined in the configuration file. Milestones are defined on a per server basis, outside Location. Access to milestones is tracked by a dedicated session cookie.
• QS_MileStoneTimeout <seconds>
  Defines the time in seconds within a client must reach the next milestone. Default are 3600 seconds.

# four milestones:
# 1) client must start with /app/index.html
# 2) and then read some images
# 3) before posting data to /app/register
# 4) afterwards, the user may download zip files
QS_MileStone          deny       "^GET /app/index.html"
QS_MileStone          deny       "^GET /app/images/.*"
QS_MileStone          deny       "^POST /app/register*"
QS_MileStone          deny       "^GET /app/.*\.zip HTTP/..."

Connection Level Control

• QS_SrvMaxConn <number>
  Defines the maximum number of concurrent TCP connections for this server (virtual host).
• QS_SrvMaxConnClose <number>[%]
  Defines the maximum number of connections supporting keep-alive. If the number of concurrent connections exceeds this threshold, the TCP connection gets closed after each request. You may specify the number of connections as a percentage of MaxClients if adding the suffix '%' to the specified value.
• QS_SrvMaxConnPerIP <number>
  Defines the maximum number of connections per source IP address.
• QS_SrvMaxConnExcludeIP <address>
  Defines an IP address or address range to be excluded from connection level control restrictions. An address range must end with a ".".
• QS_SrvMinDataRate <bytes per second> [<max bytes per second>]
  Defines the minimum upload/download throughput a client must generate (the bytes send/received by the client per seconds). This bandwidth is measured while receiving request data (request line, header fields, or body). The client connection get closed if the client does not fulfill this required minimal data rate and the IP address of the causing client get marked in order to be handled with low priority (see the QS_ClientPrefer directive). The "max bytes per second" activates dynamic minimum throughput control: The required minimal throughput is increased in parallel to the number of concurrent clients sending/receiving data. The "max bytes per second" setting is reached when the number of sending/receiving clients is equal to the MaxClients setting. No limitation is set by default.
• QS_SrvRequestRate <bytes per second> [<max bytes per second>]
  Same as QS_SrvMinDataRate but enforcing a minimal upload throughput only.
• QS_SrvDataRateOff
  Disables the QS_SrvMinDataRate and QS_SrvMinDataRate enforcement for a virtual host.
• QS_SrvMinDataRateOffEvent '+'|'-'<env-variable>
  Disables the QS_SrvMinDataRate and QS_SrvMinDataRate enforcement for a connection when the defined process environment variable is set. The '+' prefix is used to add a variable to the configuration while the '-' prefix is used to remove a variable. Directive may be used on a per Location basis.

# minimum request rate (bytes/sec at request reading):
QS_SrvRequestRate                                 120

# limits the connections for this virtual host:
QS_SrvMaxConn                                     800

# allows keep-alive support till the server reaches 600 connections:
QS_SrvMaxConnClose                                600

# allows max 50 connections from a single ip address:
QS_SrvMaxConnPerIP                                 50

# disables connection restrictions for certain clients:
QS_SrvMaxConnExcludeIP                    172.18.3.32
QS_SrvMaxConnExcludeIP                    192.168.10.

Client Level Control

• QS_ClientEntries <number>
  Defines the number of individual clients managed by mod_qos. Default is 50'000 concurrent IP addresses. Each client requires about 68 bytes of shared memory on a 32bit system or 104 bytes on a 64bit system respectively. Client IP source address store survives graceful server restart.
• QS_ClientEventRequestLimit <number>
  Defines the allowed number of concurrent requests comming from the same client source IP address having the QS_EventRequest variable set.
• QS_ClientEventPerSecLimit <number>
  Defines how often a client may cause a QS_Event per second. Such events are requests having the QS_Event variable set, e.g. defined by mod_setenvif or using the QS_SetEnvIf directive. The rule is enforced by adding a delay to requests causing the event (similar to the QS_LocRequestPerSecLimit directive.
• QS_ClientEventBlockCount <number> [<seconds>]
  Defines the maximum number of QS_Block events allowed within the defined time (default is 600 seconds). Client IP is blocked when reaching this counter for the specified time (blocked at connection level).
• QS_ClientPrefer [<percent>]
  Accepts only VIP and high priority clients when the server has less than 80% (or the defined percentage) of free TCP connections. Use the QS_VipHeaderName or QS_VipIPHeaderName directive in order to identify VIP clients. The distinction between high and low priority clients is made based on the client data transfer behavior (clients sending slow, using small data packets, or accessing "unusual" content types (see QS_ClientTolerance), get marked as low priority clients, look for "r;" events within the access log or use the status viewer to determine which client addresses are identified as log priority clients). A low priority flag is cleared after 24h hours. Clients identified by QS_SrvMaxConnExcludeIP are excluded from connection restrictions. Filter is applied on connection level.
• QS_ClientTolerance <percent>
  Defines the allowed variation from a "normal" client (average). Default is 500.

# don't allow a client IP to access /app/start.html 20 or
# more times within 10 minutes:
SetEnvIf     Request_URI /app/start.html          QS_Block=yes
QS_ClientEventBlockCount                          20

# don't allow more than 20 "403" status code responses
# (forbidden) for a client within 10 minutes:
QS_SetEnvIfStatus        403                      QS_Block

Log Messages

Error Log

mod_qos writes messages to Apache's error log when enforcing a rule. Each error messages is prefixed by an id: mod_qos(<number>). These error codes (number only) are also written to the error notes in order to be processed within error pages using server-side includes (SSI).

mod_qos(00x): initialisation event mod_qos(01x): request level control event mod_qos(02x): vip session event mod_qos(03x): connection level event mod_qos(04x): generic filter event mod_qos(05x): bandwidth limitation event mod_qos(06x): client control event

Access Log

mod_qos adds event variables to the request record which may be added to access log messages.

• mod_qos_ev
  Status event message of mod_qos. It's a single letter which is used to signalize an event: "D"=denied, "S"=pass due an available VIP session, "V"=create VIP session, "K"=connection closed (no keep-alive), "T"=dynamic keep-alive, "r"=IP has marked as a slow/bad client, and "L"=means a request slow down.
• mod_qos_cr
  The number of concurrent requests to a location matching the QS_LocRequestLimit, QS_LocRequestLimitMatch, QS_LocRequestPerSecLimit, QS_LocRequestPerSecLimitMatch, QS_LocKBytesPerSecLimit, QS_LocKBytesPerSecLimitMatch, QS_CondLocRequestLimitMatch, or QS_EventRequestLimit directive.
• mod_qos_con
  This event shows the number of concurrent connections to this server. Only available if the directive QS_SrvMaxConn is used.
• mod_qos_user_id
  The user id which is available when enabling the user tracking. User tracking is based on a unique identifier generated by mod_unique_id which is stored as a cookie. This feature is enabled by setting the QS_UserTrackingCookieName <cookie name> directive.
• UNIQUE_ID
  This is a unique request id generated by mod_unique_id. mod_qos uses this id to mark messages written to the error log. So it might be useful to log the UNIQUE_ID environment variable as well in order to correlate error to access log messages.

LogFormat "%h %t \"%r\" %>s %b %T \"%{User-Agent}i\" id=%{UNIQUE_ID}e \
            %{mod_qos_ev}e %{mod_qos_cr}e %{mod_qos_con}e %{mod_qos_user_id}e #%P"

Request Statistics

The qslog tool, which is part of the support utilities of mod_qos, may be used to gather request statistics from Apache's access log data. This includes data such as the number of denied requests or new VIP session creations per minute but also total requests per second and other data. Refer to the usage text of the qslog utility for further details.

TransferLog "| ./bin/qslog -o logs/qs_log -f I..R.BT..Q..U"

Status Viewer

mod_qos features a handler showing the current connection and request status.

<Location /qos> SetHandler qos-viewer </Location>

Utilities

mod_qos provides optional tools for log data processing and analysis:

• qslog
  A real time TransferLog/CustomLog data analyzer. It reads the per request log data from stdin and generates statistic records every minute.
• qspng
  Creates graphics (png images) from the output of qslog.
• qsrotate
  Log rotation tool similar to Apache's rotatelogs.
• qssign
  A log data integrity check tool. It reads log data from stdin (pipe) and writes the signed data to stdout.
• qsfilter2
  Rule generator for QS_Permit* directives.
• qscheck
  Monitoring tool for tcp connectivity tests to remote servers.

Use Cases

The following use cases may give you an idea about how to use mod_qos.

Slow Application

In the case an application is very slow (e.g. at location /ccc), requests wait until a timeout occurs. Due many waiting requests, the number of free TCP connections goes out and the web sever is not able to process other requests to applications still working fine, e.g. to /aaa, /bbb /dd1, and /dd2. mod_qos limits the concurrent requests to an application in order to assure the availability of other resources.

Example:

# maximum number of active TCP connections is limited to 256: # (limited by the available memory, adjust the settings according to the # used hardware): MaxClients 256 # limits the maximum of concurrent requests per application to 100: QS_LocRequestLimit /aaa 100 QS_LocRequestLimit /bbb 100 QS_LocRequestLimit /ccc 100 QS_LocRequestLimitMatch "^(/dd1/|/dd2/).*$" 100

HTTP Keep-Alive

The keep-alive extension of HTTP 1.1 allows persistent TCP connections for multiple request/responses. This accelerates access to the web server due less and optimized network traffic. The disadvantage of these persistent connections is, that server resources are blocked even no data is exchanged between client and server. mod_qos allows a server to support keep-alive as long as sufficient connections are free but stopping the keep-alive support when reaching a defined connection threshold.

Example:

# maximum number of active TCP connections is limited to 256: # (limited by the available memory, adjust the settings according to the # used hardware): MaxClients 256 # disables keep-alive when 70% of the TCP connections are occupied: QS_SrvMaxConnClose 70%

Client Opens Many Concurrent Connections

A single client may open many TCP connections simultaneously in order to download different content from the web server. So the client gets many connections while other users may not be able to access the server due no free connections remain for them. mod_qos can limit the number of concurrent connections for a singe IP source address.

Example:

# maximum number of active TCP connections is limited to 896 # (limited by the available memory, adjust the settings according to the # used hardware): MaxClients 896 # don't allow a single client to open more than 50 TCP connections: QS_SrvMaxConnPerIP 50

Many Requests to a Single URL

If you have to limit the number of requests to an URL, mod_qos can help with that too. You may limit the number ot requests per second to an URL by adding a delay to requests accessing this resource.

Example:

# does not allow more than 150 requests/sec: QS_LocRequestPerSecLimit /download/mod_qos.so.gz 150 # but do not allow more than 600 concurrent requests: QS_LocRequestLimit /download/mod_qos.so.gz 600

Too Many Client Connections

mod_qos may prefer "known" client IP addresses in the case that too many clients access the server. "Known" clients are those which has once been identified by the application by setting the corresponding HTTP response header. Such identification may happen at successful user login. Connections from clients which are not known to mod_qos (never marked by the corresponding response header) are denied if the server runs on low TCP connection resources (20% or fewer free connections). mod_qos prefers also those clients which communicates instantaneous and fast with the server and denies access to slow clients sending data irregular in the case the server has not enougth resources.

A minimal request bandwidth should be enforced too in order to close the connections comming from idle clients. The QS_SrvMinDataRate does this. You may want to combine this with the QS_SrvMaxConnPerIP directive as shown above in the "Client Opens Many Concurrent Connections" example.

Example:

# maximum number of active TCP connections is limited to 896 (limited # by the available memory, adjust the settings according to the used # hardware): MaxClients 896 # idle timeout: Timeout 30 # keep alive (for up to 70% of all connections): KeepAlive on MaxKeepAliveRequests 100 KeepAliveTimeout 5 QS_SrvMaxConnClose 70% # name of the HTTP response header which marks preferred clients (this # may be used to let the application decide which clients are "good", # e.g. authenticated users, you may also use QS_VipUser when using an # Apache authentication module such as mod_auth_basic or mod_auth_oid): QS_VipIPHeaderName mod-qos-login # enables the known client perfer mode (server allows new TCP connections # from known/good clients only when is has more than 716 open TCP connections): QS_ClientPrefer 80 # handles two hundred thousand different client IP addresses: QS_ClientEntries 200000 # minimum request/response speed (deny slow clients blocking the server, # e.g. defending slowloris): QS_SrvMinDataRate 150 1200 # and limit request header and body: LimitRequestFields 30 QS_LimitRequestBody 102400 # don't allow more than 30 TCP connections per client source address: QS_SrvMaxConnPerIP 30