This report gives a tour of the concrete architecture of the Apache web server (release 1.3.4). The goal is to provide support for anyone who wants to modify a subsystem, or add extra functionality.
The main components of the concrete architecture of the Apache server are the core and the modules.
This paper covers the details of Apache core architecture, the essential data structures with their uses, and gives an extended insight into the concrete architecture of a module. The concurrency approach employed in Apache is also detailed.
In general, anyone who wants to add extra functionality only has to write a new module. This usualy means providing one or more handlers (functions) for one of the phases of processing an HTTP request. In fact, even an important part of the Apache core has the "look and feel" of a module, although it is not a proper one (it shares information with other core sub-components).
The way a module's handlers are called is transparent to the module, and all communication with a module is done through pointers to functions. Because of this, fact extractors cannot capture the interaction between core and modules.
To extract the concrete architecture we have used a variety of sources: fact extractors (Portable Book Shelf), papers on Apache, read me files, and to a large extent, analyzing relevant parts of the source code.

Apache, concrete architecture, design, web server

http://www.grad.math.uwaterloo.ca/~oadragoi/CS746G/a2/caa.html

1. Introduction

A webserver provides users of a web a means of retrieving data, usually documents, through web browsers. The user clicks on a link in the browser and a request is sent from the browser to the server. The server retrieves the document from storage and returns it to the browser, which presents it to the user.

The Apache server divides the handling of requests into separate phases. These request phases are:

1. URI -> filename translation
   • determine what information is wanted, and where in storage it is
2. Auth ID checking
   • is the user who they say they are?
3. Auth access checking
   • is the user authorized here?
4. Other access checking
   • access checking other than authorization
5. Determining MIME type of requested object
   • information can be in many different forms, including text, images, sound, or video. We will use the term "document" for any type of information available on the web.
6. Sending response to client
   • what is returned to the client, or what action the server should take, is determined by the method specified in the request
7. Logging the request
   (Apache Group)

In Apache, each phase is handled by a module or set of modules. Each module is looked at in succession, to see if it has a handler for the phase. This results in a flow of control and data that is similar to a pipeline.

Figure 1.Apache pipe-line of request handling phases

The above figure illustrates the conceptual movement of the data structure request_rec and the flow of control with the broken arrows. The process starts and ends in the core, where request_rec is created and where the cleanup is done after the request has been handled.

Actually, control moves from the core to each phase and then back to the core, as is shown by the solid arrow lines. As well, request_rec is first created by the core, then passed to each phase and back to the core in turn.

2. High level concrete architecture of Apache

The high level concrete architecture of the Apache web server is not very different from the high level conceptual architecture, in the sense that we still find the same splitting of functionality between the core and the modules. In addition to the Apache core and the modules we can identify some new small modules. Two of them (ap, regex) are essentially libraries of utility functions, used by both the core and the modules. The third one (os) is the one that ensures the independence of the operating system from the Apache core and the standard modules.

Figure 2. High level concrete architecture of Apache.

The next section presents in more detail each component of Apache.

2.1. Overview of Apache components

2.2. Concrete description of the Apache core

All the files forming the Apache core are grouped in the main/ directory. The designers of Apache wanted as much functionality of the web server as possible to be implemented as separate modules and therefore there are many interactions between the sub-components of the Apache core. The idea is that someone extending Apache should not have to modify anything in the core. The only sub-component that might need changing in order to extend the server is the one that implements the HTTP protocol (which is part of the core). Although (on the good side) the HTTP protocol is a separate sub-component of the core, there is no well define API.

Figure 3. The concrete architecture of the Apache core.

2.3. The main server loop (http_main.c)

Figure 3 shows the interaction of this file with the other sub-components of the Apache core. This sub-component also controls the number of active child processes, through a special shared data structure (score board) that holds information on the status of each children. More on this data structure and the way Apache manages concurrency is presented in the data structures section and the section on concurrency.

2.4. The sub-component implementing the basic functionality of a web server(http_core.c)

This file could almost have been mod_core.c . In fact it defines a module structure as any module. http_core.c defines the command table for all the standard configuration commands. It also implements handlers for self-initialization and for some of the phases of the HTTP request cycle:

• a very rudimentary URI to file name translation
• "do-nothing" handler for MIME type checking
• "do-nothing" handler for fix-ups
• a very rudimentary default handler for all the MIME types (only delivery of a document as it is, with no support for all special operations defined by HTTP).

One reason why http_core.c is not a separate module is related to the legacy configuration commands (from NCSA web server) that Apache must implement. These commands, and in particular Options, are more powerful than the typical Apache configuration commands in the sense that they affect more than one module. In order to implement this kind of behavior http_core.c must have access to some Apache core data structures that are not accessible to ordinary modules.

2.5. The configuration sub-component (http_request.c)

2.6. The request flow control sub-component (http_request.c)

2.7. The protocol sub-component (http_protocol.http, http_rfc1413.c)

2.8. Resources management sub-component (buff.c, alloc.c) and utilities sub-component (util*.c)

2.9. Logging sub-component (http_log.c)

2.10 Virtual hosts handling component(http_chost.c)

3. Concrete description of an Apache module

3.1. How to write a module?

First a module must be initialized (when it is loaded). As the core has no information on the internal structure of the module, the module must export a handler for this purpose.

The idea behind Apache is to be as versatile as possible, therefore each module can define its own commands to be used by users in configuration files. However, the core is the one that actually reads from the configuration files, so there must be a way to match a command with a module, or more precisely with the handler exported by the module to execute that command. This matching is done through the command table memorized in the module structure.

The purpose of a module, as have been said, is to add new functionality to the way the web server is servicing HTTP request. However the flow of executing a request is controlled by the core (http_request.c). A module is not required to export handlers for all the distinct phases of the HTTP request cycle. Figure 4 shows a module that implements handlers for all of the phases but in fact most of the modules will implement one or a few of them.

A special case of handler is the one for the phase that actually delivers the object to the client. Those handlers are called content handlers or response handlers. They are special because a module often implements several content handlers, one for each type of object the module knows how to deliver (e.g. a module might know how to deliver disk directories, but might also know how to deliver lists of people formated in the same way as directories). In order to discover which content handler must be called for the current request the core will use the content handler table in the module's modules structure. It consists of pairs of content type(i.e. MIME type) and a pointer to the handler. Of course a module can export the same content handler for more than one MIME type.

An important element to be taken into account when writing a module is that although handlers are invoked implicitly (or might not be invoked at all if handlers of other modules, for the same phase , inform the core that they have completed that phase) different handlers can communicate between data structures private to the module (usually static ones).

Figure 4. The detailed concrete architecture of a complex Apache module .

The most simple module is one that does not provide (i.e. does not need) any initialization or configuration handlers, does not define any custom configuration commands and implements only one handler (usually the content handler). A complex module (like the one in Figure 4) will implement all or most of the possible handlers.

4 Description of the Apache distribution modules

4.1 Standard modules

When the configuration scripts are run a special .c file modules.c is automatically generated in the root of the source code. modules.c defines special arrays of pointers to module structures, called ap_prelinked_modules[], for the modules that are linked with the core, and module *ap_preloaded_modules[] for those that will be preloaded.

It is interesting that even mod_core, which is the module structure defined by the http_code.c sub-component is listed in the array generated in modules.c. The following list gives an overview of some of the standard modules. The mod_core (the pseudo module define in the apache core (subcomponent http_core.c) is also included.

• For URI to file name translation phase:
  • mod_userdir: translate the user home directories into actual paths mod_rewrite Apache 1.2 and up
  • mod_rewrite: rewrites URLs based on regular expressions, it has additional handlers for fix-ups and for determining the mime type
• For authentication / authorization phases:
  • mod_auth, mod_auth_anon,mod_auth_db, mod_auth_dbm : User authentication using text files, anonymous in FTP-style, using Berkeley DB files, using DBM files.
  • mod_access: host based access control.
• For determining the MIME type of the requested object (the content type, the encoding and the language):
  • mod_mime: determines document types using file extensions.
  • mod_mime_magic: determines document types using "magic numbers" (e.g. all gif files start with a certain code)
• For fix-ups phase:
  • mod_alias: replace aliases by the actual path
  • mod_env: fix-up the environment (based on information in configuration files)
  • mod_speling: automatically correct minor typos in URLs
• For sending actual data back to the client: to chose the appropriate module for this phase the mime type or the pseudo mime type (e.g. for a CGI-script) is used.
  • mod_actions: file type/method-based script execution
  • mod_asis: send the file as it is
  • mod_autoindex: send an automatic generated representation of a directory listing
  • mod_cgi: invokes CGI scripts and returns the result
  • mod_include: handles server side includes (documents parse by server which includes certain additional data before handing the document to the client)
  • mod_dir: basic directory handling.
  • mod_imap: handles image-map file
• For logging the request phase:
  • mod_log_*: various types of logging modules

A summary of what each standard modules has to offer (i.e. what phases handles is given in the following. Note that a module can define handlers for more than one phase. Again mod_core has been included.

4.2 The proxy module

• mod_proxy.h mod_proxy.c is in a sense the main files of this module since they define the module structure. The handlers implemented by this module are:
  • URI to filename translation this phase is necessary because request might refer to file that are not on the local disk but o different machines.
  • fix-up handler this only put the requested URL in a canonical for (so it can be used as a search key in the cache.
  • content handler: this is implemented because the proxy does not simply delivers files from the local disk, it might contact an web server in order to obtain the file and it might save the file into a cache.
• proxy_cache.c manages the cache implemented by the proxy module. The cache is an example of a private data structure that survive between implicit invocation to the handlers of the proxy module.
• proxy_connect.c implements the code that connect this server to an web server. The proxy acts as a client for the web server and as a server for the HTTP client.
• proxy_ftp.c, proxy_http.c implement utility routines specific for HTTP and FTP protocols.
• proxy_util.c implements various routines that main deal with matching symbolic host name, host Internet address, etc.

5. Concurrency

Each request that the server receives is actually handled by a copy of the httdp program. Rather than creating a new copy when it is needed, and killing it when a request is finished, Apache maintains at least 5 and at most 10 inactive children at any given time. The parent process runs a periodic check on a structure called the scoreboard, which keeps track of all existing server processes and their status. If the scoreboard lists less than the minimum number of idle servers, then the parent will spawn more. If the scoreboard lists more than the maximum number of idle servers, which is by default 10, then the parent will proceed to kill off the extra children.

When it receives a request, the parent process passes it along to the next idle child on the scoreboard. Then the parent goes back to listening for the next request.

Figure 5.The parent passes a request to a child to be served

There is also a default limit of 256 on the total number of servers that can exist at one time. The authors of Apache provided this upper bound in order to keep the machine that the software is running on from being swamped by servers and crashing. The default was picked to keep the scoreboard file small enough so that it can be scanned by the processes without causing overhead concerns.

Since the number of requests that can be processed at any one time is limited by the number of processes that can exist, there is a queue provided for waiting requests. The maximum number of pending requests that can sit on the queue is 511.

5.1. Keep-alive or Persistent Connections

Apache uses the persistent connection to allow multiple requests from a client to be handled by one connection, rather than opening and closing a connection for each request. The default maximum number of requests allowed over one connection is 100. The connection is closed by a timeout.

6. Data Structures

opening line - some data structures that are central to the functioning of the Apache server

6.1. request_rec

Once a request has been read in, http_request.c is the code which handles the main line of request processing. It finds the right per-directory configuration, building it if necessary. It then calls all the module dispatch functions in the right order.

Figure 6. Calls to modules for the request processing cycle

When the module handlers are called, the only argument passed to them is request_rec. The pieces of this structure which are public to the modules allow them to learn what the request is and how it should be handled. Most of the handlers complete their part of the request cycle by changing some fields in the request rec. But the response handlers must actually return something to the client. Sometimes these handlers need to direct a server to return some other file instead of the one that the client originally requested. This is a redirected request.

Some handlers can farm out part of their job to another process in the form of a sub-request.

Request_rec can be a linked list if the request is redirected by a handler. The structure can contain pointers to the request_rec the request is redirected to and the request_rec it is redirected from. Or, if it is a sub-request, it can contain a pointer to the original request_rec.

6.2. Internal Redirected Requests

The response handler may find that the request to be served is better handled as another type of request. If the request is to an imagemap, a type map, or a CGI script, then the actual resource the user requested is in some other URI than the one originally used. In this case, the module's handler generates a new request and passes it to another process.

The handler invokes ap_internal_redirect, which initiates a new request_rec. The chain of redirects is placed in a list of request_recs which is linked by pointers. The results of the final response handler is passed back up the chain to the one that caught the original request, and is then sent back to the client.

6.3. Sub-requests

The sub_request mechanism allows a response handler to look up files and URIs without actually sending a response. This is done using the functions ap_sub_req_lookup_file or ap_sub_req_lookup_uri. These construct a new request_rec which is processed up to the point of the response.

6.4. Request_rec

6.4.1. Structure of request_rec

Here is a partial list of the fields that are contained in a request_rec.

• pointers to other request_rec, as described above
• pointer to resource pool
• object being requested
  • URI
  • filename
  • path
  • status of information - e.g. set to zero if no such file
• document content information
  • content type
  • encoding
• MIME header tables - in, out, and error headers
• information about the request
  • protocol to use
  • method - e.g. GET, HEAD, POST
• information for logging
  • bytes sent
  • request description

6.5. compool

Rather than keeping track of which files are opened and where allocated memory is, and then explicitly tracking it all done to deallocate it, Apache uses the idea of resource pools. A resource pool is a data structure which keeps track of all allocations of finite resources that are associated with a request. When the request cycle is finished, all the resources held in the pool are released at one time.

This provides the advantages of garbage collection without the extensive code, and small amounts of space can be allocated without adding large amounts of record keeping.

One disadvantage of this method is that resources that are not being actively used cannot be released until the pool is cleared. This can create problems, especially with memory. So the modules can establish private resource pools that they can clear or destroy as they want.

6.6. Command Tables

The core's command table is held in http_core.c. - example - (Thau, pg 3)XS

Each module may have its own command table, which allows it to handle commands read from configuration files. The entries for each command listed in the table are:

1. the name of the command
2. a pointer to the command handler - This is a C function which processes the command.
3. an argument which is passed to the command handler - A command handler may process many commands.
4. where the command may appear -
5. the type and number of arguments the command takes
6. a description of the arguments that the command takes (Thau, pg 3)

6.7. Scoreboard

The scoreboard structure is used to keep track of the child processes. The information is kept brief, basically just the status value and the pid, the process id number. The creators of Apache have plans to add a separate set of longer score structures that will give the number of requests serviced, and data on the current or most recent request.

The status values defined in scoreboard.h are the following:

 SERVER_DEAD 0
 SERVER_STARTING 1       /* Server Starting up */
 SERVER_READY 2          /* Waiting for connection (or accept() lock) */
 SERVER_BUSY_READ 3      /* Reading a client request */
 SERVER_BUSY_WRITE 4     /* Processing a client request */
 SERVER_BUSY_KEEPALIVE 5 /* Waiting for more requests via keepalive */
 SERVER_BUSY_LOG 6       /* Logging the request */
 SERVER_BUSY_DNS 7       /* Looking up a hostname */
 SERVER_GRACEFUL 8       /* server is gracefully finishing request */
 SERVER_NUM_STATUS 9     /* number of status settings */

6.8. Module Structure

The specific contents of a module are determined by the type of function the module performs. Figure 4 shows a generalized picture of a module.

7. Fact extraction and visualization using PBS

As an example Figure 7 shows the result for the Apache core structure. The content of the sub-components is hidden in order to increase clarity. The arrows that point down (at the bottom of the picture), go to the utilities component described earlier (ap, regex, os).

The following table shows what has been grouped under each sub-system (file.* means both file.c and file.h, prefix*.h means all files with that prefix).

Another major difficulty in using PBS, or any other automatic fact extractor is that there is no way to extract relations between the Apache core and the modules. All the calls and references are done through pointers to functions or data structures. These kinds of interactions are difficult, if not impossible, to extract at compile time. Even most of the interactions between the http_core.c and the rest of the sub-components of the Apache core are done through the same mechanism (see the section on http_core.c).

Figure 7. Facts extracted using the Portable Bookshelf Tools

It should be noted that the suppliers and users of http_config have not been drawn. That is because nearly everybody references this module since many .h files have been included in it. The above figure is more a validation of the concrete architecture depicted in Figure 3, rather than a source for it.

8. Conclusions

This report has offered a tour of the concrete architecture of the Apache web server (release 1.3.4). The modular architecture of the server seems to offer great opportunity for extending the code. Designers of Apache strove to move as much of the functionality as possible into the modules. Therefore, modules must implement a well defined API.

Communication between the core and the external modules is done through the modules handler functions. The module handlers are invoked to perform certain phases of processing a request. Handlers receive a reference to the request_rec, which contains the information about the request and the resources the handlers need.

We did not observe the same independence and well defined API between the components of the core. Of course there are some clear utility components that offer services to the other components of the core and to the modules, but the important parts of the core are tightly linked together. One example of this inter-dependence is the http_request.c which controls the flow of processing a request, the http_config.c component which performs the actual invocation of handlers, and the http_protocol.c which communicates with the HTTP client. This linking of the core components makes it somewhat more difficult to change the behavior of Apache by modifying the core. Fortunately, modules can do the same jobs as well, if not better, and they are usually easier to write.

Since the method of calls to its handlers is transparent to the module and all communication with a module is done through pointers to functions, we have found that fact extractors do not capture the interaction between core and modules.

9. Dictionary of terms