Optimization and Scalability

Optimizing Apache and Tomcat

Altering Apache and Tomcat Parameters

Apache and Tomcat Parameters should only be modified with support or consulting as these parameters can have severe and catastrophic performance impacts on your server when improperly adjusted. SUSE will not be able to provide support for catastrophic failure when these advanced parameters are modified without consultation. Tuning values for Apache httpd and Tomcat requires that you align these parameters with your server hardware. Furthermore testing of these altered values should be performed within a test environment.

Apache’s httpd MaxClients Parameter

The MaxClients setting determines the number of Apache httpd processes, and thus limits the number of client connections that can be made at the same time (SUSE Manager uses the pre-fork MultiProcessing Modules). The default value for MaxClients in SUSE Manager is 150. If you need to set the MaxClients value greater than 150, Apache httpd’s ServerLimit setting and Tomcat’s maxThreads must also be increased accordingly (see below).

The Apache httpd MaxClients parameter must always be less or equal than Tomcat’s maxThreads parameter!

If the MaxClients value is reached while the software is running, new client connections will be queued and forced to wait, this may result in timeouts. You can check the Apache httpd’s error.log for details:

[error] Server reached MaxClients setting, consider increasing the MaxClients setting

The default MaxClients parameter can be overridden on SUSE Manager by editing the server-tuning.conf file located at /etc/apache2/. For example server-tuning.conf file:

# prefork MPM
   <IfModule prefork.c>
           # number of server processes to start
           # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#startservers
           StartServers         5
           # minimum number of server processes which are kept spare
           # http://httpd.apache.org/docs/2.2/mod/prefork.html#minspareservers
           MinSpareServers      5
           # maximum number of server processes which are kept spare
           # http://httpd.apache.org/docs/2.2/mod/prefork.html#maxspareservers
           MaxSpareServers     10
           # highest possible MaxClients setting for the lifetime of the Apache process.
           # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#serverlimit
           ServerLimit        150
           # maximum number of server processes allowed to start
           # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients
           MaxClients         150
           # maximum number of requests a server process serves
           # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild
           MaxRequestsPerChild  10000
   </IfModule>

Whenever the Apache httpd MaxClients parameter is changed, the ServerLimit must also be updated to the same value, or the change will have no effect.

Tomcat’s maxThreads Parameter

Tomcat’s maxThreads represents the maximum number of request processing threads that it will create. This value determines the maximum number of simultaneous requests that it is able to handle. All HTTP requests to the SUSE Manager server (from clients, browsers, XMLRPC API scripts, etc.) are handled by Apache httpd, and some of them are routed to Tomcat for further processing. It is thus important that Tomcat is able to serve the same amount of simultaneous requests that Apache httpd is able to serve in the worst case. The default value for SUSE Manager is 200 and should always be equal or greater than Apache httpd’s MaxClients. The maxThreads value is located within the server.xml file located at /etc/tomcat/.

Example relevant lines in server.xml:

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="200" connectionTimeout="20000"/>
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="::1" maxThreads="200" connectionTimeout="20000"/>
Tuning Notes

When configuring Apache httpd’s MaxClients and Tomcat’s maxThreads parameters you should also take into consideration that each HTTP connection will need one or more database connections. If the RDBMS is not able to serve an adequate amount of connections, issues will arise. See the following equation for a rough calculation of the needed amount of database connections:

((3 * java_max) +  apache_max + 60)

Where:

  • 3 is the number of Java processes the server runs with pooled connections (Tomcat, Taskomatic and Search)

  • java_max is the maximum number of connections per Java pool (20 by default, changeable in /etc/rhn/rhn.conf via the hibernate.c3p0.max_size parameter)

  • apache_max is Apache httpd’s MaxClients

  • 60 is the maximum expected number of extra connections for local processes and other uses

Big Scale Deployment (1000 Minions or More)

In the following sections find considerations about a big scale deployment. In this context, a big scale compromises 1000 minions or more.

General Recommendations

SUSE recommends the following in a big scale SUSE Manager deployment:

  • SUSE Manager servers should have at least 8 recent x86 cores, 32 GiB of RAM, and, most important, fast I/O devices such as at least an SSD (2 SSDs in RAID-0 are strongly recommended).

  • Proxies with many minions (hundreds) should have at least 2 recent x86 cores and 16 GiB of RAM.

  • Use one SUSE Manager Proxy per 500-1000 clients. Keep into account that download time depends on network capacity. Here is a rough example calculation with physical link speed of 1 GB/s:

    400 Megabytes  *        3000       /      119 Megabyte/s        / 60
= 169 Minutes

    This is:

    Size of updates * Number of minions / Theoretical download speed / 60

  • Depending on hardware you can accept hundreds of minion keys.

  • Plan time for onboarding minions- at least one hour per 1000 minions.

  • It is not recommended onboarding more than approx. 1000 minions directly to the SUSE Manager server- proxies should be used instead. This is because every minion can use up to 3 TCP connections simultaneously, and too many TCP connections can cause performance issues.

  • If the following error appears in output of dmesg, you probably have an excessive number of minions attached to a single SUSE Manager server or proxy for the ARP cache to contain all of their addresses:

    kernel: neighbour table overflow

    In that case, increase the ARP cache values via sysctl, for example, by adding the following lines to /etc/sysctl.conf:

    net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 8192
net.ipv4.neigh.default.gc_thresh3 = 16384
net.ipv4.neigh.default.gc_interval = 60
net.ipv4.neigh.default.gc_stale_time = 120
Start Small and Scale Up

Always start small and scale up gradually. Keep the server monitored in order to identify possible issues early.

Tuning Proposals

SUSE proposes the following tuning settings in a big scale SUSE Manager deployment:

  • Increase the maximum Tomcat heap memory to face a potentially long queue of Salt return results. Set 8 GiB instead of the current default 1 GiB: parameter Xmx1G in /etc/sysconfig/tomcat (affects onboarding and Action execution).

  • Increase the number of Taskomatic workers, allowing to parallelize work on a high number of separate jobs. Set parameter org.quartz.threadPool.threadCount = 100 in /etc/rhn/rhn.conf (affects onboarding and staging).

  • Allow Taskomatic to check for runnable jobs more frequently to reduce latency. Set parameter org.quartz.scheduler.idleWaitTime = 1000 in /etc/rhn/rhn.conf (affects onboarding, staging and Action execution).

  • Increase Tomcat’s Salt return result workers to allow parallelizing work on a high number of Salt return results. Set parameter java.message_queue_thread_pool_size = 100 in /etc/rhn/rhn.conf (affects patching).

  • Increase the number of PostgreSQL connections available to Java applications (Tomcat, Taskomatic) according to the previous parameters, otherwise extra workers will starve waiting for a connection. Set parameter hibernate.c3p0.max_size = 150 in /etc/rhn/rhn.conf (affects all minion operations). Make sure enough PostgreSQL connections are configured before changing this parameter - refer to smdba system-check autotuning --help to get automatic tuning of the PostgreSQL configuration file while changing the number of available connections. Additional manual tuning is usually not necessary but might be required depending on scale and exact use cases.

  • Increase the number of Taskomatic’s minion-action-executor worker threads allowing to parallelize the scheduling of Actions to minions. Set parameter taskomatic.com.redhat.rhn.taskomatic.task.MinionActionExecutor.parallel_threads = 8 in /etc/rhn/rhn.conf (affects all minion operations, especially staging).

  • Increase Salt’s presence ping timeouts if responses might come back later than the defaults. Set parameters java.salt_presence_ping_timeout = 20 and java.salt_presence_ping_gather_job_timeout = 20 in /etc/rhn/rhn.conf (affects all minion operations).

  • Increase the number of Salt master workers so that more requests can run in parallel (otherwise Tomcat and Taskomatic workers will starve waiting for the Salt API, and Salt will not be able to serve files timely). Set parameter worker_threads: 100 in /etc/salt/master.d/susemanager.conf (affects onboarding and patching).

    • Increase this parameter further if file management states fail with the error "Unable to manage file: Message timed out"

    • Note that Salt master workers can consume significant amounts of RAM (typically about 70 MB per worker). It is recommended to keep usage monitored when increasing this value and to do so in relatively small increments (eg. 20) until failures are no longer produced.

  • Disable daily comparison of configuration files. Click on Admin  Task Schedules, then on the compare-configs-default link, then on the Disable Schedule button and finally on Delete Schedule.

  • Increase the maximum heap memory for the search daemon to be able to index many minions. Set 4 GiB instead of the current default 512 MB: add rhn-search.java.maxmemory=4096 in /etc/rhn/rhn.conf (affects background indexing only).

Note that increasing the number of Postgres connections will require more RAM, make sure the SUSE Manager server is monitored and swap is never used.

Also note the above settings should be regarded as guidelines-they have been tested to be safe but care should be exercised when changing them, and consulting support is highly recommended.