Technology
Stefan Damian, , , , , Leave a Comment on MidTier Optimizations
, , , , ,

MidTier Optimizations

Virtual Environment Notes

  • For VMWare based virtual environment, consider mapping 1 CPU core to 1 virtual CPU.
  • Each Mid-Tier instance can handle up to 150 concurrent Users per virtual CPU and up to 300 concurrent Users per dual CPU virtual machine (2 CPU virtual machine).
  • BMC recommends maximum 300 concurrent Users / Servlet engine instance for most Servlet engines deployed on VM’s.
  • Refer to the following VMWare guide

Enterprise Java Applications on VMWare Best Practices Guide

Please consult VMWare regarding recommendations for VM configuration / performance tuning.

  • The Mid-Tier instances should be hosted on different ESX Servers or otherwise ESX Server can become a single point of failure.

 

Physical Environment Notes

  • Mid-Tier can support maximum of 200 concurrent Users per CPU.
  • BMC recommends maximum 400 concurrent Users / Servlet engine instance for most Servlet engines deployed on physical machines.
  • With any growth in number of concurrent Users, horizontal scaling will be required for the Mid-Tier layer, the AR Server layer and the Database layer.
  • If there is significant Web Services Traffic, dedicated Mid-Tier Instance(s) for handling Web Services Integration and dedicated AR Integration Server Instance(s) for handling Web Services Integration may be required.

 

Recommendation on Backup of Files

When backing up any files related to Mid-Tier, Web Server and Application Server / Servlet Engine, the recommendation is to backup these files to a different backup Folder.

Backing up xml files, Jar files, etc., to the same Folder as the original files can cause unpredictable behavior with Mid-Tier in certain scenarios.

This recommendation is a good preventive measure to avoid such issues. Such issues are hard to identify and correct, while the preventive measure is simple to implement.

 

Mid-Tier Cache Folder Size

  • When using an AR System 7.6.04 ITSM Stack for a single locale (typically English locale), the size of Mid-Tier cache folder is typically around 1.8 GB assuming Preload has been performed.
  • When using an AR System 7.6.04 ITSM Stack for all locales, the size of Mid-Tier cache folder is typically more than 2.0 GB+ assuming Preload has been performed.
  • MS Ticket 110120867604748: On Windows 2008, Windows Explorer does not show the correct File size and Timestamp when an Application has an open File handle. Hence the size of Mid-Tier cache folder reported when Mid-Tier is running will be inaccurate. After Mid-Tier is shutdown, the Mid-tier cache folder size will be reported correctly by Windows Explorer.

 

Cache Corruption

The typical cause of Cache corruption is multiple Mid-Tier applications getting deployed in the same Servlet container.

Following are some recommendations for preventing Cache corruption and Cache size issues.

  • In Tomcat (or your Servlet container), make sure that multiple contexts for arsys (Mid-Tier) are not present. Having multiple contexts for the Mid-Tier application can lead to cache corruption, abnormally large cache folder size and inconsistent behavior in Mid-Tier. This issue is typically observed with when Tomcat is used as a Servlet container.

Mid-Tier application can be deployed in any of the following ways:

  • ${TOMCAT_INSTALL_DIR}Tomcat<Tomcat Version>\conf\Catalina\localhost\arsys.xml or any arsys*.xml* files. These are usually created by Installer, but xml backup files may be manually created by Customer as well.
  • ${TOMCAT_INSTALL_DIR}Tomcat6<Tomcat Version>\conf\server.xml or some other xml file under this directory. Some Customer’s manually add this entry in order to be able to deploy Mid-Tier under a context path other than arsys.
  • Do make sure that multiple contexts for arsys (Mid-Tier) are not present when deploying Mid-Tier using either of the above ways.
  • In Midtier/WEB-INF/lib, make sure that multiple copies of same jar file are not present. For example – When we apply any hot-fix we copy Midtier.jar in WEB-INF/lib and back the older Midtier.jar to Midtier_backup.jar. Due to this Tomcat loads the libraries / jar and this could result to inconsistent behavior. To resolve this, the recommendation is to rename Midtier.jar to a non-Jar file i.e. Midtier.jar.bak or remove it from the lib folder completely.
  • Double check that Mid-Tier is not getting deployed twice due to adding it to webapps folder of the Servlet container.
  • If Tomcat (or your Servlet container) has deployed Mid-Tier and is already running and you attempt to start another instance of Tomcat (or your Servlet container), a second instance of Mid-Tier application can get deployed thereby leading to cache corruption.  This issue is specifically observed with Mid-Tier deployment on UNIX based Operating Systems. This is a subtle way of causing Mid-Tier cache corruption and should be avoided.
  • We recommend the use of only the short name of the AR Server in the Mid-Tier AR Server List. Do not use both the short and long name because Midtier will build 2 separate caches and that will take up twice the resources. Additionally using the short name of the AR Server also reduces the Mid-Tier URL length and marginally reduces the number of bytes sent over the wire.

 

Recommended Tuning

Mid-Tier Config.properties

  • In a deployment environment where the AR System applications are not modified, turn off Definition change checks. Otherwise, map this to the frequency of your AR System application modification.
    • arsystem.cache_update_interval=86400
    • In a deployment environment where the AR System applications are not modified, set the following to 604800 (1 week) or higher. The minimum recommended value is 86400 (1 day).
      • arsystem.formhtmljs_expiry_interval=86400
      • arsystem.resource_expiry_interval=86400
      • arsystem.pooling_max_connections_per_server=80

Increasing arsystem.pooling_max_connections_per_server is not generally recommended as this can have an adverse impact on Java Heap requirement.

This setting is usually not changed from its default value, because it represents a pool of connections and not the number of users who can connect to an AR System server.

If Mid-Tier uses large heap (> 1.4 GB) and AR Server List and Fast threads are properly configured, then only consider increasing arsystem.pooling_max_connections_per_server.

  • arsystem.log_level=Severe
  • arsystem.ehcache.overflowToDisk=true
  • arsystem.ehcache.overflowToDiskTemp=false

Tomcat 6 server.xml

  • The recommended Tomcat 6 HTTP connector settings are specified below. They may need to be changed on a case by case basis.

<Connector URIEncoding=”UTF-8″ acceptCount=”250″ connectionTimeout=”90000” disableUploadTimeout=”true” enableLookups=”false” maxHttpHeaderSize=”8192″ maxKeepAliveRequests=”-1″ maxThreads=”300″ minSpareThreads=“50”  port=”80″ protocol=”HTTP/1.1″ redirectPort=”8443″/>

  • If hardware available is high end, maxThreads in HTTP Connector entry can be increased to 400 or 500.
    • Tomcat instance (or Servlet Engine Instance) used for hosting Mid-Tier should not be used for hosting any Application other than Mid-Tier.
    • If any other application is hosted in the same Tomcat instance (or Servlet Engine Instance), this can have an adverse effect on Heap requirements and scalability of Mid-Tier.

Heap Settings

The following are the recommended JVM parameter settings when the Servlet engine hosts only the Mid-Tier web application.

  • Initial Memory Pool – 1024MB (-Xms parameter for Java)
  • Maximum Memory Pool – 1400MB (-Xmx parameter for Java)
  • The Maximum Memory Pool size of 1400 MB is the minimum recommended value. Depending on the Deployment scenario, this may need to be revised. Viz. In a Deployment scenario with heavy searches of up to 5000 objects being performed by Users, Maximum Memory Pool may need to be revised to 2048 MB or more.

GC Settings

The GC tuning parameters vary based on the environment. It depends on many factors such as number of processors, cores, OS, heap sizes etc.

GC pauses are the times when an application appears unresponsive because garbage collection is occurring.

In order to achieve low / shorter GC pauses, the generally recommended GC tuning parameters are as follows for a 64 bit JVM running on a multi processor CPU machine.

Note: The GC tuning parameters recommended are applicable for Java 5 and Java 6 (Java 1.5 and Java 1.6).

-XX:+UseCompressedOops

-XX:+UseConcMarkSweepGC

-XX:+UseParNewGC

-XX:NewRatio=3

-XX:PermSize=256m

-XX:+HeapDumpOnOutOfMemoryError

-XX:ErrorFile=<Path>\java_hs_err.log

-XX:HeapDumpPath=<Path>

Note: <Path> should be replaced with Correct Folder path. Please ensure sufficient disk space for the heap dump.

Note:  When using the GC tuning parameters

  • “-XX:+UseParNewGC -XX:NewRatio=3 -XX:PermSize=256m“, maximum heap size should be set to 2 GB or greater.
  • -Xincgc parameter should be removed from the JVM parameters list because we are recommending the option -XX:+UseConcMarkSweepGC
  • The last 3 settings listed above are for obtaining a heap dump on out of memory. In case the JVM (Java Virtual machine) runs out of memory, then the heap dump obtained should be sent to BMC Software for further analysis.

A brief description of the recommended GC Tuning parameters is as follows (refer to Java documentation for the details).

  • -XX:+UseCompressedOops : The -XX:+UseCompressedOops option can improve performance of the 64-bit JRE by compressing object references to 32 bits, thus reducing the amount of data that it must process.
  • -XX:+UseConcMarkSweepGC : The -XX:+UseConcMarkSweepGC option enables the concurrent Garbage Collector (also known as Concurrent Mark and Sweep collector or CMS). The concurrent collector performs most of its work concurrently (i.e., while the application is still running) to keep garbage collection pauses short. It is designed for applications with medium- to large-sized data sets for which response time is most important. This collector performs concurrent mark and sweep Garbage Collection on the tenured generation (or old generation).
  • -XX:+UseParNewGC : The -XX:+UseParNewGC option enables the parallel young generation collector which performs efficient Garbage Collection of the young generation. The parallel young generation collector (-XX:+UseParNewGC) is integrated with the concurrent low pause collector (-XX:+UseConcMarkSweepGC)
  • -XX:NewRatio : NewRatio denotes the relative size of the tenured generation to the young generation. Setting -XX:NewRatio=3 means that the ratio between the young and tenured generation is 1:3. In other words, the combined size of the young generation (eden and survivor spaces) will be one fourth of the total heap size. This setting is observed to be optimal for Mid-Tier.

Guidelines for Running preload

It is the optimal procedure to run preload once:

  1. In the Mid Tier Configuration Tool, turn on Enable Cache Persistence in the Mid Tier Configuration Tool.
  2. Turn on preload.
  3. Allow preload to finish preloading all user facing AR System forms.
  4. Turn off preload.

Using this optimal procedure, the statistics service loads only the objects that correspond to the actual usage of the system into the mid tier memory. After the preload service has run once, all of the relevant objects are written to disk (because Cache persistence is enabled). By turning off the preload service, the statistics service has full memory access to load only those objects which are collected in actual usage. You can repeat this procedure if the applications on the AR System server have changed.

Unqualified Searches (AR configuration settings)

  • Allow-Unqual-Queries – Specifies whether unqualified searches can be performed on the AR System server. Valid values are T (allows unqualified searches) or F (disallows unqualified searches). The default value is T. The recommended value is F (disallow). Unqualified searches are ARGetListEntry orARGetListEntryWithFields calls, in which thequalifier parameter is NULL or has an operationvalue of 0 (AR_COND_OP_NONE). Such searches might cause performance problems because they return all requests for a form. This is especially problematic for large forms. It is recommended to turn off unqualified searches otherwise the searches could be enormous resulting in unpredictable system behavior.
  • Max-Entries-Per-Query – The maximum number of requests returned by a search. The default value is no server-defined maximum (entry is not defined). The recommended value is 2000 (recommendation may need to be changed on a case by case basis). Because users can also specify the maximum number of requests returned through Search Preferences in the AR System User Preference form or the Options dialog box in BMC Remedy User, the actual maximum is the lower of these values. BMC recommends always setting a value for this parameter as unqualified searches can yield enormous results resulting in unpredictable system behavior. Increasing it would affect the heap size requirement of Midtier and cause Midtier to go out of memory.

Archiving (archiving AR data)

Defining and implementing an Archiving policy is recommended for reducing the size of the Operational data. Smaller sized operational data can lead to better performance of the AR based implementation. When implementing an Archiving policy, maintaining referential integrity is essential.

AR Server Tuning

For AR Server Tuning, refer to the following White Paper

BMC Remedy AR System Server 7.6 Performance Tuning for Business Service Management

 

Reporting

  • Running Reports can put a significant load on Mid-Tier and can require significant amount of memory. Hence Reporting should be done on a dedicated box using a dedicated Mid-Tier instance / BOXI instance.
  • AR Reports should have a query specified, and the Override option should be unchecked when running the report. With the Override option checked the report runs by overriding the query. It is similar to unqualified search. Viz. – Incident Report on “HPD:Help Desk” form can pull 100,000’s of entries, if the query is not provided or if it is overridden thereby affecting Midtier scalability.

 

Load Balancers

General notes:

  • A Load Balancer is a Third party component in the Deployment Architecture not owned by BMC Software. Configuring the Load Balancer correctly is the Customer’s responsibility. BMC Software advises the Customer to work with the Load Balancer Vendor for correctly configuring the Load Balancer to work with Mid-Tier.
  • Using a Load Balancer between Browser and Mid-Tier requires configuring the Load Balancer appropriately for maintaining a Sticky session.
  • Typical Load Balancer configurations use Cookie based persistence and round-robin or least connections algorithm for load balancing. Configuring the Load Balancer correctly is the Customer’s responsibility.
  • The Web Load Balancer Timeout should be set to a value marginally higher than Session Timeout setup in Mid-Tier. Viz. If Mid-Tier Session Timeout is 90 minutes, the Web Load Balancer Session Timeout can be set to 91 minutes or 92 minutes. Note that setting Web Load Balancer Timeout to a value smaller than the Session Timeout setup in Mid-Tier can result in Timeout issues for Mid-Tier.
  • The Mid-Tier does not support failover. Failover to a backup mid tier invalidates the user’s HTTP session on the failed Mid-Tier. (Failover is not transparent to the user. They would see a 9201 “Session timeout or invalid” error in their browsers and would need to log in again.)
  • If there is a Load Balancer or Firewall between the Mid-Tier and the AR System server, configure it to not sever idle connections from the Mid-Tier.  Reestablishing severed connections takes additional time and can impact Mid-Tier performance.
  • Please refer to the following F5 KB article regarding Session persistence.

Notes on F5 Load Balancer

http://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/ltm_configuration_guide_10_0_0/ltm_persist_profiles.html

  • Refer to the following known issue when F5 is used with a Proxy.

The Request-URI header in an HTTP request stores certain session data. Occasionally, however, for Cookie and Universal persistence types specifically, the BIG-IP system ignores the session data in this header, and sends requests to an unexpected node. For example, this issue can occur when clients send requests to a virtual server through an internet proxy device. You can prevent this problem by creating an OneConnect profile, and assigning both the OneConnect profile and the persistence profile to the virtual server.

  • The suggested F5 solution for making F5 Load Balancer work correctly with a Proxy is to create an OneConnect profile and applying both the OneConnect profile and the persistence profile to the virtual server.

Notes on use of a DNS Proxy

  • If a DNS Proxy between the Browser and the Load Balancer, the Customer needs to configure the DNS Proxy correctly so that the Mid-Tier Sticky session requirement is satisfied.
  • A DNS Proxy is a Third party component in the Deployment Architecture not owned by BMC Software. Configuring the DNS Proxy correctly is the Customer’s responsibility.
  • Usually Load Balancer issues manifest in the form of Timeout issues observed with Mid-Tier.
  • Understand the Customer Deployment Architecture including Third party components like Load Balancers, DNS Proxy, Firewalls, etc.
  • Eliminate the Third party components completely or one by one and test Mid-Tier to narrow down the problem area. If Timeouts occur after eliminating all Third party components, it could be a possible Mid-Tier product issue. If Timeouts do not occur after eliminating all Third party components, it could be a possible non BMC issue with some other component in the Deployment architecture.
  • Setup arsystem.response.hostip=true in Mid-Tier config.properties and restart Mid-Tier. This setting prints the Mid-Tier Host IP Address as the ARRESPONSEHOSTIP header in the HTTP Servlet Response. This can be used along with Fiddler for examining the HTTP Response Headers and for making sure that all Requests for a User Session are directed to the same Host IP Address. If all Requests for a User Session are not directed to the same Host IP Address, there can be a problem with Load Balancer settings.

Notes on load balancer between Web Server and AR Server

For versions 7.6.04 or later, BMC recommends configuring the load balancer that is located between the web servers and AR System servers without setting a “Sticky” bit. In versions earlier than 7.6.04, BMC recommended setting the Sticky bit to activate session affinity to route all connections from one web server to the same AR System server.

For more information, please see Using a Hardware Load Balancer with BMC Remedy Action Request System 7.6.04

 

Source: BMC Knowledge Base

Leave a Reply