Tuning a Oracle Coherence Application

The following is the first of five posts which will list the best practices and performance suggestions for tuning one’s Oracle Coherence environment.  This is a general guide that will evolve as new product features are added to Oracle Coherence.   In general the key performance killers for any data-grid are the:  Network, Java Virtual Machine (JVM) configuration and grid configuration.  Network is critical and running a multicast test will help validate the environment.

Tuning a Oracle Coherence Application

  Network Topology:

1. Check the Network 
2. Check the MTU size.   Oracle Coherence uses a packet size based on the standard 1500 byte MTU.  If one is on Windows, this operating system includes a fast I/O path for "small" packets, where small is defined as 1024 bytes.  Increasing this limit to match the MTU can significantly improve network performance. FastSendDatagramThreshold is the registry value for this on Windows. To make these changes to your registry, run the included "optimize.reg" registry file in the Oracle Coherence product installation.
3. Run the Datagram test to measure network throughput and check the success rate.  The practical max on a well-tuned gigabit Ethernet link is ~115MB/sec.
4. Make sure there is a 1 GB network between all servers (avoid mixed speed networks)
5. Network Switch tuning (avoid multiple switches)
6. Are Cisco switches involved?  The solution is to make sure unicast is being used with WKA (Well-Known Addresses).  See the following link with Cisco Switches around packet pauses (visible when coherence logging is turned up).
7. Check the JMX MBean for packet delivery/receiver success rates when running these basic tests.

Operating System:

1. Adjust UDP buffer size on the operating systems where the Oracle Coherence Servers are running.  

JVM Tuning:

• Start with a Java Heap of 1 GB (-Xms=1g -Xmx=1g) for each Oracle Coherence Server and tune from there based on performance tests, application profile, and the operating system (32 bit or 64 bit OS). A JVM with 1 GB heap uses 1.2 GB of physical RAM.
• Do not configure your JVMs to exceed physical RAM since this will cause swapping and bad performance.

1. Run “swap –l”, “top”, or “vmstat” to verify the system is not swapping and RAM is available
2. Allow ~400 MB for the operating system
3. Take into account other software running on the system

• If one is using Oracle HotSpot VM, make sure to use the “-server” argument.   This is a link to all HotSpot VM Options.
• If one is using Oracle JRockit RealTime VM, the following parameters are a good starting point

-Xms=1g -Xmx=1g -XgcPrio:deterministic -XpauseTarget=10ms -Xverbose:opt,memory,gc,gcpause,memdbg,compaction,starttime,load,cpuinfo,systemgc -Xverbosetimestamp –Xgcreport -Xverboselog:/full/path/logfile.log

• If one is running into strange Network issues, make sure to first add the following flag to the Oracle Coherence Server nodes: -Djava.net.preferIPv4Stack=true

Oracle Coherence Application Tuning

1. Use getall() and putall() APIs which will result in a huge performance improvement
2. Serialization:  Use either the ExternalizableLite interface or POF (Portable Object Framework) from Coherence.  POF will result in the best performance gains (up to seven times (7x) compared to java.io.Serializable) however there is more initial code to with POF.
3. Entry Processors -  can be used to update data instead of doing:  lock(id), value=getID, setValue
4. Set once across the wire instead of multiple locks/etc in order to execute business rules where the data lives.  Three times (3x) improvement from less network hops.  Use invokeAll() API and look into setting the thread count in cache configuration to higher than one for this.
5. Use “lite” events such as event listeners which can be configured to receive or not receive old/new values.
6. Database integration – Caching Strategy 
1. Coherence Behind - Use Oracle Coherence as L2 Cache for ORM (Oracle TopLink JPA)
2. Coherence to the side - Application manages Data Crud in Oracle Coherence next to OR/M
3. Coherence on Top - Coherence is the system of record, use cacheloaders and cache stores to integrate with Data Sources

Review the Production Checklist which contains information on the following topics:

1. Network
2. Hardware
3. Operating System
4. JVM
5. Java Security Manager
6. Application Instrumentation
7. Coherence Editions and Modes
8. Coherence Operational Configuration
9. Coherence Cache Configuration
10. Large Cluster Configuration
11. Other Resources

 

Oracle Coherence Networking Links

Coherence TCMP Network Protocol Explanation  TCMP stands for “Tangosol Cluster Management Protocol” which is based on UDP Unicast.

Important Links

Best Practices 
Performance Tuning
Production Checklist
Coherence Technical FAQ
Testing and Tuning Coherence
Sample Cache Configuration Files
Platform-Specific Deployment Considerations
Patterns for Pre-Loading the Cache