FANDOM


Readings

General

Tuning HotSpot VM

Tuning IBM Java SDK

Oracle JDBC Driver Tuning

JBoss Tuning

WebSphere Tuning

WebLogic Tuning

Linux Tuning

Windows Tuning

JVM Tuning Foundation

Basic Glossary or Definition

  • Profiling
    • a form of dynamic program analysis that measures, for example, the usage of memory, the usage of particular instructions, or frequency and duration of function calls. (from Wikipedia)
  • Tuning
    • the improvement of system performance. (from Wikipedia)
  • Optimization
    • improving software to make it work more efficiently or use fewer resources. (from Wikipedia)
  • Diagnosis
    • the identification of the nature and cause of anything. (from Wikipedia)
  • Debugging
    • a methodical process of finding and reducing the number of bugs, or defects, in a computer program or a piece of electronic hardware, thus making it behave as expected. (from Wikipedia)

Types of Diagnostic Information in IBM JVM

  • from the diagnostic guide of IBM Java SDK which you can find from here.

Javadump (Thread dump)

The Javadump is sometimes referred to as a Javacore or thread dump in some JVMs. This dump is in a human-readable format produced by default when the JVM terminates unexpectedly because of an operating system signal, an OutOfMemoryError exception, or when the user enters a reserved key combination (for example, Ctrl-Break on Windows®). It can also be generated by calling com.ibm.jvm.Dump.JavaDump() from inside the application. A Javadump summarizes the state of the JVM at the instant the signal occurred. Much of the content of the Javadump is specific to the IBM JVM.

Heapdump

The JVM can generate a Heapdump at the request of the user (for example by calling com.ibm.jvm.Dump.HeapDump() from inside the application) or (by default) when the JVM terminates because of an OutOfMemoryError exception. You can specify finer control of the timing of a Heapdump with the -Xdump:heap option. For example, you could request a Heapdump after a certain number of full garbage collections have occurred. The default Heapdump format (phd files) is not human-readable and you process it using available tools such as Heaproots.

System dump (Core dump)

System dumps (also known as core dumps on Linux® platforms) are platform-specific files that contain information about the active processes, threads, and system memory. System dumps are usually large. By default, system dumps are produced by the JVM only when the JVM fails unexpectedly because of a GPF (general protection fault) or a major JVM or system error. You can also request a system dump by calling com.ibm.jvm.Dump.SystemDump() from your application. You can use the -Xdump:system option to produce system dumps when other events occur

Garbage collection data

A JVM started with the -verbose:gc option produces output in XML format that can be used to analyze problems in the Garbage Collector itself or problems in the design of user applications. Numerous other options affect the nature and amount of Garbage Collector diagnostic information produced. See Garbage Collector diagnostics for more information.

Trace data

The IBM JVM tracing allows execution points in the Java code and the internal JVM code to be logged. The -Xtrace option allows the number and areas of trace points to be controlled, as well as the size and nature of the trace buffers maintained. The internal trace buffers at a time of failure are also available in a system dump and tools are available to extract them from a system dump. Generally, trace data is written to a file in an encoded format and then a trace formatter converts the data into a readable format. However, if small amounts of trace are to be produced and performance is not an issue, trace can be routed to STDERR and will be pre-formatted.


Generating and Analyzing Dump and Log

  • VisualVM can generate heap dump and thread dump for local or remote Sun JVM, but only thread dump for IBM JVM.
  • Eclipse Memory Analyzer can analyze heap dump of both Sun JVM and IBM JVM.
  • ThreadLogic can analyze thread dump of both Sun JVM and IBM JVM.

Heap Dump

Sun JDK

Generating
Analyzing
  • Eclipse Memory Analyzer

IBM Java SDK

Generating

Getting heap dump with IBM Java SDK is explained in the following page.

You can't generate heap dump of IBM Java SDK using jconsole or VisualVM even though JMX agent is enabled. But, using IBM Health Center you can generate heap dump or other types of dumps on the fly. For more, refer the following page.

Unlike the explanation in above page, no dialog box appears finishing dump. The dump file seems to be located where the VM started and the name starts with 'heapdump.'.

Analyzing
  • Eclipse Memory Analyzer

Thread Dump

Sun JDK

Generating

The following is quoted from J2SE 5.0 Troubleshooting and Diagnostic Guide.

1.16 Ctrl-Break Handler

On Solaris or Linux the HotSpot VM will print a thread dump to the application's standard output if the Ctrl and \ keys are pressed. The thread dump consists of the thread stack for all Java threads in the VM. On Windows the equivalent key sequence is the Ctrl and Break keys. The thread dump doesn't terminate the application – it continues after the thread information is printed.
On Solaris and Linux a thread dump is also printed if the J2SE process receives a QUIT signal. So kill -QUIT <pid> causes the process with id <pid> to print a thread dump to its standard output.
In addition to the thread stacks, the ctrl-break handler also executes a deadlock detection algorithm. If any deadlocks are detected then it prints out additional information on each deadlocked thread.

Analyzing
  • ThreadLogic

IBM Java SDK

Generating

The following is quoted from IBM Java SDK Diagnostics Guide 6.

Chapter 25. Using Javadump

...

The signal for Linux, AIX and z/OS is SIGQUIT. Use the command kill -QUIT n to send the signal to a process with process id (PID) n. Alternatively, press CTRL+\ in the shell window that started Java.(CTRL+V on z/OS.)
The signal for AIX is SIGQUIT. Use the command kill -QUIT n to send the signal to a process with process id (PID) n. Alternatively, press CTRL+\ in the shell window that started Java.
In Windows, the dump is initiated by using CTRL+Break in the command window that started Java.
The signal for i5/OS is SIGQUIT. Use the command kill -QUIT n to send the signal to a process with process id (PID) n. The PID for a particular JVM can be found in the joblog for the job (JVAB56D: Java Virtual Machine is IBM Technology for Java. PID(x)), or using the "ps" command from qsh or an i5/OS PASE shell.
The JVM will continue operation after the signal has been handled.

Analyzing
  • ThreadLogic

GC Log

  • Garbage Collector Log Analysers
    • list of tools to analyze the garbage collection (GC) log files produced by the JVM when using -verbose:gc or -Xloggc:.. or equivalent.

Analyzing

misc

Obtaining Clue on OOM(OutOfMemory) Error

OutOfMemoryError is one of the most terrible problems. To prevent it from occurring again, it is very important to identify the cause. The heap dump at the time of error maybe provide the most exact cause to the error. With JDK 1.5 or higher, you can use the following options with JVM to obtain the heap dump at the time when OutOfMemoryError ocurrs.

 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=''path-of-dump-to-generate''

One thing that needs to be considered more is that the above options derive how much overhead or side effects. I don't have any idea about it, and so I have no confidence where the above options can be safely used in production environment. To confirm using in production environment, more thorough research is necessary.

HotSpot VM Tuning

General VM Option

  • For production system
-server -Xms2048m -Xmx2048m -XX:MaxPermSize=256M -XX:NewRatio=2 -XX:SurvivorRatio=8 -XX:+UseParallelOldGC
-verbose:gc -Xloggc:/path/to/gclog -XX:+DisableExplicitGC
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/heapdump -XX:-HeapDumpOnCtrlBreak

-XX:-HeapDumpOnCtrlBreak option doesn't work with Java 1.6 or higher.

Identifying Current/Min/Max Size of Young and Old Generation

You can identify details of memory usage such as min./max. size of old generation, current size of eden space and current size young generation on a running VM instance using jstat command.

For example, the VM launched with the following option.

 -Xms128m -Xmx256m -XX:MaxPermSize=256M -server -XX:+UseParallelOldGC -XX:NewRatio=2 -XX:SurvivorRatio=8

Shows the following information with jstat -gccapacity -h5 3680 5s.

 
 C:\lang\jdk1.6\bin>jstat -gccapacity -h5 3680 5s
  NGCMN    NGCMX     NGC     S0C   S1C       EC      OGCMN      OGCMX       OGC         OC      PGCMN    PGCMX     PGC
     PC     YGC    FGC
  43648.0  87360.0  43648.0 4480.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     48     3
  43648.0  87360.0  43648.0 4480.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     48     3
  43648.0  87360.0  43648.0 4480.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     48     3
  43648.0  87360.0  43648.0 4480.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     48     3
  43648.0  87360.0  43648.0 4480.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     48     3
  NGCMN    NGCMX     NGC     S0C   S1C       EC      OGCMN      OGCMX       OGC         OC      PGCMN    PGCMX     PGC
     PC     YGC    FGC
  43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     49     3
  43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     49     3
  43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     49     3
  43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     49     3
  43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     49     3
  NGCMN    NGCMX     NGC     S0C   S1C       EC      OGCMN      OGCMX       OGC         OC      PGCMN    PGCMX     PGC
     PC     YGC    FGC
  43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     49     3
  43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     49     3
  43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     49     3
  43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     49     3
  43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
  103936.0     49     3

You can identify the above statistics in visual form using visualgc. visualgc is provided as a plugin of VisualVM.

References

Using VisualVM with IBM JDK

You can use JMX to monitoring IBM JDK with VisualVM.
Java SE 5.0 introduced platform MBeanServer and IBM Java SDK 5.0 or higher naturally provide it. Platform MBeanServer includes various basic managed beans about memory, thread, garbage collection, class loading and so on[1]. VisualVM can show such information with visual views.

Basically remote access of platform MBeanServer is not activated. You can enable it in several ways.
If you can control the start-up of the VM, you can enable remote access of platform MBeanServer by setting com.sun.management.jmxremote.por system property in command line to launch VM. If you are in position where you can control the system properties, you can programmatically load JMX connector server and expose it using RMI registry.

Method 1. Enabling built-in JMX agent using system properties

You can enable built-in JMX agent of Java runtime with the following system properties explicitly in start-up command of JVM.

-Dcom.sun.management.jmxremote.port=3333
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

With VisualVM, you should add new local or remote application using [File > Add JMX Connection] menu item. The connection string has the form of hostname:port such as localhost:3333 or 203.201.71.31:3333.

For more about built-in JMX agent, refer readings 2 below. If more secure setup for remote access to the platform MBeanServer is necessary, it would be really helpful.

Method 2. Exposing platform MBeanServer using JMXConnectorServer programmatically.

When you can not control the start-up command of the JVM, you can expose platform MBeanServer through JMXConnectorServer instance in your code. The JMXConnectorServer should be exposed using RMI registry.

The following code sample shows core steps, acquiring platform MBeanServer, creating JMXConnectorServer instance, creating RMI registry and starting JMXConnectorServer.

...
 
//acquiring platform MBeanServer
MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
 
//creating JMXConnectorServer instance
JMXServiceURL url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://"
      + InetAddress.getLocalHost().getHostAddress() + ":9999/server");
JMXConnectorServer cs = JMXConnectorServerFactory.newJMXConnectorServer(url, null, mbs);
 
//creating RMI registry
Registry reg = LocateRegistry.createRegistry(9999);
 
//starting JMXConnectorServer
cs.start();
 
...

With VisualVM, you should add new local or remote application using [File > Add JMX Connection] menu item. The connection string has the form of service:jmx:rmi:///jndi/rmi://host-address:port/service such as service:jmx:rmi:///jndi/rmi://203.201.71.31:9999/server.
Now you can identify JVM argument or system properties, monitor memory or garbage collection, class loading statistics, thread in timeline, and so on with VisualVM.

If you want stop the remote access to the platform MBeanServer programmatically, just stop the JMXConnectorSever.

...
 
//stoping JMXConnectorServer
cs.stop();
 
//unload RMI registry
java.rmi.server.UnicastRemoteObject.unexportObject(reg,true);
 
...

As for me, the above code worked with IBM Java SDK 6.0 on Windows and IBM Java SDK 6.0 64bit on AIX 6.2.

Readings

  1. java.lang.management package API
  2. Monitoring and Management Using JMX
  3. Connecting to JMX Agents Explicitly
  4. Support for the Java Attach API of IBM Java v6 on AIX