How to Remote Manage¶

A significant amount of internal state and operations of a z2 system can be accessed via JMX using tools such as jconsole or jvisualvm. In order to access z2 via JMX some JMX related configuration should be applied.

Configuring JMX access in Z2¶

Accessing a Java VM remotely via JMX¶

It should not be necessary, but unfortunately, as JMX in based on RMI, accessing to a Java VM via JMX remotely can be non-trivial, if

a) You have port restrictions
b) The externally used hostname is not the same as the internally used hostname

Both are typical, if you need to manage machines remotely.

Here are two practical approaches:

Accessing a Remote JVM via JMX directly¶

The tricky thing about JMX over RMI is that knowing the host name and the JMX port is merely sufficient to request RMI access that then uses a different (random) port and a host name defined by the JMX endpoint. That is, when accessing to the JMX endpoint we get a redirect to a different host and port combination that may not make sense from where we are calling, either because the host name is not resolvable or wrong (e.g. localhost), or because the port is not accessible.

However, the following two system properties can be used to solve these problems:

com.sun.management.jmxremote.rmi.port	Defines the RMI port to use and can be set to be the same as the JMX port
java.rmi.server.hostname	Defines the RMI host name to use

If hostnames differ¶

So, for example, assuming your host is accessible as myhost externally, but has a different name by itself, adding

java.rmi.server.hostname=myhost

to Z2_HOME/bin/runtime.properties (which is one way to make sure the system property is set on all z2 processes), will make sure that JMX access to myhost works. However, when doing so and checking for the ports used by the JVM you might see something like this:

netstat -nltp | grep java

tcp6       0      0 :::42149                :::*                    LISTEN      2775/java
tcp6       0      0 :::8080                 :::*                    LISTEN      2775/java
tcp6       0      0 :::38291                :::*                    LISTEN      2775/java
tcp6       0      0 :::37237                :::*                    LISTEN      2735/java
tcp6       0      0 :::43767                :::*                    LISTEN      2735/java
tcp6       0      0 :::7800                 :::*                    LISTEN      2775/java
tcp6       0      0 :::7777                 :::*                    LISTEN      2735/java

Note the expected "unexpected" ports 42149 and 38291 for the web worker process and 37237 and 43767 for the home process.

Using the same port for JMX and RMI¶

By default, z2 sets uses the JMX port 7777 for the <home> process and 7800 for the web worker process. If not all ports are accessible but you are willing to allow to make dedicated ports accessible, it could
be a good idea to make sure that these ports are also used for JMX over RMI.

In that case, we need to set the property com.sun.management.jmxremote.rmi.port for each process invidually.

For the home process, that is started directly from the command line, we configure startup settings in Z2_HOME/bin/launch.properties. We enhance home.vmopts to include the new property:

home.vmopts=\
        -Xmx64M -cp z.jar \
        -Dcom.sun.management.config.file=management.properties \
        -Dworker.remoteJmx=true \
        -Djava.util.logging.config.file=logging.properties \
        -Dcom.zfabrik.home.concurrency=5 \
        -Dcom.sun.management.jmxremote.rmi.port=7700 \
        -Dcom.zfabrik.home.start=environment/home

In addition in Z2_HOME/base/environment.base/webWorker.properties we change the VM options to

worker.process.vmOptions\:JEXL3=`\
 -Xmx128m -Xms128m -XX:+HeapDumpOnOutOfMemoryError \
 -Dderby.system.home=../../data/derby \
 -Dsvnkit.symlinks=false \
 -Duser.language=en \
 -Dcom.sun.management.config.file=management.properties \
 -Dcom.sun.management.jmxremote.rmi.port=${this["worker.jmx.port"]}\
`

so that the same port is used for JMX over RMI as for the initial JMX access.

After that we will see two of the obscure ports gone. For example this:

netstat -nltp | grep java

tcp6       0      0 :::44809                :::*                    LISTEN      8980/java           
tcp6       0      0 :::35691                :::*                    LISTEN      8956/java           
tcp6       0      0 :::8080                 :::*                    LISTEN      8980/java           
tcp6       0      0 :::7700                 :::*                    LISTEN      8956/java           
tcp6       0      0 :::7800                 :::*                    LISTEN      8980/java           
tcp6       0      0 :::7777                 :::*                    LISTEN      8956/java    

The remaining two random ports are due to the Java Attach API.

Using jstatd¶

Another way of allowing access to all Java VMS on some machine is by running jstatd .

Accessing a Remote JVM via JMX via an SSH tunnel¶

Another option is to access JMX via an SSH tunnel. Given you made sure the ports for JMX and JMX RMI are identical, this works and has the additional benefits, that you could protect access to JMX completely by limiting it to SSH access.

Supposedly, all you need to do for that is to add the property com.sun.management.jmxremote.local.only=true to Z2_HOME/bin/management.properties.

In addition, it will not hurt to force the hostname used in RMI access to localhost via setting

java.rmi.server.hostname=localhost

in Z2_HOME/bin/runtime.properties.

Remote Synchronization and Log Streaming¶

Starting with version 2.8, Z2 offers a built-in command line utility to synchronize (with log output) and provide streaming access to the z2 home log. When in Z2_HOME/bin run:

java -cp z.jar com.zfabrik.launch.Manage -?

for the usage:

SYNOPSIS

java -cp z.jar Manage <command> <options>

COMMANDS

    sync        The sync performs a synchronization of a running (remote) z2 Home.  

    showlog     Continuously stream the z2 Home log of a running z2 Home to the current stderr.  

OPTIONS

    -url <url>
        JMX URL identifying the target z2 Home. Defaults to service:jmx:rmi:///jndi/rmi://localhost:7777/jmxrmi
        The URL may be shortened to <host>:<port> (e.g. localhost:7777)

    -user <username> 

        Username used for JMX authentication. Optional.

    -pass <password> 

        Password used for JMX authentication. Optional. Mandatory when a username has been set

    -b <n>

        Number of lines to read before current (if available) when running showlog. 

EXAMPLE

    java -cp z.jar com.zfabrik.launch.Manage showlog -url host:7777 -user admin -pass admin