z2-Environment

h1. How to Remote Manage

A significant amount of internal state and operations of a z2 system can be accessed via JMX using tools such as jconsole or jvisualvm. In order to access z2 via JMX some JMX related configuration should be applied.

h2. Configuring JMX access in Z2

h2. Accessing a Java VM remotely via JMX

It should not be necessary, but unfortunately, as JMX in based on RMI, accessing to a Java VM via JMX remotely can be non-trivial, if

a) You have port restrictions

b) The externally used hostname is not the same as the internally used hostname

Both are typical, if you need to manage machines remotely.

Here are two practical approaches: 

h3. Accessing a Remote JVM via JMX directly

The tricky thing about JMX over RMI is that knowing the host name and the JMX port is merely sufficient to request RMI access that then uses a different (random) port and a host name defined by the JMX endpoint. That is, when accessing to the JMX endpoint we get a redirect to a different host and port combination that may not make sense from where we are calling, either because the host name is not resolvable or wrong (e.g. localhost), or because the port is not accessible. 

However, the following two system properties can be used to solve these problems:

| com.sun.management.jmxremote.rmi.port | Defines the RMI port to use and can be set to be the same as the JMX port |

| java.rmi.server.hostname | Defines the RMI host name to use |

h4. If hostnames differ

So, for example, assuming your host is accessible as myhost externally, but has a different name by itself, adding

<pre><code class="bash">

java.rmi.server.hostname=myhost

</code></pre>

to @Z2_HOME/bin/runtime.properties@ (which is one way to make sure the system property is set on all z2 processes), will make sure that JMX access to myhost works. However, when doing so and checking for the ports used by the JVM you might see something like this:

<pre><code class="bash">

netstat -nltp | grep java

tcp6       0      0 :::42149                :::*                    LISTEN      2775/java

tcp6       0      0 :::8080                 :::*                    LISTEN      2775/java

tcp6       0      0 :::38291                :::*                    LISTEN      2775/java

tcp6       0      0 :::37237                :::*                    LISTEN      2735/java

tcp6       0      0 :::43767                :::*                    LISTEN      2735/java

tcp6       0      0 :::7800                 :::*                    LISTEN      2775/java

tcp6       0      0 :::7777                 :::*                    LISTEN      2735/java

</code></pre>

Note the expected "unexpected" ports 42149 and 38291 for the web worker process and 37237 and 43767 for the home process.

h4. Using the same port for JMX and RMI

By default, z2 sets uses the JMX port 7777 for the <home> process and 7800 for the web worker process. If not all ports are accessible but you are willing to allow to make dedicated ports accessible, it could

be a good idea to make sure that these ports are also used for JMX over RMI.

In that case, we need to set the property @com.sun.management.jmxremote.rmi.port@ for each process invidually.

For the home process, that is started directly from the command line, we configure startup settings in @Z2_HOME/bin/launch.properties@. We enhance @home.vmopts@ to include the new property:

<pre><code class="bash">

home.vmopts=\

        -Xmx64M -cp z.jar \

        -Dcom.sun.management.config.file=management.properties \

        -Dworker.remoteJmx=true \

        -Djava.util.logging.config.file=logging.properties \

        -Dcom.zfabrik.home.concurrency=5 \

        -Dcom.sun.management.jmxremote.rmi.port=7700 \

        -Dcom.zfabrik.home.start=environment/home

</code></pre>

In addition in @Z2_HOME/base/environment.base/webWorker.properties@ we change the VM options to 

<pre><code class="bash">

worker.process.vmOptions\:JEXL3=`\

 -Xmx128m -Xms128m -XX:+HeapDumpOnOutOfMemoryError \

 -Dderby.system.home=../../data/derby \

 -Dsvnkit.symlinks=false \

 -Duser.language=en \

 -Dcom.sun.management.config.file=management.properties \

 -Dcom.sun.management.jmxremote.rmi.port=${this["worker.jmx.port"]}\

`

</code></pre>

so that the same port is used for JMX over RMI as for the initial JMX access.

After that we will see two of the obscure ports gone. For example this:

<pre><code class="bash">

netstat -nltp | grep java

tcp6       0      0 :::44809                :::*                    LISTEN      8980/java           

tcp6       0      0 :::35691                :::*                    LISTEN      8956/java           

tcp6       0      0 :::8080                 :::*                    LISTEN      8980/java           

tcp6       0      0 :::7700                 :::*                    LISTEN      8956/java           

tcp6       0      0 :::7800                 :::*                    LISTEN      8980/java           

tcp6       0      0 :::7777                 :::*                    LISTEN      8956/java    

</code></pre>

The remaining two random ports are due to the "Java Attach API":https://docs.oracle.com/javase/8/docs/technotes/guides/attach/index.html.

h3. Using jstatd 

Another way of allowing access to all Java VMS on some machine is by running "jstatd":https://docs.oracle.com/javase/6/docs/technotes/tools/share/jstatd.html .

h3. Accessing a Remote JVM via JMX via an SSH tunnel

Another option is to access JMX via an SSH tunnel. Given you made sure the ports for JMX and JMX RMI are identical, this works and has the additional benefits, that you could protect access to JMX completely by limiting it to SSH access. 

Supposedly, all you need to do for that is to add the property @com.sun.management.jmxremote.local.only=true@ to Z2_HOME/bin/management.properties.

In addition, it will not hurt to force the hostname used in RMI access to localhost via setting 

<pre><code class="bash">

java.rmi.server.hostname=localhost

</code></pre>

in @Z2_HOME/bin/runtime.properties@.

h2. Remote Synchronization and Log Streaming

Starting with version 2.8, Z2 offers a built-in command line utility to synchronize (with log output) and provide streaming access to the z2 home log. When in Z2_HOME/bin run:

<pre><code class="bash">

java -cp z.jar com.zfabrik.launch.Manage -?

</code></pre>

for the usage:

<pre><code class="bash">

SYNOPSIS

java -cp z.jar Manage <command> <options>

COMMANDS

    sync        The sync performs a synchronization of a running (remote) z2 Home.  

    showlog     Continuously stream the z2 Home log of a running z2 Home to the current stderr.  

OPTIONS

    -url <url>

        JMX URL identifying the target z2 Home. Defaults to service:jmx:rmi:///jndi/rmi://localhost:7777/jmxrmi

        The URL may be shortened to <host>:<port> (e.g. localhost:7777)

    -user <username> 

        Username used for JMX authentication. Optional.

    -pass <password> 

        Password used for JMX authentication. Optional. Mandatory when a username has been set

    -b <n>

        Number of lines to read before current (if available) when running showlog. 

EXAMPLE

    java -cp z.jar com.zfabrik.launch.Manage showlog -url host:7777 -user admin -pass admin 

</code></pre>