How to configure and use the Gateway module¶

The Gateway module implements a "zero-downtime-upgrade" feature in Z2. Specifically, it uses the worker process management of Z2 in conjunction with an intermediate reverse proxy style Web handler to implement the following feature:

Upgrading a stateful Web application, i.e. a Web application that stores user data in its HTTP session typically implies downtime, and if the session state is not serializable and persisted during the upgrade, it does additionally imply that user state gets lost and typically that users need to log on again.

Using the Gateway, running sessions may be preserved and worker resources may still be assigned on the current software revision for as long as there are running sessions during a node upgrade and until all sessions have been terminated. The typical application of this feature is to roll out functional and user interface corrections without interrupting users. Users can switch over to post-upgrade software by terminating their session (e.g. via a log out) and starting a new one (e.g. by logging in again).

Note:

• This feature is relatively new and before you use it in production, you should have carefully tested your scenario.
• There are natural limitations to this feature. Upgrades that change the structure or semantics of persisted data or other resources that are shared across nodes cannot be handled this way.

A sample on how to use Gateway¶

A sample can be found in the z2-samples.gateway repository. Install as follows:

mkdir install
cd install

On Mac-OS or Linux run:

wget http://download.z2-environment.net/z2/z2-base-v2.10.zip
unzip z2-base-v2.10.zip

On Windows download the archive and unpack using the Windows explorer. Make sure to unpack into the installation folder previously created.

This will create a folder z2-base.core that contains the complete version 2.9 z2-base installation.

Check out the sample

git clone -b v2.10 https://www.z2-environment.net/git/z2-samples.gateway

On Mac OS or Linux run:

cd z2-base.core/bin
./gui.sh

On Windows run:

cd z2-base.core\bin
gui.bat

(In order to check that z2 is up, when you see "Completed home process initialization", try http://localhost:8080/adm with user "z*" and password "z".)

Check out the environment module (that typically holds all configuration regarding the "environment" of the application - such as database connection and Web server configuration). This module is holding a Gateway configuration as described below. After re-starting your Z2 installation, try the following:

1. Open a browser and navigate to http://localhost:8080/z_gateway . Use (by default) user "z*" with password "z".

You should see this:

2. Open another browser window and navigate to http://localhost:8080/adm (same user). Choose the group "Workers" and update.

Check for the current worker process and their state. You should see something like this:

Both web applications application create a HTTP session. Now go back to the Gateway user interface and press "Detach environment/webWorker@0 and sync". On your console you will see that another worker process started (called environment/webWorker@1) and if you update the worker list in the admin interface you should see something like this:

That is: One worker was detached while another one is in state started.

If you now refresh the Gateway user interface you will see that it is still served from environment/webWorker@0. Press "log off". You should find that it now switched to environment/webWorker@1.

Let's recap: Now we have a session on environment/webWorker@0 via the admin user interface and one one environment/webWorker@1 via the Gateway user interface.

3. Detach environment/webWorker@1 by clicking "Detach environment/webWorker@1 and sync" on the Gateway user interface.

When checking the worker list you will now see something like this:

Remember that it is the Gateway user interface that keeps environment/webWorker@1 alive. If you click on "log off" the session will be terminated, worker environment/webWorker@1 served its purpose and will terminate, and the Gateway user interface will be served from environment/webWorker@2. Checking the worker list you should see something like this:

Finally, if you wait 30 minutes until the session of the admin user interface has expired, also environment/webWorker@0 will terminate.

Enabling Gateway¶

The sample configuration in z2-samples.gateway has a pre-configured Gateway setup. Here is how this differs from a regular non-Gateway setup:

The Gateway Web server¶

As explained in the next section, when using Gateway, Z2 runs a Web server on the Home process in addition to Web servers in Web workers. All requests that go normally directly to the Web worker process will instead be handled on the Home process and forwarded appropriately to a Web worker process.

In order to enable this Web server, uncomment the system state participation in environment/gateway/z.properties as done in the sample.

The Gateway Web server is configured in environment/gateway/jetty.xml. This is a regular Jetty configuration. Instead of being an ordinary Java Web container all requests are handled by a special Gateway Jetty handler.

The only part of this configuration that could be of interest to be changed is the normal connector settings (ports and concurrency) - just as you normally would tune your Jetty Web server - and the name of the worker process that is to receive the forwarded requests - the Web worker process. The latter is defined in this section:

<Set name="handler">
  <New id="Handlers" class="org.eclipse.jetty.server.handler.HandlerCollection">
    <Set name="handlers">
      <Array type="org.eclipse.jetty.server.Handler">
        <Item>
          <New id="Contexts" class="org.eclipse.jetty.server.handler.ContextHandlerCollection"/>
        </Item>
        <Item> 
          <Call class="com.zfabrik.gateway.GatewayFactory" name="getHandler">
            <Set name="workerProcessComponentName">environment/webWorker</Set>
          </Call>
        </Item>
      </Array>
    </Set>
  </New>
</Set>

Unless you want to change the name of the target worker process, you do not need to touch Gateway's jetty.xml.

The Web worker Jetty configuration¶

When enabling Gateway, we want Web workers to not listen on the same port (default 8080) as Gateway. With Gateway configured, we want Web workers to only process Gateway requests - which are slightly different than normal HTTP requests due to some extra metadata passed on the proxy Web server running in the Home process.

In order to facilitate that, the Jetty configuration for the instance running on the Web worker is modified to use the GatewayServer rather than the normal Jetty Server implementation class. This GatewayServer class computes the internal localhost port for proxy to web worker communication and communicates additional metadata on session usage to the proxy server of the Home process.

In terms of configuration we change the web server component descriptor to include z2-gateway.xml and z2-gateway-http.xml rather than jetty.xml and jetty-http.xml

The main difference is a) the use of

<Configure id="Server" class="com.zfabrik.gateway.worker.GatewayServer">
...
</Configure>

(z2-gateway.xml) and b) the use of a Gateway connection factory (z2-gateway-http.xml):

 <Call name="addConnector">
    <Arg>
      <New id="httpConnector" class="org.eclipse.jetty.server.ServerConnector">

//...

        <Arg name="factories">
          <Array type="org.eclipse.jetty.server.ConnectionFactory">
            <Item>

              <!--  Use Gateway connection factory! -->
              <New class="com.zfabrik.gateway.worker.GatewayConnectionFactory">
                <Arg name="config"><Ref refid="httpConfig" /></Arg>
              </New>

            </Item>
          </Array>
        </Arg>

//...

    <!-- gateway specific settings on worker node -->
        <Set name="host">localhost</Set>
        <!-- Use Gateway port setting!s -->
        <Set name="port"><SystemProperty name="com.zfabrik.gateway.jetty.http.host"/></Set>

      </New>
    </Arg>
  </Call>

The purpose of the GatewayConnectionFactory is to use Gateway specific connection implementations GatewayConnection that allow us to add the additional metadata provided from the GatewayServer class (above).

Running the Gateway user interface¶

Finally, in order to actually trigger "Detach & Sync", you should enable the Gateway user interface by adding a dependency to com.zfabrik.gateway/web to environment/webWorkerUp.

How does Gateway work¶

Gateway consists of two parts:

The reverse proxy Gateway handler¶

When using Gateway, there is a Jetty Web server running on the Home process. This Web server maintains a mapping of sessions to Web worker nodes. Any newly created session will be assigned to the latest running Web worker process. All request will be routed to the Web worker process assigned to their session or, if there is no session id on the request, to the latest Web worker process.

On every Web worker process there is another Web server that accepts requests forwarded from the Home process and subsequently treats them as if they were ordinary HTTP requests in the first place.

Detached worker processes¶

Apart from being stopped, starting, started, or stopping, Z2 worker processes can also be in detached state. When a worker process is detached, it will terminate itself unless there is still unfinished work. Open Web application sessions is one type of such unfinished work.

In other words, if a worker has no open sessions and is being detached, it will terminate itself right away.

Finally, when the Home process runs a synchronization, detached worker processes do not count as running processes. That is, when the synchronization finds that there is only detached worker process instances of a worker process component, it will start another one.

That is why the Gateway user interface will not only detach a worker process but also trigger a synchronization. Here is a diagram illustrating the flow from detach and sync to worker process termination:

Ports of worker processes¶

Worker processes open ports to receive Web requests but also for JVM debugging and JMX access. These ports are configured in the worker process component definition. See for example environment/webWorker. These ports are actually base port numbers and the real port to be used is computed from a generation number that is also part of the worker process instance name that you saw above. I.e. the instance name environment/webWorker@2 is generation 2 of the worker process component environment/webWorker.

The port configurations in worker process components are:

Property	Description	Typical value
worker.debug.port	Base port of the JVM debug port to use. If the home process has remote debugging enabled, worker processes will also be configured for remote debugging.	5100 in environment/webWorker
worker.jmx.port	Base port of the remote JMX access. Will be ignored if no JMX access configured	7800 for environment/webWorker
com.zfabrik.gateway.port	Base port of the web worker side of Gateway. Will only listen to localhost and only be used, if Gateway is configured	8800 in environment/webWorker