z2-Environment

h1. How to configure and use the Gateway module

The Gateway module implements a "zero-downtime-upgrade" feature in Z2. Specifically, it uses the worker process management of Z2 in conjunction with an intermediate reverse proxy style Web handler to implement the following feature: 

Upgrading a stateful Web application, i.e. a Web application that stores user data in its HTTP session typically implies downtime, and if the session state is not serializable and persisted during the upgrade, it does additionally imply that user state gets lost and typically that users need to log on again.

Using the Gateway, running sessions may be preserved and worker resources may still be assigned on the current software revision for as long as there are running sessions during a node upgrade and until all sessions have been terminated. The typical application of this feature is to roll out functional and user interface corrections without interrupting users. Users can switch over to post-upgrade software by terminating their session (e.g. via a log out) and starting a new one (e.g. by logging in again).

*Note:*

* This feature is relatively new and before you use it in production, you should have carefully tested your scenario.

* There are natural limitations to this feature. Upgrades that change the structure or semantics of persisted data or other resources that are shared across nodes cannot be handled this way.

h2. A sample on how to use Gateway

A sample can be found in the "z2-samples.gateway":http://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-gateway repository. Install as follows:

{{include(Install_sample_prefix)}}

Check out the sample

<pre><code class="bash">

git clone -b v2.7 http://git.z2-environment.net/z2-samples.gateway

</code></pre>

{{include(Install_sample_postfix)}}

Check out the *environment* module (that typically holds all configuration regarding the "environment" of the application - such as database connection and Web server configuration). This module is holding a Gateway configuration as described below. After re-starting your Z2 installation, try the following:

1. Open a browser and navigate to http://localhost:8080/z_gateway. Use (by default) user "z*" with password "z".

You should see this:

!gateway1.png!

2. Open another browser window and navigate to http://localhost:8080/adm (same user). Choose the group "Workers" and update.

Check for the current worker process and their state. You should see something like this:

!admin1.png!

Both web applications application create a HTTP session. Now go back to the Gateway user interface and press "Detach environment/webWorker@0 and sync". On your console you will see that another worker process started (called environment/webWorker@1) and if you update the worker list in the admin interface you should see something like this:

!admin2.png!

That is: One worker was _detached_ while another one is in state _started_.

If you now refresh the Gateway user interface you will see that it is still served from *environment/webWorker@0*. Press "log off". You should find that it now switched to *environment/webWorker@1*.

Let's recap: Now we have a session on *environment/webWorker@0* via the admin user interface and one one *environment/webWorker@1* via the Gateway user interface.

3. Detach *environment/webWorker@1* by clicking "Detach environment/webWorker@1 and sync" on the Gateway user interface.

When checking the worker list you will now see something like this: 

!admin3.png!

Remember that it is the Gateway user interface that keeps *environment/webWorker@1* alive. If you click on "log off" the session will be terminated, worker *environment/webWorker@1* served its purpose and will terminate, and the Gateway user interface will be served from *environment/webWorker@2*. Checking the worker list you should see something like this:

!admin4.png!

Finally, if you wait 30 minutes until the session of the admin user interface has expired, also *environment/webWorker@0* will terminate.

h2. Enabling Gateway

The sample configuration in "z2-samples.gateway":http://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-gateway has a pre-configured Gateway setup. Here is how this differs from a regular non-Gateway setup:

h3. The Gateway Web server

As explained in the next section, when using Gateway, Z2 runs a Web server on the Home process in addition to Web servers in Web workers. All requests that go normally directly to the Web worker process will instead be handled on the Home process and forwarded appropriately to a Web worker process. 

In order to enable this Web server, uncomment the system state participation in "environment/gateway/z.properties":http://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-gateway/revisions/master/entry/environment/gateway/z.properties as done in the sample.

The Gateway Web server is configured in "environment/gateway/jetty.xml":http://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-gateway/revisions/master/entry/environment/gateway/jetty.xml. This is a regular Jetty configuration. Instead of being an ordinary Java Web container all requests are handled by a special Gateway Jetty handler. 

The only part of this configuration that could be of interest to be changed is the normal connector settings (ports and concurrency) - just as you normally would tune your Jetty Web server - and the name of the worker process that is to receive the forwarded requests - the _Web worker process_. The latter is defined in this section:

<pre><code class="xml">

<Set name="handler">

  <New id="Handlers" class="org.eclipse.jetty.server.handler.HandlerCollection">

    <Set name="handlers">

      <Array type="org.eclipse.jetty.server.Handler">

        <Item>

          <New id="Contexts" class="org.eclipse.jetty.server.handler.ContextHandlerCollection"/>

        </Item>

        <Item> 

          <Call class="com.zfabrik.gateway.GatewayFactory" name="getHandler">

            <Set name="workerProcessComponentName">environment/webWorker</Set>

          </Call>

        </Item>

      </Array>

    </Set>

  </New>

</Set>

</code></pre>

Unless you want to change the name of the target worker process, you do not need to touch Gateway's jetty.xml.

h3. The Web worker Jetty configuration

When enabling Gateway, we want Web workers to not listen on the same port (default 8080) as Gateway. With Gateway configured, we want Web workers to only process Gateway requests - which are slightly different than normal HTTP requests due to some extra metadata passed on the proxy Web server running in the Home process.

In order to facilitate that, the Jetty configuration for the instance running on the Web worker is modified to use the "GatewayServer":https://redmine.z2-environment.net/projects/z2-base/repository/base/revisions/master/entry/com.zfabrik.gateway/java/src.api/com/zfabrik/gateway/worker/GatewayServer.java rather than the normal Jetty Server implementation class. This *GatewayServer* class computes the internal localhost port for proxy to web worker communication and communicates additional metadata on session usage to the proxy server of the Home process.

In terms of configuration we change the web server component descriptor to include "gateway.xml":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-gateway/revisions/master/entry/environment/webServer/gateway.xml and "gateway-http.xml":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-gateway/revisions/master/entry/environment/webServer/gateway-http.xml rather than *jetty.xml* and *jetty-http.xml*

The main difference is a) the use of 

<pre><code class="xml">

<Configure id="Server" class="com.zfabrik.gateway.worker.GatewayServer">

...

</Configure>

</code></pre>

(gateway.xml) and b) the use of a Gateway connection factory (gateway-http.xml):

<pre><code class="xml">

  <Call name="addConnector">

    <Arg>

      <New id="httpConnector" class="org.eclipse.jetty.server.ServerConnector">

//...

        <Arg name="factories">

          <Array type="org.eclipse.jetty.server.ConnectionFactory">

            <Item>

              <!--  Use Gateway connection factory! -->

              <New class="com.zfabrik.gateway.worker.GatewayConnectionFactory">

                <Arg name="config"><Ref refid="httpConfig" /></Arg>

              </New>

            </Item>

          </Array>

        </Arg>

//...

	<!-- gateway specific settings on worker node -->

        <Set name="host">localhost</Set>

        <!-- Use Gateway port setting!s -->

        <Set name="port"><SystemProperty name="com.zfabrik.gateway.jetty.http.host"/></Set>

      </New>

    </Arg>

  </Call>

</code></pre>

The purpose of the "GatewayConnectionFactory":https://redmine.z2-environment.net/projects/z2-base/repository/base/revisions/master/entry/com.zfabrik.gateway/java/src.api/com/zfabrik/gateway/worker/GatewayConnectionFactory.java is to use Gateway specific connection implementations "GatewayConnection":https://redmine.z2-environment.net/projects/z2-base/repository/base/revisions/master/entry/com.zfabrik.gateway/java/src.api/com/zfabrik/gateway/worker/GatewayConnection.java that allow us to add the additional metadata provided from the GatewayServer class (above).

h3. Running the Gateway user interface

Finally, in order to actually trigger "Detach & Sync", you should enable the Gateway user interface by adding a dependency to *com.zfabrik.gateway/web* to "environment/webWorkerUp":http://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-gateway/revisions/master/entry/environment/webWorkerUp.properties.

h2. How does Gateway work

Gateway consists of two parts:

h3. The reverse proxy Gateway handler

When using Gateway, there is a Jetty Web server running on the Home process. This Web server maintains a mapping of sessions to Web worker nodes. Any newly created session will be assigned to the latest running Web worker process. All request will be routed to the Web worker process assigned to their session or, if there is no session id on the request, to the latest Web worker process.

On every Web worker process there is another Web server that accepts requests forwarded from the Home process and subsequently treats them as if they were ordinary HTTP requests in the first place.

h3. Detached worker processes

Apart from being stopped, starting, started, or stopping, Z2 worker processes can also be in _detached_ state. When a worker process is detached, it will terminate itself unless there is still unfinished work. Open Web application sessions is one type of such unfinished work. 

In other words, if a worker has no open sessions and is being detached, it will terminate itself right away. 

Finally, when the Home process runs a synchronization, detached worker processes do not count as running processes. That is, when the synchronization finds that there is only detached worker process instances of a worker process component, it will start another one.

That is why the Gateway user interface will not only detach a worker process but also trigger a synchronization. Here is a diagram illustrating the flow from detach and sync to worker process termination:

!steady_workers1.png!

h2. Ports of worker processes

Worker processes open ports to receive Web requests but also for JVM debugging and JMX access. These ports are configured in the worker process component definition. See for example "environment/webWorker":http://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-gateway/revisions/master/entry/environment/webWorker.properties. These ports are actually base port numbers and the real port to be used is computed from a _generation_ number that is also part of the worker process instance name that you saw above. I.e. the instance name *environment/webWorker@2* is generation 2 of the worker process component *environment/webWorker*.

The port configurations in worker process components are:

|_. Property |_. Description |_. Typical value |

|worker.debug.port| Base port of the JVM debug port to use. If the home process has remote debugging enabled, worker processes will also be configured for remote debugging.|5100 in environment/webWorker|

|worker.jmx.port|Base port of the remote JMX access. Will be ignored if no JMX access configured|7800 for environment/webWorker|

|com.zfabrik.gateway.port|Base port of the web worker side of Gateway. Will only listen to localhost and only be used, if Gateway is configured|8800 in environment/webWorker|