z2-Environment

h1. Reduce Repo Traffic - Pre Deploy-Build and Offline Execution

As a principle, a z2 installation is made to check for updates at startup and on-demand. After installation of the core however initial repository access may overwhelm the repository infrastructure leading to reduced scale out performance. In other cases repository access from a live installation is not desired.

h2. Example Problems

* You have large Git Repos and initial cloning by the system takes long

* You use the Maven Component Repository and do not want live-systems to download from a public maven repo

* You want to use z2 in a docker image for scale out in e.g. Kubernetes and want to avoid lengthy initialization times

* You want to always run live systems in offline mode, and rather than having z2 fetch updates, build a complete z2 installation for distribution

All of these cases are made up from a mix of the following concerns:

# Initialization time and network usage of a fresh core instance

# Avoiding access to remote systems at runtime

h2. Reduce Initialization Time and Network Usage of a Clean Z2-Home Installation

h3. Pre-Deploy Build

Assuming you want to quickly scale out and start up instances without requiring downloading of resources it is simplest to actually build a "warm" z2-core by retrieving all component resources and possibly even prebuilding all Java components. For example, this could be the last step of preparation of a docker image for distribution.

See below Appendix 1 for a code sample.

h4. Pros: 

* Simple to implement and fits nicely to a docker setup

* When using the system in online mode, this combines the best of both worlds:

** Quick initialization due to the pre-deployment build

** Automatic updates by simple restarts (of e.g. docker containers)

h4. Cons: 

* Image could still be large, if repos hold a lot of unused content/history

h3. Hub CR

Another way of generally reducing and optimizing resource retrieval for a larger distributed system is to use the Hub Component Repository (see [[How_to_use_the_hub_cr]]). In essence, your system will have different target states depending on whether it is used on a development scenario or for production. In the latter case, the Hub CR would be used and otherwise regular source code repositories would be used - controlled by e.g. environment vars. This approach can be combined with the Pre-Deploy Build.

h4. Pros:

* Only what is needed is retrieved - optionally pre-compiled

h4. Cons: 

* Non-Trivial repo or target state configuration depending on scenarios

* Requires a Hub setup and installation. Another central piece of infrastructure.

h3. SFTPSSH Repo

Finally, another approach would be to use the [[SFTPSSH_Component_Repository]] (that has not been implemented as of now).

h4. Pros:

* Nodes only download what is needed - in pre-packaged form

* Simple to setup and maintain. No active infrastructure besides some SFTP location.

h4. Cons: 

* Non-Trivial repo or target state configuration depending on scenarios

h2. Avoiding Remote Access from Live Nodes

In combination with the pre-deployment approach from above, configuring the [[V29-Reference#Using-the-Offline-Mode|offline mode]] would make sure that nodes do not attempt to access remote repositories.

This is particularly interesting when using the [[V29-Reference#Maven-Repository-Support|Maven Component Repository]] and you want to avoid unmanaged access to remote source. In that case, anyway, it would be advisable to run a local artifact repository such as Artifactory.

h2. Appendix 1: Pre-Deployment Build

A simple [[V29-Reference#Main-Program-Components-core|main program]] component to prepare all component resources and to optionally build all Java components would look like this:

<pre><code class="java">

/**

 * A simple script to prepare all repos for offline operation by retrieving all components.

 * 

 * Pass "compile" as command line parameter to also pre-compile all Java components.

 */

public class Preparer {

	private final static Logger LOG = Logger.getLogger(Preparer.class.getName());

	public static void main(String[] args) throws Exception {

		boolean compile = Arrays.stream(args).filter("compile"::equals).findAny().isPresent();

		if (compile) {

			LOG.info("Compiling Java components");

		}

		// find all components and retrieve

		IComponentsManager cm = IComponentsManager.INSTANCE;

		IResourceLookup    cl = IComponentsLookup.INSTANCE;

		int retrieved = 0;

		int compiled  = 0;

		for (String cn : cm.findComponents(X.val(true))) {

			try {

				if (cm.retrieve(cn)!=null) {

					LOG.log(Level.INFO,"Retrieved {0}",cn);

					retrieved++;

					if (compile && IJavaComponent.TYPE.equals(cm.getComponent(cn).getType())) { 

						cl.lookup(cn, IJavaComponent.class);

						LOG.log(Level.INFO,"Compiled {0}",cn);

						compiled++;

					}

				}

			} catch (Exception e) {

				LOG.log(Level.SEVERE,"Failed to handle "+cn, e);

			}

		}

		LOG.log(Level.INFO,"Retrieved resources of {0} components",retrieved);

		if (compile) {

			LOG.log(Level.INFO,"Compiled {0} Java components",compiled);

		}

	}

}

</code></pre>

Provided this is implemented with component @tools/preDeployment@, a command line executed in @$Z2_HOME/bin@ would be 

<pre><code class="bash">

java -Dcom.zfabrik.offline=false -DcomponentName=tools/preDeployment -cp z.jar com.zfabrik.launch.MainRunner compile

</code></pre>