Reduce Repo Traffic - Pre Deploy-Build and Offline Execution¶

As a principle, a z2 installation is made to check for updates at startup and on-demand. After installation of the core however initial repository access may overwhelm the repository infrastructure leading to reduced scale out performance. In other cases repository access from a live installation is not desired.

Example Problems¶

• You have large Git Repos and initial cloning by the system takes long
• You use the Maven Component Repository and do not want live-systems to download from a public maven repo
• You want to use z2 in a docker image for scale out in e.g. Kubernetes and want to avoid lengthy initialization times
• You want to always run live systems in offline mode, and rather than having z2 fetch updates, build a complete z2 installation for distribution

All of these cases are made up from a mix of the following concerns:

1. Initialization time and network usage of a fresh core instance
2. Avoiding access to remote systems at runtime

Reduce Initialization Time and Network Usage of a Clean Z2-Home Installation¶

Pre-Deploy Build¶

Assuming you want to quickly scale out and start up instances without requiring downloading of resources it is simplest to actually build a "warm" z2-core by retrieving all component resources and possibly even prebuilding all Java components. For example, this could be the last step of preparation of a docker image for distribution.

See below Appendix 1 for a code sample.

Pros:¶

• Simple to implement and fits nicely to a docker setup
• When using the system in online mode, this combines the best of both worlds:
  • Quick initialization due to the pre-deployment build
  • Automatic updates by simple restarts (of e.g. docker containers)

Cons:¶

• Image could still be large, if repos hold a lot of unused content/history

Hub CR¶

Another way of generally reducing and optimizing resource retrieval for a larger distributed system is to use the Hub Component Repository (see How_to_use_the_hub_cr). In essence, your system will have different target states depending on whether it is used on a development scenario or for production. In the latter case, the Hub CR would be used and otherwise regular source code repositories would be used - controlled by e.g. environment vars. This approach can be combined with the Pre-Deploy Build.

Pros:¶

• Only what is needed is retrieved - optionally pre-compiled

Cons:¶

• Non-Trivial repo or target state configuration depending on scenarios
• Requires a Hub setup and installation. Another central piece of infrastructure.

SFTPSSH Repo¶

Finally, another approach would be to use the SFTPSSH_Component_Repository (that has not been implemented as of now).

Pros:¶

• Nodes only download what is needed - in pre-packaged form
• Simple to setup and maintain. No active infrastructure besides some SFTP location.

Cons:¶

• Non-Trivial repo or target state configuration depending on scenarios

Avoiding Remote Access from Live Nodes¶

In combination with the pre-deployment approach from above, configuring the offline mode would make sure that nodes do not attempt to access remote repositories.

This is particularly interesting when using the Maven Component Repository and you want to avoid unmanaged access to remote source. In that case, anyway, it would be advisable to run a local artifact repository such as Artifactory.

Appendix 1: Pre-Deployment Build¶

A simple main program component to prepare all component resources and to optionally build all Java components would look like this:

/**
 * A simple script to prepare all repos for offline operation by retrieving all components.
 * 
 * Pass "compile" as command line parameter to also pre-compile all Java components.
 */
public class Preparer {
    private final static Logger LOG = Logger.getLogger(Preparer.class.getName());

    public static void main(String[] args) throws Exception {
        boolean compile = Arrays.stream(args).filter("compile"::equals).findAny().isPresent();
        if (compile) {
            LOG.info("Compiling Java components");
        }
        // find all components and retrieve
        IComponentsManager cm = IComponentsManager.INSTANCE;
        IResourceLookup    cl = IComponentsLookup.INSTANCE;
        int retrieved = 0;
        int compiled  = 0;
        for (String cn : cm.findComponents(X.val(true))) {
            try {
                if (cm.retrieve(cn)!=null) {
                    LOG.log(Level.INFO,"Retrieved {0}",cn);
                    retrieved++;
                    if (compile && IJavaComponent.TYPE.equals(cm.getComponent(cn).getType())) { 
                        cl.lookup(cn, IJavaComponent.class);
                        LOG.log(Level.INFO,"Compiled {0}",cn);
                        compiled++;
                    }
                }
            } catch (Exception e) {
                LOG.log(Level.SEVERE,"Failed to handle "+cn, e);
            }
        }
        LOG.log(Level.INFO,"Retrieved resources of {0} components",retrieved);
        if (compile) {
            LOG.log(Level.INFO,"Compiled {0} Java components",compiled);
        }
    }

}

Provided this is implemented with component tools/preDeployment, a command line executed in $Z2_HOME/bin would be

java -Dcom.zfabrik.offline=false -DcomponentName=tools/preDeployment -cp z.jar com.zfabrik.launch.MainRunner compile