Project

General

Profile

Reduce Repo Traffic - Pre Deploy-Build and Offline Execution » History » Version 2

Henning Blohm, 30.12.2021 21:53

1 1 Henning Blohm
h1. Reduce Repo Traffic - Pre Deploy-Build and Offline Execution
2
3
As a principle, a z2 installation is made to check for updates at startup and on-demand. After installation of the core however initial repository access may overwhelm the repository infrastructure leading to reduced scale out performance. In other cases repository access from a live installation is not desired.
4
5
h2. Example Problems
6
7
* You have large Git Repos and initial cloning by the system takes long
8
* You use the Maven Component Repository and do not want live-systems to download from a public maven repo
9
* You want to use z2 in a docker image for scale out in e.g. Kubernetes and want to avoid lengthy initialization times
10
* You want to always run live systems in offline mode, and rather than having z2 fetch updates, build a complete z2 installation for distribution
11
12
All of these cases are made up from a mix of the following concerns:
13
14
a) Initialization time and network usage of a fresh core instance
15
b) Avoiding access to remote systems at runtime
16
17
h2. Reduce Initialization Time and Network Usage of a Clean Z2-Home Installation
18
19
h3. Pre-Deploy Build
20
21
Assuming you want to quickly scale out and start up instances without requiring downloading of resources it is simplest to actually build a "warm" z2-core by retrieving all component resources and possibly even prebuilding all Java components. For example, this could be the last step of preparation of a docker image for distribution.
22
23
See below Appendix 1 for a code sample.
24
25
Pros: 
26
* Simple to implement and fits nicely to a docker setup
27 2 Henning Blohm
* When using the system in online mode, this combines the best of both worlds:
28
** Quick initialization due to the pre-deployment build
29
** Automatic updates by simple restarts (of e.g. docker containers)
30 1 Henning Blohm
31
Cons: 
32
* Image could still be large, if repos hold a lot of unused content/history
33
34
h3. Hub CR
35
36
Another way of generally reducing and optimizing resource retrieval for a larger distributed system is to use the Hub Component Repository (see [[How_to_use_the_hub_cr]]). In essence, your system will have different target states depending on whether it is used on a development scenario or for production. In the latter case, the Hub CR would be used and otherwise regular source code repositories would be used - controlled by e.g. environment vars. This approach can be combined with the Pre-Deploy Build.
37
38
Pros:
39
* Only what is needed is retrieved - optionally pre-compiled
40
41
Cons: 
42
* Non-Trivial repo or target state configuration depending on scenarios
43
* Requires a Hub setup and installation. Another central piece of infrastructure.
44
45
h3. SFTPSSH Repo
46
47
Finally, another approach would be to use the [[SFTPSSH_Component_Repository]] (that has not been implemented as of now).
48
49
Pros:
50
* Nodes only download what is needed - in pre-packaged form
51
* Simple to setup and maintain. No active infrastructure besides some SFTP location.
52
53
Cons: 
54
* Non-Trivial repo or target state configuration depending on scenarios
55 2 Henning Blohm
56
h2. Avoiding Remote Access from Live Nodes
57
58
In combination with the pre-deployment approach from above, configuring the offline mode would make sure that nodes do not attempt to access remote repositories.
59
60
This is particularly interesting when using the [[V29-Reference#Maven-Repository-Support|Maven Component Repository]] and you want to avoid unmanaged access to remote source. In that case, anyway, it would be advisable to run a local artifact repository such as Artifactory.