z2-Environment

h2. Install CDH4 from a preconfigured repository

This site provides a pre-configured one-check out user space installation of Cloudera's CDH4 Hadoop and HBase distributions. This page explains how to install it on your machine - which is really, really simple compared to normally suggested Hadoop installation procedures.

*Note #1:* This will only work on Linux or Mac OS. A machine with 8GB of RAM should be sufficient.

*Note #2:* The repository also contains an Eclipse project file and has Eclipse launchers for most functions required.

*Note #3:* This setup is for educational purposes only. It has no security requirements and there is no one taking any liability on anything regardings its use.

In short there are the followings steps:

# Clone the repository

# Adapt your local environment

# Format HDFS

# Start and stop

h3. Clone the repository

The pre-configured distribution is stored in the repository "z2-samples-cdh4-base":http://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-cdh4-base. We assume you install everything (including an Eclipse workspace - if you run the samples) in *install*.

<pre><code class="ruby">

cd install

git clone -b master http://git.z2-environment.net/z2-samples.cdh4-base

</code></pre>

h3. Adapt your environment

Before you can run anything two customizations are needed:

h4. Set important environment variables

There is a shell script "env.sh":http://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-cdh4-base/revisions/master/entry/env.sh that you should open and change as described. At the time of this writing it is required that you define your JAVA_HOME (please do, even if set elsewhere already) and the NOSQL_HOME, which is the absolute path to the folder that has the *env.sh* file. This script is called from elsewhere and having absolute paths in here is a safe way to make sure things will be found.

If you are a Subversion user, note the following: In order to run embedded z2 M/R jobs, the *env.sh* identifies a z2 Home location next to the CDH4 checkout (see above) either in the folder *core* or in the folder *z2-base.core*. This is due to the fact that the Subclipse plugin of Eclipse uses the project name ("core") as check out folder while the command line client uses the folder name ("z2-base.core"). So please make sure, you have a z2 Home in exactly one of these locations (as said, depending on the Subversion client you use) or customize *env.sh* accordingly to set a good Z2_HOME variable.

h4. Enable password-less SSH

Currently this is still required to have the start / stop scripts running. This requirement may be dropped in the future. 

If you have not created a unique key for SSH or have no idea what that is, run

<pre><code class="ruby">

ssh-keygen

</code></pre>

(just keep hitting enter). Next copy that key over to the machine you want to log on to without password, i.e. localhost in this case (you can get ssh-copy-id from "here":https://gist.github.com/2575680 if you don't have it):

<pre><code class="ruby">

ssh-copy-id <your user name>@localhost

</code></pre>

If this fails because your SSH works differently, or ssh will refuse to log on without password please "ask the internet". Sorry.

All that matters is that in the end

<pre><code class="ruby">

ssh <your user name>@localhost

</code></pre>

(substituting <your user name> with your actual user name of course) works without asking for a password.

In addition it may help to run <code>ssh <your user name>@0.0.0.0</code> as well to make sure the host key for that (localhost) address has been verified.

h3. Formatting HDFS

Finally, the last step before you can start up, is to prepare the local node to store data. This is done by running the *format_dfs.sh* script. Alternatively you can use the Eclipse launcher of the same name.

This should complete without any questions or errors. Otherwise please verify your settings above.

h3. Start and Stop

Depending on your sample requirements, you can start Hadoop (HDFS, Yarn, the History Server) or HBase (including all the Hadoop services) using the *start_hadoop.sh* script (or launcher) or the *start_hbase.sh* script (or launcher) respectively. Similarly you can stop everything with the stop scripts.

When you have started, after a short while, using *jps* on the command line, you should see the following Java processes (and possibly others of course):

<pre><code class="ruby">

DataNode

NodeManager

NameNode

SecondaryNameNode

JobHistoryServer

ResourceManager

</code></pre>

and additionally those, if you run HBase:

<pre><code class="ruby">

HRegionServer

HQuorumPeer

HMaster

</code></pre>

There is lots of other scripts in the distribution that you can use to start or stop single components. If you do however, please run (in the shell):

<pre><code class="ruby">

. ./env.sh

</code></pre>

(note the leading period)

If you ran the start script and it returned, here is some URLs you should check to verify everything is looking good:

* Try to reach the Namenode at http://localhost:50070

* Try to reach the Yarn Nodemanager at http://localhost:8088

and, if you are running HBase:

* Try to reach the HBase Master at http://localhost:60010

*Note:* If you notice that you cannot restart or that HBase is not stopping correctly, that is most likely exactly the case. Sometimes HBase processes do not stop. To make sure there is no process left, use *jps* from the command line and kill remaining processes.