Project

General

Profile

Install prepacked CDH4 » History » Version 10

Henning Blohm, 20.09.2012 16:23

1 1 Henning Blohm
h2. Install CDH4 from a preconfigured repository
2
3
This site provides a pre-configured one-ckeck out user space installation of Cloudera's CDH4 Hadoop and HBase distributions. This page explains how to install it on your machine - which is really, really simple compared to normally suggested Hadoop installation procedures.
4
5
*Note #1:* This will only work on Linux or Mac OS
6
7
*Note #2:* The repository also contains an Eclipse project file and has Eclipse launchers for most functions required.
8
9 9 Henning Blohm
*Note #3:* This setup is for educational purposes only. It has no security requirements and there is no one taking any liability on anything regardings its use.
10
11 5 Henning Blohm
In short there are the followings steps:
12 1 Henning Blohm
13
# Clone the repository
14
# Adapt your local environment
15 5 Henning Blohm
# Format HDFS
16
# Start and stop
17 1 Henning Blohm
18
h3. Clone the repository
19
20
The pre-configured distribution is stored in the repository "z2-samples-cdh4-base":http://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-cdh4-base. We assume you install everything (including an Eclipse workspace - if you run the samples) in *install*.
21
22 3 Henning Blohm
<pre><code class="ruby">
23 1 Henning Blohm
cd install
24
git clone -b http://git.z2-environment.net/z2-samples.cdh4-base
25
</code></pre>
26 2 Henning Blohm
27
h3. Adapt your environment
28
29
Before you can run anything really there are two customizations needed:
30
31
h4. Set important environment variables
32
33 8 Henning Blohm
There is a shell script "env.sh":http://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-cdh4-base/revisions/master/entry/env.sh that you should open and change as described. At the time of this writing it is required that you define your JAVA_HOME (please do, even if set elsewhere already) and the NOSQL_HOME, which is the absolute path to the folder that has the *env.sh* file. This script is called from elsewhere and having absolute paths in here is a safe way to make sure things will be found.
34 2 Henning Blohm
35
h4. Enable password-less SSH
36
37
Currently this is still required to have the start / stop scripts running. This requirement may be dropped in the future. 
38
39 1 Henning Blohm
If you have not created a unique key for SSH or have no idea what that is, run
40 2 Henning Blohm
41 3 Henning Blohm
<pre><code class="ruby">
42 2 Henning Blohm
ssh-keygen
43
</code></pre>
44 1 Henning Blohm
45 2 Henning Blohm
(just keep hitting enter). Next copy that key over to the machine you want to log on to without password, i.e. localhost in this case:
46
47 3 Henning Blohm
<pre><code class="ruby">
48 2 Henning Blohm
ssh-copy-id <your user name>@localhost
49
</code></pre>
50
51 1 Henning Blohm
If this fails because your SSH works differently, or ssh will refuse to log on without password please "ask the internet". Sorry.
52 2 Henning Blohm
53
All that matters is that in the end
54
55 3 Henning Blohm
<pre><code class="ruby">
56 1 Henning Blohm
ssh <your user name>@localhost
57
</code></pre>
58
59
(substituting <your user name> with your actual user name of course) works without asking for a password.
60 10 Henning Blohm
In addition it may help to run <code>ssh hb@0.0.0.0</code> as well to make sure the host key for that (localhost) address has been verified.
61 3 Henning Blohm
62
h3. Formatting HDFS
63
64
Finally, the last step before you can start up, is to prepare the local node to store data. This is done by running the *format_dfs.sh* script. Alternatively you can use the Eclipse launcher of the same name.
65
66 8 Henning Blohm
This should complete without any questions or errors. Otherwise please verify your settings above.
67 3 Henning Blohm
68
h3. Start and Stop
69 1 Henning Blohm
70
Depending on your sample requirements, you can start Hadoop (HDFS, Yarn, the History Server) or HBase (including all the Hadoop services) using the *start_hadoop.sh* script (or launcher) or the *start_hbase.sh* script (or launcher) respectively. Similarly you can stop everything with the stop scripts.
71 5 Henning Blohm
72 6 Henning Blohm
When you have started, after a short while, using *jps* on the command line, you should see the following Java processes (and possibly others of course):
73 1 Henning Blohm
74 6 Henning Blohm
<pre><code class="ruby">
75
HRegionServer
76
HQuorumPeer
77
DataNode
78
NodeManager
79
HMaster
80
NameNode
81
SecondaryNameNode
82
JobHistoryServer
83
ResourceManager
84
</code></pre>
85 1 Henning Blohm
86 3 Henning Blohm
87 4 Henning Blohm
There is lots of other scripts in the distribution that you can use to start or stop single components. If you do however, please run (in the shell):
88 1 Henning Blohm
89 4 Henning Blohm
<pre><code class="ruby">
90
. ./env.sh
91
</code></pre>
92
(note the leading period)
93
94 3 Henning Blohm
If you ran the start script and it returned, here is some URLs you should check to verify everything is looking good:
95
96 6 Henning Blohm
* Try to reach the Namenode at http://localhost:50070
97
* Try to reach the Yarn Nodemanager at http://localhost:8088
98 4 Henning Blohm
99 7 Henning Blohm
and, if you are running HBase:
100 4 Henning Blohm
101 6 Henning Blohm
* Try to reach the HBase Master at http://localhost:60010
102 4 Henning Blohm
103
*Note:* If you notice that you cannot restart or that HBase is not stopping correctly, that is most likely exactly the case. Sometimes HBase processes do not stop. To make sure there is no process left, use *jps* from the command line and kill remaining processes.