Project

General

Profile

Sample-hbase-mail-digester » History » Version 19

Henning Blohm, 03.08.2014 12:31

1 1 Henning Blohm
h1. Sample that combines HBase with full-stack Spring and Hibernate usage
2 2 Henning Blohm
3 14 Henning Blohm
(HOLD ON FOR A SECOND - STILL WORKING ON IT)
4
5 12 Henning Blohm
This sample consists of an application that loads large Mbox archive files into HBase and extracts email addresses using a map reduce job. Extracted email addresses are then written to a relational database and offered for editing.
6 3 Henning Blohm
7 4 Henning Blohm
Being a full stack sample it shows how to design a multi-module application with a service tier that can be seamlessly used from a Web app as well as from an application-level map-reduce job.
8
9 1 Henning Blohm
*Note*: This sample still uses v2.2 of z2 - so making sure the correct versions are specified below is crucial.
10 5 Henning Blohm
*Note*: Due to HBase, you will need to run this on Linux or Mac OS.
11 4 Henning Blohm
12 11 Henning Blohm
Proceed to [[Sample-hbase-mail-digester#How-this-works|how this works]], if you do not want to run the sample but only learn about it.
13 9 Henning Blohm
14 4 Henning Blohm
h2. Install
15 1 Henning Blohm
16 5 Henning Blohm
Here is the quick guide to getting things up and running. This follows closely [[How_to_run_a_sample]] and [[Install_prepacked_CDH4]].
17
18
h3. Checkout
19
20
Create some installation folder and check out the z2 core and the HBase distribution, as well as the sample application. 
21
22 3 Henning Blohm
<pre><code class="bash">
23
git clone -b v2.2 http://git.z2-environment.net/z2-base.core
24
git clone -b v2.2 http://git.z2-environment.net/z2-samples.cdh4-base
25
git clone -b master http://git.z2-environment.net/z2-samples.hbase-mail-digester
26 1 Henning Blohm
</code></pre>
27 5 Henning Blohm
28
(Note: Do not use your shared git folder, if you have any, as the neighborhood of these projects may be inspected by z2 later on).
29
30
h3. Prepare
31
32
We need to apply some minimal configuration for HBase. At first, please follow [[Install_prepacked_CDH4]] on how to configure your HBase checkout. There are a few steps that need to be taken once only but still have to.
33
34
Assuming HBase has started and all processes show as described, there is one last thing to get running before starting the actual application:
35
36
{{include(How to run Java db)}}
37
38 6 Henning Blohm
h2. Start
39 5 Henning Blohm
40 7 Henning Blohm
Now that all databases are up we can start the application simply by running (as always):
41 5 Henning Blohm
42
<pre><code class="bash">
43
# on Linux / Mac OS:
44
cd z2-base.core/run/bin
45
./gui.sh
46
47
# on Windows:
48
cd z2-base.core\run\bin
49
gui.bat
50
</code></pre>
51
52 7 Henning Blohm
At first startup this will download some significant amount of dependencies (Spring, Vaadin, etc.). So go and get yourself some coffee....
53 5 Henning Blohm
54
When started, go to http://localhost:8080/digester-admin. You should see this:
55 1 Henning Blohm
56
!start.png!
57 7 Henning Blohm
58
h2. Using the application
59
60 13 Henning Blohm
To feed data into the application, please download some mail archive in mbox format, e.g. from http://tomcat.apache.org/mail/. Upload the file to the digester application as outlined on the first tab. Mails from mail archives are imported into the HBase database.
61 7 Henning Blohm
62 8 Henning Blohm
When imported, run the analysis map-reduce job by clicking on "Start analysis job". The purpose of this job is to identify mail senders and provide some data like email address, full name, and number of mails into the relational database powered by Derby.
63 7 Henning Blohm
64 13 Henning Blohm
When there is only little data in HBase, progress reporting is rather coarse-grained. After a while you should see:
65 7 Henning Blohm
66 8 Henning Blohm
!job.png!
67 7 Henning Blohm
68 13 Henning Blohm
And work should be completed shortly after. When done, check the extracted sender data:
69 1 Henning Blohm
70
!counts.png!
71 8 Henning Blohm
72
h2. How this works
73
74 16 Henning Blohm
Let's go module by module in "z2-samples-hbase-mail-digester":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester
75 15 Henning Blohm
76 19 Henning Blohm
h4. com.zfabrik.samples.digester.mail.domain
77 15 Henning Blohm
78
This modules defines data types and storage access for HBase stored e-mail messages. The interface "EmailRepository":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.mail.domain/java/src.api/com/zfabrik/samples/digester/mail/EMailRepository.java is the main storage access interface. It's implementation can be found at "EmailRepositoryImpl":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.mail.domain/java/src.impl/com/zfabrik/samples/impl/digester/mail/EmailRepositoryImpl.java.
79
80 1 Henning Blohm
The Spring Bean "HBaseConnector":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.mail.domain/java/src.impl/com/zfabrik/samples/impl/digester/mail/HBaseConnector.java, used by the e-mail repository manages HBase connections.
81 16 Henning Blohm
82 19 Henning Blohm
h4. com.zfabrik.samples.digester.mail.admin
83 16 Henning Blohm
84
This module defines the _Mails_ tab that is loaded via an extension point (see [[Vaadin-addon]] for more details on what that is) from the main tab (defined in *com.zfabrik.samples.digester.admin* - below). Also it defines the _Mail file import_ activity (another extension point). The Mails tab is defined via the component "com.zfabrik.samples.digester.mail.admin/mailListView":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.mail.admin/mailListView.properties and the Mail file import activity via the component "com.zfabrik.samples.digester.mail.admin/actionsView":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.mail.admin/actionsView.properties.
85
86 17 Henning Blohm
As this module interacts mainly with the e-mail repository, it imports it from "com.zfabrik.samples.digester.mail.domain/emailRepository":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.mail.domain/emailRepository.properties in its "application context":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.mail.admin/java/src.impl/META-INF/applicationContext.xml.
87
88 19 Henning Blohm
h4. com.zfabrik.samples.digester.business.domain
89 17 Henning Blohm
90
This module defines the domain types and repositories for all relational data of the application or, more specifically, it defines the JPA entities for contacts and a JPA/Hibernate implemented repository for thoses. It's structure closely resembles those of for example [[sample-spring-hibernate]]. 
91
92 19 Henning Blohm
h4. com.zfabrik.samples.digester.contacts.admin
93 17 Henning Blohm
94
Similarly to *com.zfabrik.samples.digester.mail.admin* this module defines a main level tab _Contacts_ and an activity _Mail analysis_ in components "com.zfabrik.samples.digester.contacts.admin/maintenanceView":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.contacts.admin/maintenanceView.properties and "com.zfabrik.samples.digester.contacts.admin/jobControlView":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.contacts.admin/jobControlView.properties respectively. The latter makes use of the "ContactsJob":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.contacts.jobs/java/src.api/com/zfabrik/samples/digester/contacts/jobs/ContactsJobs.java service that allows to kick off an analysis map-reduce job and to retrieve job status information. 
95
96 19 Henning Blohm
h4. com.zfabrik.samples.digester.contacts.jobs
97 17 Henning Blohm
98
This module defines the "ContactsJob":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.contacts.jobs/java/src.api/com/zfabrik/samples/digester/contacts/jobs/ContactsJobs.java service and the actual map-reduce job "AggregatorJob":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/show/com.zfabrik.samples.digester.contacts.jobs/java/src.impl/com/zfabrik/samples/impl/digester/contacts/jobs.
99
100
The ContactsJob "implementation":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.contacts.jobs/java/src.impl/com/zfabrik/samples/impl/digester/contacts/jobs/ContactsJobsImpl.java interacts directly with Hadoop and the supporting function of [[Hadoop Addon]] to submit the map-reduce job and similarly to check its status. Note the (unfortunately necessary) careful class loader hygiene implemented by using "ThreadUtil":http://www.z2-environment.net/javadoc/com.zfabrik.core.api!2Fjava/api/com/zfabrik/util/threading/ThreadUtil.html . See more in ("Protecting Java in a Modular World – Context Classloaders":http://www.z2-environment.net/blog/2012/07/for-techies-protecting-java-in-a-modular-world-context-classloaders/) to learn about this necessity when using overly cleverly implemented class loader usage such as in Hadoop and HBase. 
101
102
For the principles please check out [[how to hadoop]]. The mapper task "AggregatorMapper":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.contacts.jobs/java/src.impl/com/zfabrik/samples/impl/digester/contacts/jobs/AggregatorMapper.java is straight-forward: For every email message with sender @S@ encountered emit @(s.address, 1)@. 
103
104
The combiner "AggregatorCombiner":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.contacts.jobs/java/src.impl/com/zfabrik/samples/impl/digester/contacts/jobs/AggregatorCombiner.java simply sums counts up and implements
105
106
<pre>
107
(s.address, o=(o_1, o_2, ...)) --> (s.address, sum(o_i,i))
108
</pre>
109
110
At the heart of this sample's intention is the reducer "AggregatorReducer":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.contacts.jobs/java/src.impl/com/zfabrik/samples/impl/digester/contacts/jobs/AggregatorReducer.java. 
111
112
Receiving a tuple @(s.address, o=(o_1, o_2, ...))@  it computes @(s.address, sum(o_i,i))=:(a,t)@ and then updates a contact email count by looking for the contact entity identified by the email address a and then settings its mail count to t.
113
114
To do so it simply uses the ContactRepository that was Spring injected via the exact same module re-use as in the admin user interface modules – no difference here, although this is running on a hadoop data node.
115
116
Specifically in AggregatorReducer we use Spring AspectJ configuration:
117
118
<pre><code class="java">
119
@Configurable
120
public class AggregatorReducer extends Reducer<Text, IntWritable, Text, Writable> {
121
122
	private int count;
123
	
124
	@Autowired
125
	private ContactsRepository contacts;
126
	// ...
127
</code></pre>
128 18 Henning Blohm
129
to inject a service that is reference in the module's application context as 
130
131
<pre><code class="xml">
132
<!-- import contacts repository -->
133
<bean id="contactsRepository" class="com.zfabrik.springframework.ComponentFactoryBean">
134
	<property name="componentName" value="com.zfabrik.samples.digester.business.domain/contactsRepository" />
135
	<property name="className" value="com.zfabrik.samples.digester.business.domain.ContactsRepository" />
136
</bean>
137
</code></pre>
138
139
h4. com.zfabrik.samples.digester.admin
140
141
This final module is the visual glue of the whole application. It defines the skeleton Vaadin application that loads all extension point implementations. This happens in "AdminWindowWorkArea":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.admin/java/src.impl/com/zfabrik/samples/impl/digester/web/AdminWindowWorkArea.java
142
143
<pre><code class="java">
144
// ...
145
// add all extensions
146
Collection<Component> exts = ExtensionComponentsUtil.getComponentyByExtensionForApplication(
147
  AdminApplication.current(), 
148
  EXTENSION_POINT
149
);
150
		
151
for (Component c : exts) {
152
  addTab(c);
153
}
154
// ...
155
</code></pre>
156
157
for the top level tabs and very similarly in "DigesterActions":https://redmine.z2-environment.net/projects/z2-samples/repository/z2-samples-hbase-mail-digester/revisions/master/entry/com.zfabrik.samples.digester.admin/java/src.impl/com/zfabrik/samples/impl/digester/web/actions/DigesterActions.java for the activities presented on the front tab:
158
159
<pre><code class="java">
160
// ...		
161
// search for extensions and add them here
162
for (Component c : ExtensionComponentsUtil.getComponentyByExtensionForApplication(
163
  AdminApplication.current(), 
164
  EXTENSION_POINT)
165
) {
166
  addComponent(c);
167
}
168
// ...
169
</code></pre>