Eclipse Configuration for Hadoop-1.1.2 Access

Although http://wiki.apache.org/hadoop/EclipsePlugIn page provides ready to use eclipse-plugins suitable for some of the early releases of Hadoop, these plugins may not be compatible with the Hadoop environment released recently, and the Eclipse IDE with advance features. Therefore, it could be convenient to build the eclipse-plugin on your own and plug that in the eclipse environment, instead of messy-ing up with issues like -

An internal error occurred during: "Connecting to DFS <hadoop-location-name>".
org/apache/commons/configuration/Configuration

Pre-requisite:

1. JDK-1.6 is installed and JAVA_HOME and Path are setup for Java classes.
2. Setup ANT_HOME and Path for ant build script.
3. Ensure ant build script is available as cygwin command.
4. Hadoop-1.1.2 is installed in the system
5. Eclipse – Indigo flavor is installed in the system (The newly built eclipse-plugin may not work for other versions of eclipse)

Preparation of eclipse-plugin build environment:

Navigate to the directory ${Hadoop.home}/src/contrib./eclipse-plugin

1. Update the target jar in build.xml as shown below (in italics) –

eclipse-plugin build script

2. Update the corresponding build.properties file with the following details –

Home directory where eclipse is located –
eclipse.home=C:/Users/IBM_ADMIN/Documents/eclipse/eclipse

Destination directory where eclipse-plugin jar file to be kept after successful build –
dist.dir=C:/hadoop/hadoop-1.1.2/build/contrib/eclipse-plugin

Version of the packages to be included in eclipse-plugin jar file -
version=1.1.2
commons-cli.version=1.2
commons-configuration.version=1.6
commons-httpclient.version=3.0.1
commons-lang.version=2.4
jackson-core-asl.version=1.8.8
jackson-mapper-asl.version=1.8.8

3. Update the MANIFEST.MF file kept inside META-INF directory as shown below (in italics) –

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: MapReduce Tools for Eclipse
Bundle-SymbolicName: org.apache.hadoop.eclipse;singleton:=true
Bundle-Version: 0.18
Bundle-Activator: org.apache.hadoop.eclipse.Activator
Bundle-Localization: plugin
Require-Bundle: org.eclipse.ui,
 org.eclipse.core.runtime,
 org.eclipse.jdt.launching,
 org.eclipse.debug.core,
 org.eclipse.jdt,
 org.eclipse.jdt.core,
 org.eclipse.core.resources,
 org.eclipse.ui.ide,
 org.eclipse.jdt.ui,
 org.eclipse.debug.ui,
 org.eclipse.jdt.debug.ui,
 org.eclipse.core.expressions,
 org.eclipse.ui.cheatsheets,
 org.eclipse.ui.console,
 org.eclipse.ui.navigator,
 org.eclipse.core.filesystem,
 org.apache.commons.logging
Eclipse-LazyStart: true
Bundle-ClassPath: classes/,
 lib/hadoop-core.jar,
 lib/commons-cli-1.2.jar,
 lib/commons-configuration-1.6.jar,
 lib/commons-httpclient-3.0.1.jar,
 lib/commons-lang-2.4.jar,
 lib/jackson-core-asl-1.8.8.jar,
 lib/jackson-mapper-asl-1.8.8.jar
Bundle-Vendor: Apache Hadoop

4. Execute the command – ant clean package in CYGWIN console from directory –

${Hadoop.home}/src/contrib/eclipse-plugin

5. Collect the eclipse-plugin-1.1.2.jar file from directory - 
C:/hadoop/hadoop-1.1.2/build/contrib/eclipse-plugin and drop it in directory - ${eclipse.home}/plugins

6. Start the eclipse IDE, or restart the tool if it’s already running.

Map/Reduce Locations Setup:

1. Click on the New Hadoop Location button in Map/Reduce Locations section of the Map/Reduce perspective.
2. Enter a name for the Hadoop location in Location name:

Hadoop Location Setup

3. Enter the details for Map/Reduce Master and HDFS Master
    Host: localhost (for Hadoop running in the localhost, otherwise the IP address of the system hosting Hadoop)
    Port:
           mapred.job.tracker – 9001
           fs.default.name – 9000

    No updates are required in the Advance Parameters tab.

4. Click on the Finish button.

5. Reconnect as shown below –

Eclipse connected to HDFS

Eclipse will display the HDFS files/directories on Hadoop environment.