Eclipse Configuration for Hadoop-1.1.2 Access

Although http://wiki.apache.org/hadoop/EclipsePlugIn page provides ready to use eclipse-plugins suitable for some of the early releases of Hadoop, these plugins may not be compatible with the Hadoop environment released recently, and the Eclipse IDE with advance features. Therefore, it could be convenient to build the eclipse-plugin on your own and plug that in the eclipse environment, instead of messy-ing up with issues like -

An internal error occurred during: "Connecting to DFS <hadoop-location-name>".
org/apache/commons/configuration/Configuration

Pre-requisite:

1. JDK-1.6 is installed and JAVA_HOME and Path are setup for Java classes.
2. Setup ANT_HOME and Path for ant build script.
3. Ensure ant build script is available as cygwin command.
4. Hadoop-1.1.2 is installed in the system
5. Eclipse – Indigo flavor is installed in the system (The newly built eclipse-plugin may not work for other versions of eclipse)

Preparation of eclipse-plugin build environment:

Navigate to the directory ${Hadoop.home}/src/contrib./eclipse-plugin

1. Update the target jar in build.xml as shown below (in italics) –
eclipse-plugin build script

2. Update the corresponding build.properties file with the following details –

Home directory where eclipse is located –
eclipse.home=C:/Users/IBM_ADMIN/Documents/eclipse/eclipse

Destination directory where eclipse-plugin jar file to be kept after successful build –
dist.dir=C:/hadoop/hadoop-1.1.2/build/contrib/eclipse-plugin

Version of the packages to be included in eclipse-plugin jar file -
version=1.1.2
commons-cli.version=1.2
commons-configuration.version=1.6
commons-httpclient.version=3.0.1
commons-lang.version=2.4
jackson-core-asl.version=1.8.8
jackson-mapper-asl.version=1.8.8

3. Update the MANIFEST.MF file kept inside META-INF directory as shown below (in italics) –

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: MapReduce Tools for Eclipse
Bundle-SymbolicName: org.apache.hadoop.eclipse;singleton:=true
Bundle-Version: 0.18
Bundle-Activator: org.apache.hadoop.eclipse.Activator
Bundle-Localization: plugin
Require-Bundle: org.eclipse.ui,
 org.eclipse.core.runtime,
 org.eclipse.jdt.launching,
 org.eclipse.debug.core,
 org.eclipse.jdt,
 org.eclipse.jdt.core,
 org.eclipse.core.resources,
 org.eclipse.ui.ide,
 org.eclipse.jdt.ui,
 org.eclipse.debug.ui,
 org.eclipse.jdt.debug.ui,
 org.eclipse.core.expressions,
 org.eclipse.ui.cheatsheets,
 org.eclipse.ui.console,
 org.eclipse.ui.navigator,
 org.eclipse.core.filesystem,
 org.apache.commons.logging
Eclipse-LazyStart: true
Bundle-ClassPath: classes/,
 lib/hadoop-core.jar,
 lib/commons-cli-1.2.jar,
 lib/commons-configuration-1.6.jar,
 lib/commons-httpclient-3.0.1.jar,
 lib/commons-lang-2.4.jar,
 lib/jackson-core-asl-1.8.8.jar,
 lib/jackson-mapper-asl-1.8.8.jar
Bundle-Vendor: Apache Hadoop

4. Execute the command – ant clean package in CYGWIN console from directory –

${Hadoop.home}/src/contrib/eclipse-plugin

5. Collect the eclipse-plugin-1.1.2.jar file from directory - 
C:/hadoop/hadoop-1.1.2/build/contrib/eclipse-plugin and drop it in directory - ${eclipse.home}/plugins

6. Start the eclipse IDE, or restart the tool if it’s already running.

Map/Reduce Locations Setup:

1. Click on the New Hadoop Location button in Map/Reduce Locations section of the Map/Reduce perspective.
2. Enter a name for the Hadoop location in Location name:

Hadoop Location Setup
3. Enter the details for Map/Reduce Master and HDFS Master
    Host: localhost (for Hadoop running in the localhost, otherwise the IP address of the system hosting Hadoop)
    Port:
           mapred.job.tracker – 9001
           fs.default.name – 9000

    No updates are required in the Advance Parameters tab.

4. Click on the Finish button.

5. Reconnect as shown below –
Eclipse connected to HDFS
Eclipse will display the HDFS files/directories on Hadoop environment.

38 comments:

Unknown said...

Hi,

I've downloaded the Hadoop 1.1.2, but it has not contained eclipse-plugin anymore, I read one of your post in which you described how to build the plugin and use it with eclipse.
So, can you please let me have that jar plugin or even the eclipse plugin source which is resided in "${Hadoop.home}/src/contrib./eclipse-plugin"?

Thank you so much

Surajit Paul said...

Hey, I will certainly share the files you need.

Did you download hadoop-1.1.2.tar.gz?

When you untar the tarball, you will find the source code in -
hadoop-1.1.2/src/contrib/eclipse-plugin

You will need to modify the build.xml, build.properties and META-INF/MANIFEST.MF, as stated above in the blog to have correct hadoop-eclipse-plugin-1.1.2.jar

Unknown said...

Dear Surajit,
I made a mistake and I was downloading the "hadoop-bin-1.1.2.tar.gz" which was not contained the "src" and "eclipse-plugin" folder! I had to download the "hadoop-1.1.2.tar.gz".
anyway, I followed your instruction but when I run "ant clean package" command, I will get a long list of errors like:

compile:
[echo] contrib: eclipse-plugin
[javac] /Users/morteza/Desktop/hadoop-1.1.2/src/contrib/eclipse-plugin/build.xml:61: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 45 source files to /Users/morteza/Desktop/hadoop-1.1.2/build/contrib/eclipse-plugin/classes
[javac] /Users/morteza/Desktop/hadoop-1.1.2/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSFolder.java:28: package org.apache.hadoop.fs does not exist
[javac] import org.apache.hadoop.fs.FileStatus;
[javac] ^
[javac] /Users/morteza/Desktop/hadoop-1.1.2/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSFolder.java:29: package org.apache.hadoop.fs does not exist
[javac] import org.apache.hadoop.fs.Path;
[javac] ^
[javac] /Users/morteza/Desktop/hadoop-1.1.2/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSPath.java:25: package org.apache.hadoop.hdfs does not exist
[javac] import org.apache.hadoop.hdfs.DistributedFileSystem;
[javac] ^
[javac] /Users/morteza/Desktop/hadoop-1.1.2/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSPath.java:29: package org.apache.hadoop.fs does not exist
[javac] import org.apache.hadoop.fs.FileSystem;
[javac] ^
.
.
.
.
[javac] /Users/morteza/Desktop/hadoop-1.1.2/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/HadoopJob.java:302: package Counters does not exist
[javac] Counters.Group group = counters.getGroup(groupName);
[javac] ^
[javac] Note: /Users/morteza/Desktop/hadoop-1.1.2/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/JarModule.java uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 100 errors

Can you please let me know what's the problem?
I also read other resources, but I got the same error, even I used Apache Forrest...
anyway, if you could share your "hadoop-eclipse-plugin-1.1.2.jar", I really appreciate your kindliness...

Surajit Paul said...

Preparation of eclipse-plugin build environment is the tricky part. Before you build the eclipse-plugin, you may like to crosscheck the prerequisites once again.

What's the email address you would like to receive the eclipse-plugin jar to?

Unknown said...
This comment has been removed by the author.
Unknown said...

I'm getting the following error, whild i run ant

[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\server\HadoopJob.java:60: cannot find symbol
[javac] symbol : variable JobStatus
[javac] location: class org.apache.hadoop.eclipse.server.HadoopJob.JobState
[javac] case JobStatus.SUCCEEDED:
[javac] ^
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\server\HadoopJob.java:146: cannot find symbol
[javac] symbol : class FileSystem
[javac] location: class org.apache.hadoop.eclipse.server.HadoopJob
[javac] FileSystem fs = location.getDFS();
[javac] ^
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\server\HadoopJob.java:148: cannot find symbol
[javac] symbol : class Path
[javac] location: class org.apache.hadoop.eclipse.server.HadoopJob
[javac] if (FileUtil.copy(fs, new Path(jobFile), tmp, false, location
[javac] ^
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\server\HadoopJob.java:148: cannot find symbol
[javac] symbol : variable FileUtil
[javac] location: class org.apache.hadoop.eclipse.server.HadoopJob
[javac] if (FileUtil.copy(fs, new Path(jobFile), tmp, false, location
[javac] ^
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\server\HadoopJob.java:150: cannot find symbol
[javac] symbol : class JobConf
[javac] location: class org.apache.hadoop.eclipse.server.HadoopJob
[javac] this.jobConf = new JobConf(tmp.toString());
[javac] ^
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\server\HadoopJob.java:302: package Counters does not exist
[javac] Counters.Group group = counters.getGroup(groupName);
[javac] ^
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\server\HadoopJob.java:305: package Counters does not exist
[javac] for (Counters.Counter counter : group) {
[javac] ^
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\dfs\DFSFile.java:74: cannot find symbol
[javac] symbol : class FileStatus
[javac] location: class org.apache.hadoop.eclipse.dfs.DFSFile
[javac] FileStatus fs = getDFS().getFileStatus(path);
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 100 errors
G8 blog.
Can you send me the eclipse_plugin.jar to mail Id at kaasiein@yahoo.co.in. it will be helpful to other developers if you can upload in your site.

Unknown said...

Hi Can you please send me the Eclipse plugin for Hadoop i am using a mac os X

Surajit Paul said...

What's the email address you would like to receive the eclipse-plugin jar to?

Unknown said...
This comment has been removed by the author.
Unknown said...
This comment has been removed by the author.
Surajit Paul said...

Gautam, you should be able to build eclipse plugin following the above steps. You may also save the section for build.xml on your laptop if its difficult to view in here.

Unknown said...

Thank you

Anonymous said...

Thank you Surajit for such an enlightening post! As has been reported by Kasi above, I too ran into the same issue but was able to resolve it by using "ant jar -lib /Users/xxx/hadoop-1.1.2".

Anonymous said...

Hi Surajit, I was the one who posted the previous comments about the alternate way to use ant to build the eclipse plugin.

I am using Mac OS X Lion 10.7.5 (11G63) and can see the Map/Reduce perspective in Eclipse (STS). However, when I click the New Location, nothing happens! No dialog launches!!

Would appreciate if you can directly email me at cibsbui at hotmail dot com if you have some suggestions. If you plan to post my comment, please remove my email address for obvious reasons. Thanks!

Anonymous said...

Just add something like



to the classpath so that it can compile with the external libraries.

Anonymous said...

can you please share across eclipse-plugin.jar . I am using Mac
kdvenkata@gmail.com

Anonymous said...

I am getting "Unhandled event loop exception" when I try to add a New Hadoop Location with the following trace. The Configuration class is in the hadoop-core jar which is part of the jars in the lib. Tried adding $HADOOP_HOME, $HADOOP_HOME/lib, $HADOOP_HOME/conf to the path but still no luck! Any suggestions would be greatly appreciated.

java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at org.apache.hadoop.eclipse.server.HadoopServer.(HadoopServer.java:224)
at org.apache.hadoop.eclipse.servers.HadoopLocationWizard.(HadoopLocationWizard.java:88)
at org.apache.hadoop.eclipse.actions.NewLocationAction$1.(NewLocationAction.java:41)
at org.apache.hadoop.eclipse.actions.NewLocationAction.run(NewLocationAction.java:40)
at org.eclipse.jface.action.Action.runWithEvent(Action.java:498)
at org.eclipse.jface.action.ActionContributionItem.handleWidgetSelection(ActionContributionItem.java:584)
at org.eclipse.jface.action.ActionContributionItem.access$2(ActionContributionItem.java:501)
at org.eclipse.jface.action.ActionContributionItem$5.handleEvent(ActionContributionItem.java:411)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4136)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1458)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1481)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1466)
at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1271)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:3982)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3621)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1053)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:942)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:86)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:588)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:543)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:124)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:353)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:629)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:584)
at org.eclipse.equinox.launcher.Main.run(Main.java:1438)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:501)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:421)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:412)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 37 more

Random Tamil Guy said...

Hi Surajit,

I am using HortonWorks Sandbox for Hadoop 1.1.2 and would like to connect to this Sandbox using Eclipse. I already have an Eclipse plugin which connects to Hadoop 0.18.0. Can I use the same plugin and jar files to connect to 1.1.2? If not, from where can I get the plugin and the necessary jar files? I would appreciate your help.

Thanks

Surajit Paul said...

@Random Tamil Guy, unfortunately the hadoop-eclipse plugin must be compliant with the versions used for both hadoop and eclipse. You may like to build your own plugin using the relevant versions, just to avoid weird behavior or strange exceptions while working with Hortonworks. Follow the steps stated above to build the plugin. If you face any issue, let me know.

Random Tamil Guy said...

Thanks Surajit for the quick reply.

I am able to create hadoop-eclipse-plugin-1.1.2.jar successfully using your steps. However, when I copy the eclipse plugin into the plugin folder of eclipse indigo and try to create a Map-reduce Project, I am getting the following error:
Problem Opening Wizard and if I look into error details I get the following:
The selected wizard could not be started.
Plug-in org.apache.hadoop.eclipse was unable to load class org.apache.hadoop.eclipse.NewMapReduceProjectWizard.
org.apache.hadoop.eclipse.NewMapReduceProjectWizard

Could you please let me know why this error is happening? If not, could you please send me the eclipse plugin jar for 1.1.2?

Surajit Paul said...

Which version of eclipse are you using?

umesh said...

hi Surajit
your post is really helpful...
i followed the steps but could not generate the plugin successfully..
I got a list of error similar to what Kasi Ramanathen has got..
Can you please send me the plugin to umeshgowda5@gmail.com

Thanks a lot..

Anonymous said...

Surjit i m not able to run ant command from my cygwin
can u please help me on this.

$ - ant clean package
-bash: -: command not found



Anonymous said...

Hi Surjit i m not able to run Ant command from my cygwin i m getting
$ - ant clean package
-bash: -: command not found

Can you please help me...

Rgds
Nilesh

Anonymous said...

Hey Surajit, for one of the pre-requisites, you said to:
"Setup ANT_HOME and Path for ant build script"
Could you walk me through that, as I am lost, and also, when I downloaded Ant, it never came with a build.xml nor a build.properties file.

Gerald Park said...

Hey Surajit,

I know as a stranger, this is asking for a lot, but is it possible if you could just email me the" eclipse-plugin jar" file to my email?

My email is: geraldpark7@gmail.com

THANK YOU SO MUCH!!!

Anonymous said...

Hi Surajit,

I'm getting same error what Morteza posted above. could u please send me jar file to mahi278@gmail.com




Surajit Paul said...

Make sure you download hadoop-*.*.*.tar.gz. It has got the relevant ANT script required to build the plugin.

Unknown said...

Hello Surajit,
I have followed the step outline in the post but it is not working as expected. I would like you send me the hadoop-eclipse-plugin-1.1.2.jar
my email address is:
jacob.agyemang@gmail.com
Thank you
Jacob

Unknown said...

hi,

I am using hadoop 1.1.2 and eclipse Indigo. But i am unable to create the JAR by following your steps. please send me the JAR file reddy880@gmail.com.

Thanks in advance...

Regards
reddy

Surajit Paul said...

Hi Jacob,

Could you please tell me where did you face the trouble? Note that you need to follow all steps properly, also ensure that cygwin is properly installed.

Unknown said...

Hi Surajit,

I installed hadoop-1.1.2 on windows 7.
Now, I want to install plugin to Eclipse, so please send me .JAR hadoop-1.1.2-eclipse plugin,
zoran55zoki@gmail.com
Thanks in advance

Unknown said...

Hi Suraj,

I am failing while building in ubuntu. error - log4j#log4j;1.2.15: configuration not found in log4j#log4j;1.2.15: 'master'. It was required from org.apache.hadoop#Hadoop;working@ubuntu log4j

can u pls send the jar to my id ars.1001@gmail.com.
Thanks
Ash.

Unknown said...

Dude, I get this build failed error. Any idea on its cause. All other steps seems to be fine

BUILD FAILED
C:\cygwin64\usr\local\hadoop-1.2.1\src\contrib\eclipse-plugin\build.xml:69: C:\opt\eclipse\plugins does not exist.

Unknown said...

Hi Suraj,
My plugin build too is failing
Can you pls send me the jar to bhsahana05@gmail.com

Thanks in advance
Sahana

Unknown said...

Hi

My eclipse build too failed can u pls send the
eclipse plugin jar to bhsahana05@gmail.com

Thanks
sahana

Cindy said...

Hi, I still got an error: An internal error occurred during: "Connecting to DFS ".
org/apache/commons/configuration/Configuration. Under the location in Eclipse, it is showed: Error: failure to login.
Do you know what's the problem about this? Thanks a lot

Surajit Paul said...

@Cindy,

This looks like the plugin does not have all jar/tools compatible with Eclipse you'r using.