An internal error occurred during: "Connecting to DFS <hadoop-location-name>
org/apache/commons/configuration/Configuration
Pre-requisite:
1. JDK-1.6 is installed and JAVA_HOME and Path are setup for Java classes.
2. Setup ANT_HOME and Path for ant build script.
3. Ensure ant build script is available as cygwin command.
4. Hadoop-1.1.2 is installed in the system
5. Eclipse – Indigo flavor is installed in the system (The newly built eclipse-plugin may not work for other versions of eclipse)
Preparation of eclipse-plugin build environment:
Navigate to the directory ${Hadoop.home}/src/contrib./eclipse-plugin
1. Update the target jar in build.xml as shown below (in italics) –
![]() |
eclipse-plugin build script |
2. Update the corresponding build.properties file with the following details –
Home directory where eclipse is located –
eclipse.home=C:/Users/IBM_ADMIN/Documents/eclipse/eclipse
Destination directory where eclipse-plugin jar file to be kept after successful build –
dist.dir=C:/hadoop/hadoop-1.1.2/build/contrib/eclipse-plugin
Version of the packages to be included in eclipse-plugin jar file -
version=1.1.2
commons-cli.version=1.2
commons-configuration.version=1.6
commons-httpclient.version=3.0.1
commons-lang.version=2.4
jackson-core-asl.version=1.8.8
jackson-mapper-asl.version=1.8.8
3. Update the MANIFEST.MF file kept inside META-INF directory as shown below (in italics) –
Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: MapReduce Tools for Eclipse
Bundle-SymbolicName: org.apache.hadoop.eclipse;singleton:=true
Bundle-Version: 0.18
Bundle-Activator: org.apache.hadoop.eclipse.Activator
Bundle-Localization: plugin
Require-Bundle: org.eclipse.ui,
org.eclipse.core.runtime,
org.eclipse.jdt.launching,
org.eclipse.debug.core,
org.eclipse.jdt,
org.eclipse.jdt.core,
org.eclipse.core.resources,
org.eclipse.ui.ide,
org.eclipse.jdt.ui,
org.eclipse.debug.ui,
org.eclipse.jdt.debug.ui,
org.eclipse.core.expressions,
org.eclipse.ui.cheatsheets,
org.eclipse.ui.console,
org.eclipse.ui.navigator,
org.eclipse.core.filesystem,
org.apache.commons.logging
Eclipse-LazyStart: true
Bundle-ClassPath: classes/,
lib/hadoop-core.jar,
lib/commons-cli-1.2.jar,
lib/commons-configuration-1.6.jar,
lib/commons-httpclient-3.0.1.jar,
lib/commons-lang-2.4.jar,
lib/jackson-core-asl-1.8.8.jar,
lib/jackson-mapper-asl-1.8.8.jar
Bundle-Vendor: Apache Hadoop
4. Execute the command – ant clean package in CYGWIN console from directory –
${Hadoop.home}/src/contrib/eclipse-plugin
5. Collect the eclipse-plugin-1.1.2.jar file from directory -
C:/hadoop/hadoop-1.1.2/build/contrib/eclipse-plugin and drop it in directory - ${eclipse.home}/plugins
6. Start the eclipse IDE, or restart the tool if it’s already running.
Map/Reduce Locations Setup:
1. Click on the New Hadoop Location button in Map/Reduce Locations section of the Map/Reduce perspective.
2. Enter a name for the Hadoop location in Location name:
![]() |
Hadoop Location Setup |
Host: localhost (for Hadoop running in the localhost, otherwise the IP address of the system hosting Hadoop)
Port:
mapred.job.tracker – 9001
fs.default.name – 9000
No updates are required in the Advance Parameters tab.
4. Click on the Finish button.
5. Reconnect as shown below –
![]() |
Eclipse connected to HDFS |
38 comments:
Hi,
I've downloaded the Hadoop 1.1.2, but it has not contained eclipse-plugin anymore, I read one of your post in which you described how to build the plugin and use it with eclipse.
So, can you please let me have that jar plugin or even the eclipse plugin source which is resided in "${Hadoop.home}/src/contrib./eclipse-plugin"?
Thank you so much
Hey, I will certainly share the files you need.
Did you download hadoop-1.1.2.tar.gz?
When you untar the tarball, you will find the source code in -
hadoop-1.1.2/src/contrib/eclipse-plugin
You will need to modify the build.xml, build.properties and META-INF/MANIFEST.MF, as stated above in the blog to have correct hadoop-eclipse-plugin-1.1.2.jar
Dear Surajit,
I made a mistake and I was downloading the "hadoop-bin-1.1.2.tar.gz" which was not contained the "src" and "eclipse-plugin" folder! I had to download the "hadoop-1.1.2.tar.gz".
anyway, I followed your instruction but when I run "ant clean package" command, I will get a long list of errors like:
compile:
[echo] contrib: eclipse-plugin
[javac] /Users/morteza/Desktop/hadoop-1.1.2/src/contrib/eclipse-plugin/build.xml:61: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 45 source files to /Users/morteza/Desktop/hadoop-1.1.2/build/contrib/eclipse-plugin/classes
[javac] /Users/morteza/Desktop/hadoop-1.1.2/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSFolder.java:28: package org.apache.hadoop.fs does not exist
[javac] import org.apache.hadoop.fs.FileStatus;
[javac] ^
[javac] /Users/morteza/Desktop/hadoop-1.1.2/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSFolder.java:29: package org.apache.hadoop.fs does not exist
[javac] import org.apache.hadoop.fs.Path;
[javac] ^
[javac] /Users/morteza/Desktop/hadoop-1.1.2/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSPath.java:25: package org.apache.hadoop.hdfs does not exist
[javac] import org.apache.hadoop.hdfs.DistributedFileSystem;
[javac] ^
[javac] /Users/morteza/Desktop/hadoop-1.1.2/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSPath.java:29: package org.apache.hadoop.fs does not exist
[javac] import org.apache.hadoop.fs.FileSystem;
[javac] ^
.
.
.
.
[javac] /Users/morteza/Desktop/hadoop-1.1.2/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/HadoopJob.java:302: package Counters does not exist
[javac] Counters.Group group = counters.getGroup(groupName);
[javac] ^
[javac] Note: /Users/morteza/Desktop/hadoop-1.1.2/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/JarModule.java uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 100 errors
Can you please let me know what's the problem?
I also read other resources, but I got the same error, even I used Apache Forrest...
anyway, if you could share your "hadoop-eclipse-plugin-1.1.2.jar", I really appreciate your kindliness...
Preparation of eclipse-plugin build environment is the tricky part. Before you build the eclipse-plugin, you may like to crosscheck the prerequisites once again.
What's the email address you would like to receive the eclipse-plugin jar to?
I'm getting the following error, whild i run ant
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\server\HadoopJob.java:60: cannot find symbol
[javac] symbol : variable JobStatus
[javac] location: class org.apache.hadoop.eclipse.server.HadoopJob.JobState
[javac] case JobStatus.SUCCEEDED:
[javac] ^
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\server\HadoopJob.java:146: cannot find symbol
[javac] symbol : class FileSystem
[javac] location: class org.apache.hadoop.eclipse.server.HadoopJob
[javac] FileSystem fs = location.getDFS();
[javac] ^
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\server\HadoopJob.java:148: cannot find symbol
[javac] symbol : class Path
[javac] location: class org.apache.hadoop.eclipse.server.HadoopJob
[javac] if (FileUtil.copy(fs, new Path(jobFile), tmp, false, location
[javac] ^
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\server\HadoopJob.java:148: cannot find symbol
[javac] symbol : variable FileUtil
[javac] location: class org.apache.hadoop.eclipse.server.HadoopJob
[javac] if (FileUtil.copy(fs, new Path(jobFile), tmp, false, location
[javac] ^
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\server\HadoopJob.java:150: cannot find symbol
[javac] symbol : class JobConf
[javac] location: class org.apache.hadoop.eclipse.server.HadoopJob
[javac] this.jobConf = new JobConf(tmp.toString());
[javac] ^
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\server\HadoopJob.java:302: package Counters does not exist
[javac] Counters.Group group = counters.getGroup(groupName);
[javac] ^
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\server\HadoopJob.java:305: package Counters does not exist
[javac] for (Counters.Counter counter : group) {
[javac] ^
[javac] D:\cygwin\ux\home\user\hadoop-1.1.2\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\dfs\DFSFile.java:74: cannot find symbol
[javac] symbol : class FileStatus
[javac] location: class org.apache.hadoop.eclipse.dfs.DFSFile
[javac] FileStatus fs = getDFS().getFileStatus(path);
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 100 errors
G8 blog.
Can you send me the eclipse_plugin.jar to mail Id at kaasiein@yahoo.co.in. it will be helpful to other developers if you can upload in your site.
Hi Can you please send me the Eclipse plugin for Hadoop i am using a mac os X
What's the email address you would like to receive the eclipse-plugin jar to?
Gautam, you should be able to build eclipse plugin following the above steps. You may also save the section for build.xml on your laptop if its difficult to view in here.
Thank you
Thank you Surajit for such an enlightening post! As has been reported by Kasi above, I too ran into the same issue but was able to resolve it by using "ant jar -lib /Users/xxx/hadoop-1.1.2".
Hi Surajit, I was the one who posted the previous comments about the alternate way to use ant to build the eclipse plugin.
I am using Mac OS X Lion 10.7.5 (11G63) and can see the Map/Reduce perspective in Eclipse (STS). However, when I click the New Location, nothing happens! No dialog launches!!
Would appreciate if you can directly email me at cibsbui at hotmail dot com if you have some suggestions. If you plan to post my comment, please remove my email address for obvious reasons. Thanks!
Just add something like
to the classpath so that it can compile with the external libraries.
can you please share across eclipse-plugin.jar . I am using Mac
kdvenkata@gmail.com
I am getting "Unhandled event loop exception" when I try to add a New Hadoop Location with the following trace. The Configuration class is in the hadoop-core jar which is part of the jars in the lib. Tried adding $HADOOP_HOME, $HADOOP_HOME/lib, $HADOOP_HOME/conf to the path but still no luck! Any suggestions would be greatly appreciated.
java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at org.apache.hadoop.eclipse.server.HadoopServer.(HadoopServer.java:224)
at org.apache.hadoop.eclipse.servers.HadoopLocationWizard.(HadoopLocationWizard.java:88)
at org.apache.hadoop.eclipse.actions.NewLocationAction$1.(NewLocationAction.java:41)
at org.apache.hadoop.eclipse.actions.NewLocationAction.run(NewLocationAction.java:40)
at org.eclipse.jface.action.Action.runWithEvent(Action.java:498)
at org.eclipse.jface.action.ActionContributionItem.handleWidgetSelection(ActionContributionItem.java:584)
at org.eclipse.jface.action.ActionContributionItem.access$2(ActionContributionItem.java:501)
at org.eclipse.jface.action.ActionContributionItem$5.handleEvent(ActionContributionItem.java:411)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4136)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1458)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1481)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1466)
at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1271)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:3982)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3621)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1053)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:942)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:86)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:588)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:543)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:124)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:353)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:629)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:584)
at org.eclipse.equinox.launcher.Main.run(Main.java:1438)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:501)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:421)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:412)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 37 more
Hi Surajit,
I am using HortonWorks Sandbox for Hadoop 1.1.2 and would like to connect to this Sandbox using Eclipse. I already have an Eclipse plugin which connects to Hadoop 0.18.0. Can I use the same plugin and jar files to connect to 1.1.2? If not, from where can I get the plugin and the necessary jar files? I would appreciate your help.
Thanks
@Random Tamil Guy, unfortunately the hadoop-eclipse plugin must be compliant with the versions used for both hadoop and eclipse. You may like to build your own plugin using the relevant versions, just to avoid weird behavior or strange exceptions while working with Hortonworks. Follow the steps stated above to build the plugin. If you face any issue, let me know.
Thanks Surajit for the quick reply.
I am able to create hadoop-eclipse-plugin-1.1.2.jar successfully using your steps. However, when I copy the eclipse plugin into the plugin folder of eclipse indigo and try to create a Map-reduce Project, I am getting the following error:
Problem Opening Wizard and if I look into error details I get the following:
The selected wizard could not be started.
Plug-in org.apache.hadoop.eclipse was unable to load class org.apache.hadoop.eclipse.NewMapReduceProjectWizard.
org.apache.hadoop.eclipse.NewMapReduceProjectWizard
Could you please let me know why this error is happening? If not, could you please send me the eclipse plugin jar for 1.1.2?
Which version of eclipse are you using?
hi Surajit
your post is really helpful...
i followed the steps but could not generate the plugin successfully..
I got a list of error similar to what Kasi Ramanathen has got..
Can you please send me the plugin to umeshgowda5@gmail.com
Thanks a lot..
Surjit i m not able to run ant command from my cygwin
can u please help me on this.
$ - ant clean package
-bash: -: command not found
Hi Surjit i m not able to run Ant command from my cygwin i m getting
$ - ant clean package
-bash: -: command not found
Can you please help me...
Rgds
Nilesh
Hey Surajit, for one of the pre-requisites, you said to:
"Setup ANT_HOME and Path for ant build script"
Could you walk me through that, as I am lost, and also, when I downloaded Ant, it never came with a build.xml nor a build.properties file.
Hey Surajit,
I know as a stranger, this is asking for a lot, but is it possible if you could just email me the" eclipse-plugin jar" file to my email?
My email is: geraldpark7@gmail.com
THANK YOU SO MUCH!!!
Hi Surajit,
I'm getting same error what Morteza posted above. could u please send me jar file to mahi278@gmail.com
Make sure you download hadoop-*.*.*.tar.gz. It has got the relevant ANT script required to build the plugin.
Hello Surajit,
I have followed the step outline in the post but it is not working as expected. I would like you send me the hadoop-eclipse-plugin-1.1.2.jar
my email address is:
jacob.agyemang@gmail.com
Thank you
Jacob
hi,
I am using hadoop 1.1.2 and eclipse Indigo. But i am unable to create the JAR by following your steps. please send me the JAR file reddy880@gmail.com.
Thanks in advance...
Regards
reddy
Hi Jacob,
Could you please tell me where did you face the trouble? Note that you need to follow all steps properly, also ensure that cygwin is properly installed.
Hi Surajit,
I installed hadoop-1.1.2 on windows 7.
Now, I want to install plugin to Eclipse, so please send me .JAR hadoop-1.1.2-eclipse plugin,
zoran55zoki@gmail.com
Thanks in advance
Hi Suraj,
I am failing while building in ubuntu. error - log4j#log4j;1.2.15: configuration not found in log4j#log4j;1.2.15: 'master'. It was required from org.apache.hadoop#Hadoop;working@ubuntu log4j
can u pls send the jar to my id ars.1001@gmail.com.
Thanks
Ash.
Dude, I get this build failed error. Any idea on its cause. All other steps seems to be fine
BUILD FAILED
C:\cygwin64\usr\local\hadoop-1.2.1\src\contrib\eclipse-plugin\build.xml:69: C:\opt\eclipse\plugins does not exist.
Hi Suraj,
My plugin build too is failing
Can you pls send me the jar to bhsahana05@gmail.com
Thanks in advance
Sahana
Hi
My eclipse build too failed can u pls send the
eclipse plugin jar to bhsahana05@gmail.com
Thanks
sahana
Hi, I still got an error: An internal error occurred during: "Connecting to DFS ".
org/apache/commons/configuration/Configuration. Under the location in Eclipse, it is showed: Error: failure to login.
Do you know what's the problem about this? Thanks a lot
@Cindy,
This looks like the plugin does not have all jar/tools compatible with Eclipse you'r using.
Post a Comment