Big Data Engine - Hadoop

This blog will list various exceptions/warnings I encountered while configuring Hadoop environment, and the relevant fixes I applied.


Exception in thread "main" java.lang.IncompatibleClassChangeError:
Found class org.apache.hadoop.mapreduce.JobContext, but interface was
expected at org.apache.sqoop.config.ConfigurationHelper.getJobNumMaps(ConfigurationHelp
er.java:53)


-- Clean build fixes the issue.


Exception in thread "main" java.lang.NoSuchMethodError: org/apache/thrift/EncodingUtils.setBit(BIZ)B


-- Ensure libthrift.jar in HIVE, SQOOP, and HBASE are same

Exception in thread "main" java.lang.NoClassDefFoundError: com.sun.security.auth.UnixPrincipal

-- Need Oracle JDK as Hadoop is built using Oracle JDK



Command: $start-hbase.sh

java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused


-- HDFS listens to port 9000, hence the hbase should point to port 9000.



FATAL zookeeper.ZKConfig: The server in zoo.cfg cannot be set to localhost in a fully-distributed setup because it won't be reachable. See "Getting Started" for more information.


-- Its a warning.

org.apache.hadoop.hbase.util.FileSystemVersionException: File system needs to be upgraded.  You have version null and I want version 7.  Run the '${HBASE_HOME}/bin/hbase migrate' script.
   at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:349)
   at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:354)
   at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:130)
   at org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:115)
   at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:513)
   at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:376)
   at java.lang.Thread.run(Thread.java:780)
2013-05-30 19:56:16,050 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2013-05-30 19:56:16,050 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
2013-05-30 19:56:16,050 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000

--Following command fixes the issue:
$ bin/hadoop fs -rmr /hbase
Deleted hdfs://localhost:9000/hbase



13/05/30 22:04:45 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x13ef63ab322000d, negotiated timeout = 40000
java.lang.IllegalArgumentException: Not a host:port pair:

-- This issue is caused by version mismatch of the jar files loaded in the client. The issue can be avoided by replacing all client jar with server specific jar files.



2013-05-31 09:33:23,583 WARN  conf.HiveConf (HiveConf.java:(75)) - hive-site.xml not found on CLASSPATH
2013-05-31 09:33:24,273 ERROR tool.ImportTool (ImportTool.java:run(484)) - Encountered IOException running import job: java.io.IOException: Exception thrown in Hive
   at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:335)

Exception in thread "main" java.lang.NullPointerException
   at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:131)
   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
   at hadoop.sqoop.HiveSqooper.cleanUp(HiveSqooper.java:145)
   at hadoop.sqoop.HiveSqooper.main(HiveSqooper.java:47)
ERROR tool.ImportTool (ImportTool.java:run(484)) - Encountered IOException running import job: java.io.IOException: Exception thrown in Hive
Caused by: java.lang.NoClassDefFoundError: org/antlr/runtime/tree/TreeAdaptor
Caused by: java.lang.ClassNotFoundException: org.antlr.runtime.tree.TreeAdaptor

------- hadoop-env.sh
export HIVE_CONF_DIR=/home/surajit/hadoop/hive-0.10.0/conf
# Extra Java CLASSPATH elements.  Optional.
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_CONF_DIR


2013-05-31 12:16:01,938 ERROR ql.Driver (SessionState.java:printError(401)) - FAILED: SemanticException [Error 10028]: Line 2:17 Path is not legal ''file:/home/surajit/workspace/infolytics/bse30'': Move from: file:/home/surajit/workspace/infolytics/bse30 to: hdfs://localhost:9000/user/hive/warehouse/bse30 is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
org.apache.hadoop.hive.ql.parse.SemanticException: Line 2:17 Path is not legal ''file:/home/surajit/workspace/infolytics/bse30'': Move from: file:/home/surajit/workspace/infolytics/bse30 to: hdfs://localhost:9000/user/hive/warehouse/bse30 is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.

SqoopOptions.setWarehouseDir("hdfs://localhost:9000/user/hive/warehouse/sensex.db"); --- fixes the issues.



PIG Exceptions:
-------------------------------------------------------------------------------------
ERROR 1070: Could not resolve HBaseStorage using imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Failed to parse: Pig script failed to parse:
pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve HBaseStorage using imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]

-- Full path required - org.apache.pig.backend.hadoop.hbase.HBaseStorage
FMCG1 = LOAD 'hbase://bse30' using org.apache.pig.backend.hadoop.hbase.HBaseStorage('firstdecade:*') as (id:bytearray, histdate:chararray, index_name:chararray, open:double, high:double, low:double, close:double);



ERROR 2998: Unhandled internal error. org.apache.zookeeper.ZooKeeper

java.lang.NoClassDefFoundError: org.apache.zookeeper.ZooKeeper
   at org.apache.pig.backend.hadoop.hbase.HBaseStorage.initialiseHBaseClassLoaderResources(HBaseStorage.java:712)
   at org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:667)
   at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:473)
   at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:294)
   at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
   at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
   at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
   at org.apache.pig.PigServer.storeEx(PigServer.java:931)
   at org.apache.pig.PigServer.store(PigServer.java:898)
   at org.apache.pig.PigServer.openIterator(PigServer.java:811)
   at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696)
   at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320)
   at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
   at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
   at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
   at org.apache.pig.Main.run(Main.java:538)
   at org.apache.pig.Main.main(Main.java:157)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
   at java.lang.reflect.Method.invoke(Method.java:613)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.ZooKeeper

---- Copy hbase-*.jar and zookeeper*.jar into $PIG_HOME/lib folder




1 comment:

Sandeep Amilineni said...

Thanks, the following solution worked

Exception in thread "main" java.lang.NoSuchMethodError: org/apache/thrift/EncodingUtils.setBit(BIZ)B


-- Ensure libthrift.jar in HIVE, SQOOP, and HBASE are same