I am trying to import data from sqoop to hive
MySQL
use sample;
create table forhive( id int auto_increment,
firstname varchar(36),
lastname varchar(36),
primary key(id)
);
insert into forhive(firstname, lastname) values("sample","singh");
select * from forhive;
1 abhay agrawal
2 vijay sharma
3 sample singh
This is the Sqoop command I'm using (version 1.4.7)
sqoop import --connect jdbc:mysql://********:3306/sample
--table forhive --split-by id --columns id,firstname,lastname
--target-dir /home/programmeur_v/forhive
--hive-import --create-hive-table --hive-table sqp.forhive --username vaibhav -P
This is the error I'm getting
Error Log
18/08/02 19:19:49 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Enter password:
18/08/02 19:19:55 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
18/08/02 19:19:55 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
18/08/02 19:19:55 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/08/02 19:19:55 INFO tool.CodeGenTool: Beginning code generation
18/08/02 19:19:56 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM
forhiveAS t LIMIT 118/08/02 19:19:56 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM
forhiveAS t LIMIT 118/08/02 19:19:56 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/programmeur_v/softwares/hadoop-2.9.1
Note: /tmp/sqoop-programmeur_v/compile/e8ffa12496a2e421f80e1fa16e025d28/forhive.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details. 18/08/02 19:19:58 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-programmeur_v/compile/e8ffa12496a2e421f80e1fa16e025d28/forhive.jar
18/08/02 19:19:58 WARN manager.MySQLManager: It looks like you are importing from mysql.
18/08/02 19:19:58 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
18/08/02 19:19:58 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
18/08/02 19:19:58 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
18/08/02 19:19:58 INFO mapreduce.ImportJobBase: Beginning import of forhive
18/08/02 19:19:58 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/08/02 19:19:59 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/08/02 19:19:59 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/08/02 19:20:02 INFO db.DBInputFormat: Using read commited transaction isolation
18/08/02 19:20:02 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(
id), MAX(id) FROMforhive18/08/02 19:20:02 INFO db.IntegerSplitter: Split size: 0; Num splits: 4 from: 1 to: 3
18/08/02 19:20:02 INFO mapreduce.JobSubmitter: number of splits:3
18/08/02 19:20:02 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabl ed
18/08/02 19:20:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1533231535061_0006
18/08/02 19:20:03 INFO impl.YarnClientImpl: Submitted application application_1533231535061_0006
18/08/02 19:20:03 INFO mapreduce.Job: The url to track the job: http://instance-1:8088/proxy/application_1533231535061_0006/
18/08/02 19:20:03 INFO mapreduce.Job: Running job: job_1533231535061_0006
18/08/02 19:20:11 INFO mapreduce.Job: Job job_1533231535061_0006 running in uber mode : false
18/08/02 19:20:11 INFO mapreduce.Job: map 0% reduce 0%
18/08/02 19:20:21 INFO mapreduce.Job: map 33% reduce 0%
18/08/02 19:20:24 INFO mapreduce.Job: map 100% reduce 0%
18/08/02 19:20:25 INFO mapreduce.Job: Job job_1533231535061_0006 completed successfully
18/08/02 19:20:25 INFO mapreduce.Job: Counters: 31
File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=622830 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=295 HDFS: Number of bytes written=48 HDFS: Number of read operations=12 HDFS: Number of large read operations=0 HDFS: Number of write operations=6 Job Counters Killed map tasks=1 Launched map tasks=3 Other local map tasks=3 Total time spent by all maps in occupied slots (ms)=27404 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=27404 Total vcore-milliseconds taken by all map tasks=27404 Total megabyte-milliseconds taken by all map tasks=28061696 Map-Reduce Framework Map input records=3 Map output records=3 Input split bytes=295 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=671 CPU time spent (ms)=4210 Physical memory (bytes) snapshot=616452096 Virtual memory (bytes) snapshot=5963145216 Total committed heap usage (bytes)=350224384 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=4818/08/02 19:20:25 INFO mapreduce.ImportJobBase: Transferred 48 bytes in 25.828 seconds (1.8584 bytes/sec)
18/08/02 19:20:25 INFO mapreduce.ImportJobBase: Retrieved 3 records.
18/08/02 19:20:25 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table forhive
18/08/02 19:20:25 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM
forhiveAS t LIMIT 118/08/02 19:20:25 INFO hive.HiveImport: Loading uploaded data into Hive
18/08/02 19:20:25 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
18/08/02 19:20:25 ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50) at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392) at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379) at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337) at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44) ... 12 more
After I did google for the same error I added HIVE_CONF_DIR also to my bashrc
export HIVE_HOME=/home/programmeur_v/softwares/apache-hive-1.2.2-bin
export HIVE_CONF_DIR=/home/programmeur_v/softwares/apache-hive-1.2.2-bin/conf
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HIVE_HOME/bin:$SQOOP_HOME/bin:$HIVE_CONF_DIR
All my Hadoop services are also up and running.
6976 NameNode
7286 SecondaryNameNode
7559 NodeManager
7448 ResourceManager
8522 DataNode
14587 Jps
I'm just unable to figure out what mistake I'm making here. Please guide!
