How to add an hard disk to hadoop

Question

I installed Hadoop 2.4 on Ubuntu 14.04 and now I am trying to add an internal sata HD to the existing cluster.

I have mounted the new hd in /mnt/hadoop and assigned its ownership to the hadoop user

Then I tried to add it to the configuration file as follow:

<configuration>
   <property>
       <name>dfs.replication</name>
       <value>2</value>
   </property>

   <property>
       <name>dfs.name.dir</name>
       <value>file:///home/hadoop/hadoopdata/hdfs/namenode, file:///mnt/hadoop/hadoopdata/hdfs/namenode</value>
   </property>

   <property>
       <name>dfs.data.dir</name>
       <value>file:///home/hadoop/hadoopdata/hdfs/datanode, file:///mnt/hadoop/hadoopdata/hdfs/datanode</value>
   </property>
</configuration>

Afterwards, I started the hdfs:

Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-namenode-hadoop-Datastore.out
localhost: starting datanode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-datanode-hadoop-Datastore.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-secondarynamenode-hadoop-Datastore.out

It seems that it does not fire up the second hd

This is my core-site.xml

<configuration>
   <property>
       <name>fs.default.name</name>
       <value>hdfs://localhost:9000</value>
   </property>
</configuration>

In addition I tried to refresh the namenode and I get a connection problem:

Refreshing namenode [localhost:9000]
refreshNodes: Call From hadoop-Datastore/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
Error: refresh of namenodes failed, see error messages above.

In addition, I can't connect to the Hadoop web interface. It seems that I have two related problems:

 1) A connection problem
 2) I cannot connect to the new installed hd

Are these problem related? How can I fix these issues?

Thanks

EDIT

I can ping the localhost and I can access localhost:50090/status.jsp

However, I cannot access 50030 and 50070

Answer 1

<property>
   <name>dfs.name.dir</name>
   <value>file:///home/hadoop/hadoopdata/hdfs/namenode, file:///mnt/hadoop/hadoopdata/hdfs/namenode</value>
</property>

This is documented as:

Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy.

Are you sure you need this? Do you want your fsimage to be copied in both locations, for redundancy? And if yes, did you actually copy the fsimage on the new HDD before starting the namenode? See Adding a new namenode data directory to an existing cluster.

The new data directory (dfs.data.dir) is OK, the datanode should pick it up and start using it for placing blocks.

Also, as a general troubleshooting advice, look into the namenode and datanode logs for more clues.

Answer 2

Regarding your comment: "sudo chown -R hadoop.hadoop /usr/local/hadoop_store." The owner has to be hdfs user. Try: sudo chown -R hdfs.hadoop /usr/local/hadoop_store.