Hadoop permission issue

Question

I've homebrew installed hadoop but now having permission control problems when doing hadoop namenode -format and ./start-all.sh command.

I think it's because I put settings in "core-site.xml". The "hadoop.tmp.dir" I put "/tmp/${name}" under.

Now it's giving me error in namenode -format as: can't create folder, permission denied. Even I sudo this command, but in the start-all.sh command, still a lot of permissions are denied. I tried to sudo start-all.sh but the password (I only use this pass for my admin on mac) but denied also.

I think it's because of the permission issues. Is there anyway I can fix it?

Thanks!

Answer 1

On your local system, it looks like you do not have the hduser user created.

As a typical setup process, it is a good process to create a hadoop group and a hduser user added to that group.

You can do that with the root/super user account with the following command:

$ sudo adduser --ingroup hadoop hduser

This assumes you have the hadoop group setup. If that is not setup, you can create a group with:

$ sudo addgroup hadoop

Answer 2

So when you run Hadoop it stores things in the data, name, and tmp dirs that you configure in the hdfs-site.xml file. If you don't set these settings they will point to ${hadoop.tmp.dir}/dfs/data, in your case the /tmp dir. This is not where you want your data stored. You will first need to add these to your hdfs config file, among other settings.

On master :

<property>
    <name>dfs.data.dir</name>
    <value>/app/hadoop/data</value>
</property>
<property>
    <name>dfs.name.dir</name>
    <value>/app/hadoop/name</value>
</property>

On slaves :

<property>
    <name>dfs.data.dir</name>
    <value>/app/hadoop/data</value>
</property>
<property>
    <name>dfs.name.dir</name>
    <value>master:/app/hadoop/name</value>
</property>

Now once this is done you must actually make those directories. So create the following dirs on master : /app/hadoop/name, /app/hadoop/data, and /app/hadoop/tmp. Create the same on slaves except the name dir.

Now you need to set the permissions so that they can be used by Hadoop. The second line just to be sure.

sudo chown <hadoop user>:<hadoop user> /app/hadoop/name /app/hadoop/data /app/hadoop/tmp
sudo chmod 0777 /app/hadoop/name /app/hadoop/data /app/hadoop/tmp

Try that, see if it works. I can answer questions if it's not the whole answer.