How to Authenticate WebHDFS with c#

Question

I have been attempting to upload files using c# into Hadoop using the WebHDFS REST API. This code works fine:

    using (var client = new System.Net.WebClient())
    {
       string result = client.DownloadString("http:/ /host:50070/webhdfs/v1/user/myuser/?op=LISTSTATUS");
       client.DownloadFile("http:/ /host:50070/webhdfs/v1/user/myuser/tbible.txt?user.name=myuser&op=OPEN","d:\tbible.txt");
    }

This code gets a 403 Forbidden:

    using (var client = new System.Net.WebClient())
    {

       client.UploadFile("http:/ /host:50070/webhdfs/v1/user/myuser/?user.name=myuser&op=CREATE", "PUT", "d:\bible.txt");
    }

I have tried adding a network credential, with no luck. How do I authenticate to our Cluster from .NET? The Cluster is Hortonworks HDP1.3 on RHEL5. (The extra spaces in this post are to keep http:/ / from being a link) Also, I would have liked to use Microsoft's hadoop SDK, but it is alpha and wont compile in my environment :(

Answer 1

Make sure that you are writing to a directory that is under the group which WebHDFS operates under. By default this is hdfs.

A quick way to check this doing hadoop fs -ls on the directory's parent directory to get the group permission settings (the second column that may look like a username).