how to connect to HBase / Hadoop Database using C#

Question

Recently, Exploring Microsoft HDInsight Hadoop for Windows.But don't know where to began and start using apache hadoop with c# / asp.net mvc.

i know http://hadoopsdk.codeplex.com/ is best available resource to start, but can't find documentation to start from scratch? like creating cluster,database and then connecting it to C# app.

Answer 1

check this out https://github.com/hdinsight/hbase-sdk-for-net from HDInsight team

Answer 2

Microsoft released a preview of their .NET ProtoBuf client for Phoenix/Hbase on Nuget.org in June. I've found it worked well but the API was unfamiliar to me. As I learned it, I implemented a .NET Framework System.Data IDbConnection, IDbCommand et al wrapper API around it, named Garuda.Data. It makes access to Phoenix/Hbase from .NET code feel almost like SqlConnection, SqlCommand, etc for SQL Server - Check it out: https://www.nuget.org/packages/Garuda.Data/

Here is the github project repo: https://github.com/dwdii/GarudaUtil

And some example code:

using (IDbConnection phConn = new PhoenixConnection())
{
    phConn.ConnectionString = cmdLine.ConnectionString;

    phConn.Open();

    using (IDbCommand cmd = phConn.CreateCommand())
    {
        cmd.CommandText = "SELECT * FROM GARUDATEST";
        using (IDataReader reader = cmd.ExecuteReader())
        {
            while(reader.Read())
            {
                for(int i = 0; i < reader.FieldCount; i++)
                {
                    Console.WriteLine(string.Format("{0}: {1}", reader.GetName(i), reader.GetValue(i)));
                }
            }
        }
    }                        
}

Answer 3

You might find the project HBase-sharp useful. You can visit their bitbucket page for detailed info. Looks promising to me. It also includes an example. You can find it here.

HTH

Answer 4

The easiest way to get started is to use the HDInsight service on Azure (which is still in preview, but works well). That way you can just log into your azure portal and spin up a cluster, which will be linked to your azure storage.

If you really don't want to go to the cloud, then you can download the HDInsight developer preview package through the web platform installer, which will create a single node local cluster. Alternatively, if you're interested in 'just hadoop' then you could try using the VMs provided by Hortonworks or Cloudera.

After that, it's probably worth starting with Hive if you are familiar with SQL. The O'Reilly Definitive Guide to Hadoop is pretty good on that.

From the .NET point of view, the hadoopsdk on codeplex is a good place to start.

As far as very basic connection examples go, try this blog for an example, but note that the connection for HDInsight is slightly different now it's all using the templeton interface, so this will get you going:

var db = new HiveConnection(
        webHCatUri: new Uri("http://localhost:50111"),
        userName: (string) "hadoop", password: (string) null);
var result = db.ExecuteHiveQuery("select * from w3c");

If you are looking to do full on MapReduce on HDInsight, then you probably want to take a look at the C# MapReduce examples with the sdk on codeplex.

Note that the default HDInsight install also comes with some good samples, which include a bit of data to play with and some powershell scripts and .NET code to get you started.

Enjoy!