I have been playing with Cloudera and I define the number of clusters before I start my job then use the cloudera manager to make sure everything is running.
I’m working on a new project that instead of using hadoop is using message queues to distribute the work but the results of the work are stored in HBase. I might launch 10 servers to process the job and store to Hbase but I’m wondering if I later decided to add a few more worker nodes can I easily (read: programmable) make them automatically connect to the running cluster so they can locally add to clusters HBase/HDFS?
Is this possible and what would I need to learn in order to do it?
It can be done without restarting the hadoop cluster. As per this document, You can add nodes in the includes file and and making some changes in the hdfs-site.xml and mapred-site.xml files, you can achieve this. Detailed instructions are given in the this document
Here is the documentation for adding a node to Hadoop and for HBase. Looking at the documentation, there is no need to restart the cluster. A node can be added dynamically.
If I understand you correctly you have workers you coordinate by yourself that connect to HBase to save their data. you can have as many of those as you need and they can connect to Hbase as they're added (as long as they can see the zookeeper quorum).
If you are talking about increasing the Hadoop cluster. Since you already use Cloudera you can do that via the cloudera Manager REST API or the the Java client someone implemented for it
Following steps should help you launch the new node into the running cluster.
If you don't want to restart the services on the NN, when you add a new node. I would say add the names ahead to the slaves configuration file. So they report as decommission/dead nodes until they are available. Following the above DataNode only steps. Again this not the best practice.