HDFS federation

Question

I have few basic questions regarding HDFS Federation.

Is it possible to read file created on one name node from another name node which is in the cluster federation?

Does current version of Hadoop supports this feature?

Answer 1

No. It is not possible to do that.

Answer 2

Let me explain how Name node federation works as per Apache web site

NameNode:

In order to scale the name service horizontally, federation uses multiple independent Namenodes/namespaces.

The Namenodes are federated; the Namenodes are independent and do not require coordination with each other.

The Datanodes are used as common storage for blocks by all the Namenodes. Each Datanode registers with all the Namenodes in the cluster. Datanodes send periodic heartbeats and block reports. They also handle commands from the Namenodes.

In Summary,

Name nodes are mutually exclusive and does not require communication between them. Data nodes can be shared across multiple name nodes.

To answer your question, It's not possible. if the data is written one name node, you have to contact that name node only to fetch the data. You can't ask other name node.

Regarding your updated comments on data replication,

When the replication factor is three, HDFS’s placement policy is to put one replica on one node in the local rack, another on a different node in the local rack, and the last on a different node in a different rack - as per official documentation.

You can use this feature and get the data from other data centre if you have failures in local RAC. But note that you are reading data from one Federated Namenode and not from other Federated Namenode.

One Federated Namenode can't read data from other Federated Namenode. But they can share same set of Datanodes for read and write operations.

EDIT:

With-in each Federation, you can have automatic fail over of Namenode. If Active NameNode fails in a Federation, Stand-by Namenode will take over Active Namenode responsibilities.

Refer to below SE post for more details.

How does Hadoop Namenode failover process works?

Answer 3

The default behaviour of the block replication policy in hadoop can be modified by extending the BlockPlacementPolicy interface and pointing the class to the dfs.block.replicator.classname property in the Hadoop configuration files.

Please research on BlockPlacementPolicy to get a better picture.

You can actually modify where your blocks can be placed in the cluster.