Hadoop: …be replicated to 0 nodes instead of minRe

问题:

I'm getting the following error when attempting to write to HDFS as part of my multi-threaded application

could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and no node(s) are excluded in this operation.

I've tried the top-rated answer here around reformatting but this doesn't work for me: HDFS error: could only be replicated to 0 nodes, instead of 1

What is happening is this:

1. My application consists of 2 threads each one configured with their own Spring Data PartitionTextFileWriter
2. Thread 1 is the first to process data and this can successfully write to HDFS
3. However, once Thread 2 starts to process data I get this error when it attempts to flush to a file

Thread 1 and 2 will not be writing to the same file, although they do share a parent directory at the root of my directory tree.

There are no problems with disk space on my server.

I also see this in my name-node logs, but not sure what it means:

2016-03-15 11:23:12,149 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 1 to reach 1 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
2016-03-15 11:23:12,150 WARN org.apache.hadoop.hdfs.protocol.BlockStoragePolicy: Failed to place enough replicas: expected size is 1 but only 0 storage types can be selected (replication=1, selected=[], unavailable=[DISK], removed=[DISK], policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]})
2016-03-15 11:23:12,150 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 1 to reach 1 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) All required storage types are unavailable:  unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
2016-03-15 11:23:12,151 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 10.104.247.78:52004 Call#61 Retry#0
java.io.IOException: File /metrics/abc/myfile could only be replicated to 0 nodes instead of [2016-03-15 13:34:16,663] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)

What could be the cause of this error?

Thanks

回答1:

This error is caused by the block replication system of HDFS since it could not manage to make any copies of a specific block within the focused file. Common reasons of that:

1. Only a NameNode instance is running and it's not in safe-mode
2. There is no DataNode instances up and running, or some are dead. (Check the servers)
3. Namenode and Datanode instances are both running, but they cannot communicate with each other, which means There is connectivity issue between DataNode and NameNode instances.
4. Running DataNode instances are not able to talk to the server because of some networking of hadoop-based issues (check logs that include datanode info)
5. There is no hard disk space specified in configured data directories for DataNode instances or DataNode instances have run out of space. (check dfs.data.dir // delete old files if any)
6. Specified reserved spaces for DataNode instances in dfs.datanode.du.reserved is more than the free space which makes DataNode instances to understand there is no enough free space.
7. There is no enough threads for DataNode instances (check datanode logs and dfs.datanode.handler.count value)
8. Make sure dfs.data.transfer.protection is not equal to “authentication” and dfs.encrypt.data.transfer is equal to true.

Also please:

• Verify the status of NameNode and DataNode services and check the related logs
• Verify if core-site.xml has correct fs.defaultFS value and hdfs-site.xml has a valid value.
• Verify hdfs-site.xml has dfs.namenode.http-address.. for all NameNode instances specified in case of PHD HA configuration.
• Verify if the permissions on the directories are correct

Ref: https://wiki.apache.org/hadoop/CouldOnlyBeReplicatedTo

Ref: https://support.pivotal.io/hc/en-us/articles/201846688-HDFS-reports-Configured-Capacity-0-0-B-for-datanode

Also, please check: Writing to HDFS from Java, getting "could only be replicated to 0 nodes instead of minReplication"

回答2:

I had a similar issue recently. As my datanodes (only) had SSDs for storage, I put [SSD]file:///path/to/data/dir for the dfs.datanode.data.dir configuration. Due to the logs containing unavailableStorages=[DISK] I removed the [SSD] tag, which solved the problem.

Apparently, Hadoop uses [DISK] as default Storage Type, and does not 'fallback' (or rather 'fallup') to using SSD if no [DISK] tagged storage location is available. I could not find any documenation on this behaviour though.

回答3:

Check if the jps command on the computers which run the datanodes show that the datanodes are running. If they are running, then it means that they could not connect with the namenode and hence the namenode thinks there are no datanodes in the hadoop system.

In such a case, after running start-dfs.sh, run netstat -ntlp in the master node. 9000 is the port number most tutorials tells you to specify in core-site.xml. So if you see a line like this in the output of netstat

tcp        0      0 120.0.1.1:9000        0.0.0.0:*               LISTEN       4209/java

then you have a problem with the host alias. I had the same problem, so I'll state how it was resolved.

This is the contents of my core-site.xml

<configuration>
   <property>
       <name>fs.default.name</name>
       <value>hdfs://vm-sm:9000</value>
   </property>
</configuration>

So the vm-sm alias in the master computer maps to the 127.0.1.1. This is because of the setup of my /etc/hosts file.

127.0.0.1       localhost
127.0.1.1       vm-sm
192.168.1.1     vm-sm
192.168.1.2     vm-sw1
192.168.1.3     vm-sw2

Looks like the core-site.xml of the master system seemed to have mapped on the the 120.0.1.1:9000 while that of the worker nodes are trying to connect through 192.168.1.1:9000.

So I had to change the alias of the master node for the hadoop system (just removed the hyphen) in the /etc/hosts file

127.0.0.1       localhost
127.0.1.1       vm-sm
192.168.1.1     vmsm
192.168.1.2     vm-sw1
192.168.1.3     vm-sw2

and reflected the change in the core-site.xml, mapred-site.xml, and slave files (wherever the old alias of the master occurred).

After deleting the old hdfs files from the hadoop location as well as the tmp folder and restarting all nodes, the issue was solved.

Now, netstat -ntlp after starting DFS returns

tcp        0      0 192.168.1.1:9000        0.0.0.0:*               LISTEN ...
...

回答4:

I had the same error, re-starting hdfs services solved this issue. ie re-started NameNode and DataNode services.

回答5:

In my case it was a storage policy of output path set to COLD.

How to check settings of your folder:

hdfs storagepolicies -getStoragePolicy -path my_path

In my case it returned

The storage policy of my_path
BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}   

I dumped the data else where (to HOT storage) and the issue went away.

回答6:

You may leave HDFS safe mode:

hdfs dfsadmin -safemode forceExit