Content of the fsimage hdfs

问题:

I have a question on what is the metadata in the fsimage all about. I read that All mutations to the file system namespace, such as file renames, permission changes, file creations, block allocations are inside the fsimage. But the block location data as well? Does it contain the information about where (on which datanode) the blocks are stores as well? I get from this source: http://bradhedlund.com/2011/09/10/understanding-hadoop-clusters-and-the-network/ that the metadata on where blocks is stored is build by the block repots of the datanodes. Is this true? So the Fsimage does not contain information about the block locations?

回答1:

Namenode maintains two type of data

Block Location data : Since files are chopped into blocks, NN should know which piece is where. This data is kept in memory and never persisted on disk, DNs talk to NN periodically and share the blockreport.

file system (metadata) : such as the file system hierarchy, permissions, etc. This info is persisted to the disk

when namenodes starts up it loads "snapshot" of filesystem from fsimage and applies the edit logs from edits onto it, after this process we get a new snapshot. from this point on namenode can accept files system requests from clients / DNs

回答2:

Yes as far as I know fsimage does not contains any information about blocks. This information is stored by data nodes. Namenode gets this information when it starts up from datanodes.

回答3:

Hadoop provides a tool that converts the fsimage file into human readable formats. http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html

Sample output:

bin/hdfs oiv -i fsimagedemo -p Indented -o fsimage.txt

   FSImage
     ImageVersion = -19
     NamespaceID = 2109123098
     GenerationStamp = 1003
     INodes [NumInodes = 12]
       Inode
         INodePath =
         Replication = 0
         ModificationTime = 2009-03-16 14:16
         AccessTime = 1969-12-31 16:00
         BlockSize = 0
         Blocks [NumBlocks = -1]
         NSQuota = 2147483647
         DSQuota = -1
         Permissions
           Username = theuser
           GroupName = supergroup
           PermString = rwxr-xr-x
   ...remaining output omitted...