From the book "Hadoop The Definitive Guide", under the topic Namenodes and Datanodes it is mentioned that:
The namenode manages the filesystem namespace. It maintains the filesystem tree and the metadata for all the files and directories in the tree. This information is stored persistently on the local disk in the form of two files: the namespace image and the edit log.
secondary namenode, which despite its name does not act as a namenode. Its main role is to periodically merge the namespace image with the edit log to prevent the edit log from becoming too large.
I am having some confusion with these files namespace and edit log.
Namespace image is for storing the metadata.
So, my questions are
- What is the edit log? And what is its role?
- Can you explain the statement "Its main role is to periodically merge the namespace image with the edit log to prevent the edit log from becoming too large."?
Initially when the NameNode first starts up the
fsimage
file will itself be empty. When ever NameNode receives a create/update/delete request then that request is first recorded toedits
file for durability once persisted in theedits
file an in-memory update is also made. Because all read requests are served from in-memory snapshot of the metadata.So, you see the
edits
file keeps on growing with out bounds at this point. Now if the NameNode is restarted or for some reason went down and brought back up, it has no memory representation of the metadata so, it has to read theedits
file and rebuild the snapshot in-memory, which might take a while based on theedits
file size.As
edits
itself is a WAL (write ahead log) all the events have to written one after another (append only), there could be no updates in the file to prevent random disk seeks.To prevent this overhead (or to keep
edits
file manageable) SecondaryNameNode was introduced. The sole purpose of the SNN is to make sure theedits
file does not grow out of bounds. So, by default SNN triggers a process called ascheckpointing
when everedits
file reaches 64MB or for every one hour (which ever comes first).Checkpointing process it self is simple, the SNN tells the NN to role its current
edits
log and create a new edits files callededits.new
, SNN then copies over the fsimage and edits file from NN and starts applying the events in the edits file to already existing fsimage file (brought from NN), once completed the new fsimage file is sent back to NN and the NN replaces the existing fsimage with the new one sent over by SNN and renames theedits.new
toedits
. The NN now has a current version offsimage
which has events applied from theedits
file.So, that if the NameNode is restarted after checkpointing has been completed, NameNode has to just load the
fsimage
to memory and apply just the recents updates fromedits
log (which got filled after the checkpoint has been completed) to make sure it has the up to date view of the namespace which more efficient.