Namespace Manager

Namespace Manager

Overview

The namespace manager will handle all the metadata regarding the namespace of the file system. It will basically use a tree of INode's to track the directory structure of the filesystem. Within this directory structure, we will track namespace, access permissions, access times, and disk utilization. The namespace manager must also properly synchronize namespace operations such as file creation or deletion among concurrent accesses.

The namespace manager will work closely with the Operation Log to ensure that all its in memory data structures are persisted to disk and can be recovered in the case of server restarts.

Interactions with Operation Log

The namespace information and file->chunk mapping must be durable, which means that we must coordinate with the Operation Log. A set of possible Edits will have to be determined for us to correctly rebuild the metadata on NameNode server restart.

Use Cases

Use Cases Overview

Most use cases of FSNamesystem (the primary class for the Namespace Manager) follow a similar pattern:

1. Acquire a read/write lock on the FSNamesystem
2. Throw an exception if the operation is not allowed with the HA context
3. Throw an exception if the NameNode is still in safe mode
4. Run the actual operation: createSnapshot, delete, mkdir, etc...
   1. Acquire the corresponding directory read/write lock
   2. Run the operation
   3. Release the corresponding directory read/write lock
5. Release FSNameSystem read/write lock
6. Sync the edit log which was likely modified by task 4.2 (Run the operation)

Use Cases Examples

• Client wants to create a file F with replication factor R and block size B.

• FSNamesystem::startFileInt

  • acquires a write lock on FSNamesystem
  • checks that R is within the acceptable bounds set by BlockManager
  • checks that B is not smaller than minBlockSize
  • acquires a write lock on FSDirectory
  • creates ancestor directories specified in the path, if they don't exist
  • creates a new INodeFile for F and adds it to FSDirectory
  • LeaseManager creates a lease for F and client
  • releases the write lock on FSDirectory
  • releases the write lock on FSNamesystem

• Client wants to locate a segment of File F from position P to P + L.

  • FSNameSystem::getBlockLocations
    • acquires read lock on FSNamesystem
    • acquires read lock on FSDirectory
    • asks the BlockManager for the blocks corresponding to the segment to be read in F
    • releases read lock on FSDirectory
    • releases read lock on FSNamesystem
    • returns the blocks, sorted by distance to the client

• Client wants to delete a Source S.

  • FSNamesystem::delete
  • acquire write lock on FSNamesystem
  • recursively delete the directory from the lowest files/directories and moving up the directory structure
    • delete Directory:
      • acquire Directory write lock
      • check if we have permissions to delete
      • modify snapshot tracking for the deleted directory
      • delete the INode
      • update the edit logs with the delete operation
      • remove the files from BlockManager:
      • invalidate the blocks
      • remove leases and INodes for this directory
    • delete File:
      • acquire File write lock
      • modify snapshot tracking for the deleted files
      • update the edit logs with the delete operation
      • invalidate the files in BlockManager:
      • remove leases and INodes for this directory
  • release FSNamesystem write lock
  • lazy removal of blocks with removeBlocks()

• Snapshot system decides to create a snapshot of the namespace. Note that this only captures the NameSpace component of snapshotting. The actual data snapshot is performed by SnapshotManager.

• FSNamesystem::createSnapshot

  • acquire a write lock on FSNamesystem
  • acquire a write lock on the top level directory
  • call SnapshotManager::createSnapshot on the top level directory
  • release the write lock on the top level directory

INode Data Structure

The directory state is stored in a tree-like structure of INodes. The basic INode stores the following information:

1. Id
2. Name
3. Parent
4. Mod time
5. Access time
6. Permissions
7. Quota (2 types of quotas: name and space. Name quota: limit on number of files and directories rooted at any given inode. Space quota: as it sounds, a limit on the number of bytes consumed by files residing in the tree rooted at this directory)

There are two primary types of INodes. INodeFile, which tracks:

1. BlockInfo List
2. Replication factor
3. Block sizes
4. Which block to currently append to
5. Whether the file is under construction (do more work on figuring out what this means)

INodeDirectory, which tracks:

1. Children

INodeFile stores an array of BlockInfo about a given file BlockInfo contains information on:

1. Replication factor
2. Block collection id
3. DataNode Storage Info: (which nodes it is held on)
4. Block UC (under construction) state
   1. http://blog.cloudera.com/blog/2015/02/understanding-hdfs-recovery-processes-part-1/

Question: Do INodes/BlockInfo contain info on which files are currently open/closed

Questions

Question

If a client attempts to write to a file F, you should maybe interact with LeaseManager to either

1. make sure this client has a lock for the file or
2. try and get the client a lock for the file, or is this not your job?

Response

All namespace ops that also touch data will need to revoke outstanding leases.

##References

https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html