Checksums

##Overview## Checksums are used to verify the integrity of data in HDFS.

##Use Cases##

###File Creation### Checksums are created by the client when they create a file. The create() method is called with a ChecksumOpt object, which specifies the options for the checksum. The options are then passed to the createChecksum() method on the DfsClientConfig object held by the DFSClient. The DFSOutputStream used by the client to write data extends FSOutputSummer. The FSOutputSummer is used to generate checksums for chunks of data, then write the checksums along with the data to the underlying output stream. The DataNode stores the checksums in the metadata files corresponding to each block.

###Appends### A checksum is created for each chunk in each new block that is appended to the file.

###Reads### When a client wants to read a data block, the relevant DataNode will return the checksum as well. When the DataNode calls BlockSender.sendBlock(), the data sent is in the form of a 5 byte checksum header followed by packets containing the actual data. The checksum header contains the checksum type (1 byte) followed by the number BYTES_PER_CHECKSUM, which specifies the length of each chunk of data. (4 bytes). Each packet consists of a header, followed by the checksums for the data, followed by the actual data. The client uses it to verify that the data is still correct. If it is not, the client alerts the NameNode that this copy is corrupt.

###Scanning### Occasionally, the DataNode runs a scanner that recomputes the checksums and makes a log of scanner runtime. If the stored checksum doesn't match the computed, the DataNode notifies the NameNode that this copy is corrupt.

##Main Data Structures/Relevant Classes## Checksums are included in the block sent to the client through the use of a DataTransferProtocol object's writeBlock method.
The data of a block is read from the disk and sent to the client using a BlockSender object. This object can be constructed with booleans specifying whether the DataNode should verify the checksum, whether the DataNode should send the checksum to the client, and whether a corrupt checksum can be accepted. A checksum is represented as a [DataChecksum] (https://github.com/apache/hadoop/blob/d5f5a6475b19a52bf54f97cd8843408265b888f0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java) object. In addition to holding a checksum, this object includes convenience methods such as one to verify a block has the correct checksum. Apache's implementation uses the java.util.zip.Checksum as the object that actually calculates the checksum. Is a checksum fails verification, a ChecksumException is thrown with a message giving the position of the failure.