Transformation md5 needs to be implemented inside libmodsecurity.
The documentation of the transformation can be found here:
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#md5
The file already exist in the repository:
https://github.com/SpiderLabs/ModSecurity/blob/libmodsecurity/src/actions/transformations/md5.cc