Extremely slow parsing for a specific email (2 minutes slow, I can share EML privately)

[valeriansaliou]

Hello Andris,

We've been happy users of mailparser for about 8 years now (time flies!). Running a large scale email ingestion production system on it.

2 days ago, we started experiencing event loops blockages that lasted from 2 to 6 minutes per micro-service running mailparser. Recurringly. We've managed to extract the recurring email that causes this issue, in EML format.

The email has an enormous text content, we're talking about 2MB encoded in this part:

Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8;
 format=flowed

The email eventually gets parsed by the library, but it takes a tremendous time.

Could I suggest adding some configurable/opt-in safety limits on the text/HTML parts size that get parsed?

I unfortunately cannot enforce those limits based on the total size of the raw mail Buffer being passed to the library, before running the actual parsing, since most emails would be MB heavy due to attachments. I believe those limits will have to be enforced within the mailparser library.

Let me know if I should send you the original EML causing this hang issue over email for reproduction (I cannot share it here since it comes from one of our user, and thus is a private email).

Adding some more context, parsing of this EML has been tested to take:

• 15s on 1x Apple M1 Pro core
• 2 minutes on loaded 1x Intel Xeon core (circa 2019)

Valerian.

[andris9]

Yes, format=flowed can cause slowness. The un-wrapping is done with a single regex, and its worst-case complexity grows exponentially with the input length. Huge flowed blocks are rare, but when they appear they will block the event loop exactly as you saw. I don't have a good solution for now. Avoid parsing such EML files or try to improve the format=flowed handling in Mailparser's source code.

[valeriansaliou]

What solution would you agree on?

1. Adding some safety limit in the library to reject or cut off the text content before it is passed to the libmime library.
2. Complete rewrite of the decodeFlowed() function, but I'm not sure it'll help given the email I currently have here

I can work on a fix. Just need your input on that :)

[valeriansaliou]

Here you go, here's my optimized function for decodeFlowed(). On my test data set, it produces the same exact result, in only 11ms (vs 9098ms with the previous decodeFlowed()):

/**
 * Decodes a string from a format=flowed soft wrapping.
 *
 * @param {String} str Plaintext string with format=flowed to decode
 * @param {Boolean} [delSp] If true, delete leading spaces (delsp=yes)
 * @return {String} Mime decoded string
 */
decodeFlowedFast(str, delSp) {
    str = (str || '').toString();

    let lines = str.split(/\r?\n/);

    let result = [],
        buffer = null;

    // remove soft linebreaks
    // soft linebreaks are added after space symbols
    for (let i = 0; i < lines.length; i++) {
        let line = lines[i];

        let isSoftBreak = buffer !== null && / $/.test(buffer) && !/(^|\n)-- $/.test(buffer);

        if (isSoftBreak) {
            if (delSp) {
                // delsp adds space to text to be able to fold it
                // these spaces can be removed once the text is unfolded
                buffer = buffer.slice(0, -1) + line;
            } else {
                buffer += line;
            }
        } else {
            if (buffer !== null) {
                result.push(buffer);
            }

            buffer = line;
        }
    }

    if (buffer) {
        result.push(buffer);
    }

    // remove whitespace stuffing
    // http://tools.ietf.org/html/rfc3676#section-4.4
    return result.join('\n').replace(/^ /gm, '');
}

libmime PR open: nodemailer/libmime#25

[andris9]

Thanks, I merged it, but I'm not sure when I can release the fix for mailparser - mailsplit had pending changes that got released with this update, but now mailparser tests are failing and I'm not sure when I have time to resolve these.

[valeriansaliou]

Thank you for the fast merge!

I've ran some tests and found out that:

• 5 of the failing tests of mailparser are coming from this bump: zone-eu/mailsplit@16eb658 ; trailing line breaks are now appearing in parsed text results, which matches this commit description fix message-splitter. follow line breaks consistenly
• 1 of the failing test comes from my libmime fix, which looks OK: the difference is a trailing \n at the end of the parsed email text that does not appear anymore with my fix, and used to appear before. The rest of the content matches. I think it's acceptable.
• 8 remaining failed tests are about Attachment info - Included integrity (checksum mismatch), I can't say for sure if those ones are safe to resolve w/o further checking.

[andris9]

Yes, it seems we still need to work with zone-eu/mailsplit@16eb658

[andris9]

The failing test with your fix is ok, the trailing newline is not required and the test can be altered. The other failing attachment tests are also from the large mailsplit change, these are not from your fix, so you do not have to worry about these. I'm not sure when we can deal with the mailsplit issue because the guy who did the actual change went on vacation for the next 3 weeks.

[valeriansaliou]

No problem, I've upgraded my microservices to npm install mailparser from the Git patch URL and it does the job for now. Thanks for the fast help on this!

Closing this for now.