Should an email address be able to begin with these characters?

[troyhunt]

This is more of an open question than something I think we should immediately do:

I've just run this across a breach I'm loading now that has about 2.2M addresses found. Over 1k of them at the end of the file begin with one of the following characters:

1. \
2. /
3. `
4. '
5. {
6. }
7. |
8. !
9. ~

My get feel is that these chars should be stripped and are no valid use cases where they should legitimately exist at the beginning of the address (or probably anywhere in the address). Just by way of example:

I'm more inclined to strip these than include them, what do we all think?

[NeilHanlon]

Those are all completely valid characters to be used in email addresses, per RFC 5322. The only requirement for the local part of the domain with respect to first character is that it MUST not be a ..

That said, I am not really aware of any providers which permit this as an address or alias, so.. perhaps it's OK to drop them?

[junderw]

I might be inclined to include both stripped and non-stripped.

They are valid characters, but most email address validation will fail them.

[djcrabhat]

I believe only the . is restricted from being the first or last character of the local-part, so I think you've gotta take the weirdos.

"Without quotes, local-parts may consist of any combination of alphabetic characters, digits, or any of the special characters

! # $ % & ' * + - / = ? ^ _ ` . { | } ~

period (".") may also appear, but may not be used to start or end the local part, nor may two or more consecutive periods appear."

• https://datatracker.ietf.org/doc/html/rfc3696#section-3

[nemec]

Is there a pattern in the domain portion of the address? If it's all some set of custom domains, maybe they are valid. If it's gmail/yahoo/hotmail/outlook check the service's sign up rules, where those names are likely invalid.

[troyhunt]

Those are all completely valid characters to be used in email addresses, per RFC 5322.

Perhaps, but I suspect that in this case those chars aren't a valid part of the address and are instead delimiters that have been inadvertently included. That leaves us with trading off between either strictly adhering to the RFC and excluding those addresses, or parsing them out and including them despite the strict RFC definition. Which is better?

[troyhunt]

I might be inclined to include both stripped and non-stripped.

That gets very messy as it doubles up on the addresses which then affects all sorts of counts and stats. I'm really reticent to do that and would prefer to pick one approach over the other.

[troyhunt]

Is there a pattern in the domain portion of the address? If it's all some set of custom domains, maybe they are valid. If it's gmail/yahoo/hotmail/outlook check the service's sign up rules, where those names are likely invalid.

No, lots of different domains, I'm highly confident we're looking at parsing issues, not deliberately funky addresses.

[thompsonus]

I just sent myself an email starting with { from Gmail to Exim. All of the servers involved (including gmail, exim, rspamd, and dovecot) had no issues whatsoever.

I'd also advocate for adhering to RFC 3696 wherever possible. I've seen issues when systems reject perfectly legitimate but rarely seen email addresses. One of my emails has the super-short format of x@x.yy. It's my real email, but is often rejected by web form validation not respecting the RFC.

[thompsonus]

For good measure, I also tested with Exchange (MS365) and Roundcube. No issues with either one. It seems like a definite edge case, but still worthwhile to allow all valid email addresses.

[oxguy3]

If my email address is john.doe@example.com, I would like to be alerted if {john.doe@example.com appears in a data breach. That said, there is the tiny possibility that {john.doe@example.com is the correct address, so I think including both versions would be the best option.

[jaimevisser]

Even though these are valid characters, I feel more often then not they are probably going to be delimiters. If including them means I won't be alerted on a@b.c when you hit {a@b.c it'd be best to strip them.

I do think the question again becomes; should you handle this in this tool or somewhere else in the HIBP infrastructure?

[kingthorin]

I’m inclined to rephrase this: is it better to include characters that are almost certainly intended to be delimiters therefor excluding the correct address from HIBP, or strip the characters and include the correct address regardless of the RFC allowing them?

👆This. I think we have about five nines certainty that these are other peoples' failure to parse lists properly.

[accidentaldeveloper]

Do the email providers for these addresses support creating an address with the leading character?
If not, then as written they could not possibly be correct and you can strip the character without fear.

[nhairs]

That leaves us with trading off between either strictly adhering to the RFC and excluding those addresses, or parsing them out and including them despite the strict RFC definition. Which is better?

Timely, I've just been asking similar questions on email around "do I respect the RFCs or do I act sensibly / force others to act sensibly". Before doing research I was of the opinion of "force them to act sensibly".

Do the email providers for these addresses support creating an address with the leading character?

Using https://mailchimp.com/resources/most-used-email-service-providers/ as a reference.

Google: No

Yahoo: No

Outlook: No

AOL: No

Proton: No

Apple: No

Mail.com: No

GMX: No

Zoho: No

Personal Conclusion

tldr: strip them

Given that none of the above major email providers let you signup using these characters, I think it is a fair assumption that these are fail delimiters. In the (probably rare) case that it is not a failed delimiter then you should have been sensible and unfortunately you will miss out on this HIBP notification.

That said, I think there might be cases were it legitimately appears later in the address (i.e. where email providers allow the <mailbox>(+<label>)@<domain> format).

Aside: this ends up being a rabbit hole if you try to normalise it as some providers let you use . in the address which are stripped out when determining the actual mailbox to deliver to - but not all providers.
The RFC basically says it's up to the provider to how they handle it, so unless you are going to start checking MX records or probing SMTP server headers you can't really make any assumptions of what you can and can't normalise in an email address for sending.

The sending part is important in this context because (I assume) it will be used for notifications. Contrast that with searching on the web UI where you could have an index of normalised email addresses to help users who have taken advantage of things like . stripping or + labels from their email provider.

Questions

That gets very messy as it doubles up on the addresses which then affects all sorts of counts and stats. I'm really reticent to do that and would prefer to pick one approach over the other.

If there is a concern about users not receiving notifications could we (and by we I mean @troyhunt):

1. check if there are any subscribed emails that have such special characters
2. if there are funky emails, send notifications using both the stripped and raw emails, but only add the stripped emails to the database

[troyhunt]

That was a really useful set of screen caps @nhairs, thank you!

I think the only reasonable conclusion here is that due to the lack of support for these chars by mainstream email providers, the unlikelihood of anyone legitimately using them and the real world impact of an email address being missed, the chars should be stripped and the RFC be damned 🙂

That now makes this issue a feature request. I'll add some tests for this later, I need to go back an analyse precisely how those strings were formatted originally.