Not sure exactly how this would work but currently only initial messages from users who you have not talked to before will send you notifications.
Ideally this should be configurable. Although the current implementation (in my opinion) covers most use cases.