Edit: Can no longer reproduce with same setup, issue seems fixed at googles side.

TL;DR: See title. How can I tell Google they're probably processing their mail wrong?

After setting up the Matrix Authentication Service (MAS) and exim-relay as mail server, I noticed verification mails sent from the service are often in the spam directory.

When digging deeper, I found out the mails are failing DKIM authentication. This was weird because DKIM is set up correctly, as verified by other mail providers and online DKIM test tools such as DMARC Tester.

Searching online for "gmail fails DKIM authentication, while other providers pass", I found regular reports, posts or similar without resolution, or unrelated resolutions such as DKIM alignment.

Using meld, I compared the original source of mails as received by gmail with those of other providers, and found a difference:

In other providers, the header for "From:" and "Reply-To:" fields are presented with double-quotes:

From: "John Smith" <j.smith@example.com>
Reply-To: "John Smith" <j.smith@example.com>

In gmail, where DKIM fails, there are no double-quotes:

From: John Smith <j.smith@example.com>
Reply-To: John Smith <j.smith@example.com>

As this should be the raw source each, I ruled out presentation issues and digged deeper.

I found out, that specifically the rust crate lettre, as used by the MAS, encodes names with whitespace using double-quotes. Further, from researching a bit more and reading RFC 2822 sections 3.2.4 and 3.2.5, I come to the conclusion that whitespace needs no quoting in mail headers.

I created issues upstream and downstream to report the issue at lettre and MAS, particularly that their mails are failing DKIM checks at gmail:

If you've read that far, you probably wonder why I post all of that? For one, to provide another data point for people scratching their heads over mail issues.

But other than that: I'm pretty sure the google mail servers should not strip the quotes before doing the DKIM check. I assume they have some kind of decode -> process -> encode workflow, that then simply encodes the headers again, this time without the quotes. But IMHO a correctly signed message should not lead to an authentication error, even if the contents are not perfectly encoded.

I would be curious on getting some feedback from some mail experts on what is happening here. This is not my field of expertise and I'm going by what I've learned over the past 48h.

top 34 comments

sorted by: hot top controversial new old
[+] 12 points 8 months ago* (14 children)
  • [–] [S] 3 points 8 months ago* (13 children)

    Ah interesting, I'm sending from my own domain and IP with DMARC set up to quarantine.

    Yes, the mail server is accepting it because gmail accepts the mail if either SPF or DKIM passes (not AND). My observation is that google for some reason sometimes puts the mail into spam, and sometimes into the regular mailbox. In both cases the headers show dkim=fail and spf=pass and I have no idea why it's not deterministic. I've also tested this with same/similar mail contents.

    Edit: To be honest, I also don't think that mail from my domain should be "sometimes" in the regular mailbox, if DKIM fails and DMARC has adkim=quarantine.

  • source
  • parent
  • hideshow 13 child comments
  • [+] 3 points 8 months ago* (last edited 8 months ago) (12 children)
  • [–] [S] 3 points 8 months ago (11 children)

    Yes, I've seen one other header change: gmail seems to (again sometimes?) enforce Message-ID fields in the header, and may add or change it if it doesn't match it's internal requirements. Interestingly, I've seen both my mailserver getting "rejected because missing message-id" messages, and messages passing to the mailbox, but with a google-added Message-ID in the raw source.

    For my specific case of DKIM failures, I've not noticed other differences.

    If I take the specific raw source from gmail, i.e. after processing by google, re-add the quotes, and manually check the DKIM signature, the signature passes. With other words, the quotes are literally the only relevant change in my case.

  • source
  • parent
  • hideshow 11 child comments
  • [+] 3 points 8 months ago* (last edited 8 months ago) (10 children)
  • [–] [S] 3 points 8 months ago (9 children)

    I think it's technically an encoding bug in lettre, which is used in matrix authentication services: https://github.com/lettre/lettre . As far as I can tell, exim is relaying the messages "correctly" or at least without altering them.

    I.e. lettre should not add quotes for whitespace. But also google shouldn't alter messages before authenticating. In an ideal world, both sides are fixed ^^

  • source
  • parent
  • hideshow 9 child comments
  • [+] 2 points 8 months ago (6 children)
  • [–] [S] 4 points 8 months ago (4 children)

    Yes indeed! Changing the sender name to Matrix Authentication Service @ <my_homeserver> works, with successful dkim: pass.

    From original source:

    Authentication-Results: mx.google.com;
           dkim=pass [...]
    [...]
    From: "Matrix Authentication Service @ <my_homeserver>" <matrix@matrix.<my_homeserver>>
    Reply-To: "Matrix Authentication Service @ <my_homeserver>" <matrix@matrix.<my_homeserver>>
    

    So sending mails with quoted names that do not have to be quoted leads to DKIM failures at gmail.

    Nice idea!

  • source
  • parent
  • hideshow 4 child comments
  • [+] 1 point 8 months ago* (2 children)
  • [–] [S] 2 points 8 months ago (1 child)

    Interesting, thanks for reporting back.

    I went to the config and changed my sender to Matrix Authentication System again, without @.

    It works, I have a DKIM pass now. As seen, the quotes are still stripped:

    Authentication-Results: mx.google.com;
           dkim=pass
    [...]
    From: Matrix Authentication Service <matrix@matrix.<homeserver>>
    Reply-To: Matrix Authentication Service <matrix@matrix.<homeserver>>
    

    I did not touch my setup otherwise; I'm sure it failed the DKIM check before at gmail. Given the DMARC and SPF checks passed, I also don't think it was a DNS issue.

    I'm sure it was not my setup, I manually copied the gmail-processed source and verified the DKIM cert manually, with and without re-adding the quotes. I was only able to verify locally with the quotes added. This, and DKIM consistently passing at other mail servers leads me to believe it actually was an issue at google side.

    So either they fixed the issue in light-speed, or I was loosing my mind, which of course is possible as well. Will update the original post and github issues.

  • source
  • parent
  • hideshow 1 child comment
  • load more comments (1 reply)
  • load more comments (1 reply)
  • [–] 8 points 8 months ago

    Good luck getting Google to do anything about it

  • source
  • [–] 6 points 8 months ago (16 children)

    Wait, so when you download what is supposed to be the original email from Gmail, it’s not actually the original? How is that ok?

  • source
  • hideshow 16 child comments
  • [–] [S] 5 points 8 months ago (13 children)

    Usually, the important parts of the mail, such as subject, sender and contents are protected by DKIM authentication. Unfortunately this is usually not visible to the end-user, i.e. as in my case, where mails fail DKIM, but are still presented in my inbox.

    Mail servers and relays add headers to the mail as it goes, for example their own IPs to trace the mail, or authentication results if authentication happens at various endpoints.

    In the end, the mail as in the gmail postbox is the result of the original mail, and all these additions of the mail relays. In an ideal world only DKIM authenticated would be presented to the end-user, but the world of mail seems to be so broken, that many sending servers just do not apply DKIM/DMARC correctly, and thus many receivers accept broken mail.

  • source
  • parent
  • hideshow 13 child comments
  • [–] 5 points 8 months ago (12 children)

    Yeah, I understand how the mail that Gmail receives is not necessarily the mail that the user sent (in regards to headers), and I understand Gmail can add headers (like auth results, spam scores, other sorts of records), but is Gmail really changing the “From” header or other headers included in the DKIM signature?? I would think that would be absolutely unacceptable.

  • source
  • parent
  • hideshow 12 child comments
  • [–] [S] 2 points 8 months ago (11 children)

    Yes, it seems it does that. I assume it has some processing chain involving decode -> process -> encode of the whole mail, and usually that works out and the DKIM check passes. Apparantly, if you send something non-standard the decode and encode are not symmetrical and this happens. IMHO it shouldn't, so I agree. Especially, showing users mails with broken authentication seems broken as well to me.

  • source
  • parent
  • hideshow 11 child comments
  • [–] 2 points 8 months ago* (last edited 8 months ago) (10 children)

    Quotes around the display name in a header certainly conforms to the standard (and in fact, the way Gmail rewrites it does not conform to the standard when it contains a space, but is the obsolete form), and I would expect any decent mail program to leave it alone. Then again, Gmail is not a decent mail program, and hasn’t been for a long time.

    Edit: @w2xel@gehirneimer.de points out below that Gmail’s rewrite does follow the spec, so it is conforming to the standard when white space is in the display name. It’s only when there is a dot (.) that it would be using the obsolete form.

  • source
  • parent
  • hideshow 10 child comments
  • [–] [S] 2 points 8 months ago* (5 children)

    I had to think about this for a while, but the standard is only obsoleting folding white space, i.e. white space that wraps lines, such as:

    Subject: This
     is a folded line
    

    which is equivalent to

    Subject: This is a folded line
    

    As I understand it, white space is allowed before and after obsoletion. Or do I understand it wrong?

    Edit:

    I think in the obsoleted language the following would have been allowed for a From: field as well:

    From: John
     Smith <j.smith@example.com>
    
  • source
  • parent
  • hideshow 5 child comments
  • [–] 1 point 8 months ago* (last edited 8 months ago) (4 children)

    Folding white space is a different issue. It’s the atom / quoted-string part. That’s the standard form of a display name. Meaning if it’s more than one word (separated by white space), it should use the quoted string form. It does list obs-phrase as an alternative, but using obsolete syntax should be avoided when possible.

  • source
  • parent
  • hideshow 4 child comments
  • [–] [S] 2 points 8 months ago (3 children)

    phrase is 1*word, not word. The difference to obs-phrase is that it allows dots (".") and folding whitespace. Not that it allows whitespace.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 2 points 8 months ago* (last edited 8 months ago) (2 children)

    Atom is defined as such:

    atext           =       ALPHA / DIGIT / ; Any character except controls,
                            "!" / "#" /     ;  SP, and specials.
                            "$" / "%" /     ;  Used for atoms
                            "&" / "'" /
                            "*" / "+" /
                            "-" / "/" /
                            "=" / "?" /
                            "^" / "_" /
                            "`" / "{" /
                            "|" / "}" /
                            "~"
    
    atom            =       [CFWS] 1*atext [CFWS]
    

    So since atom does not contain white space, there should be no white space. Only comments can separate 1*atom.

    If they wanted to allow separating atoms by a space, it would be written:

    atom *(SP atom)

    Edit: wait, I’m wrong. CFWS includes white space. You’re right.

  • source
  • parent
  • hideshow 2 child comments
  • [–] [S] 1 point 8 months ago (3 children)

    The unquoted names with whitespace are legal, as stated in A.5: https://www.rfc-editor.org/rfc/rfc2822#appendix-A.5

    The section of obsolete syntax in the appendix starts just below A.5., unfortunately.

    I agree though, that a quoted string seems to be perfectly legal as well, so there is no good reason for google to change it.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 8 months ago* (last edited 8 months ago) (2 children)

    The section right below shows that it’s obsolete:

    A.6.1. Obsolete addressing
    
       Note in the below example the lack of quotes around Joe Q. Public,
       the route that appears in the address for Mary Smith, the two commas
       that appear in the "To:" field, and the spaces that appear around the
       "." in the jdoe address.
    
    ----
    From: Joe Q. Public <john.q.public@example.com>
    To: Mary Smith <@machine.tld:mary@example.net>, , jdoe@test   . example
    Date: Tue, 1 Jul 2003 10:52:37 +0200
    Message-ID: <5678.21-Nov-1997@example.com>
    
    Hi everyone.
    ----
    

    And in the spec itself, that syntax is named as “obs-phrase”.

    But yes, though obsolete, it is still legal syntax. So I guess I shouldn’t say it “does not conform to the standard”, but rather “just barely conforms to the standard”.

  • source
  • parent
  • hideshow 2 child comments
  • [–] [S] 1 point 8 months ago (1 child)

    In the same example, the Mary Smith is not the issue, rather the @machine.tld, as is written in the description.

    John Q. Public is an issue because of the dot, not the spaces. The spaces are an issue in jdoe@test . example, as that's actually an address, not a name

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 4 months ago

    If the issue is resolved server-side, verify whether Google updated their header normalization logic to comply with RFC 5322 regarding From header quoting. It would be valuable to see if they published a changelog entry or a specific commit hash explaining the fix to prevent future regressions in their DKIM validation pipeline.

  • source
  • [–] 1 point 8 months ago

    I can think of a few avenues for this complaint. First is EU, I’m sure they’d love to go after Google about bad security. Second is lawyers, if lawyers have lots of discovery with Gmail or litigation against Google this could impact legal cases. Search for big ongoing Google legal cases and contact the plaintiffs about this bug. Third is just emailing them but you’ll probably get better results screaming into the void, no one paying less than seven figures gets in touch with people at a major tech company

  • source
  • [–] 1 point 8 months ago

    That explains my problem :/

    It’s so heinous that their little oopsies or „genuine effort to combat spam“ basically force self hosters to use their services.

  • source