I've known for a while that Gmail sometimes silently drops incoming email -- it doesn't bounce or go to the spam folder; it just disappears. [...]

How can I debug (and ideally fix) this problem?

For me, these are the jumping-off points: the places you know messages are not coming from.

Vague suspicions that mail is vanishing are very difficult to act on, but if you have non-gmail contact with somebody who says "you didn't get my mail?" then this is the place to debug.

Some ISPs do blackhole mail, I've seen it happen. I have not seen GMail do that so I believe there would have been a bounce generated; but if the sender's mail is broken then the bounce may not reach the sender.

Are you able to update the question with some more specific information?

Now my understanding is that mail M comes from outside to your domain D, then you try to forward it to your gmail account G and that last is failing.

Problems with forwarding mail

There is a problem with mail forwarding, like this

1. spammer chooses you (monica@D) and a random sender (patsy@P1)
2. they have somebody's mail server do the dirty work of delivering that to your mailserver D, maybe from a mailserver in P1 or somewhere else P2. They don't care.
3. if D realises it's spam before accepting it, the message is rejected and that is somebody else's responsibility. You're saying that isn't the case here.
4. your domain D has accepted the message and tried to pass it on to G.
5. if G has any suspicion that it's spam, I expect they would reject it. It's possible they blackhole it (which would cause your symptoms) but I believe they're better than that.
   • I haven't checked what they actually do. An actionable debugging point would be to try finding out by use of the GTUBE string
6. when D is left holding a message which G refused to accept, D has responsibility for generating the bounce.
   • This is a new message from nobody (bounces should have an empty sender, so they cannot also bounce) to patsy@P1, which is an address that may not exist or might be flooded with similar bounces from other places.
   • Note that this is a new message - and spammers have abused mail servers to do this.
   • If you allow D to do this, you may get D tainted on the block lists; this may impact G's evaluation of other mails later

Solutions to this problem can include

1. messages coming from trusted sources must not be rejected, even when they are clearly spam
   • I do this for domains which I requested to forward to me. The connection gets a "friendly" mark on it and even spam will not be rejected, because I don't want to put that burden of generating bounces on the service provider.
2. cut-through mail delivery.
   1. D knows it will be forwarding to G, so before accepting from patsy@P1 it attempts the delivery immediately through to G.
   2. sender P2 is kept waiting on the line
   3. if G rejects the mail, then D also rejects the mail. No bounce is generated.
3. maybe you can mess with the envelope sender on D before forwarding to G, but... here be dragons.

The do-nothing solution of allowing bounces to be generated off D is... not a good way to run a mail server, and likely to cause loss of reputation.

It is unclear to me whether this is happening for monica@D, but you should definitely check.

Being aware of G -> D -> G transfer

Continuing from the problems of mail forwarding, we have to consider

1. G delivers a mail, from src@G to D
2. D tries to deliver it again, from src@G to monica@G
   • note that D is not permitted to generate mail from: anybody@gmail, as documented by the SPF records, and as GMail servers are fully aware
   • there will be exceptions, possibly if your Exim is configured to authenticate as you before delivering back to GMail? That isn't exactly a normal thing to do but I'm starting to suspect it is the answer in your specific case.
3. if G refuses the mail to monica@G, then
   • a bounce to src@G will be immediately generated by D
   • perhaps even a delivery attempted down the same connection (logs may not tell us that detail). This may trigger heuristics on the GMail side.
   • if the bounce to src@G was also refused by G, this would explain the blackholing.
   • Exim's logs would tell you. <> is the empty sender used for bounces.

Getting hold of MTA logs from domain D (1)

Are you able to update the question with some more information about the nature of the forwarding service? Did you pay an ISP to forward mail? Did you install a cloud VM and configure your own MTA? Or other?

Whichever it is, debugging will be much easier if you have the transfer logs.

If you can't get those, there may be clues in the full headers of a mail delivered successfully - telling what MTAs are involved.

The important question to answer is for a message known to have not arrived at G, what did D say about it? Having the Message-ID and asking for logs within a few days will probably be helpful.

Interpreting the email headers

Thank you for adding redacted headers. There is a lot to redact these days!

Let's list addresses,

• sender src@gmail
  • I believe that in places you have deleted, rather than replacing? Else this appears to have been a bounce message (no envelope sender) which could explain the loss.
• first delivery by gmail to ME@MYDOMAIN
  • I'm puzzled by the top of the headers. Did you retain a copy on MYDOMAIN? i.e. forwarding to two addresses, one local and one gmail?
  • If so, then there was local delivery to a ME@MYDOMAIN mailbox, and that it how you come to have these headers
• second onwards delivery attempt to you@gmail, lost

We're looking mostly at Received: ... lines, and we interpret these headers backwards because new ones are inserted at the top. (In the case of spams, we stop trusting them to be honest somewhere towards the bottom, after the last trustworthy machine tells where it got the message from.)

The duplicated headers Return-Path: <missing> .. Return-Path: <missing>(again) are odd. If it's not a paste error then I'm inclined to blame whatever did the local delivery (LMTP) for being strange. Anyway that may be incidental - you have the relevant headers.

Return-Path: 
Delivered-To: ME@MYDOMAIN.org
Received: from redacted.my-hosting-provider.com
	by redacted.my-hosting-provider.com with LMTP
	id YAYMLtaa82cvqQAAflZDjA
	(envelope-from )
	for ; Mon, 07 Apr 2025 04:28:54 -0500
Return-path: 
Envelope-to: ME@MYDOMAIN.org
Delivery-date: Mon, 07 Apr 2025 04:28:54 -0500
Received: from mail-qt1-f175.google.com ([209.85.160.175]:52619)
	by redacted.my-hosting-provider.com with esmtps  (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
	(Exim 4.98.1)
	(envelope-from )
	id 1u1imd-00000000Tfi-3sXU
	for ME@MYDOMAIN.org;
	Mon, 07 Apr 2025 04:28:54 -0500
Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-4774ce422easo42549131cf.1
        for ; Mon, 07 Apr 2025 02:28:47 -0700 (PDT)

The Received:...by redacted.my-hosting...(Exim 4.98.1)...1u1imd-00000000Tfi-3sXU...for ME@ is the one to focus on now. That is your Exim, or one working on your behalf, and it's a recent & sane piece of software.

There will be logs somewhere near you for this. Those are the logs to see next, 1u1imd... is the text to search for in those logs, and exigrep is a handy tool to do it. You may have (+7d) until 14th Apr to do this, depending on log rotation.

I don't have cPanel experience and in general there are different places the log may be,

1. it's possible that the Exim doing your work is shared across multiple domains. If so, you would need to file a ticket to ask the admin to search on your behalf, because such logs would be private to other people also. I think that is not the case for you.
2. the logs are on "your" machine, but there is a small hurdle to jump, like a "yes I know what I'm doing" button to elevate your privileges on the cloud VPS running your cPanel instance.
   • in general, the file you want is often called /var/log/exim/mainlog, then renamed to mainlog.1 and compressed to mainlog.2.gz.
3. or our lucky day, cPanel provides documentation
   • your file is /var/log/exim_mainlog and they guide you to run the command yourself as root

You should get a few lines from the command exigrep 1u1imd-00000000Tfi-3sXU /var/log/exim_mainlog* or equivalently exigrep 1AD5A4C5-B698-46B8-BD4C-7200813FDCB0@gmail.com /var/log/exim_mainlog*

The exigrep tool finds matches for the string you give it, then it pulls out from the searched files all events relating to the message(s) involved.

More general notes

on redaction

In general, a safer way to redact can be

1. gather the thing in a text file
2. pipe it through a set of replacements (a bit technical for some)
3. check for remaining matches of things you would want removed
4. then paste the text, and the list of replacement strings

It could give a more consistent redaction without destroying necessary information.

This is a tool that could be webified easily enough, but I would rarely recommend trusting a third party to redact!

on pasting gibberish into root shells

In general, a note on trusting instructions to run as root,

• although cPanel docs are an authority you can trust about exigrep .. mainlog, the bit in the middle is also important.
• The id you're searching for is a short piece of text from an external email,
  • it's not text just made up by me, so you have less need to trust my instruction
  • these ids were generated by reputable mail transfer agents
  • however in the most general we must know that carefully designed text stuffed in here could compromise the machine
• To those of us know would know a shellcode injection attack when we see one,
  • We must remember that "making it OK to paste a string of gibberish into a root shell" is not good for the world.
  • We should be clear why that specific gibberish is safe and relevant.

(Am I hoping for too much here?)

posted 9 days ago

CC BY-SA 4.0

8d ago

mcast‭

1 reputation 0 4 0 2

Google is notorious for being very picky, having significant false positives, and not caring.

Most likely they aren't discarding these mail messages, but refusing to accept them in the first place. This can happen easily when the sender doesn't have all that latest anti-spam technology turned on at their end, like DKIM, SPF, etc.

There are still a lot of old mail servers out there. Google is a big organization and has the ability to jump on the latest technology quickly. Small companies with their own mail server (for various reasons) don't know much about these things and don't want to spend time being mail server administrators, so just let them run.

For quite a while, mail from my domain to gmail would sometimes get to a recipient, sometimes not. Mail to all other domains seemed to work normally. Sometimes, but now always, I'd get a bounce message from gmail. I had many things to do, so this didn't get much attention. Real companies don't use gmail, so only message to a few select individuals weren't getting thru. Most of the time, it was more to their advantage than mine to receive mail I sent. I developed the attitude "Screw this, you're the one who picked a bad mail service".

Eventually I looked into it and found that one of those settings (SPF ?) for my domain wasn't right or properly turned on. After fixing that, gmail seems to be accepting all my mail now.

After the next new anti-spam technology emerges and gmail gets uppity again, this will probably happen all over.

So, the answer is this is probably due to lagging technology by the sender together with an overly-aggressive and holier-than-thou attitude by Google. There is probably nothing you can do about it other than not use gmail.

posted 10 days ago

CC BY-SA 4.0

Olin Lathrop‭

929 reputation 5 28 78 21