Welcome to the Power Users community on Codidact!

Power Users is a Q&A site for questions about the usage of computer software and hardware. We are still a small site and would like to grow, so please consider joining our community. We are looking forward to your questions and answers; they are the building blocks of a repository of knowledge we are building together.

Post History

50%

+0 −0

Q&A How do I stop Gmail from silently dropping legitimate email?

I've known for a while that Gmail sometimes silently drops incoming email -- it doesn't bounce or go to the spam folder; it just disappears. [...] How can I debug (and ideally fix) this problem?...

posted 10d ago by mcast‭ · edited 10d ago by mcast‭

Answer

#3: Post edited by

mcast‭ · 2025-04-09T19:04:28Z (10 days ago)
mail header analysis, and in general about security in advice from strangers

Copy Link

Raw

Markdown

> I've known for a while that Gmail sometimes silently drops incoming email -- it doesn't bounce or go to the spam folder; it just disappears. [...]
>
> How can I debug (and ideally fix) this problem?
For me, these are the jumping-off points: the places you know messages are not coming from.
Vague suspicions that mail is vanishing are very difficult to act on, but if you have non-gmail contact with somebody who says "you didn't get my mail?" then this is the place to debug.
Some ISPs do blackhole mail, I've seen it happen. I have not seen GMail do that so I believe there would have been a bounce generated; but if the sender's mail is broken then the bounce may not reach the sender.
~~Are you able to update the question with some more specific information?~~
Now my understanding is that mail M comes from outside to your domain D, then you try to forward it to your gmail account G and that last is failing.
## Problems with forwarding mail
There is a problem with mail forwarding, like this
1. spammer chooses you (monica@D) and a random sender (patsy@P1)
2. they have somebody's mail server do the dirty work of delivering that to your mailserver D, maybe from a mailserver in P1 or somewhere else P2. They don't care.
3. if D realises it's spam before accepting it, the message is rejected and that is somebody else's responsibility. You're saying that isn't the case here.
4. your domain D has accepted the message and tried to pass it on to G.
5. if G has any suspicion that it's spam, I expect they would reject it. It's possible they blackhole it (which would cause your symptoms) but I believe they're better than that.
* I haven't checked what they actually do. An **actionable debugging** point would be to try finding out by use of the [GTUBE string](https://en.wikipedia.org/wiki/GTUBE)
6. when D is left holding a message which G refused to accept, D has responsibility for generating the bounce.
* This is a new message from nobody (bounces should have an empty sender, so they cannot also bounce) to patsy@P1, which is an address that may not exist or might be flooded with similar bounces from other places.
* Note that this is a new message - and spammers have abused mail servers to do this.
* If you allow D to do this, you may get D tainted on the block lists; this may impact G's evaluation of other mails later
Solutions to this problem can include
1. messages coming from trusted sources must not be rejected, even when they are clearly spam
* I do this for domains which _I requested to forward to me._ The connection gets a "friendly" mark on it and even spam will not be rejected, because I don't want to put that burden of generating bounces on the service provider.
2. cut-through mail delivery.
1. D knows it will be forwarding to G, so before accepting from patsy@P1 it attempts the delivery immediately through to G.
2. sender P2 is kept waiting on the line
3. if G rejects the mail, then D also rejects the mail. No bounce is generated.
3. maybe you can mess with the envelope sender on D before forwarding to G, but... here be dragons.
The do-nothing solution of allowing bounces to be generated off D is... not a good way to run a mail server, and likely to cause loss of reputation.
_It is unclear to me whether this is happening for monica@D, but you should definitely check._
~~## Getting hold of MTA logs from domain D~~
Are you able to update the question with some more information about the nature of the forwarding service? Did you pay an ISP to forward mail? Did you install a cloud VM and configure your own MTA? Or other?
Whichever it is, debugging will be much easier if you have the transfer logs.
If you can't get those, there may be clues in the full headers of a mail delivered successfully - telling what MTAs are involved.
~~The important question to answer is for a message known to have not arrived at G, what did D say about it? Having the `Message-ID` and asking for logs within a few days will probably be helpful.~~

> I've known for a while that Gmail sometimes silently drops incoming email -- it doesn't bounce or go to the spam folder; it just disappears. [...]
>
> How can I debug (and ideally fix) this problem?
For me, these are the jumping-off points: the places you know messages are not coming from.
Vague suspicions that mail is vanishing are very difficult to act on, but if you have non-gmail contact with somebody who says "you didn't get my mail?" then this is the place to debug.
Some ISPs do blackhole mail, I've seen it happen. I have not seen GMail do that so I believe there would have been a bounce generated; but if the sender's mail is broken then the bounce may not reach the sender.
~~Are you able to update the question with some more specific information?~~
Now my understanding is that mail M comes from outside to your domain D, then you try to forward it to your gmail account G and that last is failing.
## Problems with forwarding mail
There is a problem with mail forwarding, like this
1. spammer chooses you (monica@D) and a random sender (patsy@P1)
2. they have somebody's mail server do the dirty work of delivering that to your mailserver D, maybe from a mailserver in P1 or somewhere else P2. They don't care.
3. if D realises it's spam before accepting it, the message is rejected and that is somebody else's responsibility. You're saying that isn't the case here.
4. your domain D has accepted the message and tried to pass it on to G.
5. if G has any suspicion that it's spam, I expect they would reject it. It's possible they blackhole it (which would cause your symptoms) but I believe they're better than that.
* I haven't checked what they actually do. An **actionable debugging** point would be to try finding out by use of the [GTUBE string](https://en.wikipedia.org/wiki/GTUBE)
6. when D is left holding a message which G refused to accept, D has responsibility for generating the bounce.
* This is a new message from nobody (bounces should have an empty sender, so they cannot also bounce) to patsy@P1, which is an address that may not exist or might be flooded with similar bounces from other places.
* Note that this is a new message - and spammers have abused mail servers to do this.
* If you allow D to do this, you may get D tainted on the block lists; this may impact G's evaluation of other mails later
Solutions to this problem can include
1. messages coming from trusted sources must not be rejected, even when they are clearly spam
* I do this for domains which _I requested to forward to me._ The connection gets a "friendly" mark on it and even spam will not be rejected, because I don't want to put that burden of generating bounces on the service provider.
2. cut-through mail delivery.
1. D knows it will be forwarding to G, so before accepting from patsy@P1 it attempts the delivery immediately through to G.
2. sender P2 is kept waiting on the line
3. if G rejects the mail, then D also rejects the mail. No bounce is generated.
3. maybe you can mess with the envelope sender on D before forwarding to G, but... here be dragons.
The do-nothing solution of allowing bounces to be generated off D is... not a good way to run a mail server, and likely to cause loss of reputation.
_It is unclear to me whether this is happening for monica@D, but you should definitely check._
### Being aware of G -> D -> G transfer
Continuing from the problems of mail forwarding, we have to consider
1. G delivers a mail, from src@G to D
2. D tries to deliver it again, from src@G to monica@G
* note that D is not permitted to generate mail `from: anybody@gmail`, as documented by the SPF records, and as GMail servers are fully aware
* there will be exceptions, possibly if your Exim is configured to authenticate as you before delivering back to GMail? That isn't exactly a normal thing to do but I'm starting to suspect it is the answer in your specific case.
3. if G refuses the mail to monica@G, then
* a bounce to src@G will be immediately generated by D
* perhaps even a delivery attempted down the same connection (logs may not tell us that detail). This may trigger heuristics on the GMail side.
* if the bounce to src@G was also refused by G, this would explain the blackholing.
* Exim's logs would tell you. `<>` is the empty sender used for bounces.
## Getting hold of MTA logs from domain D (1)
~~Are you able to update the question with some more information about the nature of the forwarding service? Did you pay an ISP to forward mail? Did you install a cloud VM and configure your own MTA? Or other?~~
Whichever it is, debugging will be much easier if you have the transfer logs.
If you can't get those, there may be clues in the full headers of a mail delivered successfully - telling what MTAs are involved.
The important question to answer is for a message known to have not arrived at G, what did D say about it? Having the `Message-ID` and asking for logs within a few days will probably be helpful.
## Interpreting the email headers
Thank you for adding redacted headers. There is a lot to redact these days!
Let's list addresses,
* sender `src@gmail`
* I believe that in places you have deleted, rather than replacing? Else this appears to have been a bounce message (no envelope sender) which could explain the loss.
* first delivery by gmail to `ME@MYDOMAIN`
* I'm puzzled by the top of the headers. Did you retain a copy on MYDOMAIN? _i.e._ forwarding to two addresses, one local and one gmail?
* If so, then there was local delivery to a `ME@MYDOMAIN` mailbox, and that it how you come to have these headers
* second onwards delivery attempt to `you@gmail`, lost
We're looking mostly at `Received: ...` lines, and we interpret these headers backwards because new ones are inserted at the top. (In the case of spams, we stop trusting them to be honest somewhere towards the bottom, after the last trustworthy machine tells where it got the message from.)
The duplicated headers `Return-Path: <missing>` .. `Return-Path: <missing>`(again) are odd. If it's not a paste error then I'm inclined to blame whatever did the local delivery (LMTP) for being strange. Anyway that may be incidental - you have the relevant headers.
```
Return-Path:
Delivered-To: ME@MYDOMAIN.org
Received: from redacted.my-hosting-provider.com
by redacted.my-hosting-provider.com with LMTP
id YAYMLtaa82cvqQAAflZDjA
(envelope-from )
for ; Mon, 07 Apr 2025 04:28:54 -0500
Return-path:
Envelope-to: ME@MYDOMAIN.org
Delivery-date: Mon, 07 Apr 2025 04:28:54 -0500
Received: from mail-qt1-f175.google.com ([209.85.160.175]:52619)
by redacted.my-hosting-provider.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
(Exim 4.98.1)
(envelope-from )
id 1u1imd-00000000Tfi-3sXU
for ME@MYDOMAIN.org;
Mon, 07 Apr 2025 04:28:54 -0500
Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-4774ce422easo42549131cf.1
for ; Mon, 07 Apr 2025 02:28:47 -0700 (PDT)
```
The `Received:`...`by redacted.my-hosting`...`(Exim 4.98.1)`...`1u1imd-00000000Tfi-3sXU`...`for ME@` is the one to focus on now. That is your Exim, or one working on your behalf, and it's a recent & sane piece of software.
There will be logs somewhere near you for this. Those are the logs to see next, `1u1imd...` is the text to search for in those logs, and `exigrep` is a handy tool to do it. You may have (+7d) until 14th Apr to do this, depending on log rotation.
I don't have cPanel experience and in general there are different places the log may be,
1. it's possible that the Exim doing your work is shared across multiple domains. If so, you would need to file a ticket to ask the admin to search on your behalf, because such logs would be private to other people also. I think that is not the case for you.
2. the logs are on "your" machine, but there is a small hurdle to jump, like a "yes I know what I'm doing" button to elevate your privileges on the cloud VPS running your cPanel instance.
* in general, the file you want is often called `/var/log/exim/mainlog`, then renamed to `mainlog.1` and compressed to `mainlog.2.gz`.
3. or our lucky day, [cPanel provides documentation](https://support.cpanel.net/hc/en-us/articles/5528247629463-How-to-use-exigrep-to-search-the-Exim-logs)
* your file is `/var/log/exim_mainlog` and they guide you to run the command yourself as root
You should get a few lines from the command `exigrep 1u1imd-00000000Tfi-3sXU /var/log/exim_mainlog*` or equivalently `exigrep 1AD5A4C5-B698-46B8-BD4C-7200813FDCB0@gmail.com /var/log/exim_mainlog*`
The [exigrep tool](https://www.exim.org/exim-html-current/doc/html/spec_html/ch-exim_utilities.html) finds matches for the string you give it, then it pulls out from the searched files all events relating to the message(s) involved.
# More general notes
## on redaction
In general, a safer way to redact can be
1. gather the thing in a text file
2. pipe it through a set of replacements (a bit technical for some)
3. check for remaining matches of things you would want removed
4. then paste the text, and the list of replacement strings
It could give a more consistent redaction without destroying necessary information.
This is a tool that could be webified easily enough, but I would rarely recommend trusting a third party to redact!
## on pasting gibberish into root shells
In general, a note on trusting instructions to run as root,
* although cPanel docs are an authority you can trust about `exigrep .. mainlog`, the bit in the middle is also important.
* The id you're searching for is a short piece of text from an external email,
* it's not text just made up by me, so you have less need to trust my instruction
* these ids were generated by reputable mail transfer agents
* however in the most general we must know that carefully designed text stuffed in here could compromise the machine
* To those of us know would know a shellcode injection attack when we see one,
* We must remember that "making it OK to paste a string of gibberish into a root shell" is not good for the world.
* We should be clear why that specific gibberish is safe and relevant.
(Am I hoping for too much here?)

#2: Post edited by

mcast‭ · 2025-04-09T16:13:38Z (10 days ago)
about mail forwarding

Copy Link

Raw

Markdown

> I've known for a while that Gmail sometimes silently drops incoming email -- it doesn't bounce or go to the spam folder; it just disappears. [...]
>
> How can I debug (and ideally fix) this problem?
For me, these are the jumping-off points: the places you know messages are not coming from.
Vague suspicions that mail is vanishing are very difficult to act on, but if you have non-gmail contact with somebody who says "you didn't get my mail?" then this is the place to debug.
Some ISPs do blackhole mail, I've seen it happen. I have not seen GMail do that so I believe there would have been a bounce generated; but if the sender's mail is broken then the bounce may not reach the sender.
~~Are you able to update the question with some more specific information?~~

> I've known for a while that Gmail sometimes silently drops incoming email -- it doesn't bounce or go to the spam folder; it just disappears. [...]
>
> How can I debug (and ideally fix) this problem?
For me, these are the jumping-off points: the places you know messages are not coming from.
Vague suspicions that mail is vanishing are very difficult to act on, but if you have non-gmail contact with somebody who says "you didn't get my mail?" then this is the place to debug.
Some ISPs do blackhole mail, I've seen it happen. I have not seen GMail do that so I believe there would have been a bounce generated; but if the sender's mail is broken then the bounce may not reach the sender.
~~Are you able to update the question with some more specific information?~~
Now my understanding is that mail M comes from outside to your domain D, then you try to forward it to your gmail account G and that last is failing.
## Problems with forwarding mail
There is a problem with mail forwarding, like this
1. spammer chooses you (monica@D) and a random sender (patsy@P1)
2. they have somebody's mail server do the dirty work of delivering that to your mailserver D, maybe from a mailserver in P1 or somewhere else P2. They don't care.
3. if D realises it's spam before accepting it, the message is rejected and that is somebody else's responsibility. You're saying that isn't the case here.
4. your domain D has accepted the message and tried to pass it on to G.
5. if G has any suspicion that it's spam, I expect they would reject it. It's possible they blackhole it (which would cause your symptoms) but I believe they're better than that.
* I haven't checked what they actually do. An **actionable debugging** point would be to try finding out by use of the [GTUBE string](https://en.wikipedia.org/wiki/GTUBE)
6. when D is left holding a message which G refused to accept, D has responsibility for generating the bounce.
* This is a new message from nobody (bounces should have an empty sender, so they cannot also bounce) to patsy@P1, which is an address that may not exist or might be flooded with similar bounces from other places.
* Note that this is a new message - and spammers have abused mail servers to do this.
* If you allow D to do this, you may get D tainted on the block lists; this may impact G's evaluation of other mails later
Solutions to this problem can include
1. messages coming from trusted sources must not be rejected, even when they are clearly spam
* I do this for domains which _I requested to forward to me._ The connection gets a "friendly" mark on it and even spam will not be rejected, because I don't want to put that burden of generating bounces on the service provider.
2. cut-through mail delivery.
1. D knows it will be forwarding to G, so before accepting from patsy@P1 it attempts the delivery immediately through to G.
2. sender P2 is kept waiting on the line
3. if G rejects the mail, then D also rejects the mail. No bounce is generated.
3. maybe you can mess with the envelope sender on D before forwarding to G, but... here be dragons.
The do-nothing solution of allowing bounces to be generated off D is... not a good way to run a mail server, and likely to cause loss of reputation.
_It is unclear to me whether this is happening for monica@D, but you should definitely check._
## Getting hold of MTA logs from domain D
Are you able to update the question with some more information about the nature of the forwarding service? Did you pay an ISP to forward mail? Did you install a cloud VM and configure your own MTA? Or other?
Whichever it is, debugging will be much easier if you have the transfer logs.
If you can't get those, there may be clues in the full headers of a mail delivered successfully - telling what MTAs are involved.
The important question to answer is for a message known to have not arrived at G, what did D say about it? Having the `Message-ID` and asking for logs within a few days will probably be helpful.

#1: Initial revision by

mcast‭ · 2025-04-09T09:11:15Z (10 days ago)

Copy Link

Raw

Markdown

> I've known for a while that Gmail sometimes silently drops incoming email -- it doesn't bounce or go to the spam folder; it just disappears. [...]
>
> How can I debug (and ideally fix) this problem?

For me, these are the jumping-off points: the places you know messages are not coming from.

Vague suspicions that mail is vanishing are very difficult to act on, but if you have non-gmail contact with somebody who says "you didn't get my mail?" then this is the place to debug.

Some ISPs do blackhole mail, I've seen it happen.  I have not seen GMail do that so I believe there would have been a bounce generated; but if the sender's mail is broken then the bounce may not reach the sender.

Are you able to update the question with some more specific information?

Communities

Post History