System Administration & Network Administration
ubuntu-12.04 zimbra
Updated Thu, 18 Aug 2022 07:33:37 GMT

Zimbra 7.2 OSE Mail Queue Disappeared


System Info:

Description: Ubuntu 10.04.4 LTS
Linux mail 2.6.32-42-server #95-Ubuntu SMP Wed Jul 25 16:10:49 UTC 2012 x86_64 GNU/Linux
H/W path          Device      Class      Description
====================================================
                              system     Bochs
/0                            bus        Motherboard
/0/0                          memory     96KiB BIOS
/0/401                        processor  QEMU Virtual CPU version 1.0.50
/0/402                        processor  CPU
/0/1000                       memory     2GiB System Memory
/0/1000/0                     memory     2GiB DIMM RAM
/0/100                        bridge     440FX - 82441FX PMC [Natoma]
/0/100/1                      bridge     82371SB PIIX3 ISA [Natoma/Triton II]
/0/100/1.1        scsi1       storage    82371SB PIIX3 IDE [Natoma/Triton II]
/0/100/1.1/0.0.0  /dev/cdrom  disk       DVD reader
/0/100/1.2                    bus        82371SB PIIX3 USB [Natoma/Triton II]
/0/100/1.3                    bridge     82371AB/EB/MB PIIX4 ACPI
/0/100/2                      display    GD 5446
/0/100/a                      storage    Virtio block device
/0/100/12                     network    Virtio network device
/1                eth0        network    Ethernet interface

I accidentally installed Postfix from APT onto our Zimbra email server virtual machine (KVM/Proxmox VE 2.1). I purged it instantly, however, it did not help. The following are the steps I took in a desperate attempt to fix it:

  • Reinstalled Zimbra
  • Stopped Zimbra service

  • Reset permissions

    chown -R zimbra:zimbra /opt/zimbra

  • Fixed permissions

    /opt/zimbra/libexec/zmfixperms -verbose -extended

  • Restarted Zimbra service

  • Renewed the Zimbra SSH keys

    zmsshkeygen

  • Updated Zimbra SSH keys

    zmupdateauthkeys

Following the steps above, Zimbra now works again, the problem is that all the emails that were in the mail queue from a whole day of broken Zimbra, are now GONE! And I don't just mean that you can't see them in the GUI, they're not even on the file system anymore.

There were at least 75 emails that were in the queue when I attempted to fix Zimbra, and this is the state of the spool now:

zimbra@mail:~$ postqueue -p
Mail queue is empty

Where the heck did all my emails go!?



Solution

If anyone has this problem in the future, this is how you repair it. You may end up with more than a few duplicates, be that's better than losing an important email...

To find the messages that got queued up, run: 

grep -iRE '<recipient_email_address>' /var/log/mail.log* | grep -i 'queued as' | awk '{print $6}' | cut -d ':' -f 1 | xargs | tr ' ' '|'

To find the actual email files on the file system, run: 

grep -iRE '<output_from_previous_command>' /opt/zimbra/store | grep -iv 'Message-Id' | cut -d ':' -f 1 | xargs

"Inject" email back to the user account they came from: 

su -l zimbra
/opt/zimbra/bin/zmlmtpinject <output_from_previous_command> -r <recipient_email>

Keep in mind that you will lose all date/time information. The only way around that is by adding an "X-Zimbra-Received" header to each message containing a copy of the "Date:" header. Use your imagination for how to do this, Perl comes to mind...


References