System Info:
Description: Ubuntu 10.04.4 LTS
Linux mail 2.6.32-42-server #95-Ubuntu SMP Wed Jul 25 16:10:49 UTC 2012 x86_64 GNU/Linux
H/W path Device Class Description
====================================================
system Bochs
/0 bus Motherboard
/0/0 memory 96KiB BIOS
/0/401 processor QEMU Virtual CPU version 1.0.50
/0/402 processor CPU
/0/1000 memory 2GiB System Memory
/0/1000/0 memory 2GiB DIMM RAM
/0/100 bridge 440FX - 82441FX PMC [Natoma]
/0/100/1 bridge 82371SB PIIX3 ISA [Natoma/Triton II]
/0/100/1.1 scsi1 storage 82371SB PIIX3 IDE [Natoma/Triton II]
/0/100/1.1/0.0.0 /dev/cdrom disk DVD reader
/0/100/1.2 bus 82371SB PIIX3 USB [Natoma/Triton II]
/0/100/1.3 bridge 82371AB/EB/MB PIIX4 ACPI
/0/100/2 display GD 5446
/0/100/a storage Virtio block device
/0/100/12 network Virtio network device
/1 eth0 network Ethernet interface
I accidentally installed Postfix from APT onto our Zimbra email server virtual machine (KVM/Proxmox VE 2.1). I purged it instantly, however, it did not help. The following are the steps I took in a desperate attempt to fix it:
Reset permissions
chown -R zimbra:zimbra /opt/zimbra
Fixed permissions
/opt/zimbra/libexec/zmfixperms -verbose -extended
Restarted Zimbra service
Renewed the Zimbra SSH keys
zmsshkeygen
Updated Zimbra SSH keys
zmupdateauthkeys
Following the steps above, Zimbra now works again, the problem is that all the emails that were in the mail queue from a whole day of broken Zimbra, are now GONE! And I don't just mean that you can't see them in the GUI, they're not even on the file system anymore.
There were at least 75 emails that were in the queue when I attempted to fix Zimbra, and this is the state of the spool now:
zimbra@mail:~$ postqueue -p
Mail queue is empty
Where the heck did all my emails go!?
If anyone has this problem in the future, this is how you repair it. You may end up with more than a few duplicates, be that's better than losing an important email...
To find the messages that got queued up, run:
grep -iRE '<recipient_email_address>' /var/log/mail.log* | grep -i 'queued as' | awk '{print $6}' | cut -d ':' -f 1 | xargs | tr ' ' '|'
To find the actual email files on the file system, run:
grep -iRE '<output_from_previous_command>' /opt/zimbra/store | grep -iv 'Message-Id' | cut -d ':' -f 1 | xargs
"Inject" email back to the user account they came from:
su -l zimbra
/opt/zimbra/bin/zmlmtpinject <output_from_previous_command> -r <recipient_email>
Keep in mind that you will lose all date/time information. The only way around that is by adding an "X-Zimbra-Received" header to each message containing a copy of the "Date:" header. Use your imagination for how to do this, Perl comes to mind...