Update: I’m now hosting this file here in the hopes that it can be useful to others since the original link doesn’t work anymore. Please note: I cannot provide ANY support for this. I didn’t write it. You should read through the options carefully before running this on your machine. That being said, it did work for me at the time, so hopefully it will work for you as well.
After copying an IMAP account from one server to another I ran into this same problem. The bash script that is supplied worked perfectly for me to fix the problem. They give example scripts for both BSD and Linux.
Fixing mail.app’s IMAP date problem
There is one last problem with mail.app that I finally got around to fixing today. When I upgraded to Tiger my IMAP accounts didn’t upgrade well so I deleted them and added them again. That fixed the connection and synchronization problems but introduced the IMAP Date problem.
The IMAP Date problem is the result of how mail.app figures out the Date Received time for an email. Rather than using the Date: header in the email it uses the time the file was written to the file system. This becomes a problem when files are copied to a new location on the server and the creation time of the file is changed. When the entire contents of a Maildir is copied to a new location this can cause all emails to display with the wrong date and time!
The solution is to change the time stamp of each email message that is wrong. The process is:
- Get the Date: header from the email message
- Compare it to the file system time stamp
- If they are different, change the time stamp to the date in the email
Here is the shell script that accomplishes this. All that needs to be done is to point it at a user’s Maildir and it will handle the rest. A word of warning, this script is likely far from perfect. I did run it on my own Maildir and it fixed about 16,000 emails in about 3 minutes.
Old link (broken): Fixing mail.app’s IMAP date problem | MostlyGeek
New link to download the script hosted here: fix_imap_time_for_apple_mail_app.sh
Here is the contents of the script:
#!/bin/sh
#
# Date : July 4th, 2005
# Author: Benson Wong
# tummytech@gmail.com
#
# This shell script corrects email messages where the file system
# date does not match the Date: header in the email.
#
# This will fix problems with mail clients like Apple's mail.app
# which uses the file system timestamp resulting in emails with the
# wrong file system timestamp to display the wrong received date
#
# This script has to be run by a user [root] with the
# necessary privileges to read/modify files in a user's Maildir.
#
MDIR_PATH="$1"
if [ $# -lt 1 ]
then
echo "Usage: $0 /path/to/user/Maildir"
exit 1
fi
if [ ! -d "$MDIR_PATH" ]
then
echo "Error: $MDIR_PATH does not exist"
echo "Usage: $0 /path/to/user/Maildir"
exit 1
fi
if [ ! -r "$MDIR_PATH" ]
then
echo "Error: $MDIR_PATH is not readable"
echo "Usage: $0 /path/to/user/Maildir"
exit 1
fi
if [ ! -w "$MDIR_PATH" ]
then
echo "Error: $MDIR_PATH is not writable"
echo "Usage: $0 /path/to/user/Maildir"
exit 1
fi
# set the internal field separator to the newline character
# instead of the default "".
# This is required for handling filenames and directories with spaces
IFS="
"
set -f
echo "start"
# Find all emails
for i in `find $MDIR_PATH -type f | \
egrep -v "(courierimap|maildirsize|maildirfolder)"`
do
EDATE=`awk '/^Date: [A-Za-z]*,/ {print $4,$3,$5,$6}' "$i" | head -1`
if [ -z "$EDATE" ]
then
continue
fi
FDATE=`ls -l --time-style=long-iso "$i" | awk '{print $6,$7}'`
# Reformat the date for touch.
NDATE=`date -d "$EDATE" "+%Y%m%d%H%M"`
ODATE=`date -d "$FDATE" "+%Y%m%d%H%M"`
if [ "$NDATE" -eq "$ODATE" ]
then
# Skip it if the times are correct.
echo -n "_"
continue
fi
echo `basename $i` "from $ODATE to $NDATE"
touch -c -t "$NDATE" "$i"
done
echo "done"
21 responses to “Fixing mail.app’s IMAP date problem | MostlyGeek”
> Fixing mail.app’s IMAP date problem | MostlyGeek
I’ve just transferred a couple of accounts with Mail.app and they’re all “received on 5 Jan ’06”. It looks like I need your script but it’s no longer on this website. Could you please send a copy to the Gmail address and a backup to ehondrick@sbamug.com? Thank you very much.
Hi
I just copied my Maildir’s from a freebsd box to a mac mini setup with ECM2 on os x and have exactly the problem you had. All the files in the Maildir have the same date (ie the day I copied them) and mail.app displays that date. Unfortunately the link to the script you wrote no longer exists. Would it be possible to either email it to me or re link it ?
Thanks in advance.
Eric
Note to author: There’s an extra <del> tag in the markup on this page causing the script to appear crossed-out in Firefox.
@Ted – Thanks for pointing that out. Fixed! 🙂
Thanx for this great script!
I’ve my lesson learned: when doing a mail migration, use rsync and not scp 😡
How do I run this script? I’m desperately needing to fix this date issue. But I don’t know how to run this script.
Do i copy it into some directory?
THen how do I run it?
How do I tell it what folder or mail server to access?
Hello,
Thanks for you script, I used it wehen i migrated from Cyrus to Dovecot and from mbox to Maildir. In the first run about 95% went ok and the rest fell through because:
– The header said date: instead of Date:
– There was a tab after Date: instead of a space
– The starting day (Mon, Tue, Wed) was missing from the Date: header.
I adapted your script a little and now I have a 99,9% hit rate.
To run this on Mac OS X Server (10.5 and possibly earlier versions), a few changes are required which I have pasted below in case anyone else needs them (indicated below by #### in the script). You also need to update the cyrus files in each mailbox. The full process is
1) Shutdown the Mail server using Server Admin, or with “sudo serveradmin stop mail” from the commandline
2) Run the modified script (below)
3) Delete all the cyrus.index, cyrus.header and cyrus.cache files from the affected users mailbox (don’t forget subfolders)
from the commandline, in the folder of the user to be corrected use: “sudo find . -name “cyrus.*” > ~/Desktop/CyrusFiles.txt” to get the names of every file.
4) Run cyrus reconstruct on the user. (something like: Sudo /usr/bin/cyrus/bin/reconstruct -r user/USERNAME) (where you should leave user as it is and change USERNAME as appropriate).
If you don’t update the cyrus files from scratch Mail.app will still have the wrong dates.
Modified Script:
#!/bin/sh
#
# Date : July 4th, 2005
# Author: Benson Wong
# tummytech@gmail.com
#
# This shell script corrects email messages where the file system
# date does not match the Date: header in the email.
#
# This will fix problems with mail clients like Apple’s mail.app
# which uses the file system timestamp resulting in emails with the
# wrong file system timestamp to display the wrong received date
#
# This script has to be run by a user [root] with the
# necessary privileges to read/modify files in a user’s Maildir.
#
MDIR_PATH=”$1″
if [ $# -lt 1 ]
then
echo “Usage: $0 /path/to/user/Maildir”
exit 1
fi
if [ ! -d “$MDIR_PATH” ]
then
echo “Error: $MDIR_PATH does not exist”
echo “Usage: $0 /path/to/user/Maildir”
exit 1
fi
if [ ! -r “$MDIR_PATH” ]
then
echo “Error: $MDIR_PATH is not readable”
echo “Usage: $0 /path/to/user/Maildir”
exit 1
fi
if [ ! -w “$MDIR_PATH” ]
then
echo “Error: $MDIR_PATH is not writable”
echo “Usage: $0 /path/to/user/Maildir”
exit 1
fi
# set the internal field separator to the newline character
# instead of the default “”.
# This is required for handling filenames and directories with spaces
IFS=”
”
set -f
echo “start”
# Find all emails
for i in `find $MDIR_PATH -type f | \
#### Modify next line for cyrus mailbox strucuture
egrep -v “(cyrus.cache|cyrus.header|cyrus.index)”`
do
EDATE=`awk ‘/^Date: [A-Za-z]*,/ {print $4,$3,$5,$6}’ “$i” | head -1`
if [ -z “$EDATE” ]
then
continue
fi
####Modify for Mac OS X version of “ls”
FDATE=`ls -l -T “$i” | awk ‘{print $6,$7,$9,$8}’`
# Reformat the date for touch.
####Modify for Mac OS X Version of “date”
NDATE=`date -j -f “%b %d %Y %T” “$EDATE” “+%Y%m%d%H%M”`
ODATE=`date -j -f “%b %d %Y %T” “$FDATE” “+%Y%m%d%H%M”`
if [ “$NDATE” -eq “$ODATE” ]
then
# Skip it if the times are correct.
echo -n “_”
continue
fi
echo `basename $i` “from $ODATE to $NDATE”
touch -c -t “$NDATE” “$i”
done
echo “done”
Another (simpler but unintuitive and silly) fix is to remove the “Date Received” column and display “Date Sent” instead. Date sent uses the “Date” header in the email. Only works if you control the client of course.
Hello,
Thank you for producing this great script. If I run it on an IMAP directory that is synced between Mail.app and GMail, will the changes migrate to GMail via the IMAP connection? I am in the middle of a nightmare that resulted from copying files from my laptop to GMail with IMAP, not noticing the systematic data changes…
Many thanks.
@Brad, I’m not sure what the will happen in that scenario. You may just have to try it (maybe back up your data before you do it though.)
@Gabriel Thanks for your script. Would it work on OS X Lion Server 10.7. Thanks a lot for your help!
Your script got me in the right direction, but I could not get it to work on my OS X 10.6.2 Server. I wrote my own in python.
I got this to work by doing the following:
1. shutting down the mail-server:
sudo serveradmin stop mail:
2. running the script below.
3. delete your Mail.app data and caches
4. restart the mail server – and running Mail.app
Script:
#!/usr/bin/env pythonimport os,sys
import re,time
ignore_files = ['dovecot-uidlist.lock','dovecot-keywords','dovecot-uidlist','dovecot.index','dovecot.index.cache','dovecot.index.log','maildirfolder']
delete_files = ['dovecot.index','dovecot.index.cache','dovecot.index.log']
def ticks2mysqlTimeDate(ticks):
"""docstring for ticks2mysqlTimeDate"""
date_obj=MySQLdb.times.DateFromTicks(ticks)
time_obj=MySQLdb.times.TimeFromTicks(ticks)
return date_obj.isoformat()+" "+time_obj.isoformat()
def month_number(mmm):
"""docstring for month_number"""
if mmm=='Jan' or mmm=='jan':
mm=1
elif mmm=='Feb' or mmm=='feb':
mm=2
elif mmm=='Mar' or mmm=='mar':
mm=3
elif mmm=='Apr' or mmm=='apr':
mm=4
elif mmm=='May' or mmm=='may' or mmm=='Mai' or mmm=='mai':
mm=5
elif mmm=='Jun' or mmm=='jun':
mm=6
elif mmm=='Jul' or mmm=='jul':
mm=7
elif mmm=='Aug' or mmm=='aug':
mm=8
elif mmm=='Sep' or mmm=='sep':
mm=9
elif mmm=='Oct' or mmm=='oct' or mmm=='Okt' or mmm=='okt':
mm=10
elif mmm=='Nov' or mmm=='nov':
mm=11
elif mmm=='Dec' or mmm=='dec' or mmm=='Des' or mmm=='des':
mm=12
return mm
def fix_date(fullpathfile):
"""docstring for fix_data"""
sres = os.stat(fullpathfile)
time_created=sres[9]
time_modified=sres[8]
last_opened=sres[7]
utc_time = time.gmtime(sres[8])
if (utc_time.tm_year==2020 and utc_time.tm_mon==11 and utc_time.tm_mday==26):
lines = open(fullpathfile).readlines(4000)
#print 'Checking: %s' % fullpathfile
for l in lines:
m = re.search('^Date:\s+[A-Za-z]{0,3}\,?\s*(\d+)\s+([A-Za-z]{3})\s+(\d{4})\s+(\d+)\:(\d{2})\:?(\d{0,2})',l,re.I)
if m:
mday=int(m.group(1))
month=month_number(m.group(2))
year=int(m.group(3))
hour=int(m.group(4))
minute=int(m.group(5))
second=0 #int(m.group(6))
correct_time=time.mktime((year,month,mday,hour,minute,second,0,0,0))
print 'File: %s <== Header: %s' % (time.ctime(time_modified), l[5:-1])
os.utime(fullpathfile,(correct_time,correct_time))
return
print '*** Unable to parse file: %s' % fullpathfile
for l in lines:
if l.lower().find('date')==0:
print l
return
pass
def dig(dir_fifo):
while dir_fifo:
curdir = dir_fifo.pop(0)
flist=os.listdir(curdir)
for f in flist:
fullpathfile=curdir+'/'+f
if f in delete_files:
os.unlink(fullpathfile)
if not f in ignore_files:
if not os.path.islink(fullpathfile):
if os.path.isdir(fullpathfile):
print "Adding dir: %s" % fullpathfile
dir_fifo.insert(0,fullpathfile)
else:
fix_date(fullpathfile)
def main(mdirs):
"""docstring for main"""
dir_fifo=[]
for d in mdirs[1:]:
rootdir=os.path.realpath(d)
print "Adding root dir: %s" % rootdir
dir_fifo.insert(0,rootdir)
dig(dir_fifo)
pass
if __name__ == '__main__':
if (len(sys.argv)<2):
print "Usage:\n%s /path/to/mailDir1 /path/to/mailDir2 ..." % sys.argv[0]
sys.exit(1)
main(sys.argv)
@Will Thanks for writing this script and sharing it with all. I’m having errors while I try to run it because it has some indentation errors. Can you please send me your python script file with good indentation so it’ll work for me. I have had so many sleepless nights because of this date issue in my emails.
FYI, there's a much simpler way to fix the received dates on both sent and received emails.
Problem: Using Outlook 2011 beta 6 I made the mistake of transferring about 1000 emails from an Exchange server (Account A) to Gmail (Account B) via IMAP. Outlook didn't send the received dates along with the emails, as the IMAP spec says it should (naughty Outlook! but you are in beta, so I'll forgive you), so Gmail used the current time on the server, as the IMAP spec says it should (stupid spec! it should use the email header date, IMO).
Solution:
These steps assume you're transferring a mix of email you sent *and* received all together in one go, but it doesn't really matter.
Step 1 – Transfer the emails from Account B back to Account A using Apple Mail. This should fix the received dates on emails sent *to* you – as Apple Mail will check the email headers and fix the received dates as it sends them to Account A. But the emails sent *from* you will have your current system time (because they were never really "received" by you).
Step 2 – Transfer them back to your Mac.
Step 3 – Close Mail and rename your Envelope Index database (mv ~/Library/Mail/Envelope Index ~/Library/Mail/Envelope Index.bak). Open Mail again – you will be presented with an Import dialogue – this is good! Mail will rebuild your index, including getting the received times of your sent emails from the email headers.
Step 4 – Transfer the emails back to Account B. N.B. If this is Gmail, you may need to Trash all of the emails first, otherwise Gmail will see the email is identical and discard it without changing the timestamp.
Further thoughts: In hindsight, you may not even need steps 1 & 2 – the Envelope Index rebuild probably takes care of both. Oh well, I'd recommend transferring to another account anyway, so you have at least 2 copies at all times!
Good luck!
@Tim
You so saved my, Tim. I could hug your, kiss you… I was using 3 devices to access my IMAP email (around 6.000 mails) using Outlook 2010, Apple Mail and the iPhone. Everything was working perfectly until I installed Thunderbird on my business laptop. It replaced all the received dates with the current date of the mail download. So all my mails had the same date as mentioned in this thread.
I thought about writing a small C# tool, that would open Outlook and replace the received date with the sent date using MAPI.
However, your advise of using Apple Mail and replacing the Envelope Index was a lot faster, easier and probably a lot more reliable. You could probably do the same using Onyx, however renaming should be equally safe.
Thanks a lot dude.
–Step 3 – Close Mail and rename your Envelope Index database (mv ~/Library/Mail/Envelope Index ~/Library/Mail/Envelope Index.bak). Open Mail again – you will be presented with an Import dialogue – this is good! Mail will rebuild your index, including getting the received times of your sent emails from the email headers.–
This is all I had to do, move that file somewhere else and restart mail.. it did the import (Which is really a rebuild, not a true import of new data) and that was all it took to fix the date problem. No other scripts/steps needed.
Just a note for anyone else trying to get Mac OS X Mail.app to properly set message “Date Sent” columns; None of the procedures described above work. However, the “Date Received” column values are corrected using just step 3 as Tim describes it.
Not sure how Mail can possibly have logic that allows the “Date Sent” to be years later than the “Date Received”, but it does…and actually attempting to correct received dates corrupts good sent dates! After performing step 3 I have a lot of “Date Sent” dates of “Today” for email with a “Date Received” of 2005-2007.
Anyone know how to force Apple Mail to use the date field (“Date”) in each message’s header? What is it using as the “Date Sent” source instead?
UPDATE: I wanted to say as well, that the “Date” field in the headers is _NOT_ the received date, it is the date of the message, implying that it is the received date (but could be the date the message was created or saved on either the user’s computer or the originating smtp server).
In any case the received date is the earliest “Received” date available, at least in every case I looked into (a few dozen across multiple servers, including OS X and Yahoo). Seems to be the first entry as well, but I am not sure that can be counted on.
Example:
Date: December 29, 2007 10:31:32 PM CST
Received: from web58714.mail.re1.yahoo.com ([66.196.100.191]:48993) by mail.XXX.com with smtp (Exim 4.60 #1) id 1J8pkr-0001Kj-CP for ; Sat, 29 Dec 2007 22:36:34 -0600
Received: (qmail 54288 invoked by uid 60001); 30 Dec 2007 04:31:33 -0000
Received: from [72.177.119.253] by web58714.mail.re1.yahoo.com via HTTP; Sat, 29 Dec 2007 20:31:32 PST
Notice that the email was received for transit by Yahoo’s servers with the save date as the “Date” header, confirming that the “Date” field is the sent date (the sender used a yahoo.com email address in the example given). A second later that server delivered the email to yahoo’s qmail server for queueing to delivery to the destination address.
Finally my server received the email 5 minutes later, as indicated by the first Received field.
Anyway, all that to say: I can see how the Date Received could be variable, after all just because my email server received the email 5 minutes after it was sent does not mean I received it in my mailbox at that time, I could have not checked my email for another month. So using the file system’s date for the mail, as stored when I check my email, makes sense in this case. HOWEVER; there is, imnsho, no valid reason to use any other date than the actual sent date (in this case either the Date field, or possibly the oldest Received field date).
@Tim
This solution worked like a charm! Thanks Tim!
Hi there @Tim (OR ANYONE ELSE WHO CAN HELP?)
Can you please explain a little clearer how to do your step #3? Do you mean just rename the folder inside my home/library/mail folder to something different and mail will think it is new and therefore update it? Or Is this folder somewhere else? I don;t understand the “envelope index” terminology and this problem has been driving me mad…. step 3 seams to work for others in this forum.
thanks so much!