CIX scratchpad to MBox mail store file conversion Howto, mark 2 Comments and feedback sought! -- Liam P. *Migrating from Ameol to Thunderbird* A preliminary how-to guide on converting a messagebase of email in Ameol or another CIX offline reader into MBox format for import into Mozilla Thunderbird, Eudora or other Mbox-based email clients. Based on the notes at http://www.livejournal.com/users/lproven/65227.html and cix:bikers/14nonbike_qury:4149 ========================== This guide will help you convert your email store from a CIX offline reader into a dedicated email client. _Notes_ 1. The source format is a CIX scratchpad, not your OLR’s internal storage format. You must export your messages to a scratchpad then work on that file, not on the messagebase itself. 2. I am assuming that you are using Windows, starting from Ameol & going to Thunderbird. However, the same procedure will work on any OS – you will just need appropriate tools 3. Do not start with your entire email store. Start with a small subset of just a few messages – perhaps a dozen or two. Once you know what you’re doing, try a larger number or if you confident, all of them. 4. Ameol strips attachments out of mail messages when it decodes them. This leaves the messages malformed and they will not survive the translation, appearing as blank messages. I welcome any suggestions as to how to resolve this. 5. If you have sorted your Ameol inbox into subfolders, I suggest exporting the whole thing to one big scratchpad & recreating the hierarchy afterwards. 6. Note that Ameol message flags – read/unread, priority, keep, lock etc. – will not survive the conversion. I believe, in fact, that they don’t survive exportation to a scratchpad. While Microsoft Word is perfectly capable of doing the search and replace operations, it struggles with large file sizes, such as an email archive of hundreds of thousands of messages. It can bring Windows to its knees with swapping. I have found that the shareware TextPad editor is capable of handling files of around the size of the PC’s total memory without causing excessive thrashing and it has the search & replace capabilities required. Recommended, but if you have a preferred editor, by all means use it. http://www.textpad.com/ _Steps_ [1] Create a scratchpad To begin, select the email messages you want to work on and export them to a scratchpad. In Ameol, to do this, open the File menu and choose Export. Select the “CIX Scratchpad” format. (This will be written File | Export | CIX Scratchpad from now on for brevity.) [2] Make a backup copy of your scratchpad. Work on the copy. You will not get this perfectly right first time! [3] Open the scratchpad in your editor. You might also wish to create a small MBox file in your email program – perhaps by sending yourself a few messages from a webmail account, for example – and open it side-by-side for comparison. [4] Convert the message headers The only difference between a scratchpad and an MBox file are the delimiters which separate messages from one another. The email headers are in standard Internet mail formats which do not change. To accomplish the conversion, it’s merely necessary to convert CIX’s message separator lines into MBox-standard ones. This can be done with a couple of careful search & replace commands, but it requires an editor whose search & replace commands can handle strings containing carriage returns. [5] Copy the converted file into the mail program’s store directory. This must be done with your email program shut down and not running. Use Task Manager to ensure that it is not still in memory but invisible. [6] Re-open the mail program and open the new folder Most email programs today maintain separate index files for performance reasons. The first time a newly-converted mail store file is opened, these indices must be created. This takes some time. [7] After moving around messages to taste, compact the MBox files. MBox files are inefficient. If a message is moved or deleted, a blank gap is left in the file; the files do not shrink. To prevent the email program needing vast amounts of RAM to open these “holey” files, it’s necessary to compress them after sorting and tidying messages. _The file formats_ MBox mail messages are delimited by a line starting with the word “From” followed by a space. The Thunderbird "From " header is just a date & it's always the same one, so the value is clearly not used. CIX scratchpad messages begin with !MF or !MFA Followed by a header containing the CIX Memo number. If we wish to preserve that information, since throwing data away is seldom a good idea, we can make it into a extra MBox header by reformatting “Memo [number]” as “X-CIX-Memo [number]”. So what you need to do is replace pairs of lines containing: !MF Memo or !MFA Memo With pairs saying: From CIX scratchpad X-CIX-Memo: (I used this phrasing, in place of Mozilla’s date, just as a marker for myself). You need to search for “exclamation mark M F, newline, Memo, space” and replace it with “From CIX Scratchpad, newline, X-CIX-Memo, space”, or similar. The point is to replace both lines at once with the new pair of lines. Once you have done this search & replace, your working-copy scratchpad should be an MBox format file. For Thunderbird users, you can just copy the file into Thunderbird’s mail folders and next run the program should find it, display it as a message folder and, when you click on it, index it and display the messages within. Thunderbird keeps its message store under a randomly-generated profile name in the Application Data folder in a user’s Documents and Settings folder (assuming WinXP). Under this is a folder for each configured email service, named after the server itself, with a suffixed number for multiple instances of the same server. E.g. mine is: C:\Documents and Settings\Liam Proven\Application Data\Thunderbird\Profiles\97jxmos5.import\Mail\mail.cix.co.uk This is a suitable directory to place the converted file into. If Thunderbird does not recognize it, try moving an existing file out of there – such as “Inbox” – and replacing it with your converted file, renamed appropriately. N.B. This process can be slow. I converted some 325,000 messages totalling some 300MB on a 512MB dual-processor 2GHz machine. Saving the modified file out of my editor took some ten minutes and Thunderbird took several hours to index and display the resultant email folder. Start with small test folders and be patient. After a successful import, choose “File | Compact Folders” to reorganize the MBox files for efficient use. I think you need an extra step before this step "So what you need to do is replace pairs of lines containing: !MF Memo or !MFA Memo With pairs saying: From CIX scratchpad X-CIX-Memo: " That finds the message delimiter used by the mbox format i.e. [blank line] From[space] and replaces it with before the above step [blank line] >From[space] Otherwise and messages with that sequence in will get split into two (bit before the from and bit after) See cix:email/clients:153 for a fuller description. Andy I don't quite follow. > I think you need an extra step before this step [...] All right. But then: > That finds the message delimiter used by the mbox format i.e. > > [blank line] > From[space] > > and replaces it with before the above step > > [blank line] > >From[space] The phrasing of that is giving me difficulties. You're saying that before replacing CIX message delimiters with MBox ones, it's necessary to quote any lines starting with "From" after a blank line? (Where "quote" means insert a greater-than symbol.) > Otherwise and messages with that sequence in will get split into two > (bit before the from and bit after) Right. I can see that. I've not had this problem, but I can see that. > See cix:email/clients:153 for a fuller description. -- Liam P. >You're saying that before replacing CIX message delimiters with MBox ones, >it's necessary to quote any lines starting with "From" after a blank line? >(Where "quote" means insert a greater-than symbol.) Yep, that's exactly what I'm saying, but perhaps not very clearly! This might be a better explanation: http://email.about.com/cs/standards/a/mbox_format.htm Cheers Andy Right. Gotcha. TFTI! -- Liam P.