Editing Maildir

{{Short description|E-mail format}}
The '''Maildir''' [[e-mail]] format is a common way of storing [[email]] messages on a [[file system]], rather than in a database. Each message is assigned a [[Computer file|file]] with a unique name, and each mail folder is a file system directory containing these files. Maildir was designed by [[Daniel J. Bernstein]] circa 1995, with a major goal of eliminating the need for program code to handle file locking and unlocking through use of the local filesystem.<ref name="qmail-manpage"/> Maildir design reflects the fact that the only operations valid for an email message is that it be created, deleted or have its status changed in some way.

[[File:Maildir.png|frame|Internal structure]]

__TOC__

== Specifications ==

A Maildir [[directory (file systems)|directory]] (often named <code>Maildir</code>) usually has three subdirectories named <code>tmp</code>, <code>new</code>, and <code>cur</code>.<ref>{{Cite book |last=Blum |first=Richard |url=https://books.google.com/books?id=vPv-jIhyowsC&dq=maildir-+how+it+works&pg=PA224 |title=Postfix |date=2001 |publisher=Sams Publishing |isbn=978-0-672-32114-6 |language=en}}</ref> 

* The <code>tmp</code> subdirectory temporarily stores e-mail messages that are in the process of being delivered. This subdirectory may also store other kinds of temporary files.
* The <code>new</code> subdirectory stores messages that have been delivered, but have not yet been seen by any mail application.
* The <code>cur</code> subdirectory stores messages that have already been seen by mail applications.<ref name="courier-maildir" />

=== Maildir++ ===

Sam Varshavchik, the author of the [[Courier Mail Server]] and other software, defined the '''Maildir++''' extension<ref name="courier-maildir"/><ref name="courier-plusplus"/> to the Maildir format to support subfolders and mail quotas. Maildir++ directories contain subdirectories with names that start with a '.' (dot) which are also Maildir++ folders.  The extension complies with the original Maildir specification, which allows for subdirectories in addition to ''tmp'', ''new'' and ''cur''.

== Technical operation ==

A [[mail delivery agent]] is a [[Computer process|program]] that delivers an email message into a Maildir. The mail delivery agent creates a new file with a unique filename in the <code>tmp</code> directory.<ref name="djbspec-2000"/><ref name="dovecot"/><ref name="courier-maildir"/> At the time of its invention guaranteeing unique filenames efficiently was difficult. The original [[qmail]]<ref name="qmail-manpage"/> algorithm for unique names was:
# read the current [[Unix time]]
# read the current [[process identifier]] (PID)
# read the current [[hostname]]
# concatenate the above three values into a string separated by the [[Full stop|period character]]; this is the new filename
# if [[stat (system call)|<code>stat()</code>]] reports that the filename exists, then wait two seconds
# go to previous step until the filename does not exist
# create a file with the unique filename and write the message contents to the new file

By 2000, the qmail author recommended in an updated specification<ref name="djbspec-2000"/> to append the value of a per-process counter to the PID, whose value should be incremented after each delivery. The rate-limiting recommendation to "wait two seconds" was dropped. 

By 2003, the recommendations had been further amended to require that instead of the PID and counter, the middle part of the filename should be created by "concatenating enough of the following strings to guarantee uniqueness" even in the face of multiple simultaneous deliveries to the same maildir from one or more processes:<ref name="djbspec-2003"/>
<blockquote>
* #''n'', where ''n'' is (in hexadecimal) the output of the operating system's <code>unix_sequencenumber()</code> system call, which returns a number that increases by 1 every time it is called, starting from 0 after reboot.
* X''n'', where ''n'' is (in hexadecimal) the output of the operating system's <code>unix_bootnumber()</code> system call, which reports the number of times that the system has been booted. Together with #, this guarantees uniqueness; unfortunately, most operating systems don't support <code>unix_sequencenumber()</code> and <code>unix_bootnumber()</code>.
* R''n'', where ''n'' is (in hexadecimal) the output of the operating system's <code>unix_cryptorandomnumber()</code> system call or an equivalent source, such as <code>/dev/urandom</code>. Unfortunately, some operating systems don't include cryptographic random number generators.
* I''n'', where ''n'' is (in hexadecimal) the UNIX [[inode]] number of this file. Unfortunately, inode numbers aren't always available through [[Network File System|NFS]].
* V''n'', where ''n'' is (in hexadecimal) the UNIX device number of this file. Unfortunately, device numbers aren't always available through NFS. (Device numbers are also not helpful with the standard UNIX filesystem: a maildir has to be within a single UNIX device for <code>link()</code> and <code>rename()</code> to work.)
* M''n'', where ''n'' is (in decimal) the microsecond counter from the same <code>gettimeofday()</code> used for the left part of the unique name.
* P''n'', where ''n'' is (in decimal) the process ID.
* Q''n'', where ''n'' is (in decimal) the number of deliveries made by this process.
</blockquote>

This 2003 algorithm was criticised<ref name="dovecot-2006-crit"/> in 2006 as being unnecessarily complex by [[Timo Sirainen]], the creator of [[Dovecot (software)|Dovecot]].

As of November 2023, qmail author [[Daniel J. Bernstein|Daniel Bernstein]] had made no further changes to the 2003 filename generation recommendations.<ref name="djbspec-hist"/> On modern POSIX systems, [[temporary file]]s can be safely created with the <code>[[mkstemp]]</code> C library function.

The delivery process stores the message in the maildir by creating and writing to <code>tmp/''uniquefilename''</code>, and then moving this file to <code>new/''uniquefilename''</code>. The moving can be done using <code>[[rename (computing)|rename]]</code>, which is atomic in many systems.<ref name="opengroup-rename"/> Alternatively, it can be done by [[hard link|hard-linking]] the file to <code>new</code> and then unlinking the file from <code>tmp</code>. Any leftover file will eventually be deleted. This sequence guarantees that a maildir-reading program will not see a partially written message. There can be multiple programs reading a maildir at the same time. They range from [[mail user agent]]s (MUAs), which access the server's file system directly, through [[Internet Message Access Protocol]] or [[Post Office Protocol]] servers acting on behalf of remote MUAs, to utilities such as [[Biff (Unix)|biff]] and [[rsync]], which may or may not be aware of the maildir structure. Readers should never look in <code>tmp</code>.

When a cognizant maildir-reading process (either a [[Post Office Protocol|POP]] or [[Internet Message Access Protocol|IMAP]] server, or a mail user agent acting locally) finds messages in the <code>new</code> directory, it ''must'' move them to <code>cur</code>.  It is just a means to notify the user "you have ''X'' new messages".<ref name="courier-msg38512"/> This moving needs to be done using the atomic filesystem <code>rename()</code>, as the alternative ''link-then-unlink'' technique is non-atomic and may result in duplicated messages. An informational suffix is appended to filenames at this stage. It consists of a [[colon (punctuation)|colon]] (to separate the unique part of the filename from the actual information), a "2", a [[comma]] and various [[flag (computing)|flag]]s. The "2" specifies the version of the information that follows the comma. "2" is the only currently officially specified version, "1" being an experimental version. The specification defines flags that show whether the message has been read, deleted and so on: the initial (capital) letter of "Passed", "Replied", "Seen", "Trashed", "Draft", and "Flagged".<ref name="djbspec-2003"/> Applications often choose to supplement this very limited set of flags, for example notmuch<ref>{{Cite web|url=https://notmuchmail.org/|title=Notmuch mail system homepage|website=notmuchmail.org|access-date=2019-06-22}}</ref> offers flag synchronization in addition to arbitrary user-defined flags,<ref name="notmuch-config d147">{{cite web | title=notmuch 0.38.3 documentation | website=notmuch-config | url=https://notmuchmail.org/doc/latest/man1/notmuch-config.html#nmconfig-maildir.synchronize_flags | access-date=2024-04-17}}</ref> while Dovecot uses lowercase letters to match 26 IMAP keywords,<ref name="dovecot"/> which may include keywords such as $[[Message Disposition Notification|MDNSent]] or user-defined flags.

Although Maildir was intended to allow lockless usage, in practice some software that uses Maildirs also uses locks, such as Dovecot.<ref name="dovecot-2006-crit-2"/>

== File-system compatibility issues ==

The Maildir standard can only be implemented on systems that accept colons in filenames. <ref>{{Cite web |title=mailbox — Manipulate mailboxes in various formats |url=https://docs.python.org/3/library/mailbox.html |access-date=2023-06-19 |website=Python documentation}}</ref>

Systems that don't allow colons in filenames (this includes [[Microsoft Windows]] and some configurations of [[Novell Storage Services]]) can use a non-standard alternative separator, such as ";" or "-". It is often trivial to patch [[free and open-source software]] to use a different separator.<ref name="mutt-colons"/>

As there is currently no agreement on what character this alternative separator should be, there can be interoperability difficulties between different Maildir-supporting programs on these systems. However, not all Maildir-related software needs to know what the separator character is, because not all Maildir-related software needs to be able to read or modify the flags of a message ("read", "replied to" etc.); software that merely delivers to a Maildir or archives old messages from it based only on date, should work no matter what separator is in use. If only the [[email client|MUA]] needs to read or modify message flags, and only one MUA is used, then non-standard alternative separators may be used without interoperability problems.

==Software that supports Maildir directly==

=== [[Mail Transfer Agent|Mail servers]] ===

*[[Dovecot (software)|Dovecot]] IMAP server
*[[Courier Mail Server]] SMTP and IMAP server, for which the Maildir++ format was invented
*[[Sendmail]] The original SMTP server
*[[Exim]] SMTP server
*[[Postfix (software)|Postfix]] SMTP server
*[[qmail]] SMTP server, for which the Maildir format was invented
*[[MeTA1]] SMTP server
*[[OpenSMTPD]] SMTP server
*[https://stalw.art/ Stalwart Mail Server], SMTP and IMAP server implemented in [[Rust (programming language)|Rust]]

=== [[Mail delivery agent|Delivery agents]] ===
*[[procmail]]
*[[Dovecot (software)|Dovecot]] delivery agent
*[[maildrop]]
*[[getmail]], a Maildir-aware mail-retrieval and delivery agent alternative to [[Fetchmail]]
*[[fdm (software)|fdm]]
*[https://www.muchsync.org/ muchsync], synchronising [https://notmuchmail.org notmuch mail] mailboxes between any number of replicas
*[[OfflineIMAP]]
*[https://isync.sourceforge.io/ isync], synchronises mailboxes, supporting Maildir and IMAP4
*[https://github.com/phlummox/attomail Attomail], a minimal Maildir-aware MDA implemented in Haskell

=== [[Mail user agent|Mail readers]] ===

*aerc<ref>{{Cite web|url=https://aerc-mail.org/|title=aerc - the world's best email client homepage|website=aerc-mail.org}}</ref> (efficient and extensible email client)
*[[Balsa (e-mail client)|Balsa]] previously the official GNOME mail reader (prior to Evolution)
*[[Cone (software)|Cone]] a curses-based mail reader
*[[Evolution (software)|Evolution]], official GNOME mail client
*[[GNUMail]]
*[[Gnus]]
*[[KMail]], KDE mail reader
*[[mailx]]
*[[Mutt (email client)|Mutt]]
*Notmuch<ref>{{Cite web|url=https://notmuchmail.org/|title=Notmuch mail system homepage|website=notmuchmail.org|access-date=2019-06-22}}</ref> (fast, global-search and tag-based email system)
*[[Alpine (email client)|Pine/Alpine]]
*[[Mozilla Thunderbird]] – experimental and “disabled by default because there are still many bugs” <ref name=thunderbird>{{Cite web|url=https://support.mozilla.org/en-US/kb/maildir-thunderbird|title=Maildir in Thunderbird|website=mozilla.org|access-date=2020-12-06}}</ref>

==Notes and references==
{{reflist|refs=
<ref name="djbspec-2000">{{cite web |first=Daniel J. |last=Bernstein. |author-link=Daniel J. Bernstein |year=c. 2000 |url=http://cr.yp.to/proto/maildir.html |title=Using maildir format |archive-url=https://web.archive.org/web/20000902121438/http://cr.yp.to/proto/maildir.html |archive-date=2000-09-02 |orig-year=First published 2000 or earlier |url-status=live |access-date=2018-11-23 }}</ref>
<ref name="djbspec-2003">{{cite web |first=Daniel J. |last=Bernstein. |author-link=Daniel J. Bernstein |year=2003|url=http://cr.yp.to/proto/maildir.html |title=Using maildir format |archive-url=https://web.archive.org/web/20030401082238/https://cr.yp.to/proto/maildir.html |archive-date=2003-04-01 |orig-year=The earliest version of this document was first published in 2000 or earlier |url-status=live |access-date=2018-11-23}}</ref>
<ref name="djbspec-hist">{{cite web |url=https://web.archive.org/web/20230503185934/cr.yp.to/proto/maildir.html |publisher=[[Internet Archive]] |access-date=2023-11-23 |title=Wayback Machine snapshots of cr.yp.to/proto/maildir.html |year=2023}}</ref>
<ref name="courier-maildir">{{cite web
 |url=http://www.courier-mta.org/maildir.html
 |title=maildir(5)
 |first=Sam
 |last=Varshavchik
 |year=2009
 |access-date=2024-08-09
 |url-status=live
 |archive-url=https://web.archive.org/web/20240417195248/https://www.courier-mta.org/maildir.html
 |archive-date=2024-04-17
}}</ref>
<ref name="courier-msg38512">{{cite mailing list |url=http://www.mail-archive.com/courier-users@lists.sourceforge.net/msg38512.html |title=Management of maildir structures |date=25 July 2016 |access-date=26 July 2016 |mailing-list=courier-users |author=Sam Varshavchik }}</ref>
<ref name="courier-plusplus">{{cite web
 |url=http://www.courier-mta.org/imap/README.maildirquota.html
 |title=Maildir++
 |first=Sam
 |last=Varshavchik
 |year=2011
 |access-date=2024-08-09
 |url-status=live
 |archive-url=https://web.archive.org/web/20240529081454/https://www.courier-mta.org/imap/README.maildirquota.html
 |archive-date=2024-05-29
}}</ref>
<ref name="dovecot">[https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/ Dovecot Wiki: maildir format]</ref>
<ref name="mutt-colons">[http://marc.info/?l=mutt-users&m=131644437917477&w=2 mutt maildir support: workaround for filesystems that don't accept colons]</ref>
<ref name="dovecot-2006-crit">{{cite web
 |url=https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/#mail-delivery
 |title=Maildir Mailbox Format: Mail Delivery'
 |first=Timo
 |last=Sirainen
 |author-link=Timo Sirainen
 |access-date=2024-08-09
 |orig-date=2006-12-05<!--
      The diff for the original 2006-12-05 critique added in the Dovecot wiki is archived at:
      https://web.archive.org/web/20210820172330/https://wiki2.dovecot.org/MailboxFormat/Maildir?action=diff&rev1=11&rev2=12 -->
 |url-status=live
 |archive-url=https://web.archive.org/web/20240624123136/https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/#mail-delivery
 |archive-date=2024-06-24
 |quote="All this trouble is rather pointless. Only the first step is what really guarantees that the mails won’t get overwritten, the rest just sounds nice. Even though they might catch a problem once in a while, they give no guaranteed protection and will just as easily pass duplicate filenames through and overwrite existing mails.¶ Step 2 is pointless because there’s a race condition between steps 2 and 3. PID/host combination by itself should already guarantee that it never finds such a file. If it does, something’s broken and the stat() check won’t help since another process might be doing the same thing at the same time, and you end up writing to the same file in tmp/, causing the mail to get corrupted.¶ In step 4 the link() would fail if an identical file already existed in the Maildir, right? Wrong. The file may already have been moved to cur/ directory, and since it may contain any number of flags by then you can’t check with a simple stat() anymore if it exists or not.¶ Step 2 was pointed out to be useful if clock had moved backwards. However, this doesn’t give any actual safety guarantees because an identical base filename could already exist in cur/. Besides if the system was just rebooted, the file in tmp/ could probably be even overwritten safely (assuming it wasn’t already link()ed to new/).¶ So really, all that’s important in not getting mails overwritten in your Maildir is step 1: Always create filenames that are guaranteed to be unique. Forget about the 2 second waits and such that the Qmail’s man page talks about."
}}</ref>
<ref name="dovecot-2006-crit-2">{{cite web
 |url=https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/#locking
 |title=Maildir Mailbox Format: Locking
 |first=Timo
 |last=Sirainen
 |author-link=Timo Sirainen
 |access-date=2024-08-09
 |orig-date=2006-12-05<!--
      The diff for the original 2006-12-05 description of locking in the Dovecot wiki is archived at:
      https://web.archive.org/web/20181123154140/https://wiki2.dovecot.org/MailboxFormat/Maildir?action=diff&rev1=13&rev2=14  -->
 |url-status=live
 |archive-url=https://web.archive.org/web/20240624123136/https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/#locking
 |archive-date=2024-06-24
}}</ref>
<ref name="opengroup-rename">{{cite web |url=http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html |title=rename |author=<!--Staff writer(s); no by-line.--> |year=2013 |publisher=[[The Open Group]] |access-date=23 July 2016 |quote=That specification requires that the action of the function be atomic.}}</ref>
<ref name="qmail-manpage">{{cite web |archive-url=https://web.archive.org/web/19971012032244/http://www.qmail.org/qmail-manual-html/man5/maildir.html |url=http://www.qmail.org/qmail-manual-html/man5/maildir.html |first=Daniel J. |last=Bernstein |author-link=Daniel J. Bernstein |title=maildir(5) |archive-date=1997-10-12 |year=1995 |access-date=2018-11-23 |url-status=dead }}</ref>
}}

==See also==
*[[mbox]]
*[[MH Message Handling System]]
*[[MIX (email)]]

== External links ==
* [https://manpages.debian.org/stretch/qmail/maildir.5.en.html manual page for maildir]
* [http://cr.yp.to/proto/maildir.html maildir specifications]

[[Category:Email storage formats]]