feat(providers): let the host supply a mail message's Message-ID

CHANGELOG.md

@@ -18,6 +18,8 @@ What counts as a breaking change — and what does not — is defined in [API st
 
 - **`MailMessage.list_unsubscribe`** adds RFC 8058 one-click unsubscribe: an `https` URL emitted as `List-Unsubscribe` together with `List-Unsubscribe-Post: List-Unsubscribe=One-Click`, which Gmail and Yahoo require of bulk senders. The URL is written on one line, never folded or RFC 2047-encoded, and is limited to 978 printable ASCII characters. See [One-click unsubscribe](docs/providers-and-backends.md#one-click-unsubscribe). ([#56](https://github.com/endurain-project/jasil/issues/56))
 
+- **`MailMessage.message_id`** lets the host supply the `Message-ID`, so it can store the key before sending and find the message in the relay's logs even if the process dies before recording the result. It must be an RFC 5322 `<left@right>` dot-atom `msg-id` of at most 250 characters, is sent and returned unchanged, and must be an opaque value, never derived from personal data. Without it, an ID is generated as before. See [Choosing the Message-ID](docs/providers-and-backends.md#choosing-the-message-id). ([#55](https://github.com/endurain-project/jasil/issues/55))
+
 ### Changed
 
 - `SmtpMail` sends `MAIL FROM`, `RCPT TO` and `DATA` itself instead of through `smtplib`'s `send_message`, so it knows which step failed. The conversation on the wire is unchanged, including the `SIZE` declaration and success with a warning when only some recipients are refused. A refused `MAIL FROM` is now reported as a sender refusal, no longer as a rejected message.

docs/providers-and-backends.md

@@ -570,7 +570,7 @@ a verified sender. A per-message sender lets one platform send as several
 identities (per team, brand or campaign) through the same relay; JASIL keeps no
 allowlist, so take it from your own configuration, never from user input. The
 `From` header keeps any display name; the envelope sender is the bare address,
-and the `Message-ID` uses the sender's domain. If a server refuses *some* recipients the send succeeds and a
+and a generated `Message-ID` uses the sender's domain. If a server refuses *some* recipients the send succeeds and a
 warning logs the refused count, because failing would make a retry re-send to
 recipients that already received it. Every recipient of a message is visible in
 `To`, and a refused one is only counted in that warning — send one message per
@@ -605,6 +605,34 @@ with no login or cookie (that is what `One-Click` promises), and the relay must
 DKIM-sign the message covering `List-Unsubscribe`, which RFC 8058 requires.
 `mailto:` unsubscribe is not supported.
 
+### Choosing the Message-ID
+
+`send` generates a `Message-ID` on the sender's domain unless the message brings
+its own. Set `MailMessage.message_id` to store the key before sending, so a crash
+between `send` and recording its result still leaves you a key to look the message
+up in the relay's or provider's logs:
+
+```python
+platform.mail.send(
+    MailMessage(to=["a@example.com"], subject="Hi", text="Hello", message_id="<9f2c41d07a8e@mail.example.com>")
+)
+```
+
+- The format is `<left@right>`, each side dot-separated printable ASCII without
+  whitespace, quotes, brackets or parentheses, at most 250 characters (the RFC 5322
+  `msg-id` without its obsolete and domain-literal forms). Anything else is a
+  `ValueError` that does not echo it. `send` returns the value unchanged.
+- Make it an opaque value, such as a random token stored with the delivery, and
+  put a domain you control on the right. The `Message-ID` is visible to the
+  recipient, every relay and anyone the message is forwarded to: never derive it
+  from an address or other personal data, and do not use an unsalted hash of one.
+- Keeping it unique is the host's job. Reuse it only to retry the same message,
+  never for different content.
+- Some mailbox providers show two copies with the same `Message-ID` as one, but
+  nothing guarantees it: a retry with the same ID can still be delivered twice.
+- Some relays replace the `Message-ID` with their own (Amazon SES does). Check
+  what yours does before relying on the ID appearing in its logs or in bounces.
+
 ### Runtime-configured mail
 
 `build_platform(settings, sources=RuntimeSources(mail=load_mail))` makes

docs/security/integration-checklist.md

@@ -80,6 +80,7 @@ through.
 - [ ] Put credentials in `MailSettings.username` / `password`, never in the URI.
 - [ ] Set `from_address` to an identity your relay has verified.
 - [ ] Take any per-message `MailMessage.from_address` from your own configuration, never from user input.
+- [ ] If you supply `MailMessage.message_id`, make it a random token, never a value derived from an address or other personal data.
 - [ ] Remember the SMTP host is **not** SSRF-checked — it is expected to be an
       internal relay. Keep that setting out of attacker-influenced configuration.
 - [ ] Do not put a rendered body or secrets in an event payload that drives a

docs/security/threat-model.md

@@ -106,7 +106,10 @@ own timeouts only.
 **Personal data and secrets in logs and job rows.** The job runner persists
 `str(error)` and logs `exc_info`, so `MailError` carries only a category and an
 SMTP status code and is raised `from None`. Log lines record the recipient count,
-`Message-ID` and outcome, never addresses, subjects or bodies. The password is
+`Message-ID` and outcome, never addresses, subjects or bodies. A host-supplied
+`MailMessage.message_id` is logged and travels with the message to every relay and
+recipient, so it must be opaque and never derived from personal data; JASIL checks
+its format only. The password is
 excluded from `repr(settings)`, and credentials are refused in the URI so they
 cannot end up in a logged string.
 

jasil/_core/mail_message.py

@@ -11,6 +11,7 @@
 a durable-job handler is persisted and logged, and an address is personal data.
 """
 
+import re
 from collections.abc import Sequence
 from datetime import datetime
 from email import policy
@@ -49,6 +50,13 @@ def fold(self, *, policy: Policy) -> str:
 # not declare ``BODY=8BITMIME``, so a raw 8bit body is not valid for a strict relay.
 _MIME_POLICY = policy.SMTP.clone(cte_type="7bit", header_factory=_HEADERS)
 
+#: Longest host-supplied ``Message-ID``, angle brackets included.
+MAX_MESSAGE_ID_LENGTH = 250
+
+# RFC 5322 ``msg-id`` without its obsolete and domain-literal forms: ``<`` dot-atom-text ``@`` dot-atom-text ``>``.
+_DOT_ATOM_TEXT = r"[A-Za-z0-9!#$%&'*+\-/=?^_`{|}~]+(?:\.[A-Za-z0-9!#$%&'*+\-/=?^_`{|}~]+)*"
+_MESSAGE_ID = re.compile(rf"<{_DOT_ATOM_TEXT}@{_DOT_ATOM_TEXT}>", re.ASCII)
+
 
 class MailMessageLike(Protocol):
     """The shape :func:`build_mime` reads; ``jasil.providers.MailMessage`` fits.
@@ -77,6 +85,10 @@ def html(self) -> str | None:
     def reply_to(self) -> str | None:
         """Single ``Reply-To`` address, if any."""
 
+    @property
+    def message_id(self) -> str | None:
+        """Host-supplied ``Message-ID``, if any."""
+
     @property
     def list_unsubscribe(self) -> str | None:
         """One-click unsubscribe URL, if any."""
@@ -164,6 +176,20 @@ def parse_address(value: str, *, role: str) -> Address:
         raise invalid from None
 
 
+def _message_id(value: object) -> str:
+    """Return ``value`` when it is a ``<left@right>`` msg-id that the header carries unchanged.
+
+    Raises:
+        ValueError: Otherwise; the message does not echo the value.
+    """
+    if not isinstance(value, str) or len(value) > MAX_MESSAGE_ID_LENGTH or not _MESSAGE_ID.fullmatch(value):
+        raise ValueError(
+            "The mail message_id must be '<left@right>', each side dot-separated printable ASCII without "
+            f"whitespace, quotes, brackets or parentheses, at most {MAX_MESSAGE_ID_LENGTH} characters"
+        )
+    return value
+
+
 def _unsubscribe_url(value: object) -> str:
     """Return ``value`` when it is an ``https`` URL that can be written verbatim on one header line.
 
@@ -198,7 +224,9 @@ def build_mime(message: MailMessageLike, *, sender: str, now: datetime) -> Email
         now: The ``Date`` header value, from the platform clock.
 
     Returns:
-        A complete message with ``From``, ``To``, ``Date`` and ``Message-ID`` set.
+        A complete message with ``From``, ``To``, ``Date`` and ``Message-ID`` set; the
+        ``Message-ID`` is the message's own when it has one, otherwise generated
+        on the sender's domain.
 
     Raises:
         ValueError: When the message or sender is invalid.
@@ -217,6 +245,7 @@ def build_mime(message: MailMessageLike, *, sender: str, now: datetime) -> Email
         raise ValueError("The mail subject must be a single line")
     from_address = parse_address(sender, role="sender")
     reply_to = parse_address(message.reply_to, role="reply-to") if message.reply_to else None
+    message_id = None if message.message_id is None else _message_id(message.message_id)
     unsubscribe = None if message.list_unsubscribe is None else _unsubscribe_url(message.list_unsubscribe)
 
     mime = EmailMessage(policy=_MIME_POLICY)
@@ -226,7 +255,7 @@ def build_mime(message: MailMessageLike, *, sender: str, now: datetime) -> Email
         mime["Reply-To"] = reply_to
     mime["Subject"] = message.subject
     mime["Date"] = format_datetime(now)
-    mime["Message-ID"] = make_msgid(domain=from_address.domain)
+    mime["Message-ID"] = make_msgid(domain=from_address.domain) if message_id is None else message_id
     if unsubscribe is not None:
         # RFC 8058 one-click: the mailbox provider POSTs ``List-Unsubscribe=One-Click`` to the URL.
         mime["List-Unsubscribe"] = f"<{unsubscribe}>"

jasil/providers.py

@@ -965,15 +965,26 @@ class MailMessage:
         reply_to: Optional single ``Reply-To`` address.
         from_address: The sender of this message, ``addr`` or ``"Name <addr>"``,
             in place of the configured ``mail.from_address``: the ``From``
-            header, the envelope sender (its bare address) and the
-            ``Message-ID`` domain. The relay decides which senders it accepts;
+            header, the envelope sender (its bare address) and the domain of
+            a generated ``Message-ID``. The relay decides which senders it accepts;
             never derive this from untrusted input.
         list_unsubscribe: Optional RFC 8058 one-click unsubscribe URL: an
             absolute ``https`` URL of printable ASCII, at most 978 characters,
             without spaces, ``<``, ``>`` or ``,``. The message then carries
             ``List-Unsubscribe`` and ``List-Unsubscribe-Post``. The URL must
             accept a POST with no login, and the relay must DKIM-sign the
             ``List-Unsubscribe`` header.
+        message_id: Optional ``Message-ID``, so the host can store the key it
+            will find in the relay's logs before sending. ``"<left@right>"``,
+            each side dot-separated printable ASCII without whitespace, quotes,
+            brackets or parentheses, at most 250 characters; ``right`` is
+            normally a domain the host controls. ``send`` returns it unchanged.
+            When ``None`` one is generated on the sender's domain. The host keeps
+            it unique: reuse it only to retry the same message, never for
+            different content. It is visible to the recipient, every relay and
+            anyone the message is forwarded to, so make it an opaque value (a
+            random token), never one derived from personal data. A relay may
+            replace it with its own.
     """
 
     to: Sequence[str]
@@ -983,12 +994,22 @@ class MailMessage:
     reply_to: str | None = None
     from_address: str | None = None
     list_unsubscribe: str | None = None
+    message_id: str | None = None
 
     def __hash__(self) -> int:
         # ``to`` is usually a list, which the generated hash cannot digest.
         recipients = (self.to,) if isinstance(self.to, str) else tuple(self.to)
         return hash(
-            (recipients, self.subject, self.text, self.html, self.reply_to, self.from_address, self.list_unsubscribe)
+            (
+                recipients,
+                self.subject,
+                self.text,
+                self.html,
+                self.reply_to,
+                self.from_address,
+                self.list_unsubscribe,
+                self.message_id,
+            )
         )
 
 

tests/test_mail.py

@@ -507,6 +507,71 @@ def test_the_url_is_part_of_the_message_identity(self):
         assert hash(_message(list_unsubscribe=self.URL)) != hash(_message())
 
 
+class TestHostSuppliedMessageId:
+    """``MailMessage.message_id`` lets the host store the key before sending; it is sent and returned unchanged."""
+
+    MESSAGE_ID = "<7f3a9c1e.d41d8cd9@mail.example.com>"
+
+    def test_the_supplied_id_is_the_header_and_the_return_value(self, backend):
+        returned = backend.send(_message(message_id=self.MESSAGE_ID))
+
+        assert returned == self.MESSAGE_ID
+        assert _last_built(backend)["Message-ID"] == self.MESSAGE_ID
+
+    def test_an_id_is_generated_when_none_is_supplied(self):
+        mime = build_mime(_message(), sender=SENDER, now=NOW)
+
+        assert mime["Message-ID"].endswith("@example.com>")
+
+    @pytest.mark.parametrize(
+        "value",
+        [
+            "<a.b!#$%&'*+-/=?^_`{|}~@x.example>",
+            "<" + "a" * (mail_message.MAX_MESSAGE_ID_LENGTH - 20) + "@mail.example.com>",
+        ],
+    )
+    def test_every_dot_atom_form_survives_the_wire_unchanged(self, value):
+        mime = build_mime(_message(message_id=value), sender=SENDER, now=NOW)
+
+        raw = mime.as_bytes()
+        unfolded = raw.split(b"\r\n\r\n")[0].replace(b"\r\n ", b" ").split(b"\r\n")
+        assert f"Message-ID: {value}".encode() in unfolded
+        assert email.message_from_bytes(raw, policy=email.policy.default)["Message-ID"] == value
+
+    @pytest.mark.parametrize(
+        "value",
+        [
+            "",
+            "a@example.com",
+            "<a@example.com",
+            "<a.example.com>",
+            "<a@b@example.com>",
+            "<@example.com>",
+            "<a@>",
+            "<a..b@example.com>",
+            "<.a@example.com>",
+            "<a.@example.com>",
+            "<a@[192.0.2.1]>",
+            "<a b@example.com>",
+            '<"q"@example.com>',
+            "<a(c)@example.com>",
+            "<olá@example.com>",
+            " <a@example.com>",
+            "<a@example.com>\r\nBcc: victim@example.com",
+            "<" + "a" * mail_message.MAX_MESSAGE_ID_LENGTH + "@example.com>",
+            42,
+        ],
+    )
+    def test_anything_but_a_dot_atom_msg_id_is_refused_without_echoing_it(self, value):
+        with pytest.raises(ValueError, match="message_id") as caught:
+            build_mime(_message(message_id=value), sender=SENDER, now=NOW)
+
+        assert not isinstance(value, str) or not value or value not in str(caught.value)
+
+    def test_the_id_is_part_of_the_message_identity(self):
+        assert hash(_message(message_id=self.MESSAGE_ID)) != hash(_message())
+
+
 class TestMemoryMail:
     def test_messages_are_recorded_in_order(self):
         mail = _memory()