diff --git a/CHANGELOG.md b/CHANGELOG.md index 475eb3f..1725e44 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,8 @@ What counts as a breaking change — and what does not — is defined in [API st - **`MailMessage.list_unsubscribe`** adds RFC 8058 one-click unsubscribe: an `https` URL emitted as `List-Unsubscribe` together with `List-Unsubscribe-Post: List-Unsubscribe=One-Click`, which Gmail and Yahoo require of bulk senders. The URL is written on one line, never folded or RFC 2047-encoded, and is limited to 978 printable ASCII characters. See [One-click unsubscribe](docs/providers-and-backends.md#one-click-unsubscribe). ([#56](https://github.com/endurain-project/jasil/issues/56)) +- **`MailMessage.message_id`** lets the host supply the `Message-ID`, so it can store the key before sending and find the message in the relay's logs even if the process dies before recording the result. It must be an RFC 5322 `` dot-atom `msg-id` of at most 250 characters, is sent and returned unchanged, and must be an opaque value, never derived from personal data. Without it, an ID is generated as before. See [Choosing the Message-ID](docs/providers-and-backends.md#choosing-the-message-id). ([#55](https://github.com/endurain-project/jasil/issues/55)) + ### Changed - `SmtpMail` sends `MAIL FROM`, `RCPT TO` and `DATA` itself instead of through `smtplib`'s `send_message`, so it knows which step failed. The conversation on the wire is unchanged, including the `SIZE` declaration and success with a warning when only some recipients are refused. A refused `MAIL FROM` is now reported as a sender refusal, no longer as a rejected message. diff --git a/docs/providers-and-backends.md b/docs/providers-and-backends.md index 85fd457..a61b944 100644 --- a/docs/providers-and-backends.md +++ b/docs/providers-and-backends.md @@ -570,7 +570,7 @@ a verified sender. A per-message sender lets one platform send as several identities (per team, brand or campaign) through the same relay; JASIL keeps no allowlist, so take it from your own configuration, never from user input. The `From` header keeps any display name; the envelope sender is the bare address, -and the `Message-ID` uses the sender's domain. If a server refuses *some* recipients the send succeeds and a +and a generated `Message-ID` uses the sender's domain. If a server refuses *some* recipients the send succeeds and a warning logs the refused count, because failing would make a retry re-send to recipients that already received it. Every recipient of a message is visible in `To`, and a refused one is only counted in that warning — send one message per @@ -605,6 +605,34 @@ with no login or cookie (that is what `One-Click` promises), and the relay must DKIM-sign the message covering `List-Unsubscribe`, which RFC 8058 requires. `mailto:` unsubscribe is not supported. +### Choosing the Message-ID + +`send` generates a `Message-ID` on the sender's domain unless the message brings +its own. Set `MailMessage.message_id` to store the key before sending, so a crash +between `send` and recording its result still leaves you a key to look the message +up in the relay's or provider's logs: + +```python +platform.mail.send( + MailMessage(to=["a@example.com"], subject="Hi", text="Hello", message_id="<9f2c41d07a8e@mail.example.com>") +) +``` + +- The format is ``, each side dot-separated printable ASCII without + whitespace, quotes, brackets or parentheses, at most 250 characters (the RFC 5322 + `msg-id` without its obsolete and domain-literal forms). Anything else is a + `ValueError` that does not echo it. `send` returns the value unchanged. +- Make it an opaque value, such as a random token stored with the delivery, and + put a domain you control on the right. The `Message-ID` is visible to the + recipient, every relay and anyone the message is forwarded to: never derive it + from an address or other personal data, and do not use an unsalted hash of one. +- Keeping it unique is the host's job. Reuse it only to retry the same message, + never for different content. +- Some mailbox providers show two copies with the same `Message-ID` as one, but + nothing guarantees it: a retry with the same ID can still be delivered twice. +- Some relays replace the `Message-ID` with their own (Amazon SES does). Check + what yours does before relying on the ID appearing in its logs or in bounces. + ### Runtime-configured mail `build_platform(settings, sources=RuntimeSources(mail=load_mail))` makes diff --git a/docs/security/integration-checklist.md b/docs/security/integration-checklist.md index 6ddb6b2..f720d24 100644 --- a/docs/security/integration-checklist.md +++ b/docs/security/integration-checklist.md @@ -80,6 +80,7 @@ through. - [ ] Put credentials in `MailSettings.username` / `password`, never in the URI. - [ ] Set `from_address` to an identity your relay has verified. - [ ] Take any per-message `MailMessage.from_address` from your own configuration, never from user input. +- [ ] If you supply `MailMessage.message_id`, make it a random token, never a value derived from an address or other personal data. - [ ] Remember the SMTP host is **not** SSRF-checked — it is expected to be an internal relay. Keep that setting out of attacker-influenced configuration. - [ ] Do not put a rendered body or secrets in an event payload that drives a diff --git a/docs/security/threat-model.md b/docs/security/threat-model.md index 565164a..de461a8 100644 --- a/docs/security/threat-model.md +++ b/docs/security/threat-model.md @@ -106,7 +106,10 @@ own timeouts only. **Personal data and secrets in logs and job rows.** The job runner persists `str(error)` and logs `exc_info`, so `MailError` carries only a category and an SMTP status code and is raised `from None`. Log lines record the recipient count, -`Message-ID` and outcome, never addresses, subjects or bodies. The password is +`Message-ID` and outcome, never addresses, subjects or bodies. A host-supplied +`MailMessage.message_id` is logged and travels with the message to every relay and +recipient, so it must be opaque and never derived from personal data; JASIL checks +its format only. The password is excluded from `repr(settings)`, and credentials are refused in the URI so they cannot end up in a logged string. diff --git a/jasil/_core/mail_message.py b/jasil/_core/mail_message.py index 89fbda4..5ee66dd 100644 --- a/jasil/_core/mail_message.py +++ b/jasil/_core/mail_message.py @@ -11,6 +11,7 @@ a durable-job handler is persisted and logged, and an address is personal data. """ +import re from collections.abc import Sequence from datetime import datetime from email import policy @@ -49,6 +50,13 @@ def fold(self, *, policy: Policy) -> str: # not declare ``BODY=8BITMIME``, so a raw 8bit body is not valid for a strict relay. _MIME_POLICY = policy.SMTP.clone(cte_type="7bit", header_factory=_HEADERS) +#: Longest host-supplied ``Message-ID``, angle brackets included. +MAX_MESSAGE_ID_LENGTH = 250 + +# RFC 5322 ``msg-id`` without its obsolete and domain-literal forms: ``<`` dot-atom-text ``@`` dot-atom-text ``>``. +_DOT_ATOM_TEXT = r"[A-Za-z0-9!#$%&'*+\-/=?^_`{|}~]+(?:\.[A-Za-z0-9!#$%&'*+\-/=?^_`{|}~]+)*" +_MESSAGE_ID = re.compile(rf"<{_DOT_ATOM_TEXT}@{_DOT_ATOM_TEXT}>", re.ASCII) + class MailMessageLike(Protocol): """The shape :func:`build_mime` reads; ``jasil.providers.MailMessage`` fits. @@ -77,6 +85,10 @@ def html(self) -> str | None: def reply_to(self) -> str | None: """Single ``Reply-To`` address, if any.""" + @property + def message_id(self) -> str | None: + """Host-supplied ``Message-ID``, if any.""" + @property def list_unsubscribe(self) -> str | None: """One-click unsubscribe URL, if any.""" @@ -164,6 +176,20 @@ def parse_address(value: str, *, role: str) -> Address: raise invalid from None +def _message_id(value: object) -> str: + """Return ``value`` when it is a ```` msg-id that the header carries unchanged. + + Raises: + ValueError: Otherwise; the message does not echo the value. + """ + if not isinstance(value, str) or len(value) > MAX_MESSAGE_ID_LENGTH or not _MESSAGE_ID.fullmatch(value): + raise ValueError( + "The mail message_id must be '', each side dot-separated printable ASCII without " + f"whitespace, quotes, brackets or parentheses, at most {MAX_MESSAGE_ID_LENGTH} characters" + ) + return value + + def _unsubscribe_url(value: object) -> str: """Return ``value`` when it is an ``https`` URL that can be written verbatim on one header line. @@ -198,7 +224,9 @@ def build_mime(message: MailMessageLike, *, sender: str, now: datetime) -> Email now: The ``Date`` header value, from the platform clock. Returns: - A complete message with ``From``, ``To``, ``Date`` and ``Message-ID`` set. + A complete message with ``From``, ``To``, ``Date`` and ``Message-ID`` set; the + ``Message-ID`` is the message's own when it has one, otherwise generated + on the sender's domain. Raises: ValueError: When the message or sender is invalid. @@ -217,6 +245,7 @@ def build_mime(message: MailMessageLike, *, sender: str, now: datetime) -> Email raise ValueError("The mail subject must be a single line") from_address = parse_address(sender, role="sender") reply_to = parse_address(message.reply_to, role="reply-to") if message.reply_to else None + message_id = None if message.message_id is None else _message_id(message.message_id) unsubscribe = None if message.list_unsubscribe is None else _unsubscribe_url(message.list_unsubscribe) mime = EmailMessage(policy=_MIME_POLICY) @@ -226,7 +255,7 @@ def build_mime(message: MailMessageLike, *, sender: str, now: datetime) -> Email mime["Reply-To"] = reply_to mime["Subject"] = message.subject mime["Date"] = format_datetime(now) - mime["Message-ID"] = make_msgid(domain=from_address.domain) + mime["Message-ID"] = make_msgid(domain=from_address.domain) if message_id is None else message_id if unsubscribe is not None: # RFC 8058 one-click: the mailbox provider POSTs ``List-Unsubscribe=One-Click`` to the URL. mime["List-Unsubscribe"] = f"<{unsubscribe}>" diff --git a/jasil/providers.py b/jasil/providers.py index 34572e0..e6dd13e 100644 --- a/jasil/providers.py +++ b/jasil/providers.py @@ -965,8 +965,8 @@ class MailMessage: reply_to: Optional single ``Reply-To`` address. from_address: The sender of this message, ``addr`` or ``"Name "``, in place of the configured ``mail.from_address``: the ``From`` - header, the envelope sender (its bare address) and the - ``Message-ID`` domain. The relay decides which senders it accepts; + header, the envelope sender (its bare address) and the domain of + a generated ``Message-ID``. The relay decides which senders it accepts; never derive this from untrusted input. list_unsubscribe: Optional RFC 8058 one-click unsubscribe URL: an absolute ``https`` URL of printable ASCII, at most 978 characters, @@ -974,6 +974,17 @@ class MailMessage: ``List-Unsubscribe`` and ``List-Unsubscribe-Post``. The URL must accept a POST with no login, and the relay must DKIM-sign the ``List-Unsubscribe`` header. + message_id: Optional ``Message-ID``, so the host can store the key it + will find in the relay's logs before sending. ``""``, + each side dot-separated printable ASCII without whitespace, quotes, + brackets or parentheses, at most 250 characters; ``right`` is + normally a domain the host controls. ``send`` returns it unchanged. + When ``None`` one is generated on the sender's domain. The host keeps + it unique: reuse it only to retry the same message, never for + different content. It is visible to the recipient, every relay and + anyone the message is forwarded to, so make it an opaque value (a + random token), never one derived from personal data. A relay may + replace it with its own. """ to: Sequence[str] @@ -983,12 +994,22 @@ class MailMessage: reply_to: str | None = None from_address: str | None = None list_unsubscribe: str | None = None + message_id: str | None = None def __hash__(self) -> int: # ``to`` is usually a list, which the generated hash cannot digest. recipients = (self.to,) if isinstance(self.to, str) else tuple(self.to) return hash( - (recipients, self.subject, self.text, self.html, self.reply_to, self.from_address, self.list_unsubscribe) + ( + recipients, + self.subject, + self.text, + self.html, + self.reply_to, + self.from_address, + self.list_unsubscribe, + self.message_id, + ) ) diff --git a/tests/test_mail.py b/tests/test_mail.py index 41de6ac..0dc0202 100644 --- a/tests/test_mail.py +++ b/tests/test_mail.py @@ -507,6 +507,71 @@ def test_the_url_is_part_of_the_message_identity(self): assert hash(_message(list_unsubscribe=self.URL)) != hash(_message()) +class TestHostSuppliedMessageId: + """``MailMessage.message_id`` lets the host store the key before sending; it is sent and returned unchanged.""" + + MESSAGE_ID = "<7f3a9c1e.d41d8cd9@mail.example.com>" + + def test_the_supplied_id_is_the_header_and_the_return_value(self, backend): + returned = backend.send(_message(message_id=self.MESSAGE_ID)) + + assert returned == self.MESSAGE_ID + assert _last_built(backend)["Message-ID"] == self.MESSAGE_ID + + def test_an_id_is_generated_when_none_is_supplied(self): + mime = build_mime(_message(), sender=SENDER, now=NOW) + + assert mime["Message-ID"].endswith("@example.com>") + + @pytest.mark.parametrize( + "value", + [ + "", + "<" + "a" * (mail_message.MAX_MESSAGE_ID_LENGTH - 20) + "@mail.example.com>", + ], + ) + def test_every_dot_atom_form_survives_the_wire_unchanged(self, value): + mime = build_mime(_message(message_id=value), sender=SENDER, now=NOW) + + raw = mime.as_bytes() + unfolded = raw.split(b"\r\n\r\n")[0].replace(b"\r\n ", b" ").split(b"\r\n") + assert f"Message-ID: {value}".encode() in unfolded + assert email.message_from_bytes(raw, policy=email.policy.default)["Message-ID"] == value + + @pytest.mark.parametrize( + "value", + [ + "", + "a@example.com", + "", + "", + "<@example.com>", + "", + "", + "<.a@example.com>", + "", + "", + "", + '<"q"@example.com>', + "", + "", + " ", + "\r\nBcc: victim@example.com", + "<" + "a" * mail_message.MAX_MESSAGE_ID_LENGTH + "@example.com>", + 42, + ], + ) + def test_anything_but_a_dot_atom_msg_id_is_refused_without_echoing_it(self, value): + with pytest.raises(ValueError, match="message_id") as caught: + build_mime(_message(message_id=value), sender=SENDER, now=NOW) + + assert not isinstance(value, str) or not value or value not in str(caught.value) + + def test_the_id_is_part_of_the_message_identity(self): + assert hash(_message(message_id=self.MESSAGE_ID)) != hash(_message()) + + class TestMemoryMail: def test_messages_are_recorded_in_order(self): mail = _memory()