Target: Babar — Synology DS224+, DSM 7.3.2-86009 Update 3, Celeron J4125, 18GB RAM, 192.168.50.230 — via Container Manager. Goal: Mirror Gmail + Hotmail to a local Maildir, served over IMAPS by a local Dovecot 2.4 instance (with flatcurve FTS active by default in the image), with a decoupled prune step that deletes old mail from the cloud only after it’s safely archived and backed up. Status: fully built and operational — Gmail pruned to a 1-year cloud window (2026-07-29).

Environment note: Real values for this machine are filled in below (from ~/dev/notebook/knowledge/home-network.md, synology.md, projects/babar/README.md, profile.md). Still verify drift at build time for the two things that actually change over months: Azure app-registration screens and Docker image tags.

Deployment status (2026-07-29):

  • ✅ mail share created (Btrfs checksum + compression on); dockergroup ACL granted; tree owned 1029:65536 (section 1.5)
  • ✅ mail-compose project created in Container Manager; mailtools running as 1029:65536
  • ✅ Gmail app-password loaded; auth verified; initial All Mail pull complete — ~15,884 messages / ~1.4 GB (/config/state/gmail-pull.log)
  • ✅ Restructured to three tiers: secrets/config moved to docker/mail (/config), share holds Maildirs only
  • ✅ Phase C done: Hotmail via Azure device-code OAuth2 (monocularjack@hotmail.com); cyrus-sasl-xoauth2 compiled into the mailtools image; initial pull complete — 2,561 messages across 22 folders. Token auto-refreshes; nightly now runs -a (both accounts).
  • ✅ Phase D done: Dovecot 2.4.4 serving both archives over IMAPS on port 10993 via two namespaces (Gmail = inbox, Hotmail = Hotmail/ prefix); Thunderbird authenticates over TLS and browses both. Runs as the image-default vmail (UID 1000) + group_add: 65536, passwd-file auth (user shawn), drop-in config. flatcurve FTS is active — the official image enables fts_flatcurve by default (index dirs appear inside each mailbox); server-side search works, no extra config needed.
  • ✅ Phase E done: DSM Task Scheduler nightly -a pull (03:00) + Hyper Backup (04:40) + backup gate (05:30), staggered so each feeds the next.
  • ✅ Phase F done: off-site Backblaze B2 Hyper Backup (S3 connector, client-side encrypted, .pem key in 1Password) of the mail share; verified complete. backup_ok.sh gate parses DSM’s backup log and writes the marker prune reads (section 9).
  • ✅ Phase G done (2026-07-29): batched prune.py moved 12,959 Gmail messages older than 1 year to Trash (5 headerless messages safely skipped); local archive (16,005 msgs) untouched. Gmail now holds ~1 year in the cloud; the rest is LAN-only in the archive.

0. Architecture recap

Three genuinely decoupled layers, three separate scheduled jobs:

  1. Pull — mbsync, pull-only and non-destructive to the archive: Sync PullNew PullFlags (new mail + flag changes only — never propagates a remote deletion onto the local copy) with Expunge None. Cloud is never modified by this step.
  2. Backup — Hyper Backup of the mail volume to an off-box target. Writes a success marker that gates step 3.
  3. Prune — a separate script that only ever touches messages already verified present locally and covered by a fresh backup. Moves them to Gmail Trash (30-day grace), never hard-expunges.

Search is layered on top via Dovecot, so the whole archive is just an IMAP mailbox you point Thunderbird at.

Why the pull is PullNew PullFlags, not Pull: in isync, Sync Pull expands to New + ReNew + Delete + Flags. That Delete means once the prune removes a message from Gmail, the next sync would flag the local archived copy \Deleted — silently defeating the archive. Restricting to PullNew PullFlags is what actually decouples the three layers. (See the isync manual.)

1. NAS prep

  1. SSH — already enabled on Babar (port 22, ssh babar). Nothing to do.
  2. Shared folder — Control Panel → Shared Folder → Create mail. Enable data checksumming (Btrfs) — catches silent corruption over years of storage, and makes the folder a clean Hyper Backup unit.
  3. Container Manager — already installed (Babar runs Gitea, the *arr stack, nostr-relay, etc. under it).
  4. Storage — three tiers, matching Babar’s convention (compose-as-code in projects/<svc>-compose/, app config/state in docker/<svc>/, bulk data in a share — exactly like the *arr stack: arrs-compose + docker/sonarr + /volume1/data):
  /volume1/mail/                        # BULK DATA — Btrfs checksum on; the Hyper Backup unit; MAIL ONLY
├── gmail/Maildir/
└── hotmail/Maildir/

/volume1/docker/mail/                 # app CONFIG + SECRETS — NOT in the mail backup; mounted as /config
├── mbsync/mbsyncrc
├── dovecot/{99-archive.conf, users, certs/}   # drop-in conf + passwd-file + self-signed cert (see section 6)
├── scripts/{prune.py, backup_ok.sh, mutt_oauth2.py}
└── state/                            # gmail_app_pw, hotmail_oauth2, last_backup_ok, *.log (chmod 700)

/volume1/docker/projects/mail-compose/   # COMPOSE-as-code only
├── docker-compose.yml
└── mailtools/Dockerfile

/volume1/mail mounts into the containers as /mail, /volume1/docker/mail as /config (see section 4). Keep secrets and regenerables (credentials, OAuth token, FTS index, logs) in /volume1/docker/mail — never in the /volume1/mail share — so the off-box backup (section 9) carries mail only. mbsync’s SyncState * lives inside each Maildir (in the share), so it’s backed up with the mail it describes, which is correct.

  1. Container access to the share — the Synology ACL gotcha. Synology shares are root:root, mode 0000, governed by a Synology ACL, not POSIX. A container running as an arbitrary UID (e.g. Dovecot’s default 1000) is not in that ACL and gets permission-denied even on a 0777-looking folder. The working pattern on Babar (the same one the media stack uses) is to run mail containers as the dedicated dockerlimited identity 1029:65536 (dockergroup) and grant that group on the share:

    • Grant dockergroup on the mail share (UI, keeps DSM’s permission DB in sync): Control Panel → Shared Folder → mail → Edit → Permissions → Local groups → dockergroup → Read/Write. This mirrors the data share’s ACL entry group:dockergroup:allow:rwxpdDaARWc--:fd--.
    • Own the archive tree as 1029:65536: sudo chown -R 1029:65536 /volume1/mail/{gmail,hotmail}. (The docker/mail config tier is a plain folder, not an ACL-governed share, so a normal sudo chown -R 1029:65536 /volume1/docker/mail && sudo chmod 700 /volume1/docker/mail/state is all it needs — no dockergroup grant required there.)
    • Verify from inside the container: docker exec mailtools sh -c 'id; touch /mail/gmail/.w && echo OK && rm /mail/gmail/.w' → expect uid=1029 gid=65536 and OK.

    Both mail containers therefore run as user: "1029:65536" (section 4). Credentials created from inside the running container land owned 1029:65536 automatically — no host-side chown needed per file. Note the ACL gotcha applies only to the mail share; the docker/mail config tier is a plain folder with ordinary POSIX ownership.

2. Gmail credential

  1. Enable 2-Step Verification on shawn.oster@gmail.com if not already on.
  2. Generate an app password (not your account password — Google blocks that over IMAP): myaccount.google.com/apppasswords → name it “NAS mbsync.” Google shows 16 chars in 4 groups.
  3. Load it from inside the running container so it lands owned 1029:65536 at 0600 without a host chown, and never touches shell history:
      docker exec -it mailtools sh -c 'umask 077; cat > /config/state/gmail_app_pw'
    Type the 16 chars with spaces removed → Ctrl-D. (mbsync strips the trailing newline.)

3. Hotmail credential (OAuth2 device-code — as built)

Basic auth / app passwords are dead for Outlook.com IMAP — it must be OAuth2/XOAUTH2. Register a free Azure/Entra app and let mutt_oauth2.py handle the token dance and auto-refresh. We use the device-code flow with a public client (no client secret) — it’s the only flow that works cleanly headless, because the container and your browser are on different machines (the localhostauthcode listener flow can’t work across that gap).

Azure/Entra registration (portal.azure.com → App registrations → New registration):

  • Name: NAS mbsync
  • Supported account types: Accounts in any organizational directory (multitenant) and personal Microsoft accounts — required for a @hotmail.com account.
  • Redirect URI: leave blank (device-code doesn’t use one).
  • After registering, copy the Application (client) ID from Overview.
  • Authentication → Advanced settings → Allow public client flows → Yes → Save. ← this is what enables device-code. Without it, the token request fails with AADSTS7000218 (“request body must contain client_assertion or client_secret”).
  • API permissions: ideally add delegated IMAP.AccessAsUser.All under APIs my organization uses → Office 365 Exchange Online. But on a brand-new personal-account tenant that API often isn’t provisioned and won’t appear — that’s fine: skip it. The v2.0 common endpoint does dynamic consent, so the IMAP scope in the token request is approved on the device-login consent screen anyway. No client secret, no admin consent.

Azure’s portal screens shift periodically — verify against current Microsoft docs at build time rather than trusting these labels verbatim. Registering an app requires a phone/address/payment on the MS account (an Xbox account like monocularjack@hotmail.com usually already has this).

Stage the token helper (pure-stdlib, runs on the container’s python3):

  sudo mkdir -p /volume1/docker/mail/scripts
sudo curl -fsSL -o /volume1/docker/mail/scripts/mutt_oauth2.py \
  https://raw.githubusercontent.com/neomutt/neomutt/main/contrib/oauth2/mutt_oauth2.py
sudo chown 1029:65536 /volume1/docker/mail/scripts/mutt_oauth2.py
sudo chmod 755 /volume1/docker/mail/scripts/mutt_oauth2.py

Authorize once (device-code). Run it inside the container so the token lands owned 1029:65536. The token is stored plaintext via cat passthrough pipes — no GPG in the container, same posture as the plaintext Gmail app-password (/config/state is chmod 700):

  sudo docker exec -it mailtools python3 /config/scripts/mutt_oauth2.py \
  /config/state/hotmail_oauth2 \
  --authorize --provider microsoft --authflow devicecode \
  --client-id <APP_ID> \
  --email monocularjack@hotmail.com \
  --decryption-pipe cat --encryption-pipe cat --verbose

It prompts for a client secret — press Enter to leave it blank (public client). Then it prints a microsoft.com/devicelogin URL + a code: open it on any device, sign in as monocularjack@hotmail.com, enter the code, approve the IMAP + offline-access consent. On success it writes /config/state/hotmail_oauth2 (contains the refresh token) and prints an access token. Then sudo chmod 600 that file. mbsync consumes it via PassCmd (section 5); the script auto-refreshes and writes the rotated refresh token back on every run.

4. Compose project: mail-compose

One compose project defines both containers — an idle mailtools (invoked on a schedule via docker exec) and a long-running dovecot. This matches how every other Babar service is defined (/volume1/docker/projects/<svc>-compose/docker-compose.yml).

mailtools/Dockerfile — holds mbsync + Python + the oauth2 helper, plus the cyrus-sasl-xoauth2 SASL plugin compiled from source:

  FROM debian:trixie-slim

# isync + python for mbsync/oauth. Debian has no cyrus-sasl-xoauth2 package, so
# compile the plugin from source — without it mbsync only offers EXTERNAL and
# cannot do XOAUTH2 against Microsoft (outlook.office365.com). The plugin's build
# installs to /usr/lib/sasl2, but Debian's libsasl2 searches the multiarch dir,
# so we copy it into /usr/lib/x86_64-linux-gnu/sasl2/. Build deps purged in-layer.
RUN set -eux; \
    apt-get update; \
    apt-get install -y --no-install-recommends \
      isync python3 ca-certificates curl \
      libsasl2-2 libsasl2-modules \
      build-essential autoconf automake libtool pkg-config libsasl2-dev git; \
    git clone --depth 1 https://github.com/moriyoshi/cyrus-sasl-xoauth2.git /tmp/xoauth2; \
    cd /tmp/xoauth2; \
    ./autogen.sh; \
    ./configure --prefix=/usr; \
    make; \
    make install; \
    install -d /usr/lib/x86_64-linux-gnu/sasl2; \
    cp -av /usr/lib/sasl2/libxoauth2.so* /usr/lib/x86_64-linux-gnu/sasl2/; \
    ls -l /usr/lib/x86_64-linux-gnu/sasl2/libxoauth2.so*; \
    apt-get purge -y build-essential autoconf automake libtool pkg-config libsasl2-dev git; \
    apt-get autoremove -y; \
    rm -rf /tmp/xoauth2 /var/lib/apt/lists/*

WORKDIR /mail
ENTRYPOINT ["sleep", "infinity"]

⚠ Why this is not optional (cost a full debugging round). isync 1.5.1 links libsasl2 and delegates XOAUTH2 to Cyrus SASL — it does not have a usable built-in XOAUTH2 despite 1.5’s changelog. With no plugin, the Hotmail pull fails immediately: selected: XOAUTH2 / available: EXTERNAL. Two install-path gotchas the Dockerfile above already handles: (1) the plugin’s build ignores --libdir and installs to /usr/lib/sasl2, but Debian’s libsasl2 only searches the multiarch dir /usr/lib/x86_64-linux-gnu/sasl2/ — hence the cp; (2) the final ls ... libxoauth2.so* is a build guard — if the copy didn’t land, set -e fails the build instead of shipping a broken image. Gmail (app-password AuthMechs LOGIN) doesn’t need any of this — only the Microsoft XOAUTH2 path does. Verify after building: docker exec mailtools ls /usr/lib/x86_64-linux-gnu/sasl2/ | grep xoauth2.

docker-compose.yml:

  services:
  mailtools:
    build: ./mailtools
    container_name: mailtools
    user: "1029:65536"               # dockerlimited:dockergroup — matches the share ACL (see section 1.5)
    volumes:
      - /volume1/mail:/mail              # bulk data (Maildirs) — the backup unit
      - /volume1/docker/mail:/config     # config + secrets — NOT backed up with the mail
    restart: unless-stopped          # idle; driven by `docker exec` on a schedule

  dovecot:
    image: dovecot/dovecot:2.4.4     # current CE release; flatcurve FTS is in core (no plugin package)
    container_name: mail-dovecot
    # AS-BUILT: runs as the image-default vmail (UID 1000), NOT 1029. Overriding user to 1029:65536
    # cascaded into Dovecot's privilege-separation state dirs (anvil/login expect the image's own
    # vmail/dovecot/dovenull identities) and would not start. The working pattern is: keep vmail, and
    # add the Synology dockergroup as a supplementary group so vmail can READ the Maildirs mbsync wrote.
    group_add:
      - "65536"                      # dockergroup — grants vmail read access to the mail share (ACL)
    volumes:
      - /volume1/mail:/mail
      # Drop-in override, NOT a full dovecot.conf replacement — the base image's dovecot.conf does
      # `!include_try conf.d/*.conf`, so a single 99-* file layers cleanly on top (section 6).
      - /volume1/docker/mail/dovecot/99-archive.conf:/etc/dovecot/conf.d/99-archive.conf:ro
      - /volume1/docker/mail/dovecot/users:/etc/dovecot/users:ro    # passwd-file (passdb/userdb)
      - /volume1/docker/mail/dovecot/certs:/certs:ro
    networks:
      - synobridge                   # match the rest of Babar's stack (Sonarr, etc.); LAN-reachable
    ports:
      - "10993:10993"                # IMAPS on a NON-privileged port — vmail (non-root) can't bind <1024
    # WHY 10993 + published port on synobridge: vmail is unprivileged, so 993 (privileged) is out; we
    # bind 10993 inside the container and publish it. synobridge is LAN-only (not internet-exposed, never
    # router-forwarded) and cannot ride the DSM HTTP reverse proxy anyway (that's HTTP/WebSocket only).
    restart: unless-stopped
    profiles: ["search"]             # held back until Phase D; `mailtools` starts without it

networks:
  synobridge:
    external: true                   # pre-existing bridge shared by the rest of Babar's stack

Note the mbsync PassCmd for Hotmail references /config/state/... and /config/scripts/... — those resolve to the mounts above.

Deploy via Container Manager (Babar convention): the plain docker CLI needs root on Synology, and every other service here is a UI-managed project. Create the project in Container Manager → Project → Create → Set Path to /volume1/docker/projects/mail-compose → Use existing docker-compose.yml. It builds mailtools and starts it; the dovecot service is held back by its profiles: ["search"] gate until Phase D. All later mbsync/prune invocations use sudo docker exec mailtools …. (TLSType replaces the older SSLType; the newer isync in Debian 13 warns on the latter.)

Rebuilding mailtools after a Dockerfile change (e.g. the SASL plugin): DSM 7.2+ Container Manager bundles Compose v2, so sudo docker compose build mailtools works from the project dir — but build alone does not replace the running container. Follow with sudo docker compose up -d --force-recreate mailtools, or you’ll keep exec-ing into the old image and wonder why the change isn’t there. (Cosmetic caveat: a CLI rebuild leaves Container Manager’s Project page showing the project as out-of-sync; harmless. To stay fully CM-native instead: Project → mail-compose → Stop → Build → Start.)

5. mbsync config

/volume1/docker/mail/mbsync/mbsyncrc (mounted at /config/mbsync/mbsyncrc; point mbsync at it with -c):

  # --- Gmail: archive-safe, All Mail only ---
IMAPAccount gmail
Host imap.gmail.com
User shawn.oster@gmail.com
PassCmd "cat /config/state/gmail_app_pw"
TLSType IMAPS
AuthMechs LOGIN

IMAPStore gmail-remote
Account gmail

MaildirStore gmail-local
Path /mail/gmail/Maildir/
Inbox /mail/gmail/Maildir/INBOX
SubFolders Verbatim

Channel gmail-pull
Far :gmail-remote:
Near :gmail-local:
Patterns "[Gmail]/All Mail"       # All Mail only — avoids per-label duplication (each labelled msg would otherwise land 2-4x)
Create Near
Sync PullNew PullFlags            # never propagate a remote deletion onto the archive
Expunge None
SyncState *

# --- Hotmail: OAuth2 device-code (monocularjack@hotmail.com) ---
IMAPAccount hotmail
Host outlook.office365.com
User monocularjack@hotmail.com
# NOTE: no `-t` — that's --test (runs login tests). Default output is the plain
# access token, which is what mbsync's XOAUTH2 needs. The cat pipes = plaintext token.
PassCmd "python3 /config/scripts/mutt_oauth2.py /config/state/hotmail_oauth2 --decryption-pipe cat --encryption-pipe cat"
AuthMechs XOAUTH2
TLSType IMAPS

IMAPStore hotmail-remote
Account hotmail

MaildirStore hotmail-local
Path /mail/hotmail/Maildir/
Inbox /mail/hotmail/Maildir/INBOX
SubFolders Verbatim

Channel hotmail-pull
Far :hotmail-remote:
Near :hotmail-local:
# Outlook folders are REAL folders (no All-Mail union like Gmail), so pull them all
# EXCEPT the noise. Real personal-account names are short: Deleted / Junk / Notes /
# Outbox / "Sync Issues" (NOT "Deleted Items"/"Junk Email" — confirm against your account).
Patterns * !"Deleted" !"Deleted Items" !"Junk" !"Junk Email" !"Notes" !"Outbox" !"Drafts" !"Sync Issues"
Create Near
Sync PullNew PullFlags
Expunge None
SyncState *

Hotmail requires the SASL plugin (section 4 Dockerfile) — AuthMechs XOAUTH2 fails with available: EXTERNAL without it. Authorize the token (section 3) before the first hotmail-pull. The initial pull throttles like Gmail (Connection reset by peer mid-run); SyncState makes it resumable, so loop it: docker exec mailtools sh -c 'until mbsync -c /config/mbsync/mbsyncrc hotmail-pull; do sleep 15; done'. If you tighten Patterns after a pull already created a noise folder, also rm -rf that folder’s local Maildir — changing the pattern stops future syncs but doesn’t delete what’s already local.

Test one channel at a time before scheduling anything:

  docker exec mailtools mbsync -c /config/mbsync/mbsyncrc gmail-pull   # watch for auth errors

The first Gmail run pulls ~20 years of All Mail and is CPU/IO-heavy on the Celeron J4125 (it “just spins” on big filesystem work — see synology.md). Run it off-hours; incrementals afterward are light.

After the first sync, confirm the on-disk path. SubFolders Verbatim + a folder literally named [Gmail]/All Mail decides where the Maildir actually lands. The prune script’s MAILDIR constant and Dovecot’s config must both point at that same real path — reconcile all three before enabling prune.

6. Dovecot 2.4 (search layer) — as built

Uses the official dovecot/dovecot:2.4.4 image. Rather than replacing the image’s dovecot.conf, we layer a single drop-in (conf.d/99-archive.conf) — the base config ends with !include_try conf.d/*.conf, so this is the least-surprising way to add just what the archive needs.

Key as-built decisions (each cost a debugging round; documented so the next person skips them):

  • Runs as vmail (UID 1000), the image default — do NOT override user: to 1029. Forcing 1029 breaks Dovecot’s privilege-separation state dirs (anvil, login processes expect the image’s own vmail/dovecot/dovenull users). Instead, keep vmail and add group_add: ["65536"] (dockergroup) so vmail can read the Maildirs — see the compose service in section 4.
  • Port 10993, not 993. vmail is unprivileged and can’t bind a port <1024. Bind 10993 inside the container, publish it, point clients there.
  • mailbox_list_layout = fs. mbsync writes with SubFolders Verbatim (filesystem layout, e.g. a literal [Gmail]/All Mail directory). Dovecot must use the matching fs layout or it won’t see the folders.
  • passwd-file auth. A tiny static passdb/userdb off a users file — no system accounts, no LDAP. The users file must be readable by vmail (own it 1000:1000, chmod 640), or auth fails silently.
  • Two namespaces, one login. The single shawn login exposes both archives: an inbox namespace (empty prefix) → /mail/gmail/Maildir, and a hotmail namespace (prefix Hotmail/) → /mail/hotmail/Maildir. In Dovecot 2.4 each namespace sets its own mail_driver = maildir + mail_path (the old 2.3 location = maildir:… is gone) — verified against the 2.4 namespaces doc. The global mail_path becomes an unused default once namespaces are explicit.
  • FTS (flatcurve) is active by default — no config needed. The official dovecot/dovecot:2.4.4 image ships fts_flatcurve enabled, so server-side full-text search works out of the box. Confirmed by fts-flatcurve/ index directories appearing inside each mailbox on disk (e.g. [Gmail]/All Mail/fts-flatcurve/). The index is a fraction of mailbox size; /volume1 has ~1.2 TB free so it’s a non-issue here, but on a tight volume df -h /volume1 is worth a glance.

/volume1/docker/mail/dovecot/99-archive.conf (as deployed):

  protocols = imap

mailbox_list_layout = fs
mail_home = /mail/.dovecot/%{user}

namespace inbox {
  separator = /
  prefix =
  mail_driver = maildir
  mail_path = /mail/gmail/Maildir
  inbox = yes
}

namespace hotmail {
  separator = /
  prefix = Hotmail/
  mail_driver = maildir
  mail_path = /mail/hotmail/Maildir
}

passdb passwd-file {
  passwd_file_path = /etc/dovecot/users
}
userdb static {
  fields {
    home = /mail/.dovecot/%{user}
  }
}

ssl = yes
ssl_server_cert_file = /certs/dovecot.pem
ssl_server_key_file = /certs/dovecot.key

service imap-login {
  inet_listener imap { port = 0 }          # plaintext IMAP off
  inet_listener imaps {
    port = 10993
    ssl = yes
  }
}
service doveadm { inet_listener http { port = 0 } }
service stats  { inet_listener http { port = 0 } }

Editing this drop-in later is risky — a bad config crash-loops Dovecot and takes both archives offline. Always cp 99-archive.conf 99-archive.conf.bak first, sudo docker restart mail-dovecot, then sudo docker logs --tail 30 mail-dovecot — a clean starting up for imap with no Fatal: means good; otherwise restore the .bak and restart. The image is minimal (no grep), so run diagnostics host-side: sudo docker exec mail-dovecot doveconf -a 2>/dev/null | grep -E "mail_path|prefix =" should list both Maildir paths.

/volume1/docker/mail/dovecot/users — one line, user:{SCHEME}hash (generate with doveadm pw -s SHA512-CRYPT):

  shawn:{SHA512-CRYPT}$6$...redacted...

Then chown 1000:1000 + chmod 640 that file (vmail must read it). Self-signed cert at /certs/dovecot.{pem,key} (CN can be anything, e.g. babar-mail) — LAN-only, so accept the trust prompt once in the client; no DSM Let’s Encrypt cert needed.

IMAPS is LAN-only via synobridge + a published 10993:10993 port (section 4) — reachable at Babar’s LAN IP like every other service, never router-forwarded, and it cannot ride the DSM nginx reverse proxy (that’s HTTP/WebSocket only; IMAP is neither).

Client setup (Thunderbird): IMAP server 192.168.50.230, port 10993, SSL/TLS, username shawn (matches the users file), the password you hashed. One gotcha: mbsync doesn’t set IMAP subscriptions, so Thunderbird hides the archive folders by default — uncheck “Show only subscribed folders” (right-click the account → Subscribe, or Server Settings → Advanced) to see [Gmail]/All Mail and the whole Hotmail/ tree (INBOX, Archive, Sent, and your custom folders). After changing Dovecot namespaces, re-open Subscribe / refresh the account for the new tree to appear.

Remote/phone access is a separate VPN project (WireGuard-on-ASUS vs Tailscale — see notebook/projects/aya-gateway); do not expose IMAPS to the internet as a substitute.

7. Prune script (the delayed-delete step)

scripts/prune.py (deployed at /volume1/docker/mail/scripts/prune.py, run as docker exec mailtools python3 /config/scripts/prune.py …) — dry-run by default, hard-gated on a fresh backup marker, batched and resumable. Ran 2026-07-29: moved 12,959 Gmail messages older than 1 year to Trash, 5 headerless messages safely skipped.

  #!/usr/bin/env python3
"""Prune Gmail [Gmail]/All Mail messages present in the local Maildir and covered by a
verified backup. Dry-run unless --commit. Moves matched messages to [Gmail]/Trash (Gmail
treats COPY->Trash as a MOVE: 30-day grace, then auto-purged). NEVER touches a message
whose Message-ID isn't in the local archive. Batched + resumable (per-batch expunge)."""
import argparse, email, imaplib, os, re, sys, time

MAILDIR   = "/mail/gmail/Maildir/[Gmail]/All Mail"   # VERIFIED on-disk path (cur/ + new/)
BACKUP_OK = "/config/state/last_backup_ok"           # epoch ts written by backup_ok.sh
PW_FILE   = "/config/state/gmail_app_pw"
USER      = "shawn.oster@gmail.com"
BACKUP_MAX_AGE = 7 * 86400
HEADER_BYTES   = 32768       # read only the header region to find Message-ID (truncation -> safe skip)
BATCH_SIZE     = 500
UID_RE = re.compile(rb"UID (\d+)")

def local_message_id(path):
    with open(path, "rb") as fh:
        head = fh.read(HEADER_BYTES)
    return email.message_from_bytes(head).get("Message-ID")

def load_local_message_ids():
    ids = set()
    for sub in ("cur", "new"):
        d = os.path.join(MAILDIR, sub)
        if not os.path.isdir(d):
            continue
        for fn in os.listdir(d):
            mid = local_message_id(os.path.join(d, fn))
            if mid:
                ids.add(mid.strip())
    return ids

def backup_gate():
    if not os.path.exists(BACKUP_OK):
        sys.exit("ABORT: no verified backup marker.")
    ts = float(open(BACKUP_OK).read().strip())
    if time.time() - ts > BACKUP_MAX_AGE:
        sys.exit(f"ABORT: backup marker stale ({(time.time()-ts)/86400:.1f}d).")

def connect(folder, readonly):
    M = imaplib.IMAP4_SSL("imap.gmail.com")
    M.login(USER, open(PW_FILE).read().strip())
    M.select(f'"{folder}"', readonly=readonly)
    return M

def fetch_mids(M, uid_batch):
    typ, data = M.uid("FETCH", ",".join(uid_batch),
                      "(UID BODY.PEEK[HEADER.FIELDS (MESSAGE-ID)])")
    out = {}
    for item in data:
        if not isinstance(item, tuple) or len(item) < 2:
            continue
        m = UID_RE.search(item[0])
        if not m:
            continue
        mid = email.message_from_bytes(item[1]).get("Message-ID")
        out[m.group(1).decode()] = mid.strip() if mid else None
    return out

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--commit", action="store_true")
    ap.add_argument("--folder", default="[Gmail]/All Mail")
    ap.add_argument("--min-age-days", type=int, default=365)
    ap.add_argument("--limit", type=int, default=0)
    ap.add_argument("--count-only", action="store_true")
    args = ap.parse_args()

    def cutoff_str():
        return time.strftime("%d-%b-%Y", time.gmtime(time.time() - args.min_age_days*86400))

    if args.count_only:
        M = connect(args.folder, readonly=True)
        typ, data = M.uid("SEARCH", None, f"(BEFORE {cutoff_str()})")
        print(f"candidates older than {args.min_age_days}d ({cutoff_str()}): {len(data[0].split())}")
        M.logout(); return

    backup_gate()
    local = load_local_message_ids()
    print(f"local archived Message-IDs: {len(local)}", flush=True)

    M = connect(args.folder, readonly=not args.commit)
    typ, data = M.uid("SEARCH", None, f"(BEFORE {cutoff_str()})")
    uids = [u.decode() for u in data[0].split()]
    if args.limit:
        uids = uids[:args.limit]
    print(f"candidates older than {args.min_age_days}d ({cutoff_str()}): {len(uids)}"
          + (f" (limited to {args.limit})" if args.limit else ""), flush=True)

    moved = skipped = 0
    nbatches = (len(uids) + BATCH_SIZE - 1) // BATCH_SIZE
    for bi in range(nbatches):
        batch = uids[bi*BATCH_SIZE:(bi+1)*BATCH_SIZE]
        attempt = 0
        while True:
            try:
                mids = fetch_mids(M, batch)
                to_move = [u for u in batch if mids.get(u) and mids[u] in local]
                if args.commit and to_move:
                    M.uid("COPY", ",".join(to_move), '"[Gmail]/Trash"')  # Gmail: move to Trash
                    M.expunge()
                break
            except (imaplib.IMAP4.abort, imaplib.IMAP4.error, OSError) as e:
                attempt += 1
                if attempt > 3:
                    print(f"\nABORT at batch {bi+1}/{nbatches}: {e}\n"
                          f"Progress saved (moved={moved}). Re-run to resume.", file=sys.stderr)
                    sys.exit(1)
                print(f"  [batch {bi+1} error: {e}; reconnecting {attempt}/3]", flush=True)
                time.sleep(5 * attempt)
                try: M.logout()
                except Exception: pass
                M = connect(args.folder, readonly=not args.commit)
        skipped += len(batch) - len(to_move)
        moved += len(to_move)
        print(f"  batch {bi+1}/{nbatches}: {'moved' if args.commit else 'would move'} "
              f"{len(to_move)} (total {moved}, skipped {skipped})", flush=True)
        time.sleep(1)
    print(f"{'MOVED' if args.commit else 'WOULD MOVE'}: {moved}  skipped(not-local): {skipped}", flush=True)
    try: M.logout()
    except Exception: pass

if __name__ == "__main__":
    main()

Safety properties: hard-aborts without a fresh backup marker; matches by Message-ID against the local Maildir; never touches a message absent locally; dry-run opens the mailbox read-only (readonly=not args.commit); Trash-move (not hard delete) keeps Gmail’s 30-day grace; per-batch expunge makes a throttled run resumable (re-run does a fresh SEARCH and continues). Reads only the first 32 KB of each local file for the Message-ID — a truncated header can only cause a safe skip, never a wrongful delete.

Staged rollout that was actually used (do it this way):

  1. --limit 200 dry-run → confirm skipped(not-local): 0 (matching logic sound).
  2. --count-only --min-age-days {365,730,1095,1825,3650} → survey how many messages each cutoff prunes, to pick the age window. (Here: mail is heavily old — ~13k >1yr, ~10k >10yr — so the cutoff mostly decides how recent stays in Gmail for phone/web, not how much is deleted. Chose 1 year.)
  3. Full dry-run (no --limit) → real total + skipped(not-local) count (got 12,964 would-move, 5 skipped).
  4. --commit --limit 5 live micro-test → verify Gmail’s COPY→Trash actually moves (All Mail candidate count must drop by exactly 5). It did — confirming COPY alone is the correct delete op; no STORE \Deleted / All-Mail EXPUNGE needed (Gmail treats Trash as a move, so those would act on stale UIDs).
  5. --commit full run.

The 5 skipped are Gmail messages with no Message-ID header — unmatchable on both sides, so they’re left in the cloud forever. Harmless. All-Mail-only also means Drafts/Spam/Trash were never archived — don’t later assume drafts were captured. Post-prune the local archive is unaffected — Sync PullNew PullFlags (section 5, correction C2) never deletes local copies when they vanish from Gmail, which is the entire point.

8. Wire up DSM Task Scheduler

Control Panel → Task Scheduler → Create → Scheduled Task → User-defined script. Owner: root (the task runs docker exec, which needs the docker socket; the work still runs as 1029:65536 inside the container — no sudo inside a scheduler task, it’s already root).

The as-built staggered pipeline (each step feeds the next):

TaskScheduleCommand
Mail pull (mail - nightly pull)Daily 03:00-a pull, see below
Hyper Backup (mail-offsite-b2)Daily 04:40Backblaze B2 (section 9)
Backup gate (mail-backup-gate)Daily 05:30bash /volume1/docker/mail/scripts/backup_ok.sh >> …/backup_ok.log 2>&1 (owner root)
Prune (optional)Manual, or weekly for a rolling windowdocker exec mailtools python3 /config/scripts/prune.py --commit

The prune was run once manually (2026-07-29) to establish the 1-year window. To keep a rolling 1-year window (mail auto-pruned as it crosses 365 days), schedule it weekly — but leave it disabled until every box in section 10 is checked, and only after the backup gate is proven.

The pull task, with timestamped append-logging so nightly runs leave a trail. Now that Hotmail is live it runs -a (all channels — Gmail + Hotmail):

  docker exec mailtools sh -c 'mbsync -c /config/mbsync/mbsyncrc -a >> /config/state/gmail-pull.log 2>&1; echo "[$(date)] nightly EXIT $?" >> /config/state/gmail-pull.log'

History: scope this to gmail-pull (not -a) until Hotmail’s token is authorized — otherwise the Hotmail channel fails-auth every night and spams the log. Switch to -a once section 3 is done (it is). Note: while an initial multi-day pull is still running, nightly runs hit mbsync’s channel lock and exit 1 (a safe no-op, not a failure) — leave “email on abnormal termination” off until initial pulls complete. This gives automatic cap-resume: any daily-throttle stop is picked back up by the next nightly run via SyncState. The nightly -a also refreshes the Hotmail OAuth token non-interactively (the mutt_oauth2.py PassCmd rotates and re-saves the refresh token each run), so the token stays alive as long as the box syncs at least every ~90 days.

Leave the prune task disabled until every box in section 10 is checked.

9. Backup (hard prerequisite for prune) — as built

Backblaze B2, off-site. Hyper Backup of the mail share to B2 via the S3-Compatible Storage connector (this DSM has no native Backblaze option). Gotchas hit:

  • Endpoint s3.us-west-004.backblazeb2.com, region us-west-004, Signature v4.
  • The B2 application key must NOT be bucket-restricted — Hyper Backup lists all buckets to populate its dropdown, and a bucket-scoped key can’t ListBuckets → “Insufficient privileges.” Create an all-buckets Read/Write key.
  • Client-side encryption ON — the .pem key is stored in 1Password. Without it the backup is unrecoverable (the password alone won’t do a full restore).
  • Data source: mail share only (not docker/mail — secrets stay out of the backup). Smart Recycle rotation, daily 04:40, weekly integrity check.

RAID is not a backup. B2 is off-site (survives box loss/theft/fire) — the right posture since the whole point is to delete the cloud copy. A SanDisk Extreme USB local copy is planned as a second (3-2-1) layer; not required for the gate.

The gate — scripts/backup_ok.sh (runs as root on the host via Task Scheduler, 05:30). DSM 7.3 has no synobackup.log; backup success is in /var/log/systemd/synobackupd.service.log as launch job [{…"action":1…"task_id":1…}] followed by job [N] exit(0). The script finds the most recent backup job for task_id=1 (the only Hyper Backup task), confirms exit(0) within 26 h, and writes the epoch to /volume1/docker/mail/state/last_backup_ok (chown 1029:65536, chmod 640) so the container’s prune.py can read it. A stale/failed backup leaves the marker untouched → prune.py aborts on its own 7-day check. The scheduled task emails only on abnormal termination, so a failed backup that closes the gate pings you automatically.

  #!/bin/bash
set -uo pipefail
LOG="/var/log/systemd/synobackupd.service.log"
MARKER="/volume1/docker/mail/state/last_backup_ok"
TASK_ID=1; MAX_AGE_HOURS=26; OWNER="1029:65536"
fail() { echo "backup_ok: GATE CLOSED — $*" >&2; exit 1; }
[ -r "$LOG" ] || fail "cannot read $LOG"
jobid=$(grep -aE '"action":1[,}].*"task_id":'"$TASK_ID"'[,}]' "$LOG" \
        | grep -a 'launch job \[{' | sed -E 's/.*"job_id":([0-9]+).*/\1/' | tail -n1)
[ -n "$jobid" ] || fail "no backup launch for task_id=$TASK_ID"
exitline=$(grep -aE "job \[$jobid\] exit\(0\)" "$LOG" | tail -n1)
[ -n "$exitline" ] || fail "job $jobid has no exit(0)"
ts=$(echo "$exitline" | awk '{print $1}')
epoch=$(date -d "$ts" +%s 2>/dev/null) \
  || epoch=$(python3 -c 'import sys,datetime;print(int(datetime.datetime.fromisoformat(sys.argv[1]).timestamp()))' "$ts" 2>/dev/null) \
  || fail "could not parse timestamp: $ts"
age_h=$(( ($(date +%s) - epoch) / 3600 ))
[ "$age_h" -le "$MAX_AGE_HOURS" ] || fail "latest success is ${age_h}h old (> ${MAX_AGE_HOURS}h)"
umask 077; echo "$epoch" > "$MARKER"; chown "$OWNER" "$MARKER"; chmod 640 "$MARKER"
echo "backup_ok: GATE OPEN — task $TASK_ID job $jobid exit(0) at $ts (${age_h}h old); marker=$epoch"

task_id=1 is hardcoded = the only Hyper Backup task on this box. If you add another Hyper Backup task, re-check which task_id is the mail one (the log JSON only carries the id, not the name).

10. Validation checklist (all met before the 2026-07-29 prune)

  • mbsync -a completes cleanly across nightly runs
  • Local count matches server (16,005 local Gmail msgs, not a multiple — All-Mail-only fix confirmed)
  • Local copies keep no \Deleted when mail leaves Gmail (PullNew PullFlags fix — proven by the prune itself: 12,959 left Gmail, all 16,005 still local)
  • Dovecot serves both archives; old mail opens in Thunderbird at 192.168.50.230:10993 (flatcurve FTS active)
  • Full Hyper Backup cycle completed (B2) and last_backup_ok fresh
  • prune.py dry-run reported skipped(not-local): 0 on the 200-sample; full dry-run 12,964 would-move / 5 skipped; --commit --limit 5 confirmed Gmail COPY→Trash is a move
  • Spot-checked old threads open from the NAS

Outcome: committed prune moved 12,959 messages older than 1 year to Gmail Trash (30-day grace), 5 headerless safely skipped. Gmail now holds ~1 year in the cloud; the full ~16k archive is preserved locally, searchable, and backed up off-site.

Remaining / optional follow-ups: SanDisk local backup (2nd copy, 3-2-1); a B2 test-restore to prove the off-site copy + .pem key actually restore (do within the 30-day Trash window); optionally schedule the prune weekly for a rolling 1-year window; Hotmail has no prune logic yet (Gmail-only); remote access is the separate VPN project.