Skip to content
Other

Email Ingestion - Engineering Architecture

Engineer-facing reference for how email actually flows into memories. The end-user guides (Email Ingestion, Background Email Polling) describe the experience; this doc describes the machinery - the two pipelines, the safety layers, and the failure modes. Ground truth as of 2026-07-26; cites gateway/src/... unless noted.

Security reviewers: use

docs/audits/email-lethal-trifecta-capability-map.md before adding an email

ingest, draft/reply, briefing, or delivery lane. It records which lanes hold

private-data access, untrusted-content exposure, and external-send capability,

plus the required structural review checklist.

The single most important fact: there are two pipelines

A feature flag decides which one runs. Read this before debugging anything.

EMAIL_CONNECTOR_ENABLED !== 'true'   ->  legacy poller runs   (email-poll.ts, cron */3)
EMAIL_CONNECTOR_ENABLED === 'true'   ->  connector pipeline runs (email-connector.ts, cron */1 discovery)

There is also a manual MCP path independent of both pollers: the ingest tool with action:"emails" (or legacy ingest_emails) calls handleIngestEmails, with a cap of 5 emails per call. This is the "Claude reads your inbox via an email MCP during a session" path.

So content enters through three doors: legacy poll, connector poll, or MCP-mediated manual read. The automatic pollers attribute messages before extraction. The manual path is already routed to a caller-selected mind, verifies write access to that mind, and then joins the extraction and storage stages.

End-to-end flow

(door 1) legacy three-minute cron --> fetch --> skip --> attribute --> legacy dedup --.
                                                                                                      |
                 (door 2) connector discovery cron --> fetch --> skip --> attribute --> queue dedup ----+--> guardrails
                                                                                                      |         |
                 (door 3) MCP manual read --> caller-selected mind --> raw-document dedup --------------'         v
                                                                                                      extract + store

Stage-by-stage

1. Scheduling and concurrency safety

The legacy poller claims a mailbox with a compare-and-swap on last_poll_at and a 30-second staleness window. The connector pipeline scans due connectors in bounded batches, then relies on durable queue uniqueness and downstream idempotency to make overlapping discovery safe. Do not assume the two schedulers use the same claim mechanism.

2. Fetch (incremental)

Gmail only. getProvider() (email-provider.ts:38) has a single case - gmail -> GmailProvider - and default: throw 'Unsupported email provider'. Outlook/IMAP are not implemented (not even stubs) on the poller path. Fetch is incremental off the Gmail historyId cursor, capped at 10 messages per invocation.

3. Skip filters (rule-based, no ML)

email-prefilter.ts shouldSkipExtraction() runs, in order: noreply-sender checks -> word-count floor (MIN_WORD_COUNT=15) -> auto-reply body checks -> newsletter detection (needs 2+ signals, such as unsubscribe markers / List-Unsubscribe / Precedence: bulk). The filters are deliberately conservative to avoid dropping real correspondence. email-poll.ts adds skips for calendar invites and Armbrain's own domain.

4. Dedup (multi-layer, durable)

The paths use related but not identical durable guards:

A past bug documented in mcp-tools-ingest.ts wrote a string message ID into a UUID column, so dedup writes silently failed and every poll re-ingested. Fixed; the comment is a landmine marker - don't reintroduce it.

5. Guardrails (detect + contain, never block)

Lives in guardrails.ts (note: not ingest-security.ts - see the naming trap below).

Naming trap: ingest-security.ts (564 lines) is not general ingest security - it's SSRF protection + zip-bomb pre-checks (IPv4-literal detection in all encodings, private/reserved/metadata-IP blocking, DNS-rebinding defense via DoH, per-hop redirect re-validation, 10 MB streaming cap). Robust, but scoped to URL/website-connector fetches, not inbound email content. The email-egress-*.ts files are a separate concern again: they govern outbound email the product sends (send gate, killswitch, authz, outbox) - not ingestion.

6. Attribution (the strongest subsystem)

email-attribution.ts (341 lines). Match priority:

  1. Exact email_contacts match, case-insensitive.
  2. company_info.domain (singular).
  3. company_info.domains[] (array).

Two hardening layers, each added after a real misfire:

Multi-mind fan-out returns human-readable "decision receipt" reasons per match. On no match, the connector path can escalate to a bounded LLM attribution pass; still nothing -> mind_id: null review queue (not dropped).

7. Extraction gate (a flag, not a human gate)

Despite the name, shouldExtractEmail() is a feature-flag resolver: per-customer extraction_enabled wins, else global EMAIL_EXTRACTION_ENABLED. Flag off -> one whole-body classify-only memory (legacy behavior). Flag on -> extractAndStoreConversational, with an empty-extraction fallback so trivial emails aren't silently dropped.

8. Store + advance

Memories are written client-isolated (mind ownership validated before every write - an email attributed to Client A cannot land in Client B). After durable work is recorded, the historyId cursor advances. Both automatic paths update connectors.last_poll_summary: the legacy poller writes {ingested, skipped, unattributed, errors, poisoned}, while connector discovery writes its generic found/queued/skipped/failed counters plus any provider-specific fields.

Failure modes (the best-engineered part)

Observability

Structured JSON log() calls cover the main branches. Per-poll summaries are stored in connectors.last_poll_summary. A cron heartbeat email_poll:last_run is written outside the flag gate, so it proves the trigger fired even when the legacy body is skipped. No StatsD/Prometheus metrics backend - logs + DB summary only. "Ingestion receipts" are a conversational, non-persisted concept. There is no published latency SLA (event -> searchable memory).

Token security

Refresh tokens are envelope-encrypted with an environment key-encryption-key and decrypted in memory per poll. The current v2 format also binds ciphertext to its storage context and customer ID, but the production diagnostic fallback still permits legacy v1 writes without that binding while issue #2124 is resolved. Gmail access uses a read-only scope. No provider credentials are stored in plaintext.

Known hazards for maintainers

  1. The two-pipeline flag split duplicates attribution/dedup logic across email-poll.ts and email-connector.ts - cross-path behavioral drift is possible. Changes to shared behavior must be checked on both paths.
  2. ingest-security.ts is misnamed (SSRF+zipbomb, not content security). The real content-security logic is in guardrails.ts + the email-egress-* files.
  3. email-prefs.ts is an additive layer over legacy KV flags - master opt-out correctness depends on every send path consulting the unified record.

Test coverage map

The main regression coverage lives in email-poll-integration.test.ts, email-connector.test.ts, email-attribution.test.ts, email-extraction-flag.test.ts, mcp-tools-ingest-consolidated.test.ts, connector-email-quarantine.test.ts, and the focused cursor/backfill suites.