Architecture · Data pipeline

Built to grow without rewriting.

Each worker is an independent Python container with its own Kafka consumer group. If one fails, the others keep running. Scale per engine according to real load.

Pipeline · 4 stages

The journey of a DMARC report.

From the perimeter SMTP to the final dashboard, every step is decoupled via Kafka. With Dead Letter Queue so no message is silently dropped.

01 · Ingest

SMTP Receiver → email.raw

SMTP receiver on port 2525 + AWS SES/S3 support. Every raw message published to the Kafka topic email.raw with an Avro schema.

02 · Normalization

normalizer-worker

SAX parsers validated against Google, Microsoft and Yahoo. Attachments to MinIO/S3. Structured output to dmarc.rua.raw.

03 · Enrichment

GeoIP · ASN · PTR

Each source IP is enriched with country, ASN, reverse DNS and reputation score. Published to dmarc.rua.enriched.

04 · Processing

Parallel workers

dmarc-worker · spf-worker · dkim-worker · mta-sts-worker · alert-engine · clickhouse-writer. DLQ for fallbacks. No bottlenecks.

Tech stack

Proven technology, no passing fads.

FastAPI, Kafka, ClickHouse and PostgreSQL — the stack that powers thousands of critical platforms in production. Every choice is documented and justified.

LayerTechnology
Backend API Python 3.12 + FastAPI + asyncpg — async end to end
Dashboard React 18 + TypeScript + Vite + Tailwind CSS — API-first SPA
Streaming Apache Kafka (3 brokers) + Confluent Schema Registry (Avro BACKWARD_TRANSITIVE)
Analytics ClickHouse 24.x — append-only, partitioned by tenant + month
Relational DB PostgreSQL 16 — RLS on every table
Cache Redis 7 — rate limiting + MFA challenge tokens
Object Storage MinIO (dev) / AWS S3 (prod) — same binary
Observability Prometheus + Grafana + CloudWatch
Auth security JWT HS256 (15 min TTL) + mandatory MFA + API Keys
Workers 15+ independent Python microservices with DLQ
Reverse Proxy Traefik v3.3 with TLS termination
Production infra AWS EC2 · EBS 150GB encrypted (KMS) · Daily S3 backups
Multi-tenant security

Real isolation, not just logical.

Row-Level Security in PostgreSQL: every query carries SET LOCAL app.tenant_id. Isolation happens at the database, not the application. Immutable audit log via Kafka → PostgreSQL append-only + S3 Object Lock (WORM).

3 tiers

Platform → MSSP → Client. Native hierarchical model. Multi-customer without overhead.

PostgreSQL RLS

Database-level isolation. No query can read cross-tenant.

WORM audit logs

S3 Object Lock + PostgreSQL append-only. Exportable forensic traceability.

Mandatory MFA

TOTP is not optional. Tokens with TTL in Redis.

CI Security

Semgrep SAST + Trivy SCA on every PR. Aikido custom rules.

Rate limiting

Sliding window in Redis per tenant. SPF: 1000 lookups/day per tenant.

Ready to secure your domains?

Start receiving DMARC reports in minutes. Get full visibility into who is sending emails using your brand.