Real-Time PII Filtering: A Reference Architecture with Regex, NER, and Managed Services

No single PII detection method is perfect on its own — the pattern that actually works in production combines three distinct techniques, each covering the blind spots of the other two.

The architecture pattern that keeps showing up

Production pipelines typically combine pattern-based rules (regular expressions for emails or credit card numbers), NER models (named entity recognition), and classifier-based detectors to flag or redact sensitive spans — no single technique alone covers the full range of PII that can show up in free-form text.

Microsoft Presidio: the open-source default

Microsoft Presidio is the open-source default: Python-native, integrates cleanly into a logging interceptor, and ships with regex-based recognizers plus a spaCy-based or custom transformer NER layer. It's the choice when you want full control and no dependency on an external managed service.

Each cloud's managed equivalent

Amazon Comprehend provides synchronous, real-time API operations to analyze personally identifiable information in a document, with support for up to 100 kilobytes of UTF-8-encoded text per request. Azure Language's text PII detection reached general availability on May 1, 2026, with quality improvements across common entity types, and now lets customers customize detection with their own regular expressions — preview entities include passwords, PINs, postal codes, and airport codes.

Layered defense, not a single checkpoint

A layered PII defense pipeline on AWS places Amazon Macie to inventory sensitive data in S3, while Amazon Comprehend detects and redacts PII in text before it gets indexed for retrieval or migrated between data stores — two distinct services covering two distinct moments in the data's lifecycle (at rest vs. in transit toward a retrieval system).

When to use which option

AWS Comprehend is the natural choice for teams already on AWS — managed NER endpoints, custom entity support, and per-request pricing that scales with traffic without managing your own infrastructure. Presidio makes sense when you need full control over the pipeline (for example, to meet data residency requirements that prevent sending text to an external service) or when your traffic volume makes a managed service's per-request cost add up significantly.