An organization processes customer data from web and mobile applications. Data includes names, emails, phone numbers, and location history. Data arrives both as batch files (from SFTP daily) and streaming JSON events (from Kafka in real-time).
To comply with data privacy policies, the following requirements must be met:
Personally Identifiable Information (PII) such as email, phone number, and IP address must be masked or anonymized before storage.
Both batch and streaming pipelines must apply consistent PII handling.
Masking logic must be auditable and reproducible.
The masked data must remain usable for downstream analytics.
How should the data engineer design a compliant data pipeline on Databricks that supports both batch and streaming modes, applies data masking to PII, and maintains traceability for audits?
Submit