data-maskingMask, redact, and anonymize sensitive data (PII, PCI, PHI) in databases, logs, and APIs. Use when protecting PII in dev/staging environments, redacting sensitive data from logs, anonymizing data for analytics, or applying k-anonymity and differential privacy for GDPR-compliant data sharing.
Independent GitHub snapshot · Automated classification · Original documentation governs
Data masking replaces real sensitive data with realistic but fake data, preserving format and structure. Essential for:
| Technique | How | When to Use |
|---|---|---|
| Static masking | Replace data at rest permanently | Dev DB copy |
| Dynamic masking | Mask on-read, original preserved | Role-based views |
| Tokenization | Replace with token that maps to real value | Payment cards |
| Format-preserving | Keep format, change values (e.g., real-looking SSN) | Testing |
| Redaction | Replace with placeholder ([REDACTED]) | Logs |
| Generalization | Replace specific value with range (age 34 → 30-40) | Analytics |
import re
PII_PATTERNS = {
"email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b',
"phone_us": r'\b(?:\+1[-.]?)?\(?[0-9]{3}\)?[-.\s]?[0-9]{3}[-.\s]?[0-9]{4}\b',
"ssn": r'\b(?!000|666|9\d{2})\d{3}-(?!00)\d{2}-(?!0000)\d{4}\b',
"credit_card": r'\b(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|3[47][0-9]{13}|6(?:011|5[0-9]{2})[0-9]{12})\b',
"ip_address": r'\b(?:(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\.){3}(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\b',
"date_of_birth": r'\b(?:0[1-9]|1[0-2])[\/\-](?:0[1-9]|[12]\d|3[01])[\/\-](?:19|20)\d{2}\b',
"passport": r'\b[A-Z]{1,2}[0-9]{6,9}\b',
"zip_code": r'\b\d{5}(?:-\d{4})?\b',
}
These regexes are a fast first pass and over-match: passport and zip_code hit ordinary numbers and product codes, phone_us hits any 10-digit run. Use them for logs; for free text or unknown columns use Presidio below. Validate card candidates with a Luhn check before redacting.
import re
from faker import Faker
fake = Faker()
Faker.seed(4471) # same seed -> same fake values on every run (stable fixtures)
def mask_email(email: str) -> str:
"""Mask email preserving domain structure."""
local, domain = email.split('@')
masked_local = local[0] + '*' * (len(local) - 2) + local[-1] if len(local) > 2 else '***'
return f"{masked_local}@{domain}"
def mask_credit_card(card_number: str) -> str:
"""Mask credit card — show only last 4 digits."""
cleaned = re.sub(r'[\s-]', '', card_number)
return '*' * (len(cleaned) - 4) + cleaned[-4:]
def mask_ssn(ssn: str) -> str:
"""Mask SSN — show only last 4."""
cleaned = ssn.replace('-', '').replace(' ', '')
return f"***-**-{cleaned[-4:]}"
def mask_phone(phone: str) -> str:
"""Mask phone — show only last 4 digits."""
digits = re.sub(r'\D', '', phone)
return f"***-***-{digits[-4:]}"
// Node.js log scrubbing (winston)
const PII_PATTERNS = {
email: /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g,
creditCard: /\b(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|3[47][0-9]{13})\b/g,
ssn: /\b(?!000|666|9\d{2})\d{3}-(?!00)\d{2}-(?!0000)\d{4}\b/g,
phone: /\b(?:\+1[-.]?)?\(?[0-9]{3}\)?[-.\s]?[0-9]{3}[-.\s]?[0-9]{4}\b/g,
password: /"password"\s*:\s*"[^"]*"/g,
token: /"(?:token|api_key|secret|authorization)"\s*:\s*"[^"]*"/gi,
};
function sanitizeLog(data) {
let sanitized = typeof data === 'string' ? data : JSON.stringify(data);
sanitized = sanitized.replace(PII_PATTERNS.email, '[EMAIL]');
sanitized = sanitized.replace(PII_PATTERNS.creditCard, '[CREDIT_CARD]');
sanitized = sanitized.replace(PII_PATTERNS.ssn, '[SSN]');
sanitized = sanitized.replace(PII_PATTERNS.phone, '[PHONE]');
sanitized = sanitized.replace(PII_PATTERNS.password, '"password":"[REDACTED]"');
sanitized = sanitized.replace(PII_PATTERNS.token, (match) => {
const key = match.split(':')[0];
return `${key}:"[REDACTED]"`;
});
return sanitized;
}
// Wrap Winston logger to auto-sanitize
const winston = require('winston');
const logger = winston.createLogger({
transports: [new winston.transports.Console()],
format: winston.format.combine(
winston.format.printf(({ level, message, ...meta }) => {
return JSON.stringify({
level,
message: sanitizeLog(message),
...JSON.parse(sanitizeLog(JSON.stringify(meta)))
});
})
)
});
-- Create masked view for dev access
CREATE OR REPLACE VIEW users_masked AS
SELECT
id,
-- Mask name: keep first letter + ***
LEFT(first_name, 1) || '***' AS first_name,
LEFT(last_name, 1) || '***' AS last_name,
-- Mask email: preserve domain
REGEXP_REPLACE(email, '^([^@])([^@]*)(@.+)#x27;, '\1***\3') AS email,
-- Mask phone: show only last 4
'***-***-' || RIGHT(phone, 4) AS phone,
-- Mask SSN: show only last 4
'***-**-' || RIGHT(ssn, 4) AS ssn,
-- Keep non-sensitive fields as-is
created_at,
status,
country
FROM users;
-- Grant dev team access to masked view only (not base table)
GRANT SELECT ON users_masked TO dev_team;
REVOKE SELECT ON users FROM dev_team;
-- Dynamic, role-aware masking: use a SECURITY DEFINER function or the
-- PostgreSQL Anonymizer extension (anon.start_dynamic_masking) instead of ad-hoc views.
# pip install presidio-analyzer presidio-anonymizer && python -m spacy download en_core_web_lg
# (Python 3.10+; the default NLP engine needs the spaCy model)
import os
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
def mask_text_presidio(text: str, masking_style: str = "replace") -> str:
"""Auto-detect and mask PII using Presidio NLP."""
results = analyzer.analyze(text=text, language="en")
if masking_style == "replace":
# Replace with type label: [EMAIL_ADDRESS]
operators = {
"DEFAULT": OperatorConfig("replace", {"new_value": "[REDACTED]"}),
"EMAIL_ADDRESS": OperatorConfig("replace", {"new_value": "[EMAIL]"}),
"PHONE_NUMBER": OperatorConfig("replace", {"new_value": "[PHONE]"}),
"PERSON": OperatorConfig("replace", {"new_value": "[NAME]"}),
"US_SSN": OperatorConfig("replace", {"new_value": "[SSN]"}),
}
elif masking_style == "hash":
# Since 2.2.361 hashes use a random salt; pass a fixed salt (from a secret)
# if the same input must map to the same output across rows
operators = {"DEFAULT": OperatorConfig("hash", {"hash_type": "sha256", "salt": os.environ["MASKING_SALT"]})}
anonymized = anonymizer.anonymize(
text=text,
analyzer_results=results,
operators=operators
)
return anonymized.text
# Example
text = "Contact Dana Whitfield at dana.whitfield@northwind-mail.org or 555-123-4567"
print(mask_text_presidio(text))
# → "Contact [NAME] at [EMAIL] or [PHONE]" (exact labels depend on detected entities)
#!/bin/bash
# mask-db-for-dev.sh — Safe production → dev data pipeline
set -e
PROD_DB="${PROD_DATABASE_URL:?set PROD_DATABASE_URL}" # read-only replica credentials
DEV_DB="${DEV_DATABASE_URL:?set DEV_DATABASE_URL}"
echo "Dumping production schema..."
pg_dump --schema-only "$PROD_DB" > schema.sql
echo "Applying schema to dev..."
psql "$DEV_DB" < schema.sql
echo "Copying and masking data..."
psql "$PROD_DB" -c "\COPY (
SELECT
id,
LEFT(first_name, 1) || 'XXXX' AS first_name,
'User' AS last_name,
'user_' || id || '@masked.invalid' AS email,
'555-000-' || LPAD((ROW_NUMBER() OVER())::TEXT, 4, '0') AS phone,
created_at,
status
FROM users
) TO STDOUT WITH CSV" | psql "$DEV_DB" -c "\COPY users FROM STDIN WITH CSV"
echo "Done. Dev database ready with masked data."
Anonymization vs Pseudonymization (GDPR Article 4):
Key techniques for true anonymization:
diffprivlib (Python) or Google DP libraries.k-anonymity alone is often insufficient for GDPR -- combine with l-diversity and/or differential privacy.
User request: "Refresh staging from production but nobody on the dev team may see real emails or phone numbers."
Run the pipeline script above with PROD_DATABASE_URL and DEV_DATABASE_URL set, then verify:
psql "$DEV_DATABASE_URL" -tAc "SELECT count(*) FROM users WHERE email NOT LIKE '%@masked.invalid'"
Expected output: 0. Any other number means unmasked rows leaked and the refresh must be discarded.
User request: "Our request logs contain customer emails and card numbers; clean them before they reach Datadog."
Wrap the logger with sanitizeLog from the middleware section. A line such as {"msg":"refund for dana.whitfield@northwind-mail.org card 4111111111111111"} is emitted as {"msg":"refund for [EMAIL] card [CREDIT_CARD]"}. Add a unit test that feeds sample PII through the logger and asserts none survives.