Skip to content

HLS PHI/PII De-Identifier (Safe Harbor) — Structured Data

Skill details

Version 0.5 · Author Zachary Phillips · License Databricks License

Browse on GitHub · Raw SKILL.md · Evaluation report

Overview

Operationalizes the HIPAA Safe Harbor method (45 CFR §164.514(b)(2)) over structured Unity Catalog tables: it detects the 18 Safe Harbor identifier classes at the column and value level, applies a defensible per-class de-identification strategy, enforces k-anonymity on the surviving quasi-identifiers, and applies the result as a dynamic Unity Catalog view over the raw table so raw PHI never moves. The output is a governed view plus an analyst-facing readout of what was removed, tokenized, generalized, suppressed, and still analyzable — with the mandatory Safe Harbor disclaimer. It operationalizes Safe Harbor; it does not certify de-identification — that is the covered entity's Privacy Officer's determination.

Runs on the execution model: Genie Code calls scripts/run_deid.py and reports its output — it does not hand-write de-identification SQL. This matters because hand-written de-id reliably forgets that derived columns are quasi-identifiers (e.g. emitting length_of_stay_days but excluding it from the k-check → a view reporting "k=5" while half its rows are uniquely re-identifiable). The entrypoint treats every quasi-identifier as in-scope and verifies actual k.

When to Use

  • "De-identify the hls.raw.patients table for the analytics team."
  • "Apply Safe Harbor to our claims table and show me what got masked."
  • "Prep encounters so a data scientist can use it without seeing PHI."
  • "Anonymize this member table before we share it with a vendor."
  • "Is this table safe to export? Make it re-identification-resistant, not just name-stripped."
  • "Mask PHI but keep length-of-stay so we can still do longitudinal analysis."
  • "De-identify these clinical PDFs / scanned documents before we share them" — unstructured documents are handled by the document entrypoints (preview_document_deid_options / apply_document_deid in scripts/deid_docs.py): ai_parse_document → layered redaction → a governed redacted derivative + a separately-governed, reversible surrogate→raw crosswalk → deterministic residual verdict. Layers: (1) member/relative name — the patient name is extracted from all patient-name fields (incl. a misspelled surname in a second field) and redacted everywhere, shared-surname relatives ("Robert James Whitfield") and relationship-cue names ("husband Robert") too; provider names are kept by design. (2) model (ai_mask, per line so recall holds on long docs) — addresses, geography, written dates, format-less IDs; no 'person name' label, so providers survive. (3) regex — SSN (incl. masked/partial), phone/fax, email, MRN, DEA (incl. dash), labeled IDs (incl. dictation/EMS/accession, digit-gated so prose is safe), URL, IP, dates. Report the deterministic residual verdict verbatim (fixed-format / member / employer, each a fixed query) — do NOT improvise a separate LLM re-scan; the model-dependent categories it lists (sub-state geography, narrative org/school names, relatives without a cue or shared surname) always require Privacy-Officer review.

Prerequisites

  • FIRST, before running anything: select serverless (Python) compute. Every entrypoint here (structured run_deid.py AND document deid_docs.py) is Python that calls the Databricks SDK — it will NOT run on a SQL-only warehouse. In Genie Code, attach serverless (Python) compute (or a Python-enabled cluster) as the first step; do not start on a SQL warehouse and switch mid-run. The SDK then runs the generated SQL against a warehouse on your behalf (auto, or pass warehouse_id=). Auth is ambient in-workspace, or a CLI profile= locally.
  • Inputs: a fully-qualified structured table (catalog.schema.table) with typed columns.
  • Consumer role: the group/role that should read the de-identified view (governance is role-driven).
  • Environment (optional): a Databricks secret scope for the tokenization salt (never a literal); a separately-governed schema if a re-identification key map is required.
  • Scope: structured/tabular tables (the run_deid path) AND unstructured clinical documents (PDF/image, the deid_docs path via ai_parse_document). For DICOM imaging, integrate the Pixels accelerator (not built into this skill).

Quick Start

Two ways to run it. Prefer the surface-and-choose flow when the analyst cares about the data-retention/utility vs anonymization tradeoff (it lets them pick k); use the one-call run_deid when the default (k=5) is fine.

import sys
sys.path.append('<this-skill-dir>/scripts')   # the scripts/ folder next to this SKILL.md

# --- Surface-and-choose (recommended): show the privacy/utility tradeoff, user picks k ---
from run_deid import preview_deid_options, apply_deid
print(preview_deid_options("<catalog.schema.table>"))   # READ-ONLY: per-column plan + k frontier
# ...user picks k after seeing utility cost...
print(apply_deid("<catalog.schema.table>", k_target=5)) # apply at the chosen k

# --- One-call default (k=5) ---
from run_deid import run_deid
print(run_deid("<catalog.schema.table>"))

preview_deid_options is READ-ONLY (the generalization search is count-only) — it shows the per-column plan (what is redacted vs tokenized vs date→year vs generalized vs kept vs suppressed) and a k frontier: for each candidate k, the k achieved, rows suppressed, and how many quasi-identifiers stay full vs coarsened vs value-suppressed. apply_deid (or run_deid) then runs the whole pipeline (profile → detect → k-anonymity generalization → view-over-raw → verify → readout). Report the tool's output; do not paraphrase or recompute it.

Workflow

⛔ EXECUTE, DO NOT REIMPLEMENT. Call run_deid and report its output; never write your own de-identification SQL. The steps below document what the entrypoint does internally — for review, not a recipe to re-code by hand.

Step 1: Profile the target

scripts/profile_table.py captures per column: name, type, sample distinct values, null rate, cardinality. This drives detection and is the first section of the audit report.

Step 2: Detect & classify PHI

scripts/detect_phi.py assigns each column to a Safe Harbor class (or "not PHI"), in order of trust: Unity Catalog PII tags → value validators (SSN/NPI-Luhn/card-Luhn/email/phone/IP/URL/date) → name heuristics → ai_classify as a tie-breaker only. Detection is recall-dominated (when unsure, flag as PHI). Surface the classification table to the user before transforming anything.

column, detected_class, method, confidence, proposed_strategy   # confirm with the user

Step 3: Choose per-class strategy

Defaults (overridable) from references/deid_strategies.md: direct identifiers → redact or tokenize (salted HMAC) if re-linkage is needed; dates → reduced to year with intervals preserved via derived fields; ages → pass through if ≤ 89 else "90+"; ZIP → first 3 digits, zeroed for the restricted prefixes (references/restricted_zip3.md).

Step 4: Enforce k-anonymity (the step baseline Genie skips)

scripts/kanon.py generalizes quasi-identifiers to a k-anonymity target (default k=5) with a utility-weighted greedy search — sacrificing low-value QIs (geo) before high-value clinical ones (age, LOS). Direct-identifier stripping is table stakes; without this step stripped data is still re-identifiable (measured baseline: k=1, ~87% of rows unique). Reports the privacy/utility trade explicitly.

Optional — redaction + tokenization only (skip k-anonymity): the surfaced apply_kanon choice (or apply_deid(..., apply_kanon=False)) builds the governed view with direct identifiers dropped/tokenized and dates reduced to year, but keeps the quasi-identifiers at full granularity (no generalization or suppression). Use the vetted path for this — do NOT hand-write the view. Because nothing is generalized, k is unmeasurable, so the verdict is REVIEW REQUIRED (re-identification resistance unverified), never a certified de-identification.

Step 5: Apply through Unity Catalog governance (no second PHI copy)

scripts/apply_uc_governance.py builds a dynamic view over the raw table that drops direct identifiers, converts dates to interval-preserving derived fields, applies the step-4 generalization, and filters suppressed sub-k rows. Consumer access is gated with is_account_group_member; the raw source must not be readable by the consumer role. Any key map lands in a separately-governed, locked schema.

Step 6: Verify (residual-risk scan) & emit the readout

The entrypoint measures actual k over ALL surviving quasi-identifiers (including derived ones) and runs a residual leak scan — any PHI pattern or near-unique free-text column is a hard FAIL regardless of k. It emits the readout, persists it to a UC audit table, and appends the mandatory Safe Harbor disclaimer naming the Privacy Officer as the certifying authority.

Key Parameters

preview_deid_options(raw_fqn, k_candidates=(2,5,10,20), profile=None, warehouse_id=None) — read-only apply_deid(raw_fqn, k_target, view_name=None, profile=None, warehouse_id=None) — applies the chosen k run_deid(raw_fqn, view_name=None, k_target=5, profile=None, warehouse_id=None) — one-call default

Parameter Default Range / Options Effect
raw_fqn — (required) catalog.schema.table The raw table to de-identify. Read-only source; never overwritten.
k_candidates (2,5,10,20) tuple of ints (preview) The k values whose privacy/utility cost is surfaced for the user to choose from.
k_target 5 (run_deid); required for apply_deid integer ≥ 2 k-anonymity target — the privacy/utility dial. Higher k = stronger re-id resistance, more generalization/suppression = less utility. apply_deid requires it (the user's choice); never pick it for a utility-sensitive table without surfacing the tradeoff.
view_name <table>_deid any valid view name Name of the governed view created over the raw table.
profile None (ambient auth) a Databricks CLI profile Local runs only; in-workspace uses ambient auth.
warehouse_id None (first available) a SQL warehouse id Pin the warehouse if the default pick is wrong.

Internals (engine-set, not arguments): generalization ladders are data-driven per column (numeric → quartile buckets → halves → suppress; categorical → keep → merge-rare-into-'other' → suppress); row suppression is capped (max_suppression_frac, default 0.10); passthrough is deny-by-default (only numeric measures pass; unknown non-numeric columns are suppressed, never leaked).

Expected Outputs

  • A governed dynamic view over the raw table (no second PHI copy) readable by the consumer role.
  • An analyst readout: removed / tokenized / generalized / suppressed / still-analyzable, with the k achieved and rows suppressed.
  • A UC audit-table record of the verified readout, plus the mandatory Safe Harbor disclaimer.
  • A residual-leak-scan result (hard FAIL on any surviving PHI pattern).

Troubleshooting

Problem Cause Solution
View over-generalizes to a huge k (utility destroyed) A raw date/high-cardinality column routed through the generic-QI path Confirm dates are year-reduced / interval-derived, not treated as raw QIs.
Readout says PASS but you suspect re-identifiability A quasi-identifier (esp. a derived one like LOS) was excluded from the k-check Cannot happen via run_deid (it checks all surviving QIs). If you see it, someone hand-wrote SQL — re-run through the entrypoint.
A free-text / name column survived unredacted Detection is recall-limited and missed it The residual leak scan should hard-FAIL it; add the column to detection and re-run. Never pass through unknown non-numeric columns.
k_target unreachable without dropping many rows Population too small for the chosen k Lower k_target, or accept the (capped) suppression the engine reports — don't silently ship sub-k rows.
Consumer can still read raw PHI The raw source isn't restricted for the consumer role Governance is a view; also ensure the consumer role cannot read the raw table (back-door exposure).
"HIPAA compliant" appears in output — Never claim it. Say "processed against Safe Harbor, pending Privacy Officer review."

Guardrails

  1. Execute, do not reimplement — call the entrypoints (preview_deid_options / apply_deid / run_deid); never hand-write de-id SQL (the LOS k-check failure cannot happen via the entrypoint).
  2. Surface the privacy/utility tradeoff; let the user choose k — for a utility-sensitive table, run preview_deid_options and let the analyst pick the k, rather than silently defaulting. (The one-call run_deid k=5 default is fine when the tradeoff isn't in question.)
  3. Never certify — the output is "processed against Safe Harbor, pending Privacy Officer review," never "de-identified / HIPAA compliant."
  4. Never transform an unconfirmed column — surface the classification table (step 2) first.
  5. Never write a re-identification key map into an analyst-accessible schema.
  6. Never echo raw PHI into chat beyond the minimum to confirm a classification, and mask it.
  7. When detection confidence is low across many columns, stop and ask — false negatives leak PHI.

Bundled Resources

  • scripts/run_deid.py — the entrypoints: preview_deid_options (read-only privacy/utility frontier), apply_deid (apply at the chosen k), and run_deid (one-call default). Pipeline: profile → detect → k-anon generalization → view-over-raw → residual leak scan → readout + UC audit log.
  • scripts/deid_docs.py — document/image de-id entrypoints (preview_document_deid_options, apply_document_deid): parse (ai_parse_document) → redact all PHI classes → governed redacted derivative + reversible locked crosswalk → residual-leak scan. Shares its PHI patterns with the SQL redactor.
  • scripts/deid_interactions.py — the surface-and-choose decision sheet (per-column strategy, date handling, utility priority, output delivery) + the notebook artifact.
  • scripts/kanon.py — utility-weighted k-anonymity generalization engine (generic, any-table).
  • scripts/apply_uc_governance.py — builds the schema-driven dynamic view over the raw table.
  • scripts/detect_phi.py — PHI classifier (UC tags → validators → name heuristics → ai_classify).
  • scripts/profile_table.py — column profiler.
  • scripts/deid_report.py — analyst-facing readout builder.
  • references/safe_harbor_classes.md — the 18 classes + default treatment.
  • references/deid_strategies.md — strategy definitions and when to use each.
  • references/restricted_zip3.md — ZIP3 prefixes that must be zeroed.
  • tests/scorers.py, tests/objective.py, tests/test_deid_scorers.py, tests/test_deid_multimodal.py — the objective function, component scorers (detection recall/F2, residual-leak gate, utility retention, k-anonymity, governance), and their pure-Python unit tests.

Disclaimer

This skill provides guidance to AI to do PHI detection and de-identification but all compliance obligations (e.g. HIPAA, GDPR, CCPA) remain with the user. We urge that you validate the results to ensure accuracy and completeness using appropriate benchmark(s). Databricks makes no guarantees that use of this skill is sufficient for regulatory compliance.

References