What we measure, and why
What counts for HIPAA compliance is the eighteen Safe Harbor categories. So that is what we measure — category by category, on each file type.
Other tools report a single figure, often around 97%, usually an F1 score: one number averaged across everything the tool found, blended from two different measures. It cannot tell you which categories were covered and which were not, and a value that should never have been masked can count toward it.
Our testing was built against the Safe Harbor requirement itself. Every identifier is planted with its category and its position known in advance, so a result is a comparison against an answer key rather than an estimate.
We also measure the opposite error. Every file contains values that look like identifiers and are not — drug names, room numbers, lot codes. Masking them does not fail a HIPAA requirement, but it costs you the use of the file. We plant those deliberately and report them separately.
How we tested
The left path is what has to be masked and the right is what has to survive. They are planted into the same files and go through the same run, and they are reported separately because they fail in opposite directions — a miss costs privacy, an over-mask costs the use of your data.
- Patient master.
- A demographic extract, one row per patient, using the column names FHIR, USCDI, OpenEMR and OpenMRS actually ship.
- Case notes.
- Agent-written call notes — claim denials, prior authorizations, refills, discharge follow-up — with identifiers in the prose and in the columns beside it.
- Clinical notes.
- Clinical prose in its own register: abbreviations, vitals, dosages, diagnosis codes, and names as often lowercase as not.
The data is synthetic. No public corpus of real case notes exists, because that data is protected health information by definition.
Coverage by category
Over-masking ledger
Over-masking is when a value that identifies nobody — a drug name, a room number, a lot code — is masked anyway: no HIPAA requirement is failed, and the data loses its use.
5,061 of 5,211 look-alike values were not masked. The 150 that were all fall in the transaction references column of the patient master.
The transaction references are the only class we mask, and why is in the notes below.
Device identifiers and serial numbers — not detectedUnder GS1, the standard governing device labeling, the application identifiers for batch/lot and for serial number specify the same data format. A serial number and a lot code are not distinguishable by format, and a lot code is not an identifier of a patient.
Neighborhood and district names — 58.0% in clinical notesHyde Park and Capitol Hill are ordinary words, and a gazetteer of place names collides with surnames and business names. Of the 169 planted, 54 were missed and 17 were masked in part.
Dates in case notes — 84.7%Dates are masked when clinical context is present: an appointment, an admission, a date of service. Most dates in a case note are not about the patient at all — callback times, filing deadlines, system timestamps.
Medical record numbers in clinical prose — 87.4%A bare record number with nothing beside it naming what it is has no shape distinguishing it from any other reference number. With a nearby label, the same number is masked.
Transaction references in the patient master — 150 of 300 maskedThese are the only look-alike values we mask. The reference format is close enough to a telephone number to be read as one, and the column is confirmed on that basis, so every value in it is masked. It costs the use of that column and nothing else.
Results reflect the datasets used for testing and may vary on your own data with format, context, and how identifiers appear.