Skip to navigation
TransformProcessors

Redact Processor

Description

With this Processor you can identify and redact Personally Identifiable Information (PII) in your data stream. This includes items like:

  • Social Security numbers
  • Email addresses
  • Credit cards
  • Phone numbers

This Processor identifies PII based on pattern detection. When the specified pattern is detected, you have the option to replace it with a specified string, or use a standard hash. When using this Processor in a Mezmo Edge Pipelines for Local Data, you can also specify a regular expression to use for pattern detection.

Configuration

OptionDescriptionExample/Options
TitleA title for the processorRedact SSN
DescriptionA description of the Processor’s function in the PipelineReplaces SSNs with a text string
Field (Optional)The field to search for the pattern. If the field is not specified, the Processor will look for the pattern across all the fields.Message
Mask Pattern
PatternCredit card number\n\n\n\nEmail address\n\n\n\nIPv4 address\n\n\n\nUS or Canada phone number\n\n\n\nUS social security number\n\n\n\nCustom Regex Pattern (Mezmo Edge Pipelines for Local Dataonly)
ActionHash\n\n\n\nReplace\n\n\n\nDetect Only
Hash Options\n\nThe value is hashed and base64 encodedmd5\n\n\n\nsha1\n\n\n\nsha2 (sha2-512_256)\n\n\nsha3 (sha3-512)
Replacement<Any text string>
PII PresenceWhen pre-set or custom PII patterns are detected in an event, a metadata field will be added to the event that can be used by down stream processors for collecting metrics or trigger an alert that this pattern is present. Metadata field will indicate one or more PII patterns detected in the event as shown below:\n\n\n”pii_presence”: {\n\n\n\n”email_address”: “yes”,\n\n\n\n “us_social_security_number”: “yes”\n\n\n\n }\n\n\n\nIf no PII patterns are detected, pii_presence field will not be added to the metadata.On or Off

Pre-defined Patterns

Social Security Number

The US social security number pattern matches any valid social security number sequence, with or without optional delimiters.

Examples:

  • XXX YY ZZZZ
  • XXX-YY-ZZZZ
  • XXX YY-ZZZZ
  • XXX-YY ZZZZ
  • XXXYYZZZZ

Phone Number

The phone number pattern matches valid phone numbers, with or without space, hypen or period (.) delimiters.

Valid phone numbers include:

  1. +1 XXX YYY ZZZZ
  2. +1 (XXX) YYY ZZZZ
  3. +1.(XXX).YYY.ZZZZ
  4. +1XXXYYYZZZZ
  5. +1-XXX YYY.ZZZZ, etc. .

The matcher will also match parts of sequences (false positives). Partially matched phone numbers includes:

  1. XXX.YYY.ZZZZ will be matched in the input +1_XXX.YYY.ZZZZ
  2. XXX.YYY.ZZZZ will be matched in the input +1.MXXX.YYY.ZZZZ

Credit Card Number

The credit card pattern currently matches Visa, Mastercard, American Express, Diners club and JCB card numbers.

Visa

13 to 16 digit numbers starting with 4. Examples: 4XXXXXXXXXXXX or 4XXXXXXXXXXXXXXX

Mastercard

16 digit numbers starting with 21-27 (new range) or 51-57 (old range).

Examples: 21XXXXXXXXXXXXXX or 56XXXXXXXXXXXXXX

American Express (AMEX)

15 digit numbers starting with 34 or 37. Examples: 34XXXXXXXXXXXXX or 37XXXXXXXXXXXXX

Diners Club

14 digit numbers starting with 300-305 or 360-389. Examples: 300XXXXXXXXXXX or 389XXXXXXXXXXX

JCB

15 digit numbers starting with 2131 or 1800 or 16 digit numbers starting with 35. Examples: 2131XXXXXXXXXXX.

Email Address

Matches RFC 5322 compliant email addresses.

Examples: user@dummy.com, user@127.0.0.1, etc

Interactive Demo

Check out an interactive demo of the Redact Processor as a component in a Compliance group, as well as instructions for building a version of the Pipette with your own sample data.

Examples

Redact Social Security Number with String

OptionValue
PatternUS social security number
ActionReplace
Replacement000-00-000

Redact Social Security Number to Hash

OptionValue
PatternUS social security number
ActionHash
Hashmd5

Custom Regex Example: Redact Canadian Social Security Number

OptionValue
PatternCustom
ActionReplace
Expression(?<ssn_canada>(\d{3})-(\d{3})-(\d{3}))
Replacement111-111-111

Redact Email Address to String

OptionValue
PatternEmail address
ActionReplace
Replacementsomeone@nowwhere.com