Skip to navigation
TransformProcessors

Dedupe Processor

Description

The Dedupe processor removes duplicate values from log data.

Use

This processor is most useful for reducing “chatter” in logs. The overlap of data across fields is the key to having this processor work effectively. This processor will emit the first matching record of the set of records that are being compared.

Configuration

There are three options to configure for this processor.

OptionDescriptionExample
Number of EventsThe number of events to compare across. Limited to 5000.5000
Comparison TypeMatch will remove duplicate records based on the specified fields. \n\n\n\nIgnore will remove duplicate records based on all fields except those specified.Match
FieldsThe field or fields to apply the Dedupe processing to..foo

Example - Match

Before

{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, 'bat': true}
{"foo": "qux", "baz": 3}
{"foo": "qux", "baz": 4}
{"foo": "qux", "baz": 5}

Dedupe Options

OptionValue
Number of Events5
Comparison TypeMatch
Fields.foo

After

{"foo": "bar", "baz": 1}
{"foo": "qux", "baz": 3}

Example - Ignore

Before

JSON
{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, "bat": true}
{"foo": "qux", "baz": 3}
{"foo": "qux", "baz": 4}
{"foo": "corge", "baz": 5}

Dedupe Options

OptionValue
Number of Events5
Comparison TypeIgnore
Fields.baz

After

{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, "bat": true}
{"foo": "qux", "baz": 3}
{"foo": "corge", "baz": 5}