> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://mezmo.ferndocs.com/dedupe-processor/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://mezmo.ferndocs.com/_mcp/server.

# Dedupe Processor

## [Description](https://docs.mezmo.com/docs/dedupe-pipeline-processor#description)

The Dedupe processor removes duplicate values from log data.

## [Use](https://docs.mezmo.com/docs/dedupe-processor#use)

This processor is most useful for reducing “chatter” in logs. The overlap of data across fields is the key to having this processor work effectively. This processor will emit the first matching record of the set of records that are being compared.

## [Configuration](https://docs.mezmo.com/docs/dedupe-processor#configuration)

There are three options to configure for this processor.

| **Option**           | **Description**                                                                                                                                                     | **Example** |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| **Number of Events** | The number of events to compare across. Limited to 5000.                                                                                                            | `5000`      |
| **Comparison Type**  | **Match** will remove duplicate records based on the specified fields. \n\n\n\n**Ignore** will remove duplicate records based on all fields except those specified. | `Match`     |
| **Fields**           | The field or fields to apply the Dedupe processing to.                                                                                                              | `.foo`      |

## [Example - Match](https://docs.mezmo.com/docs/dedupe-pipeline-processor#example---match)

### Before

```json
{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, 'bat': true}
{"foo": "qux", "baz": 3}
{"foo": "qux", "baz": 4}
{"foo": "qux", "baz": 5}
```

### Dedupe Options

| Option           | Value   |
| ---------------- | ------- |
| Number of Events | `5`     |
| Comparison Type  | `Match` |
| Fields           | `.foo`  |

### After

```json
{"foo": "bar", "baz": 1}
{"foo": "qux", "baz": 3}
```

## Example - Ignore

### Before

```json
JSON
{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, "bat": true}
{"foo": "qux", "baz": 3}
{"foo": "qux", "baz": 4}
{"foo": "corge", "baz": 5}
```

### Dedupe Options

| Option           | Value    |
| ---------------- | -------- |
| Number of Events | `5`      |
| Comparison Type  | `Ignore` |
| Fields           | `.baz`   |

### After

```json
{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, "bat": true}
{"foo": "qux", "baz": 3}
{"foo": "corge", "baz": 5}
```