Skip to navigation
Build PipelinesAI Guides

Deduplication Processor

Deduplication Processor

Purpose: Eliminates duplicate events in your pipeline, reducing data sent to the final destination.

Key Features:

  • Duplicate Detection: Compare events based on specified fields (e.g., ignore, match logic).
  • Customization: Define how many events to compare and which fields to deduplicate.

Steps to Add:

  1. Edit the pipeline and click Add Processor.
  2. Select Deduplication Processor and configure:
  • Title/Description for clarity.
  • Comparison logic (e.g., number of events to compare).
  • Fields used for deduplication.