CDC Pipeline Monitoring

Overview

The CDC (Change Data Capture) pipeline streams row-level changes from the T24/Transact source database into the operational data store in near real-time, via Debezium and Kafka. This guide covers the monitoring dashboard built into MorphTool and MorphApi: how to read pipeline health, how to interpret the error table, and how to trigger remediation without leaving the UI.

This guide covers:

  • What the dashboard shows and how to read it
  • The error table and Dead Letter Queue
  • Remediation actions: restart, replay, resnapshot
  • Troubleshooting common states

Accessing the Dashboard

Open MorphTool at https://todsdev.fintegrator.eu and navigate to MONITORING → CDC Pipeline. The page auto-refreshes every 5 seconds; use Auto-Refresh: ON/OFF to pause it, or Refresh Now for an immediate update.

What the Dashboard Shows

Summary tiles

Tile Meaning
Connectors Healthy Connectors currently RUNNING with all tasks RUNNING, out of the total configured
Total Queue Depth (Lag) Sum of unconsumed messages across all replicated topics — how far the pipeline is behind the source
Throughput (msgs/sec) Messages being processed per second, sampled between dashboard refreshes. Shows “warming up…” on the first load after a restart, since there’s no prior sample to diff against
Unresolved Errors Count of records currently in the error table that have not been replayed

Connector / Task Status

One row per connector task, showing its state (RUNNING, FAILED, PAUSED) and, for a failed task, its failure trace. A Restart button is available per task.

Queue Depth & Throughput

Per-topic breakdown of how many messages are waiting to be processed and how fast they’re draining, sorted with the most backed-up topics first.

Errors (Dead Letter Queue)

A record that fails to process (for example, a data type mismatch or constraint violation) is diverted here instead of stopping the whole pipeline. Each row shows the source topic, the processing stage where it failed, and the exception detail. A record that’s already been replayed this session is marked accordingly.

Remediation Actions

All actions ask for confirmation before running.

Restart Task

Use when a connector task shows FAILED. Restarts that specific task; it resumes from where it left off.

Replay

Use on a row in the error table to re-submit that exact record for processing. If the underlying cause was transient (e.g. a dependent record that now exists), it should process normally. If the record is structurally invalid, it will fail again and appear as a new error — check the exception detail before replaying the same class of error repeatedly.

Resnapshot

Triggers a fresh incremental read of a table from the source database — use when a table’s data looks incomplete or out of sync, rather than for a single failed record. Available as a standalone action (enter a table name, e.g. public.FBNK_ACCOUNT) or pre-filled from an error row’s source table.

Troubleshooting

A connector task is FAILED

Check its failure trace in the Connector / Task Status grid, then use Restart Task. If it fails again immediately, the underlying cause needs investigation before restarting further — contact your Fintegrator TODS administrator.

The error count keeps climbing for the same table

Repeated errors on one table usually indicate a systemic issue (a schema mismatch, or a data quality problem at the source) rather than isolated bad records. Replaying will not resolve this — the table likely needs a data or mapping fix first, then a Resnapshot.

Throughput shows “warming up…” and never updates

This is expected immediately after a deployment or restart — the dashboard needs one full refresh cycle to compute a rate. If it persists past a minute, check that Queue Depth is also updating; if neither is moving, the pipeline itself may be stalled rather than the dashboard.