clickhouse sent statistics halt

Problem

Sent statistics has stopped to update. No new data was inserted and message events are not being processed. Everything else seemed to be fine.

Detection

To detect the issue i’ve checked Kafka offsets for statistics group:

JMX_PORT=5557 && kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group statistics

It was clear that message_events topic was not consumed by Clickhouse.

Solution

Restarting Clickhouse did not solve the problem but it helped to identify the root cause since statistics started to update after the restart. Resetting the Kafka offset for message_events topic did the trick:

JMX_PORT=5557 && kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group statistics --reset-offsets --to-latest --topic message_events --execute