Logs

callback-processor

2 min read

Description

Listens to raw callback events, parse them and convert into valuable events that can be read by other services.

Data flow

Input

It is a Kafka service that provides a topic for raw callback events. The topic name is message_callbacks_raw. The group is worker

Output

Behavioral notes

Pod Scaling

The number of pods should be equal to kafka message_callbacks_raw topic partitions. Therefore required to scale pod and partitions simultaneously

JMX_PORT=5557 && kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic message_callbacks_raw
JMX_PORT=5557 && kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic message_callbacks_raw --partitions 40

To check kafka lag:

JMX_PORT=5557 && kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group deliveries | grep message_callbacks_raw | awk '{sum += $6} END {print sum}'

Post-deployment

After each deployment, the lag of the message_callbacks_raw topic should be checked. The lag should be reduced to zero after some time due to the caching process happening with source parameters buffer to keep the most relevant items in memory (is used for subscriber-sources-activities events)

We can observe how caching process helps to reduce the amount of time service needs to process the batch via logs:

Successfully processed 1000 events (1.7094056606292725s)           1000 new events has been received                                     Refreshing source params buffer for 805 subscribers

The amount of seconds in brackets should decrease after each batch, as well as the amount of subscribers that need to be refreshed (cache miss criteria)

Significant changes

After a 2024-02-05 deployment, the callback processor has been updated to support subscriber source activities events. Such update required memory and CPU scaling, as well as a new dependency in the form of a directory sources server.

  • dependencies:

Graph View

Backlinks