Example Kafka Consumer for Logging and Tracing

The blueworx-utils package contains an example Apache Kafka consumer that can read and store data from all of the Blueworx logging and tracing topics. To install the blueworx-utils package see the instructions here: Installing Blueworx Utilities

Source code for a sample consumer is also provided to give an example of how to connect to, and consume data from, the Blueworx topics. The source code is hosted on the Blueworx public Git repository:

https://github.com/blueworx/bvr-samples/tree/master/bvr-kafka-consumers

Using the example Apache Kafka Consumer

The example consumer can subscribe to all three logging and trace topics and output the resulting data to the local filesystem. The data from each topic is handled differently by the example consumer:

The blueworx.application entries are split by callIndex and written out to separate files under a callLogs subdirectory. There is no management of application log files in the example consumer and so while the consumer is running log files will be written for every call into the system. It is up to the user to manage those log files.
The blueworx.error entries are written out to error_X.out appending and wrapping according to the configuration options detailed in the table below.
The blueworx.trace entries are written out to trace_X.out appending and wrapping according to the configuration options detailed in the table below.

Specifying configuration options

All configuration options are set in the /opt/blueworx/utils/consumers/config/logging-consumer.properties properties file.

The available configuration options are:

Table 1. Apache Kafka Logging Consumer properties
Name	Mandatory	Type	Default	Description
bootstrap.servers	yes	IP address/hostname	-	The URI of the Apache Kafka broker(s) to retrieve messages from. If you specify more than one broker (in an Apache cluster setup), specify in the format host1:port,host2:port.
output_directory	no	String	logs	The root directory to use for the output from the consumer
files_per_directory	no	Number	300	How many call logging files to write out when processing blueworx.application messages before moving to a new subdirectory under callLogs. This prevents the consumer from running into maximum limits on files in a directory
trace_file_limit	no	Number	2048	How many MB the consumer should write to a trace_X.out file before moving to the next trace out file.
trace_file_count	no	Number	5	The maximum number of trace_X.out files to create before rotating back to overwrite from trace_0.out
error_file_limit	no	Number	2048	How many MB the consumer should write to an error_X.out file before moving to the next error out file.
error_file_count	no	Number	5	The maximum number of error_X.out files to create before rotating back to overwrite from error_0.out

Running the consumer

To start example consumer, use the following command:

/opt/blueworx/utils/consumers/bin/logging-consumer.sh

Note: This will start the consumer in the active shell, and so will terminate when you log off. To create a long running consumer task either:

Run the consumer with the nohup command
Run the consumer as a service

Running the consumer as a service

Note: This documentation will not detail in full how to create a service to run the consumer, and a service may need to be tailored depending on your environment.

An example service is shown below:

[Unit]
Description=BVR Kafka Logging Daemon

[Service]
ExecStart=/opt/blueworx/utils/consumers/bin/logging-consumer.sh
User=root

[Install]
WantedBy=multi-user.target

To use this example, create a new service file (for example logging-consumer.service) in /etc/systemd/system/multi-user.target.wants on your Red Hat Enterprise Linux or CentOS system.

You can now use the following commands to control your service.

Check status of the consumer service

To check the status of the logging consumer issue the following command:

systemctl status logging-consumer.service

Start the consumer service

To start the logging consumer issue the following command:

systemctl start logging-consumer.service

Stop the consumer service

To stop the logging consumer issue the following command:

systemctl stop logging-consumer.service