KafkaConnector
The KafkaConnector component is used to send messages to the Kafka Broker and retrieve messages from it. The KafkaConnector component can work either in synchronous or asynchronous mode to send the messages to the Kafka Broker.
Configuration
Component Configuration panel in the CPS (Configuration Property Sheet) allows customization in the component.
Figure 1: KafkaConnector CPS
Attributes
Connector Mode
The way in which the Kafka Connector acts; either as a Producer or a Consumer.
SEND
Selecting this mode makes the connector act as a producer, to send messages as Topics to the Kafka Broker.
RECEIVE
Selecting this mode makes the connector act as a consumer, to consume messages from the Kafka Broker.
Send Mode
This mode is enabled only when the Connector Mode is set to "Send.".
SYNCHRONOUS
When this mode is selected, the Kafka Connector acting as a producer sends requests and waits for the server reply. In this case, errors arising, if any, will be logged in the component and sent to the Exception port as per the configuration set in the Error Handling Configuration panel.
ASYNCHRONOUS
When this mode is selected, the Kafka Connector acting as a producer sends requests and does not wait for the server reply. The callback interface is used to log error messages in error logs and not sent to the Exception port.
Connection URL
The connection URL of the Kafka Connector which is being used to send or receive the messages.
Poll Interval (in ms)
The time interval in milliseconds until which the Kafka Connector acting as the consumer fetches messages from the Kafka Broker.
Topic Name
The name of the topic through which the messages are sent by the producer or through which the messages are consumed by the consumer.
Additional Properties
The default properties in the additional properties of the Kafka consumer (Connector Mode - Receive) can be changed as per requirement.
Figure 2: Additional Properties dialog box for Kafka consumer (Connector Mode - Receive) with prepopulated data
Please refer to the link below to add additional properties than the default ones to the Kafka Consumer:
https://kafka.apache.org/documentation/#newconsumerconfigs
The default properties in the additional properties of the Kafka consumer (Connector Mode - Send) can be changed as per requirement.
Figure 3: Additional Properties dialog box for Kafka consumer (Connector Mode - Send) with prepopulated data
Please refer to the link below to add additional properties than the default ones to the Kafka Producer:
https://kafka.apache.org/documentation/#producerconfigs
Expert Properties
Enable the Expert Properties view to configure these properties.
Expert properties are meant for advanced users.
Figure 4: KafkaConnector CPS with the Expert Properties enabled and highlighted
Error handling configuration
Please refer the Error Handling section in Common Configurations page for documentation.
Pre Processing XSL Configuration
Pre Processing XSL configuration can be used to transform request message before processing it. Click the ellipses button against the property to configure the properties.
Refer to the Pre/Post Processing XSL Configuration section under the Common Configurations page for details regarding Pre Processing XSL configuration and Post Processing XSL configuration (below).
Post Processing XSL Configuration
Post Processing XSL configuration can be used to transform the response message before sending it to the output port.
Process Message Based on Property
The property helps components to skip certain messages from processing.
Refer to the Process Message Based On a Property section under the Common Configurations page.
Threadpool Configuration
This property is used when there is a need to process messages in parallel within the component, still maintaining the sequence from the external perspective.
Refer to the Threadpool Configuration section under the Common Configurations page.
Functional Demonstration
Scenario
Configure two KafkaConnector microservices as producer and consumer following descriptions in the Component description and send a message. Connect a Feeder microservice to KafkaProducer and connect the KafkaProducer output and the KafkaConsumer input.
Connect Display microservices to the Kafka microservice output ports to view the respective outputs.
Figure 5: Event process to demonstrate the functions of KafkaProducer and KafkaConsumer
In this example, the input injected using Feeder is sent via the KafkaProducer, which is in turn received by the KafkaConsumer; the message will be displayed on Consumer_Display.
Input
Provide the following input in the Feder text field:
Input Text
Figure 6: Feeder input sent to the KafkaProducer
Output
KafkaProducer Display window displays the acknowledgment message.
Figure 7: Output received on the KafkaProducer display
KafkaConsumer Display window displays the message recived from the KafkaProducer.
Figure 8: Output received on the KafkaConsumer display