Enhance data ingestion and integration with Kafka and Kafka Connect. We did use Kafka for data streaming. We did particularly manage Kafka service, stream data from source to consumer, it helps. Scale our streaming services, it allows receive millions of article each day. We do scale to billions VM. Kafka streams crucial for the scalability. It is Kafka Connect, an interface provided by Kafka helps connect Kafka to various services. Retrieve, post and ingest data, it does. However, custom consumers and producers, we created directly using Kafka Connect. We did not custom Kafka Connect, we implemented providing flexibility to change connections and configuration. Thus, by using Kafka stream for scalability, custom Kafka Connect for scalability, enhance data ingestion and integration with it.
