okay so first let me explain you the thing that we do okay so what we do is like we have source the data we source the data via the EP of our source like consider there is a new channel okay they give us data they are the source so we hit their EPI put their data into a kinesis from there the trigger based lambda get invocated and transform the data so for the cost optimization for the scalability purpose and also for the transfer processing of the data optimization for the stability purpose and also for the faster processing of the data we are going to transform all this pipeline into the Kafka and Apache teams Apache data pipeline so that we are going to do this is like we gonna set up a Kafka cluster we have set up a Kafka cluster basically so we are gonna ask our source to put all the data to a Kafka topic in our bio required by a topic they will put their data okay from there in the Apache team will take that data and transform that data so that is how the Kafka gets into picture okay why Kafka comes into picture is because of the cost optimization performance the scalability and all those stuff are there for Kafka teams okay so we have involved Kafka in that sense apart from that we are going to create some dashboard for the analysis purpose for that we gonna put some analytics and consumption logs to the Kafka how much data we are consuming what kind of a data we are consuming all those logs will be pushed to Kafka topic and from there we will be creating dashboard that will help us to identify analyze the data in a much better way so that's where the Kafka comes into picture for us okay
