Yes, for sure. I have explained that about modernization in my pipelines. So first thing is like why we are modernizing our pipelines. First thing is like a cost optimization. Second is like for the performance that we have and third is like a further scalability. Right now we are using Lambdas, we are using Polars, we are using Kinesis. That is totally based on the AWS managed services, okay, and those are really, really, very costly. So what we have decided, we will use Kafka, that will be a self-managed Kafka cluster. Okay, we will ask our customer to put the data, a lot of data Kafka can handle. It can be easily scaled to a different, different level with minimum price and whenever it's not required, we can scale it down. Okay, now coming to the cost optimization, now coming to the scalability first. Okay, now consider there are like 400 Lambdas are working for the one kind of a data, let's say for the print, there are 400 executors on the Lambda are working parallelly or concurrently at a time. So for each 400 processes, you have to pay for them for sure, because Lambda works, Lambda is a serverless technology and AWS is asking for the money for each execution or each Lambda getting triggered. Okay, so that's how the AWS is making money. But sometimes what happens like we don't need to pay that much money because sometimes we need those data to be processed much faster and in a much efficient way. So we have to pay more money for it and sometimes it's like, it's very like we don't need to, we need those Lambdas and all these things to be processed and executed fast. This totally depends on what kind of a stream it is. Okay, so what we need to do is like, what we have done is like we have used Kafka, we have used Apache Beam and we have modernized pipeline, we are using Kafka, from there we are using the Apache Beam pipelines. Okay, what we will do, here we can set manually or you can say the based on auto scaling policy policies that we will set up. Okay, we can scale the executors required however we want. Okay, let's say you have received 10 GB of data in a morning. Okay, and you want that to be processed very fast right now. Okay, and with the less amount of money, you can easily create a replication factor, replication of that particular parts to 100 to 500 to 1000 at a time and that does not affect much cost to you because you are going to pay for the, you are going to pay for the machines that has been used. Okay, and that machine will be for sure cheaper in terms of Lambda. Okay, and that executor will continuously work whenever the data goes down or whenever the data flows come to down, you can easily scale it down. So, it will be faster processing as it's a big data technology, it will process your data in a much faster way. Okay, there will be cost optimization and then we are also handling the scalability with the help of Kafka and the executors. So, that's how we are doing. Okay.
