Okay, real-time data processing and the challenges that we face. So when it comes to the processing of the real-time data, we mostly face the challenge of processing time, the performance of our processors or the executors. So what we do basically for that, okay, let me give an example of what we have done, okay. So basically, we pull the data from the folders, put that data to Kinesis. So now from there, we have to make sure that data is being processed in a particular time or it should be processed in near real-time, basically. So we make sure that the concurrency of the Lambda is accurate, okay, and whatever the records is getting processed, it should be processed once, okay, and that's how we measurely say that the data is being processed at a near real-time to real-time. So let me give an example in terms of Lambdas that we are running, okay, once the data comes to the Kinesis, okay, from there, we can set the batch, okay, batch of 25, let's say, 25 articles to be processed in a single Lambda. So it will reduce the timing of like a spinning if your Lambdas and the connections that you make and the global variables that you make. So at that way, you can tackle a challenge of data-getting process in a slower process. Again, the challenge was like within this time range, we have to send it to Elasticsearch and send it to a different process like for the Kafka log consumption, okay. So what we have to do like we have to process this data, process this 25 articles to Elasticsearch and then we process like a 50 logs for this 25 articles, one log has started when we start processing the article at a time and one when we completed processing this article, we send two logs for each of the articles. So it will be around 50 logs for 25 articles and that time how we are doing like we are selecting all of those logs in a list and when pushing this log to the Kafka, we are pushing in a batches. So when we push it, we will push it in a batch but Kafka will take it as a single article, okay. So we will flush those 25 articles or 20, sorry, we will flush that 50 records or 50 logs in a batches, let's say the batch of 25 each. So at a time, 25 batches, 25 logs will be flushed and your Kafka queue, internal Kafka content, Kafka mentors and internal queue that will be pushed to the, that will get flushed at a time. So it will reduce the time of connection to the Kafka and it will also make sure the records have been pushed successfully. Apart from that, in Elasticsearch what we do is like we can update record one by one but we don't do that. What we do is like we bulk push those records in the Elasticsearch. It will reduce your time to connect to the Elasticsearch and it will also helps you reduce the repetitive way of pushing to the Elasticsearch. So basically batching thing that we have, basically the performance thing that was the major challenge and in a batching way and in a parallel way that we have optimized and that's how the challenge has been solved.
