Okay, real-time data processing and the challenges that we face so when it comes to the processing of real-time data we mostly face the challenge of processing time. The performance of our processors or the executor so what we do basically for that okay let me give an example of what we have done okay so basically we pull the data from the folders put that data to Kinesis so now from there we have to make sure that data is being processed in a particular time or it should be processed near real time basically. So we make sure that the concurrency of the lambda is accurate okay and whatever the records is getting processed it should be processed once okay and that is how we measure say that the data is being processed at a near real time to real time so let me give an example in terms of lambda that we are running okay once the data comes to Kinesis okay from there we can set the batch okay batch of 25 let us say 25 articles to be processed in a single lambda so it will reduce the timing of like a spinning if your lambdas and the connection that you made are the global variable that you make. So at that way you can tackle a challenge of data getting processed in a slower process again the challenge was like within this time range we have to send it to elastic search and send it to different process like for the Kafka log consumption okay so what we have is do we like have to process this data process is 25 articles to elastic search and then we process like a 50 logs for this 25 articles one log has started when we start processing the article at a time one and one when we completed processing this article we send two logs for each of the articles so it will be around 50 logs for 25 articles and that time how we are doing like we are selecting all those logs in a list and when pushing this log to Kafka we are pushing in batches so when you push it it will push it to a batch but Kafka so at a time 25 batches so it will reduce the time of connection to Kafka and it will also make sure the record have been pushed successfully apart from that in elastic search what do we do is like update record one by one but we do not do that what we do is like we won't push those record in elastic search it will reduce our time to connect to elastic search and it will also help you reduce the repetitive way of pushing to the elastic search so basically batching thing that we have basically the performance thing that are the major challenge in this.
