Okay, there are multiple pipelines that I have been worked on till now. So consider a complete pipeline of polling the data, transforming the data, enhancing the data and pushing that to the Elasticsearch. This is a complete one single pipeline that I have been working on. So what we do, a complete pipeline that I have been working through, there are a lot of pipelines, but this is one kind of example to look, what we do, each interval, this Python script is running on AWS ECS, that is Elastic Container Service that pulls the data at a certain interval of time, that push 500 articles to the Kinesis. Now we process that data, process that data based on the Lambda data. In that processing, we transform that data into totally different formats. That format has been completely supported by us, because we have hundreds of sources and hundreds of sources will not give us the data in a particular format that we want. So we do that at our own side, data transformation, data validation and all those stuff will be done with the base of Event-Based Triggered Lambda. Now that will firstly segregate the useful data, if any data is not useful, then we will scrap the data. And based on the field mapping and the validations, we will map the data, transform if required in a different kind of a field or in a different kind of a data type, we will transform that data. And then there are some enhancement needs to be done. So for that, what we do is we keep this data transformed to a different Kinesis, from there another Triggered-Based Lambda, Event-Based Lambda get triggered, there we do enhancement. We add publications, we add licenses, we add validations or the owner of the data, all those things we add and from there we will push that data to the Elasticsearch. So this is the pipeline that, this is one of the complex pipelines that we have worked. Again, in the enhancement of data, some of the fields are coming like an author, that is coming from the DynamoDB. We push, pull that data from the DynamoDB and append that into the articles. And this DynamoDB is not getting filled directly, we are filling that author's details from somewhere else, that is a different pipeline running in, but those Dynamo details or the DynamoDB author fields are appended internally in our own enhancement fields. And after that, we will push that data to Elasticsearch. Parallelly, that data goes for percolation, percolates from where you will get the articles. So percolate cluster works over there that creates, generates the query from your article. So there as well it works. Another pipeline that I am working on is like in a beam pipeline. So basically, when all the data, when the source push the data to a Kafka topic, on the Kafka topic of consumer or executors working, that executors are paralleling, multiple executors are working parallelly based on group IDs on all those stuff. Okay. So what we will do is like we paste the data in a pipeline, basically in a groups or in a batch. Okay. What we'll do, we do segregated data and we transform that data in a different, different way. Okay. So basically, all the things, the enhancement, all those things done in the pipeline only and this will be sent to a different Kafka topic. There will be a failure of our data. If any failure, we will send that data to DLQ. That is dead data. Okay. But we'll take example of the pass or the good example, like a good flow only. Okay. Once this data is getting processed, processed successfully, we push that data to a different topic. From there, our esync application works. That is a man, that is a flink, that is a beam again, Apache flink and Apache beam application again. Okay. That pulls the data, this data. It takes the data in which, what kind of a data it is and push that data to that kind of index. That kind of index is a elastic system. So all this thing, all this enhancement and all this transformation has been done into this pipeline. This is the complex pipeline that we are working together.
