Discuss the approach to modernizing data pipeline at Onclusive Focused on cost optimization, performance, scalability We did initially using AWS managed services like Lambda, Polar, Kinexis, Weaver Costly these services are decided to move to a self-managed Kafka cluster We did scalable and cost effective Kafka is Handle large volume of data, it can scale up or down as needed for cost optimization Replaced many Lambdas with Kafka In Apache Beam, we did Lambdas pay for execution, they are making them expensive When running many concurrently with Kafka and Apache Beam Set auto-scaling policies, we can scale executors up when needed and down when not, we do This flexibility reduce cost it does, for example, receive 10 GB of data in the morning We might scale up executor to process the data quickly, we can As data flow decreases, scale down, we do cheaper than maintaining many Lambdas This approach is thus by using Kafka and Apache Beam Improve performance, reduce cost and ensure scalability We did efficient and cost effective our modernized pipelines are
