Pricing Lyft rides with Apache Beam - a case study in migrating from a worker-based workflow to streaming

Rakesh Kumar

2019-09-12 14:00 #beam

Ride-sharing is a two-sided marketplace; balancing supply and demand with price in real time is critical to maintaining an efficient system. Dynamic pricing creates fairness for drivers (by raising rates when there is a lot of demand) and maintains good experiences for passengers (by satisfying pick-up time SLAs). This complex system makes real-time decisions using various data sources; machine learning models; and a streaming infrastructure for low latency, reliability and scalability. In this streaming infrastructure, our system consumes a massive number of events from different sources to make these pricing decisions.

Reacting to these events in a cron scheduler based legacy infrastructure with inherent latency becomes a challenge, especially where timely reactions are required to balance market conditions. By leveraging Apache Beam, Lyft’s Streaming Platform powers pricing by bringing together the best of two worlds: ML models in Python and Apache Flink on JVM as the streaming engine.Topics covered in this talk include:

• A brief discussion of dynamic pricing, including motivation and high-level problem formulation
• Comparison of legacy architecture and new streaming architecture
• Overview of streaming platform architecture and technology stack
• Major gains from streaming architecture
• Lessons learned