One of the major themes for Infrastructure Engineering over the past couple years has been higher reliability and better operational efficiency. In a recent session with the Delta Lake project I was able to share the work led Kuntal Basu and a number of other people to dramatically improve the efficiency and reliability of our online data ingestion pipeline.
Join Kuntal Basu, Staff Infrastructure Engineer, and R. Tyler Croy, Principal Engineer at Scribd, Inc. as they take you behind the scenes of Scribd’s data ingestion setup. They’ll break down the architecture, explain the tools, and walk you through how they turned off-the-shelf solutions into a robust pipeline.