Arroyo - Micah Wylde, Co-Founder and CEO

In this episode, we talk to Micah Wylde, Co-founder and CEO of Arroyo, about how they simplified stream processing for data engineers with Rust.

Show Notes

Data engineers are responsible for building and maintaining the data infrastructure of a company. They are the ones who make sure that data is collected, stored, and processed in a way that is useful for the business.

This poses a lot of challenges, especially when it comes to processing data in real-time. The data is often coming from different sources, in different formats, and at different rates. Visibility into the pipelines is often limited, and debugging is hard.

Arroyo is a new stream processing engine that aims to solve these problems. It is built in Rust and uses WebAssembly to allow users to transform, filter, aggregate, and join streams using SQL, with sub-second results. It scales to millions of events per second. The Arroyo Streaming Engine is available as open source software on GitHub.

In this episode, Micah Wylde, Founder of Arroyo, walks us through the architecture of the Arroyo Streaming Engine and explains why Rust is the best language for building data infrastructure.

About Arroyo

Arroyo was founded in 2022 by Micah Wylde and is based in San Francisco, CA. It is backed by Y Combinator (YC W23). The companies’ mission is to accelerate the transition from batch-processing to a streaming-first world.

About Micah Wylde

Micah was previously tech lead for streaming compute at Splunk and Lyft, where he built real-time data infra powering Lyft’s dynamic pricing, ETA, and safety features. He spends his time rock climbing, playing music, and bringing real-time data to companies that can’t hire a streaming infra team.

Tools and Services Mentioned

• Apache Flink
• Tokio Discord
• Clippy
• Zero to Production in Rust by Luca Palmieri
• Apache DataFusion
• Axum web framework
• sqlx crate
• log crate
• tokio tracing crate
• wasmtime - A standalone runtime for WebAssembly

References To Other Episodes

• Rust in Production Season 1 Episode 1: InfluxData

Official Links

• Arroyo Homepage
• Arroyo Streaming Engine
• Blog Post: Rust Is The Best Language For Data Infra
• Micah Wylde on LinkedIn
• Micah Wylde on GitHub
• Micah Wylde’s Personal Homepage

About corrode

"Rust in Production" is a podcast by corrode, a company that helps teams adopt Rust. We offer training, consulting, and development services to help you succeed with Rust. If you want to learn more about how we can help you, please get in touch.