Heraclitus, the Metaphysics of Change, and Streaming Data

Hailing from the Persian city of Ephesus in around 500 BC, the Greek philosopher Heraclitus is famous for his trenchant analysis of stream processing systems, saying “You never step into the same river twice.” Central to his philosophy was the idea that all things change constantly. His close readers also know him as the Weeping Philosopher—perhaps because dealing with constantly changing data at low latency is actually pretty hard. It doesn’t need to be that way.

Almost as famous as Heraclitus is Apache Kafka, the de facto standard open-source distributed stream processing system. Many of us know Kafka’s architectural and pub/sub API particulars as well as we know the philosophy of Heraclitus, but that doesn’t mean we’re equipped to build the kind of real-time streaming data systems that the next generation of business requirements are doing to demand. Like Heraclitus was telling us 2,500 years ago: we need to get on board with streams.

This talk is a thorough introduction to the Streams API, an important recent addition to Kafka that lets us build sophisticated stream processing systems that are as scalable and fault-tolerant as Kafka itself, and also happen to align quite well with the microservices sensibilities that are so common in contemporary architectural thinking. We’ll talk about how to deploy Streams applications and look at actual working code that will bring your thinking about streaming data systems from the ancient history of pub/sub paradigms into the current era.