I’m going through a major project with a client in regards to migrating to a Kafka based streaming message platform. This is a heckuva lot of fun. Previously, I’ve written about how Kafka made the message bus dumb, on purpose: find it here.
Building out a new streaming platform is an interesting challenge. There are so many ways to handle the logic. I’ll provide more details as time goes on, but there are at least three ways of dealing with a stream of data.
The Hard Wired Bus
A message bus needs various publishers and subscribers. You can very tightly couple each service by having them be aware of what’s upstream or what’s downstream. Upstream is where the message came from, downstream is where it goes next. When each component is aware of the route a message must take, it becomes brittle and hard to change over time. Imagine spaghetti.
The Smart Conductor
A conductor travels on the train. The conductor determines the best route while moving along the train tracks. The conductor can handle every message, after every service to determine which service is next in line. This cleans up the function of each service along the line but makes the conductor pretty brittle too. The more complex the system gets, the more complex the conductor gets. A rules engine would be a great component to add to the conductor if you choose this path.
The Map Maker
A map maker plots a route for a hiker to follow. In our case, the map maker is a component that sits at the very beginning of every stream. When the event comes to the map makers topic (in Kafka), the map maker determines the best route for the message to take. Using metadata or embedded data in the event, the map maker can send the route down the chain with the event itself. Each service can use a common library to read from its configured topic, allow the custom service to do some work, and then use the wrapper again to pass the message downstream. The biggest advantage here is that each service doesn’t care where the message comes from, or where it goes next. This works great for streams that are static, and the route can be determined up front. If there are decisions down stream, then it may need a ‘switch’ service that is allowed to update the route.
What’s the best path for your application? Have something better that I haven’t thought of yet?