Throughout the past 20 years of my career, I’ve seen a lot of data. Small data applications that handle a few transactions (re: blogs), large applications (multi-tenant network data collection).
Just about every single one of them has dealt with text searching in some capacity or another.
These days, if I’m moving data in work projects or fun/personal projects, I’m relying over and over on the same stack.
Logstash/Beats -> Kafka -> Logstash -> Elasticsearch with some cool visualizations in Kibana.
Want to know why?
It just freaking works. Especially well for time based events. Even if it’s not time based, it works.
… best of all? It’s all freaking free.
You can stand up a single node with all of the above on it. Easy. Sure, it won’t scale on a single node, but, each component of that list scales horizontally.
Prototype small, grow big.