Today’s landscape for software engineers is amazing.  There are open source tools to do everything!  There are commercial tools to do everything!  With high level programming languages, we can write code to do everything!   Where the heck do you start?

Building a new platform or capability can seem like a daunting task.  I’ve worked with many companies and teams that undertake new development processes with a bit of discomfort.  They know that the project will ‘take forever’ to build, will probably be over budget and won’t be easy to follow through to completion.  Even the best teams, hindered by the wrong technology choices, will struggle to complete their tasks on time and will not perform at their peak.

One really interesting example of when I ran into this problem was a few years ago.   A team was using Sharepoint to build a search tool.  They had duplicate systems (from acquisitions, etc) that had the same ‘type’ of data in them.  When a new ‘thing’ was created, they had to check all the other systems to see if that ‘thing’ already existed.  The idea was to use the full text search capability of Sharepoint to ingest the data, and serve it via a web UI.   Sharepoint checked all the boxes.  It could ingest data, it could index the data and provide a UI.   (If you were on that project, and you’re reading this, no harm meant here.  You were using the tools you had available).

What went wrong?

Sharepoint sucks!  Well at least at that point in time, that version of Sharepoint could NOT handle the volume of data being thrown at it.  Sure, it could index and search the data, but it took 24+ hours to ingest the millions of rows.  Wow!  So much for getting views of fresh data within 5 minutes (arbitrary business requirement).   Something had to give.

What was right?

After doing a little bit of research off the side of my desk, ( I wasn’t directly associated with the task, just knew what was going on), I found Apache solr.  Looking back through the solr archives, it was Apache Solr 1.4 or so that I discovered.  It seemed magical.  Configure data sources, use JDBC, execute query, ingest data, etc.   Man, did it scream!   The initial/daily data load took 5 minutes. The delta loads were near instant.  This was the right tool for the job!

As an engineer, that’s just one of the many examples of finding the right tool for the job.  It takes curiosity and a quick sense of analysis (and some intuition) when figuring out how to tackle a problem.  Let your folks experiment.  Don’t use the hammer you have, because it’s the hammer you’ve always used.  Don’t use a hammer when a rock will do.



Cross posted at:

Short URL: