De-Coder’s Ring

Software doesn’t have to be hard

Month: June 2016 (page 1 of 2)

Enough About Enough

One of my last posts was all about work life balance, “Your work life balance sucks“.  We all have issues with prioritizing our time and allocating enough in each bucket of our lives.  Family, Work, Self, Others.  Here, I want to talk about your knowledge and how it’s important to be intentional in that aspect of your life too.

I seem to be surrounded with people who either know the nitty-gritty detail about the technology or subject matter that they are engrossed in OR the stereo-typical “jack of all trades”.  You know the two types.  The first, you can’t have a discussion with about something.  They know WAY more than you do, and take the conversation very deep, very quickly.  They miss the forest for the trees.  These are your experts.   This person knows the insides and out of the tool you’re using, and their knowledge allows you to be confident in your approach and whether an idea will work or not.

The second type of person is the typical ‘jack of all trades’.  I say typical, because in this case, they know a little about a lot.  I’d even say, a little about ‘some’ things.  In the technology world, this person would be able to work around a bash shell, write database queries and make some updates to a web page.  The counter to this, is the Java developer who doesn’t know how to write a query.  The web designer who doesn’t know a lick of HTML or CSS.  My point here, is that this person is wide and shallow, as opposed with the first person, who’s super deep, but very narrow.

The mental image just came to me of the iceberg.  You know, something like this:

The first person will tell you about that little piece of ice that sits at the bottom of the iceberg.  While that’s important to someone, and is completely valid knowledge, 99% of us don’t care, and unless the conversation is about the bottom tip of the ice berg, it’s inappropriate.

The second person, they can tell you generally about ice bergs: maybe only if it’s covered in snow. If it’s just ice, they may not know about it.

The challenge a lot of us have, is how to balance in the middle of this scale.  Depending on your role, you need to find your sweet spot.   For me, as a consultant/architect/VP Engineering, I need to know “enough about enough”.  I need deep and wide.  I’d argue that I have to know 80% about an iceberg, but more importantly, know how ice works enough to be able to make some assumptions that can later be validated or experimented on.

In the world of technology, this manifests in a lot of different ways.  Mostly, it comes down to being educated enough to decide between two or more options, and picking an initial path go down.  Now, anyone can pick the path, but, the sweet spot means being able to get to the fork in the path as soon as possible to determine if it’s the right path or not.  Which database engine do we pick? Which time-series data storage do we use?  Which visualization engine will work with our python framework, etc.

There’s absolutely no way everyone can know everything about everything.  Seriously, look at the eco-system for devops these days (Back when I was first writing code, we didn’t have no automation!).  It’s amazing!  There are dozens of tools that do almost the same task.  They overlap, they have their sweet spots too, but it takes a special kind of person with a very specific set of skills (hmm.. I’ve heard that somewhere), to determine which tool to use in a specific situation.

I want to say this is another instance of the 80/20 rule, but not exactly.  Let’s go with that anyway.  Instead of learning the 100% details of something, spend 80% of the time, then keep going on to other things.  Don’t be so narrow focused.  Think about the days of Turbo Pascal.  If all you knew was 100% TP, how’s the job market these days?

Balance that with only learning 20% about something.  You will never be seen as an expert.  No matter what the subject matter is:  Technologies, development approaches, managerial styles, etc. You need to be deep enough to be an authority to make an impact on the organization you’re in if you want to excel and succeed.

Everything in life needs a balance.  Diet, exercise, hobbies, work/life, etc.  Knowledge and learning is in there too.  Be intentional about where you focus your energies in learning about something new, and figure out how much is enough.

Suricata Stats to Influx DB/Grafana

For everyone unfamiliar, Suricata is a high performance network IDS (Intrusion Detection System), IPS (Intrusion Prevention System) and NSM (Network Security Monitor).  Suricata is built to monitor high speed networks to look for intrusions using signatures.  The first/original tool in this space was Snort (by Sourcefire, acquired by Cisco).

NSM mode for Suricata accomplishes some pretty fantastic outputs.  In the early days of nPulse’s pivot to a network security company, I built a prototype ‘reassembly’ tool.  It would take a PCAP file, shove the payloads of the packets together, in order by flow, and extract a chunk of data.  Then I had to figure out what was in that jumbo payload.  Finding things like FTP or HTTP were pretty easy…. but then the possibilities became almost endless.    I’ll provide a link at the bottom with suricata and other tools in the space.  Suricata can do this extraction, in real time, on a really fast network.  It’s a multi threaded tool that scales.

Suricata can log alerts, NSM events and statistics to a json log file, eve.json.  On a typical unix box, that file will be in /var/log/suricata.  The stats event type is a nested JSON object with tons of valuable data.   The hierarchy of the object looks something like this:

Suricata Stats Organization

Stats layout for Suricata eve.json log file

For a project I’m working on, I wanted to get this Suricata stats data into a time series database.  For this build out, I’m not interested in a full Elasticsearch installation, like I’m used to, since I’m not collecting the NSM data, but only this time series data.  From my experience, RRD is a rigid pain, and graphite (as promising as it is) can be frustrating as well.  My visualization target was Grafana and it seems one of the favored data storage platforms is InfluxDB, so, I thought I’d give it a shot.  Note, Influx has the ‘tick’ stack, which included a visualization component, but, I really wanted to use Grafana.   So, I dropped Chronograf in favor of Grafana.

Getting Telegraf (machine data, CPU/Memory) injected into Influx and visualized within Grafana took 5 minutes. Seriously.  I followed the instructions and it just worked.  Sweet!  Now it’s time to get the suricata stats file working..

snippet of the above picture:

As you can see, the nested JSON is there.  I really want that “343734” “ipv4” number shown over time, in Grafana.  After I installed Logstash (duh), to read the eve.json file, I had to figure out how to get the data into Influx.  There is a nice plugin to inject the data, but unfortunately, the documentation doesn’t come with good examples. ESPECIALLY good examples using nested JSON.   Well, behold, here’s a working document, which gets all the yummy Suricata stats into Influx.

WHOAH! That’s a LOT of fields.  Are you kidding me?!  Yep, it’s excellent. Tons of data will now be ‘graphable’.  I whipped together a quick python script to read an example of the JSON object, and spit out the data points entries, so I didn’t have to type anything by hand.  I’ll set up a quick gist in github to show my work.

Let’s break it down a little bit.

This snippet tells logstash to read the eve.json file, and tell it that each line is a JSON object:

This section tells logstash to drop every event that does not have “event_type” of “stats”

Suricata has a stats log file, that could probably be used by Logstash, but I may do that on another day. It’s way more complicated than JSON.

The last section is the tough one.  I found documentation that showed the general format of “input field” => “output field”… but that was it.  It took a ton of time over the past working day to figure out exactly how to nail this.  First, fields like ‘host’, ‘user’,’password’,’db’,’measurement’ are very straight forward Influx concepts.  DB is much like a name spacing or a ‘table’ in a traditional sense.   A db contains multiple measurements.  The measurement is the thing we’re going to track over time.  In our case, these are the actual low level details we want to see, for instance, ‘stats-capture-kernel-drops’.

Here’s an example:

On the left ‘stats-decoder-ipv4’ is the measurement name that will end up in InfluxDB.  The right side is how Logstash knows where to find the value based on this event from eve.json.  %{ indicates the value will come from the record.  Then Logstash just follows the chain down the JSON document.  stats->decoder->ipv4

That’s it.   The logstash config, eve.json, little python script, and here’s a picture!

Grafana Screen Shot showing CPU, Memory and Suricata Information

Grafana Screen Shot showing CPU, Memory and Suricata Information

Pretty straight forward.  Example configuration on one of the stats:

Configure graph in grafana

Configure graph in grafana

Please leave me a comment!  What could be improved here?   (Besides my python not being elegant.. it works.. .and sorry Richard, but, I’m  a spaces guy, tabs are no bueno)

  1. GITHUB:
  2. Suricata:
  3. Snort Acquired Sourcefire:
  4. Grafana –
  5. InfluxDB/TICK stack:
  6. Chronograf:
  7. Telegraf:
  8. Logstash:

SELinux: Causing a pain, time and time again

Once again, SELinux bit me.. what a pain.  It’s good, I’m sure for something.  but dang, it’s always to blame.

Trying to set up an Apache reverse proxy.   Kept getting a 503 error,

Permission denied: AH00957: HTTP: attempt to connect to ( failed

Did some googling, and thanks to Justin Ellison @, he saved the day.

Simple command to allow the reverse proxy:

/usr/sbin/setsebool -P httpd_can_network_connect 1

Found the assist here:



Tencent buys majority Supercell stake (for more than a few cents!)

Oh my goodness.  Not since the Instagram acquisition have I been BLOWN AWAY by the valuation of a company.  I realize that I know nothing about the revenue of Supercell, but a $10B valuation?   Wow!    Their new app, Clash Royale, has definitely swept the household’s phones/ipods, etc.. and that’s ALL the kid’s are playing these days.



Older posts

© 2017 De-Coder’s Ring

Theme by Anders NorenUp ↑