De-Coder’s Ring

Software doesn’t have to be hard

Tag: cybersecurity (page 1 of 2)

Threat Hunting with Open Source Software

I’ve begun working on a new project, with a spiffy/catchy/snazzy name:
Threat Hunting: With Open Source Software, Suricata and Bro

I’ve planned out multiple chapters, from raw PCAP analysis, building with session reassembly, into full on network monitoring and hunting with Suricata and Elasticsearch.

This project will take a long time. While I work through it, I’ll be posting here regularly. I very much welcome feedback.

Here’s a little introduction video, but , more will come as I add videos.

The next video will be looking at how data is transmitted over a network… anyone ready for a super brief OSI Network model overview?

Elasticsearch Maintenance with Jenkins


Maintaining production systems is one of those unfortunate tasks that we need to deal with…  I mean, why can’t they just run themselves?   I get tired of daily tasks extremely quickly.   Now that I have a few ongoing Elasticsearch clusters to deal with, I had to come up with a way to keep them singing.

As a developer, I usually don’t have to deal with these kind of things, but in startup world, I get to do it all from maintenance, monitoring, development, etc.

Jenkins makes this kind of stuff super easy.   With a slew of python programs, that use parameters/environment variables to connect to the right Elasticsearch cluster, I’m able to perform the following tasks, in order (order is key)

  1.  Create Snapshot
  2. Monitor Snapshot until it’s done
  3. Delete Old Data ( This is especially interesting in our use case, we have a lot of intentional False Positive data for connectivity testing)
  4. Force Merge Indices

I have Jenkins set up to trigger the down stream jobs after the prior completes.

I could do a cool Jenkins Pipeline…. in my spare time.


Daily snapshots are critical in case of cluster failure.   With a four node cluster, I’m running in a fairly safe setup, but if something goes catastrophically bad, I can always restore from a snapshot.   My setup has my snapshots going to AWS S3 buckets.

Delete Old Data:

When dealing with network monitoring, network sensors and storing of NSM data (see Suricata NSM Fields ), we have determined one easy way to test end to end integration is by inserting some obviously fake False Positives into our system.   We have stood up a Threat Intelligence Platform (Soltra Edge) to serve some fake Indicator/Observables.,, etc.   They show up in everyone’s networks if there is user traffic.   Now, this is great to determine connectivity, but long term that comes to be LOTS of traffic that I really don’t need to store…. so, they get deleted.

Force Merge Indices

There is a lot of magic that happens in Elasticsearch.  Thats’s fantastic.  Force Merging allows ES to effectively shrink the number of segments in a shard, thereby increasing performance when querying it.  This is really only useful for indices that are no longer receiving data.  In our use case, that’s historical data.  I delete the old data, then force merge it.


A day in the life.. of Jenkins.





Other People’s Analysts

Over the last 6 years, I have been entrenched in Cyber Security.

  • Packet capture
  • Network Forensics
  • Identity and Access Management
  • Threat Intelligence

During my nPulse Technologies days (acquired by FireEye), I relearned all the network packet stuff that I had been taught in college.  The OSI network layers, VLANs, Q-in-Q… oh boy!    Reassembling packets (with python no less) was a REALLY fun exercise…  never made it into the product, since there were open source tools that did it better (faster?).. but I did it….   then came the challenge of using the reassembled data in an application.

Imagine this now, you’re a cyber analyst.  You’ve got some juicy intel from your ISAC (FS-ISAC? NH-ISAC?) … or maybe it’s from your industry buddies that you share intel with.   You set up your alerting mechanisms, you set up your SIEM, and you wait.

PING!  You get a hit!   You know now have an IP address that a machine in your network tried to go to.   You start your research, do a little OSINT, do some googling… find out it’s a shared host.  Oh well.

False Positive.

You tell your buddies, and that’s it for the day.

Guess what just happened?   Your group just got smarter because two of you did some work.  The first guy set up the intel, and you validated it as a false positive.    Since you both shared within your community, you just got smarter! You leveraged a few members of the community to make you all better!

This is the best scenario we have today.   Some communities share data.  Not many communities allow for automatic sharing of sightings (did you see that IOC in the wild?).  NO communities allow you to share what you did in regards to that IOC.  Did you block it in a Firewall? Did you mark it as a False Positive?

There aren’t many tools out there that can help this process..    The more we can share, the more we can attribute, the more we can automatically know what’s going on in the network of our peers, the safer we’ll all be.


Tackling Expensive and Complicated Information Security

Information Security:  It doesn’t have to be so expensive (or complicated!)


The Bad News

For Small/Medium Businesses (SMBs), you can’t approach information security the same way your bigger brothers do.  Face it, Capital One has a much larger information security (infosec) budget than the Downtown Credit Union in Powhatan, VA.   Small companies don’t have the same staffing models, technology expertise or highly specialized analysts that focus solely on protecting data.   Sure, there are free and open source tools, for example, but they still require expertise and time to get them up and running, not to mentioned tuned, maintained, updated, etc!


Here’s another challenge.  A good information security practice relies on intelligence about threats, attacks, vulnerabilities, etc.  There are open source data sets that can help your SMB know what to look for in network scans, packet matching signatures and queries in your SIEM, but that open source data tends to be stale.  Don’t get me wrong, it’s table stakes.  You NEED to be on the lookout for what Emerging Threats has, but it’s not sufficient.  That data will protect you, but it’s a tiny part of the known bad things out there.


Ok, one more ‘bad news’ comment.  There are vendors out there that will sell you cyber threat intelligence (CTI) data.  Some aggregate data from intelligence providers; they’re called TIPs, Threat Intelligence Platforms.  They provide tools and technologies to help you get known intelligence data.  Others research, probe and monitor the internet/private networks looking for ‘things’ that are bad.  They’ll either sell you the data or sell it to an aggregation company who will sell it to you.  They provide a great service, and deserve to be paid for the work they do, but again, this may be pricey and out of your budget.


The Good News!

There is a new reality out there.  There are sharing communities being formed to share this threat intelligence data (ISACs and ISAOs).  These groups are focused around specific industries (Health Care, Financial Services, Aviation, etc) and allow a platform to share more RELEVANT data.   This is data that affects your industry, and therefore has a much higher chance of being relevant to you company.   Their cyber intelligence data is target to their industry and typically much more relevant than the data served from large repositories.


Size doesn’t always matter.  With finite resources, both technical and human, it’s nearly impossible for SMBs to look out for all the bad things; and why should they?  A bank doesn’t care about a command and control channel for a botnet that is targeting manufacturing equipment.


Sharing communities are becoming the KEY source of threat intelligence data for small to mid-size business.  It’s putting the control of the infosec spend back into their hands.


By leveraging shared community data as the primary (but still not only!) source of intelligence, we substantially reduce the cost of a comprehensive cyber intelligence and threat mitigation plan.  Once we embrace this new world of industry-specific, relevant cyber intel, we’ll have new ways to connect in a USABLE way.   What’s “usable”?  In order to reap the benefits of your sharing community memberships, you need readily tools that:

  • Don’t require a skilled analyst behind the dashboard 24×7.
  • Don’t require a SIEM to use it.
  • Doesn’t require a knowledge of code.
  • Doesn’t require more than a basic understanding of CTI (STIX, TAXII) terminology


Now What

Who’s going to provide a tool like this?  Ha!  I’m not good at keeping secrets, but I’m working on something that will help bring the promise of a sharing community to reality.


Older posts

© 2017 De-Coder’s Ring

Theme by Anders NorenUp ↑