Fauie Technology

eclectic blogging, technology and hobby farming

Tag: nsm

Elasticsearch Maintenance with Jenkins


Maintaining production systems is one of those unfortunate tasks that we need to deal with…  I mean, why can’t they just run themselves?   I get tired of daily tasks extremely quickly.   Now that I have a few ongoing Elasticsearch clusters to deal with, I had to come up with a way to keep them singing.

As a developer, I usually don’t have to deal with these kind of things, but in startup world, I get to do it all from maintenance, monitoring, development, etc.

Jenkins makes this kind of stuff super easy.   With a slew of python programs, that use parameters/environment variables to connect to the right Elasticsearch cluster, I’m able to perform the following tasks, in order (order is key)

  1.  Create Snapshot
  2. Monitor Snapshot until it’s done
  3. Delete Old Data ( This is especially interesting in our use case, we have a lot of intentional False Positive data for connectivity testing)
  4. Force Merge Indices

I have Jenkins set up to trigger the down stream jobs after the prior completes.

I could do a cool Jenkins Pipeline…. in my spare time.


Daily snapshots are critical in case of cluster failure.   With a four node cluster, I’m running in a fairly safe setup, but if something goes catastrophically bad, I can always restore from a snapshot.   My setup has my snapshots going to AWS S3 buckets.

Delete Old Data:

When dealing with network monitoring, network sensors and storing of NSM data (see Suricata NSM Fields ), we have determined one easy way to test end to end integration is by inserting some obviously fake False Positives into our system.   We have stood up a Threat Intelligence Platform (Soltra Edge) to serve some fake Indicator/Observables.   Google.com, Yahoo.com, etc.   They show up in everyone’s networks if there is user traffic.   Now, this is great to determine connectivity, but long term that comes to be LOTS of traffic that I really don’t need to store…. so, they get deleted.

Force Merge Indices

There is a lot of magic that happens in Elasticsearch.  Thats’s fantastic.  Force Merging allows ES to effectively shrink the number of segments in a shard, thereby increasing performance when querying it.  This is really only useful for indices that are no longer receiving data.  In our use case, that’s historical data.  I delete the old data, then force merge it.


A day in the life.. of Jenkins.





Suricata NSM Fields

Value of NSM data

From https://suricata-ids.org

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation. Suricata is developed by the OISF and its supporting vendors.

Snort was an early IDS that matched signatures against packets seen on a network.  Suricata is the next generation of open source tools that looks to expand on the capabilities of snort.  While it continues to monitor packets, and can ‘alert’ on matches (think bad IP address, or a sequence of bytes inside of a packet), it expands the capabilities by adding Network Security Monitoring.  NSM watches a network (or PCAP file), jams packet payloads together (in order), and does further analysis.  From the payloads, Suricata can extract HTTP, FTP, DNS, SMTP, SSL certificate info etc.

This data can provide invaluable insight into what’s going on in your company or in your home.   Store the data for future searches, monitor the data actively for immediate notification of some wrong doings or anything else you want to do.  NSM data allows an analyst to track the spreading of malware.  Track how a malicious email came through.

Beyond the meta-data, Suricata can also extract the files from monitored sessions.  These files can be analyzed, replayed or shoved into a sandbox and detonated.  Build your own!

Here’s a break down of fields available, but remember they’re not always there.   Be careful in your coding.

All records contain general layer 3/4 network information:

  • timestamp
  • protocol
  • in_iface
  • flow_id
  • proto
  • src_ip
  • src_port
  • dest_ip
  • dest_port
  • event_type

This covered TCP/IP, UDP, etc.  Each event that gets logged (check out /var/log/eve.json) , has this information and more.  “event_type” indicates the ‘rest’ of the important data in this NSM record.  Values in ‘event_type’ will be one of:

  • http
  • ssh
  • dns
  • smtp
  • email
  • tls
  • fileinfo*


  • accept
  • accept_charset
  • accept_encoding
  • accept_language
  • accept_datetime
  • authorization
  • cache_control
  • cookie
  • from
  • max_forwards
  • origin
  • pragma
  • proxy_authorization
  • range
  • te
  • via
  • x_requested_with
  • dat
  • x_forwarded_proto
  • x_authenticated_user
  • x_flash_version
  • accept_range
  • age
  • allow
  • connection
  • content_encoding
  • content_language
  • content_length
  • content_location
  • content_md5
  • content_range
  • content_type
  • date
  • etag
  • expires
  • last_modified
  • link
  • location
  • proxy_authenticate
  • referrer
  • refresh
  • retry_after
  • server
  • set_cookie
  • trailer
  • transfer_encoding
  • upgrade
  • vary
  • warning
  • www_authenticate


Client/Server are child objects when this is parsed from JSON.

  • client
    •   proto_version
    •   software_version
  • server
    •   proto_version
    •   software_version


  • tx_id
  • rrtype
  • rrname
  • type
  • id
  • rdata
  • ttl
  • rcode


  • reply_to
  • bcc
  • message_id
  • subject
  • x_mailer
  • user_agent
  • received
  • x_originating_ip
  • in_reply_to
  • references
  • importance
  • priority
  • sensitivity
  • organization
  • content_md5
  • date


  • fingerprint
  • issuerdn
  • version
  • sni
  • subject


File info is special.  It can be associated with other types, like HTTP and SMTP/email.  Watch the object carefully, you’ll get a mix of fields

  • size
  • tx_id
  • state
  • stored
  • filename
  • magic
  • md5

© 2023 Fauie Technology

Theme by Anders NorenUp ↑