De-Coder’s Ring

Software doesn’t have to be hard

Category: technical (page 2 of 11)

scattered desk, scattered mind?

I’m putting a few things together.  Typically, my desk is pretty cluttered.    almost hoarder level cluttered.

yet, I know where everything is.   pencils, paperclips, bills, etc.

Reading through some of my old code, it begins nice and organized, but, over time, it gets a bit cluttered.

having lived in startup world for a long time, I tend to not have other people maintain or update my code, so it hasn’t been a super big problem…   but, it’s one of my dirty little coding secrets.

 

 

SQS Cost Optimization: Save $1684 per month

I made a bone head move.  Yes, I admit it.

Amazon SQS has always been talked about as ‘free’.  In terms of passing messages for an application, it’s supposed to be freaking cheap as can be.

I was blown away when my July 2017 SQS bill was $1720!!

What?  How’s that FREE?!

Digging into my SQS reports, I made 4.3 billion (with a B) SQS calls.    Billed at $0.0000004 per call, that adds up to $1,720!

Well, my architecture would only be scaling up from there.. I had to do something about it.

I moved to Kafka.

… but that’s not the point of this post.  I realized, later, that I could have been close to free, and optimized a ton of my downstream pipeline.

SQS messages are billed on the 64k of data chunk.    My messages were averaging 1,300 bytes (1.3k).   Doing some quick math, I could have been batching up to 49 ‘messages’ at a time per SQS call.   This would save my producer, and my consumer a ton of API calls to SQS.

If I can batch 49 ‘messages’ per API call, then my 4.3 billion calls, becomes about 87.8 million SQS calls.

87.8 million SQS calls becomes $35.10

Too late for this implementation (Kafka is better in the long run in my opinion), but if the goal were server/infrastructure-less implementation, then shoot. I could have saved $1,684 per month.

Information on SQS pricing can be found here:

https://aws.amazon.com/sqs/pricing/

TL;DR

Batch your data before pushing to SQS, save moneys…. #profit

Moving from Splunk to Elasticsearch

Splunk is magical.   It is a wonderful technology that has allowed many practitioners the ability to slice and dice data, in ways that they couldn’t do before.  I’ve been on the user side, and the application developer side of things.  I’ve worked closely with folks within the Splunk organization over the year, and want to publicly say “It’s awesome!”

The challenge I’m finding with partners, clients, etc, is that Splunk gets expensive, fast.

We need to add data, but the more data we add, the pricier in gets.

In steps Elasticsearch.

Elasticsearch is free/open source software that provides full text search capabilities in a highly scalable fashion.  Did I mention it’s free?  Just pay for hardware.

Throwing an Elastic cluster, and scaling it up, is a super simple task in a cloud provider like Elasticsearch.

One new topic for this blog, is migrating from Splunk to ES.  I’ll do a side by side to view inputs!

 

 

Threat Hunting with Open Source Software

I’ve begun working on a new project, with a spiffy/catchy/snazzy name:
Threat Hunting: With Open Source Software, Suricata and Bro

I’ve planned out multiple chapters, from raw PCAP analysis, building with session reassembly, into full on network monitoring and hunting with Suricata and Elasticsearch.

This project will take a long time. While I work through it, I’ll be posting here regularly. I very much welcome feedback.

Here’s a little introduction video, but , more will come as I add videos.

The next video will be looking at how data is transmitted over a network… anyone ready for a super brief OSI Network model overview?

Older posts Newer posts

© 2017 De-Coder’s Ring

Theme by Anders NorenUp ↑