I made a bone head move.  Yes, I admit it.

Amazon SQS has always been talked about as ‘free’.  In terms of passing messages for an application, it’s supposed to be freaking cheap as can be.

I was blown away when my July 2017 SQS bill was $1720!!

What?  How’s that FREE?!

Digging into my SQS reports, I made 4.3 billion (with a B) SQS calls.    Billed at $0.0000004 per call, that adds up to $1,720!

Well, my architecture would only be scaling up from there.. I had to do something about it.

I moved to Kafka.

… but that’s not the point of this post.  I realized, later, that I could have been close to free, and optimized a ton of my downstream pipeline.

SQS messages are billed on the 64k of data chunk.    My messages were averaging 1,300 bytes (1.3k).   Doing some quick math, I could have been batching up to 49 ‘messages’ at a time per SQS call.   This would save my producer, and my consumer a ton of API calls to SQS.

If I can batch 49 ‘messages’ per API call, then my 4.3 billion calls, becomes about 87.8 million SQS calls.

87.8 million SQS calls becomes $35.10

Too late for this implementation (Kafka is better in the long run in my opinion), but if the goal were server/infrastructure-less implementation, then shoot. I could have saved $1,684 per month.

Information on SQS pricing can be found here:

https://aws.amazon.com/sqs/pricing/

TL;DR

Batch your data before pushing to SQS, save moneys…. #profit

Short URL: http://bit.ly/2vMgVPy