I made a bone head move. Yes, I admit it.
Amazon SQS has always been talked about as ‘free’. In terms of passing messages for an application, it’s supposed to be freaking cheap as can be.
I was blown away when my July 2017 SQS bill was $1720!!
What? How’s that FREE?!
Digging into my SQS reports, I made 4.3 billion (with a B) SQS calls. Billed at $0.0000004 per call, that adds up to $1,720!
Well, my architecture would only be scaling up from there.. I had to do something about it.
I moved to Kafka.
… but that’s not the point of this post. I realized, later, that I could have been close to free, and optimized a ton of my downstream pipeline.
SQS messages are billed on the 64k of data chunk. My messages were averaging 1,300 bytes (1.3k). Doing some quick math, I could have been batching up to 49 ‘messages’ at a time per SQS call. This would save my producer, and my consumer a ton of API calls to SQS.
If I can batch 49 ‘messages’ per API call, then my 4.3 billion calls, becomes about 87.8 million SQS calls.
87.8 million SQS calls becomes $35.10
Too late for this implementation (Kafka is better in the long run in my opinion), but if the goal were server/infrastructure-less implementation, then shoot. I could have saved $1,684 per month.
Information on SQS pricing can be found here:
Batch your data before pushing to SQS, save moneys…. #profit