De-Coder’s Ring

Software doesn’t have to be hard

Month: March 2017 (page 1 of 2)

Blog Name???

Apparently I need a new blog name.

People don’t like “Chris’ Blog”

That’s understandable.  It’s boring.  It’s not content descriptive. It doesn’t show my personality.

This writing and creative stuff is hard work for me, maybe I name it something like:

“Forced Technicals”

“Reluctant Writer”

“I write cause I should”

“Technical Thoughts from a Non Writer”

Any ideas?

 

Cybersecurity Arms Race: The Nuclear Option

I see you

Sandboxing files and detecting unexpected system behaviors is one of the best approaches to finding exploits.  FireEye did it really well when they first came out with their network monitoring products.   Watch a network, extract files, shove into a sandbox, explode, see what happens.  They were credited with finding a ton of 0-day type events.  Now, we can do the same with open source software.

Then you hear about malware that can detect it’s in a sandbox or a Virtual Machine.   If it detects the virtual environment, then it doesn’t explode, doesn’t infect, doesn’t do the bad things.  Then we invest money into figuring out how to hide the virtual host or sandbox from the malware.   Arms race!   Who can do it better.   I hide, you detect, you hide, I detect.  It’s one example of the cybersecurity arms race.

In traditional warfare, the winner of the arms race has a bigger gun.   Well, a bigger stick, then a  bigger rock, then a bigger bow, gun, missile, etc.  There’s an end game there.   The nukes.  Whoever has the nukes is on top.   Even when the foe has a nuke, the arms has can’t continue.  We’re at a stalemate.  Mutually assured destruction if we all use our nuclear arms.  When we all have the biggest gun, none of us can use it.    (another blog post later about moving the traditional warfare to the cybers, but that’s for later.

What’s the nuclear option for cyber war fare?

The closes thing I can think of for mutually assured destruction would be around taking down the Internet as a whole.  It may not even be possible.   Can someone wipe out all the core routers, heck, all the routers in the world?   Is that the end?   It makes me think of my favorite definition of envy (vs Jealousy).    jealousy is “I want what you have”.. not totally bad, can help motivate someone, etc.    Envy is “I want what you have, but since I don’t, you can’t have it either”.   Going nuclear on the Internet would drastically affect every life on the planet (ok, maybe not every, but anyone who’s in a ‘civilized’ place).  If it’s even possible…

Law Enforcement Backdoor – Wide Open Spaces

ATTACK!

The news this past week has been all about the terror attacks in London.  This is a horrific event.  The pointless loss of life can never be allowed and law enforcement needs to do all it can in order to stop it.   Law enforcement / governments should be asking for a back door in encryption technology.  They should be trying to crack it.

We as citizens and technologists, can not allow them to succeed.   We certainly can’t give them a back door to read all encrypted traffic!

Some of you may be thinking, “wait, those two statements are contradictory!”.   I disagree.  We, the general population, have a different set of motivations than the federal government.   They’re not necessarily competing, but they’re not the same.   The government should be trying to break encryption.  If this stops some nut job countries who are trying to launch ICBMs at us, then yes! Stop them!

If they’re using broken encryption to spy on US citizens , then heck no, we won’t go.

What is encryption?

Encryption is  a mathematical scheme that allows data to be transferred between two or more parties.  Typically, modern encryption for the web involves a private and a public key.    Party A (me) will share my public key with you.   Party B (you) will share your public key with me.   Within apps like WhatsApp or Signal handles the keys behind the scenes.  The math allows me to alter data, using your public key.  Then, I send you the data that you can use your private key to decrypt.  It’s big math, that I definitely don’t understand, but it’s near uncrackable.  The longer the keys, the harder it is to decrypt.

That sounds bad for law enforcement, let’s give them a back door

A backdoor is a short cut. It’s a ‘universal’ key that allows the key holder to read the contents.   Think about it like a letter in the mail.  I drop a letter in the mail, and since I glued it shut, no one else can read it (bear with my on this analogy).  Now, law enforcement wants to make sure they know we’re not doing anything illegal.   They want to peek into your envelope.  They want a secret code to open a secret flap on your envelope that only they know how to open.

Genius, right?

Ultimately, putting the back door in place for Law Enforcement is an awful idea and will only open the doors to the wrong people using the back door.

Imagine this.  You’re at home, and the police department calls you.   “Hey Mr Jones, we want a back door so we can know when some one is robbing you.  We want to prevent rape, murder, pillaging etc.   Just build a new entry way, and hide it.   Only we’ll know where it is”.   They  make that call to everyone in the neighborhood, and everyone complies, because you know, “we have nothing to hide!”

All goes well for a year or so.. then all of a sudden, things go missing in everyone’s house.   Identity theft goes on the rise and we can’t figure out what happened.

Remember that law enforcement back door?  Yeah, the little hoodlum down the street found it on your house.   Then your neighbors house.  Then all the houses.

Think about it from the law enforcement agency (and this may be getting real, in today’s society).   Since they have the back door, they can now log all encrypted internet traffic for finding later.  Remember that bad thing you said about the president elect?  Now he’s president…. and doesn’t like people who say bad things against you.   Once he does a search to find who said bad things.. then.. he’ll do bad things *

Long story short, backdoors for law enforcement can sound good to some people.  If they’ll prevent terror attacks or stop crimes ‘before’ they happen, then that’s awesome.  The problem though, is that we would lose our privacy.  That’s a non-starter.

 

from passiveaggressivenotes.com – http://www.passiveaggressivenotes.com/2012/11/26/nothing-to-see-here/

 

  • This is not a real scenario in the US.. Thank God on that…. let’s pray that it doesn’t ever get to that.

Two (or Three?) Tips to Improve AWS SQS Throughput

I have software that is processing approximately 200k records per 5 minutes from SQS.   That’s about 40k/ minute, or 666 records per second.  Not bad!

The only problem is, I had to run 25 instances of my python program to get that through put.  Ummm.. that’s not cool.. it should be WAY faster than that.  Shoot, my back end is elasticsearch, which I KNOW can EASILY support dozens of thousands of inserts per second.

My code isn’t terribly so either, so I was definitely bound up on the ingest from SQS.

Even batching, reading 10 messages at a time, didn’t help.. well, it did.. reading one message at a time was abysmal.

I kept reading about ‘infinite’ throughput with SQS, and figured out how to do it. Well, computers aren’t infinite, so that’s not exactly right, but, I was able to linearly increase throughput by having multiple consumer threads per processing thread in my code… since I then output to another SQS queue, I have multiple OUTPUT threads.    Now, the code I wrote is a multi-threaded beast.

  1. Multi thread your input, Use something like python Threading and Queues, or any other threading library with 0mq
  2. Multi thread your output.. same thing.
  3. .. bonus tip, be nice to AWS

Think of your code as a pipeline, because that’s really what it is.  Input -> Processing -> Output

Pipeline that mess

If you have multiple threads reading from SQS, they can each block on IO all day long. That type of thread should be super duper light-weight.  Essentially, read from the SQS Queue and shove the record into your Thread safe work queue (0MQ or a python Queue).    I’m a huge fan of Python since I can code and get something working fast and efficiently.  Sure, I could GoLang it, but Python makes me happy.

Here’s a quick code snippet of an example SQS reading thread:

 

Like I said, super simple.  That “outbound_queue” object is a Python Queue.  This is a thread safe FIFO Queue that can be shared. In my example, I have a single consume thread that reads the objects that my ReceiveMessageThread puts in.   Once they’re read and processed, then, I have the same construct for down-streaming messages to another SQS Queue.

Oh.. here’s the second (bonus?) tip, I apparently can’t count today.

See that “WaitTimeSeconds=5”?    My first attempt didn’t have that Wait.  What would happen then, would be a lot of “Empty Receives”. that ‘messages’ array would be empty.  No big deal.  Code can handle it.. and I’m sure AWS doesn’t mind too much if you spam them, but I figured I’d try not to DDOS them…. check out the graph…. this is the difference between not having a WaitTimeSeconds and having one.

Less Empty – Full?

 

See that HUGE drop?   yeah, I stopped DDOS-ing AWS.  I’m sure that’ll save me some money to, or something like that.   OH.. Dang.. I just read the pricing for SQS. Yep.  That’ll save me cashola.  Each API call (receive_messages) counts as an action.   Don’t tell my boss that I didn’t do the timeout.. ha!

After these updates, I’m able to process the same 200k records per 5 minutes, but now, I only need 3 instances of my code running.    That’ll free up a TON of compute in my Elastic Container Services cluster.

 

Older posts

© 2017 De-Coder’s Ring

Theme by Anders NorenUp ↑