De-Coder’s Ring

Consumable Security and Technology

Author: Chris Fauerbach (page 1 of 21)

AWS Fargate: What ECS Should Have Been in the First Place

Introduce AWS Fargate

AWS has been able to manage and run Docker containers for a long time using ECS, or Elastic Container Services. I found that it’s difficult to operate unless you start from an understanding and with infrastructure as code. In startup mode, that’s not always easy, so I led myself wrong and got stuck in a manual maintenance mode of ECS.

ECS allowed you to store Docker images in a registry, like you would in Dockyard or any other Docker registry. You could then create Tasks, which was the definition of how to run the Docker image.   The task configured things like port forwarding, disk mounting and kept a link to the tagged Docker image.   Next step to actually run the Task was to set up a Service.  The service provisions a certain number of tasks to run. Then you configure which ECS Cluster to run the tasks on. Finally, the conditions to auto scale the service by adding or removing instances of the tasks/image.

Here’s a snippet from the AWS blog:

 

AWS Fargate is an easy way to deploy your containers on AWS. To put it simply, Fargate is like EC2 but instead of giving you a virtual machine you get a container. It’s a technology that allows you to use containers as a fundamental compute primitive without having to manage the underlying instances. All you need to do is build your container image, specify the CPU and memory requirements, define your networking and IAM policies, and launch. With Fargate, you have flexible configuration options to closely match your application needs and you’re billed with per-second granularity.

Fargate solves an important pain point with ECS.   The ECS cluster of EC2 instances.  You, as an administrator/devops engineer on your AWS account, needs to provision a ECS cluster.  That’s a fancy and abbreviated way of saying: “Create an autoscaling group, using a launch configuration that has User Data configured to join the ECS Cluster that you’ve defined.”….     Not difficult by any stretch, but, it’s  always felt like a layer that shouldn’t be there.

I always found auto scaling to be a challenge.  Are you autoscaling your ECS Service?  Are you auto scaling the ECS cluster?   Yeah, you kind of have to do both.

Frankly, since I was in startup mode when dealing with ECS primarily, and believe me I dealt with ECS a ton, I never took the time or bothered to figure out how to ‘get it right’…

I got it working.   In startup land, getting it working is first and foremost… getting it done “right” is secondary (as long as you aren’t getting it done awfully poorly)….

Back to the point of Fargate. This is a major simplification of the ECS/Docker process.  Now, you can configure a group of ECS tasks to run without configuring the EC2 cluster.

The magic happens behind the scenes, managed 100% by AWS.  You are even presented with a “Fargate” cluster when you look at your ECS clusters in the web user interface for ECS.

 

Amazon has taken away the need to be particular about how your tasks are running across your instances.  You don’t have to stress about making sure you’re using your ECS Cluster optimally. AWS takes care of scaling your tasks to meet your jobs’ needs.

This simplification will now make Docker containers a first class citizen within AWS.   This is a huge change and will definitely streamline administration and provisioning of your containers.

 

 

Startup Series: Should you join a startup?

Wow!

The response from my first article was huge!  I got so many DMs on Twitter that I decided to rush out the next post about the startup life.

Today, let’s talk about the decision you have to make when you leave your cushy/comfy/reliable job.

My experience has been someone ‘hired’ by the startup founders.  Starting your own gig is a very different beast.  Listen to my podcast with Nick McCarter, CEO of Chartis Federal (link below).  He gives a great, honest run through of his mindset and stage of life when he founded Chartis.  He even said he wouldn’t do the same if it were today! WHAT?!   Timing matters!

If you get approached for a role at a startup, you have a lot to consider.  There are two huge reasons that people will leave a large company and go to something tiny.

1) The lifestyle and day to day of a tiny company is agile, never stale and there typically isn’t much in the way of politics to deal with.

2) The old Risk vs Reward scenario.  If you’re in early enough, you can get a big, big payout.

Let’s break both of these down, of course, lined with my own opinions on the matter!

Startup Lifestyle

My favorite time at a startup is the first year.  In that time, I have found that roles are fast and loose.  This is the epitome of ‘GSD’.   If you aren’t familiar with GSD, well, I can’t tell you, since this is meant to be family and work friendly… I mean, it stands for: Get Stuff Done.  There is no other time in a startups life where you can have an exhilarating morning, followed by a panic stricken afternoon.  That is the wildest roller coaster.  It’s not the same as a little belt-tightening at your Fortune 500 company.   This is ‘do or die’, ‘shut the company down’ levels of panic.   Oh wait, I was talking about GSD.

Let me tell you one of my favorite stories from Perch Security.   I started on day one of Perch as VP Engineering.  Our overall task was to help companies make their community sourced intelligence actionable.  Huh?  Companies that are part of nationally important verticals (financial services, healthcare, etc) are members of communities that share cyber threat intelligence.  We built a product and service focused around enabling companies to use that data.   We built out our prototype, had a few clients, and it was all good.  One day, we woke up to the news about a report out of US-Cert regarding Grizzly Steppe, a report focused around some Russians hacking something or other..  There’s always news in cybersecurity of another threat.  It happens all the time.   If you’re not in the industry, you’d be surprised how many times new information is shared.  The point of this story though, is that our platform HIT on the Grizzly Steppe indicators without any intervention!  It was one of those ‘Ah Ha!’ moments when we truly saw our automated processes were not only working, but they were providing actionable alerts.   Unfortunately (or maybe fortunately?) in that case, they were all false positives, but it was something awesome to truly celebrate!

Risk Vs Reward

Everyone thinks they’re going to hit it big when they join a startup.  As long as they get some stock options, they’ll be set for life!  Heck, ever hear the story about the painter @ Facebook who eventually made millions?  Instead of getting paid to paint a wall, he got some stock options. .BAM.  Millions!

Guess what?  That doesn’t happen!  Ok, obviously it happened once, but, odds are that it’s not going to happen to you.  nPulse was acquired by FireEye back in 2014.  I was employee #1 after the founders.  I didn’t make $200 million like Choe.  Dang, that would have been sweet.  The money I made was fantastic.  Life changing, but I couldn’t retire!

There’s a reason why tech companies that reach $1B in valuation is called a unicorn.  The odds of it happening are slim to none.   That doesn’t mean you can’t do well, but, make sure to level your expectations vs reality.  Top 5 employees can make some good money.   Next 5 can make some money.  After that?  Who knows.   There are so many situations that can change those ranges (1-5, 5-10, etc), but if it’s a relatively small company at acquisition, it’s probably true.   If you joined Facebook in the top 20, well, you’re probably a multi-gajillionaire, so I’m not talking to you.

I’ve realized if you want to retire, if you want to make enough money to not HAVE to work, you have to start the company yourself.  If you’re successful, then you can be rewarded as you deserve.  No one deserves more than founders, and if you think your founder got too much in a liquidation event, you’re wrong.  They deserve every penny, because they put their life on the line.  Ok, not alive/dead life, but family, financial and social life.  They put all that at risk to follow an idea, a passion or a dream.  The founders of the startups I’ve worked for or advised have all worked ungodly hours.  They are stressed out, because everything is riding on them, especially in the early phases.  You may be an amazing engineer (like me, right?), but you are an employee.  You don’t get to expect the same levels of acquisition compensation that the founder of the company gets.

Does that mean you shouldn’t join a startup?

Not necessarily.  It may be a helluva lot of fun.  You may learn enough to do your own thing.  You might learn that you don’t want anything to do with a tiny company.  You may find peace, or fun at a large company.  Heck, you may be able to influence a much larger audience at a Fortune 500 company.   Over time, you might actually make more money if you’re wise with it, especially compared to taking a lower than market salary based on the hope of stock options that never pan out.

What’s your best or worst startup story?   Someone ever make more than you on an acquisition that “shouldn’t” have in your mind?   Tell me the (appropriately anonymized) story, I’d love to commiserate!

— chris.

LINKS:

Podcast: Interview with Nick McCarter – CEO, Chartis Federal

This week’s interview is with an old friend, Nick McCarter.   Nick is the CEO of Chartis Federal, a government contracting company with a heavy focus on security and secure communications.

Mr McCarter, formally, walks us through the process he went through to start his company with some advice for future entrepreneurs.

This interview really helped me answer some of the ‘what if’ scenarios, as well as alleviate some of the self doubt that we all go through.

 

Thanks Nick!

Chartis Federal

Startup Series: Life at a Startup

My colleagues know (since I talk A LOT) that I’ve had a long loving history for startups.  Like, Honest to God, hired by the founders as the first engineer (twice) kind of startups.

The first one was a very slow process to go full time at nPulse Technologies.  Randy, the founder, and I had been friends for years.  He started this packet capture company as a lifestyle company.  Something fun to do, enough to make a living, but that’s all.  For a while, I’d bill him $500/month to write a web app and some APIs to pull packets off this appliance.  We didn’t do the SaaS thing, or the PaaS thing.  We built a linux based server that had our software running.  After a few years of that, he and his co-founder decided they wanted to make nPulse a real thing.  He pulled me in as VP of Development, and it was off to the races.  By the end of the first year, we had approximately 5 full time employees.  2 more years, and we were up to 30 by the time we got acquired by FireEye.    I stuck around FireEye for a bit, but decided it wasn’t for me, and went to a big bank.  It had been approximately 10 years since I worked for a Fortune 500 company.

After two years of ups and downs (yeah, I keep it real in my blogs), I got a call out of the blue to join another company as VP Engineering.  This one was different.  I started day one with another developer at Perch Security.  The founder hadn’t quit his day job yet, but he had landed enough funding for us to get started.  I got to build a network monitoring appliance, that shipped MASSIVE amounts of data to a cloud service running at AWS.  There, data went through a pretty big orchestration to ultimately land up in Elasticsearch for storage and search by customers.   Speaking of customers, we started signing them up early, and often.  I stayed there for 14 months, until I was confident the infrastructure and code was solid, and due to many reasons, came back to the previously mentioned big bank.

Here are a few observations:

Startups are HARD

There’s no such thing as a slow day, if you have customers.  Customers demand quality (duh) and if anything goes wrong, you have to fix it immediately.  There’s no “oh, I’ll fix it when I come in in the morning”.  Small bugs, big bugs and crashed systems.  It was critical we kept everything top notch, especially when we were trying hard to find new customers, and leverage the good will and good word of our early ones.

Scaling is HARD

Five network sensors is easy.  It looks like things will scale, since all the tools are there to scale…. then one day, it stops scaling.  Add nodes, it still doesn’t scale.   Something is wrong.  Rewrite… and fast.  I switched data platforms three times with no downtime or loss of data.  Using intermediary queues like SQS, Kafka, etc is critical for scaling.

Building things is fun!

I shine when I get to build new things.  Give me a whiteboard, and I can fill it up with a pretty darned good solution.  Building an MVP is my dream job.  I get to write just enough code to prove a point, or try out a new approach.   Then it gets harder

You can make a big impact

You can make a big impact at a large company.  You don’t need a tiny company to make a big impact.  Heck, I think I make a bigger impact here.   At my last startup, Perch Security, I had a team of Cyber Security Analysts and a team of engineers.  We were up to around 30-35 customers.   That was awesome!   I could say “I built this!”..    at the bank, I’m supporting our messaging platform as the embedded technical lead, technical platform owner, whatever you want to call me.  A huge enterprise platform with over 150 developers that sends notifications and emails to every customer account holder…   Talk about an impact!

Good and Bad

Startups can be a blast.  They’re not all foosball and free lunches.  It’s collaboration at its finest, because you know everyone involved is onboard 100%, or they will lose their job.  Not by being redeployed in a down economy, but, because if they fail to deliver, the company goes under.

When a startup succeeds, and grows, and gets acquired, then it can be REALLY rewarding for those early folks (I’m still holding out hope on my Perch stock!)

Want to talk about startups? hit me up!.

Podcast – Jon Bodner

This week, we branch out a little from the previous topics of Cyber Security, which we’ll be back to soon, don’t worry.   (that was a lot of commas in that sentence)

Today is an interview with Jon Bodner.  Jon is a very popular writer and speaker on all things software development, with a new focus on the Go programming language.  I’ll add a few notes here to find Jon’s various talks, but, please subscribe and listen to this episode!

You got your engineering in my data process

GopherCon2017 – Runtime Generated, Typesafe, and Declarative: Pick Any Three

The Story Behind Capital One’s Fork Of An Open Source Project

Older posts

© 2017 De-Coder’s Ring

Theme by Anders NorenUp ↑