De-Coder’s Ring

Consumable Security and Technology

Announcing: Metrobus – A framework to simplify message bus/Kafka based microservices

July 12, 2018 / / 0 Comments

I’ve been wanting to write this code for a while.  If I were as efficient at writing in Java, I would have done so, but, I’m slower and therefore wrote it in python.  It’s not very pythonic, but, whatever.

https://github.com/chrisfauerbach/metrobus

Metrobus is a framework that allows you to focus on your microservices application logic, not logic around pulling and pushing to Kafka.    Kafka for now.  Others to come soon.  All the details you could ever want on the github page.    I’ll be adding tickets in order to track some todo items.

metrobus

Proof of concept and example for smart routing on a dumb bus.

This is a small project focused on my blog posts around routing on a message bus that’s dumb. Like Kafka.

Stateless and dumb: https://medium.com/capital-one-developers/stateless-and-dumb-microservices-on-a-message-bus-be78bca93ccb

Fast Cache: https://medium.com/capital-one-developers/blazing-fast-data-lookup-in-a-microservices-world-dd3ae548ca45

I use some of the caching ideas in here for a few of my data lookups.

The concept for the ‘test’ example application is fairly simple. The ‘pusher’ generates records, as if from a client or application. These records are simple JSON structures. For our example, we get cool things like an account number (FAKE!). The ‘pusher’ sends the message to the ‘Source’ topic on Kafka. Consider this your public entry point for upstream clients.

 

KNOWN TO DO ITEMS (Cause you know, SHIP IT!)

  • Simplify the logic or break up the code for the main handling function in metrobus.py
  • Determine best how to handle different situations like
    • Send to Error log
    • Send to dead letter queue
    • Dropped on purpose
  • Is it too simple?   what am I missing?

 

Blockchain ruins a project before it begins – Marketing is hard

June 30, 2018 / / 0 Comments

I have a genius idea (if I do say so myself) that I’m going to implement.  It’s going to be based on blockchain (#notacoin).  When I ran just a super high level past a friend of mine, it was almost shot dead in the water because it had blockchain as part of the implementation.    I guess I didn’t need to go into that detail, but alas, it is what it is.

There’s a funny thing about ‘magic bullet’ technology that makes experts scoff at a project or a product.  As a cybersecurity engineer and member of private community groups, anytime a vendor talks about their Machine Learning algorithm, they get immediately laughed at.   Even if the technology can add a value.  The opinion is that 9/10 folks that claim to use Machine Learning either don’t or don’t really add any value because of the technology.  They’re promoting the technology of machine learning more than the problems their solution solves.

Maybe it equates to the fancy whizz-bang marketing language in general.  Just because your marketing team came up with a really fun name for a feature, doesn’t mean that feature is what you promote.  Promote how you solve problems that your customers have.  It’s pretty straightforward.   You have a pain point, and my problems fixes said pain point in a demonstrable way.  Not necessarily the ‘how’ the code is implemented or what back-end tech (ML, AI, Blockchain), but how those solutions materialize to your customers.

 

The Inevitable Decline of Your Software… and how to prevent it, from inception to delivery

May 19, 2018 / / 0 Comments

Thanks for putting up with my silence!    I have a few new projects in the works, and it’s hampered my blogging.   I think that will be OK in the end.

Throughout my software engineering career, I’ve seen a lot of bad and some good.  Some of the groups I work with now do amazing work and have inspired me to keep writing a book I started a long time ago.   I’ve updated it significantly and will continue to do so.     I judge I’ve knocked out about 50% of the first draft and wanted to see judge interest to see where I need to prioritize writing of it.

Leanpub allows me to write it and sell it at the same time.   I’m putting out a big discount code here, in an effort to get some feedback.     Check it out!

http://leanpub.com/inevitabledeclineofsoftware/c/Z6fFzHxT2IHb

I hope the coupon limit gets hit quickly.  If you miss it, shoot me a note and I’ll make another one!

 

Second plug..     we’ve started a really fun podcast:

https://goaskyourdadpodcast.com

Direct link for itunes/IOS podcast app!

https://pcr.apple.com/id1381366968

 

Thanks!  Would love to hear some feedback.

 

ROS Tutorials Live – A few to start

April 5, 2018 / / 0 Comments

Dang, I’ve been quiet for a while.  Been focusing heavily on my written word at Medium    ( https://medium.com/@chrisfauerbach )

Over the past few weeks, I’ve started learning about ROS again.  The Robot Operating System.    I previously spent months looking at it for a robotic project that was left unfinished, but I want to get in on it again!    Here are a few videos of ROS tutorial walk throughs

 

 

 

 

AWS Fargate: What ECS Should Have Been in the First Place

December 5, 2017 / / 0 Comments

Introduce AWS Fargate

AWS has been able to manage and run Docker containers for a long time using ECS, or Elastic Container Services. I found that it’s difficult to operate unless you start from an understanding and with infrastructure as code. In startup mode, that’s not always easy, so I led myself wrong and got stuck in a manual maintenance mode of ECS.

ECS allowed you to store Docker images in a registry, like you would in Dockyard or any other Docker registry. You could then create Tasks, which was the definition of how to run the Docker image.   The task configured things like port forwarding, disk mounting and kept a link to the tagged Docker image.   Next step to actually run the Task was to set up a Service.  The service provisions a certain number of tasks to run. Then you configure which ECS Cluster to run the tasks on. Finally, the conditions to auto scale the service by adding or removing instances of the tasks/image.

Here’s a snippet from the AWS blog:

 

AWS Fargate is an easy way to deploy your containers on AWS. To put it simply, Fargate is like EC2 but instead of giving you a virtual machine you get a container. It’s a technology that allows you to use containers as a fundamental compute primitive without having to manage the underlying instances. All you need to do is build your container image, specify the CPU and memory requirements, define your networking and IAM policies, and launch. With Fargate, you have flexible configuration options to closely match your application needs and you’re billed with per-second granularity.

Fargate solves an important pain point with ECS.   The ECS cluster of EC2 instances.  You, as an administrator/devops engineer on your AWS account, needs to provision a ECS cluster.  That’s a fancy and abbreviated way of saying: “Create an autoscaling group, using a launch configuration that has User Data configured to join the ECS Cluster that you’ve defined.”….     Not difficult by any stretch, but, it’s  always felt like a layer that shouldn’t be there.

I always found auto scaling to be a challenge.  Are you autoscaling your ECS Service?  Are you auto scaling the ECS cluster?   Yeah, you kind of have to do both.

Frankly, since I was in startup mode when dealing with ECS primarily, and believe me I dealt with ECS a ton, I never took the time or bothered to figure out how to ‘get it right’…

I got it working.   In startup land, getting it working is first and foremost… getting it done “right” is secondary (as long as you aren’t getting it done awfully poorly)….

Back to the point of Fargate. This is a major simplification of the ECS/Docker process.  Now, you can configure a group of ECS tasks to run without configuring the EC2 cluster.

The magic happens behind the scenes, managed 100% by AWS.  You are even presented with a “Fargate” cluster when you look at your ECS clusters in the web user interface for ECS.

 

Amazon has taken away the need to be particular about how your tasks are running across your instances.  You don’t have to stress about making sure you’re using your ECS Cluster optimally. AWS takes care of scaling your tasks to meet your jobs’ needs.

This simplification will now make Docker containers a first class citizen within AWS.   This is a huge change and will definitely streamline administration and provisioning of your containers.

 

 

« Older posts

© 2018 De-Coder’s Ring

Theme by Anders Noren — Up ↑