De-Coder’s Ring

fauie.com: eclectic technology, gentleman farming and careers

Category: software development (page 1 of 3)

Python Callback Function Declaration / Definition

I’m working on a metrobus based project, and was curious how to define a flexible callback method that had a required parameter, and other keyword arguments.   I have used *args and **kwargs before, but wanted to really experiment on what I could do with them.   For metrobus, I’m only going to ‘require’ a ‘message’ parameter (single parameter callback function), but, I’ll have to use **kwargs on my declaration just so I can be a little more picky about what I receive each time.   Check out some experimentation:

 

Announcing: Metrobus – A framework to simplify message bus/Kafka based microservices

I’ve been wanting to write this code for a while.  If I were as efficient at writing in Java, I would have done so, but, I’m slower and therefore wrote it in python.  It’s not very pythonic, but, whatever.

https://github.com/chrisfauerbach/metrobus

Metrobus is a framework that allows you to focus on your microservices application logic, not logic around pulling and pushing to Kafka.    Kafka for now.  Others to come soon.  All the details you could ever want on the github page.    I’ll be adding tickets in order to track some todo items.

metrobus

Proof of concept and example for smart routing on a dumb bus.

This is a small project focused on my blog posts around routing on a message bus that’s dumb. Like Kafka.

Stateless and dumb: https://medium.com/capital-one-developers/stateless-and-dumb-microservices-on-a-message-bus-be78bca93ccb

Fast Cache: https://medium.com/capital-one-developers/blazing-fast-data-lookup-in-a-microservices-world-dd3ae548ca45

I use some of the caching ideas in here for a few of my data lookups.

The concept for the ‘test’ example application is fairly simple. The ‘pusher’ generates records, as if from a client or application. These records are simple JSON structures. For our example, we get cool things like an account number (FAKE!). The ‘pusher’ sends the message to the ‘Source’ topic on Kafka. Consider this your public entry point for upstream clients.

 

KNOWN TO DO ITEMS (Cause you know, SHIP IT!)

  • Simplify the logic or break up the code for the main handling function in metrobus.py
  • Determine best how to handle different situations like
    • Send to Error log
    • Send to dead letter queue
    • Dropped on purpose
  • Is it too simple?   what am I missing?

 

The Inevitable Decline of Your Software… and how to prevent it, from inception to delivery

Thanks for putting up with my silence!    I have a few new projects in the works, and it’s hampered my blogging.   I think that will be OK in the end.

Throughout my software engineering career, I’ve seen a lot of bad and some good.  Some of the groups I work with now do amazing work and have inspired me to keep writing a book I started a long time ago.   I’ve updated it significantly and will continue to do so.     I judge I’ve knocked out about 50% of the first draft and wanted to see judge interest to see where I need to prioritize writing of it.

Leanpub allows me to write it and sell it at the same time.   I’m putting out a big discount code here, in an effort to get some feedback.     Check it out!

http://leanpub.com/inevitabledeclineofsoftware/c/Z6fFzHxT2IHb

I hope the coupon limit gets hit quickly.  If you miss it, shoot me a note and I’ll make another one!

 

Second plug..     we’ve started a really fun podcast:

https://goaskyourdadpodcast.com

Direct link for itunes/IOS podcast app!

https://pcr.apple.com/id1381366968

 

Thanks!  Would love to hear some feedback.

 

Technology: The first class citizen

I’ve spent the last few days at Capital One’s Software Engineering conference.   How cool is that?   Hundreds of techs folks gathering for a few days to discuss areas of technology.   These are modern stacks of technology, processes and new paradigms.

For me, I’ve been able to watch about a half dozen talks on Machine Learning, the programming language Go and encryption.  The speakers were excellent, and, if I play my cards right, I’m going to work to get a few of them on here as guest bloggers!

What topics would you want to hear about?

Security: Code and Passwords

Developer’s jobs aren’t easy.  Constant deadlines, integrating new technologies… dealing with ‘Ted’ in the cube next to you that shouldn’t be eating those onion rings… you get it..  lots of issues.  #notsnowflakes

stressed out developer

stressed out developer

Now we’re forced to live in this modern world of devops.  No longer can we rely on system administrators to maintain systems.  No longer can we rely on release engineers to package and ship our code.  Now we own it all.

Some of us adapt.  Don’t get me wrong, it’s not an easy task.  Most of us don’t have enough linux-foo, or the ingrained processes to maintain a large elasticsearch cluster… but we cope.  We learn new skills, grow in breadth of knowledge… then that breadth gets deeper.  Holy cow, we’re valuable now!

Unfortunately, security still is not a top tier concern for most software engineers.  We have web exploits to worry about.  We have to worry about SQL Injection.  Stack overflows, kernel panics, all kinds of neat stuff… each of which is the beginning of a piece of vulnerable software.

The one that continues to kill me, and I have this feeling was behind a major breach in the US this week, has to do with account and environment credentials.  There are so many scenarios that require an application to know about credentials:

  1. Database connectivity
  2. External API/Service
  3. Mail servers

tons more.  how do we deal with it?

There are a few anti-patterns

… bad things.. don’t do these.

  1. Hard code the credentials in your code
  2. Use a configuration file, check it into source control
  3. Use environment variables in your public facing website to connect to your super secret database

Those are all dumb.  Don’t do anything.

What can we do?

Separation of connectivity.   Your web application shouldn’t call your database directly, especially if it’s a database with customer data, personally identifiable information or healthcare info.  That’d be dumb.   Connect your web application to an API layer , but still follow some of the ‘other’ advice below.

Supply the passwords at runtime

Use a password vault/key management system to supply passwords to an application.  Build that out into your application framework so your code doesn’t have to be aware of where the password came from.   A password vault is a high security system that allows an authorized application to make a secure request to get private information from.  For instance, your vault could store the ‘production customer database’ information. This could even be information about the host name, port, username and password of the database.

Different environments get different credentials

This one is pretty obvious, but sometimes even the best of us don’t follow this to a T..   ummm…. no, not me.. others..  yeah.. others.   Just like your web sites, always have different passwords for everything.  Don’t reuse credentials in a QA environment and a production environment.

Provision as much as you can in configuration

Putting configuration items, or items that MAY become configurable in code is a bad move.  You’re gonna have a bad time.

You're going to have a bad time

You’re going to have a bad time

Always use configuration files. In the example above, the configuration file would tell your application where to find the password vault. Not the passwords or even the database configuration.

Act like your data is exploited

This point goes kind of against the other development tips.  When building applications, always remember that there’s a chance that the database ends up on the internet.   No one wants to think about it, but, look at Equifax.  Look at Deloitte.  Look at Aetna. Target.  etc.   They got owned, and you very well may too.   Don’t live in fear, but, live in paranoia!

 

Older posts

© 2018 De-Coder’s Ring

Theme by Anders NorenUp ↑