Wait, This is All About a Vending Machine?
Tackling bottlenecks in DevOps by adding self-service through AWS Service Catalog
In this post of Cloudy With a Chance of Serverless, we will take a sip of cold soda on a hot day by seeing what a little team shuffle, and AWS Service Catalog has to offer organizations of all sizes!
Paved with Good Intentions
“Why, you stuck-up half-witted scruffy-looking nerf herder.”
- Princess Leia, The Empire Strikes Back
Overheard at the Mos Eisley Cantina: If DevOps is meant to tear down silos, then why is it that we are creating teams of DevOps Engineers - Essentially creating a new silo of all of us DevOps nerds huddled together, clicking deploy-to-prod buttons at rapid speeds? Also, we need bigger buttons, and more engineers!
While you are pondering that, let’s take a peek at what generally causes traffic jams in processes, and then possible ways on making Dev and Ops collaboration a bit easier, should an Organization find themselves reevaluating their team dynamics across boundaries.
From my experience, having a core, Cloud DevOps Team has normally been a good start: Rely on a few individuals to get up and running on AWS, put in some automation, then take requests, and eventually realize that the current automation doesn’t solve that problem, and then figure out more automation, and so on and so forth until, inevitably, this small team cannot handle the sheer number of incoming requests. Leading to stressed-out individuals, and Devs still trying to figure out what’s taking so long. If this sounds familiar to you, then keep reading.
The bottleneck doesn’t happen just because there is only a small set of highly skilled AWS Cloud DevOps individuals. It happens because the processes and tools are not in place to let the Developers become DIY-ers. And, I will share why this is not that difficult to put in place!
This Brief Intermission Is Brought To You By Our AI Overlords
“Let the past die. Kill it if you have to.”
- Kylo Ren, The Last Jedi
Okay, so Star Wars went dark, real quickly, right? Most Organizations, while they do not all represent The Empire, seem to have this insatiable need to carry on decades old team hierarchies to keep things, well, stable. However, the AWS Cloud has really disrupted the status quo because almost anyone can create a solution without needing an influx of CapEx, or for that matter highly specialized skills. As a side effect, herein also lies the problem of not adhering to standards and not following best-practices.
The previous generation of DevOps was really focused on automation, and heavy on the Operations side. Nowadays it is about adding more to enable Developers be more proficient, e.g. CICD, while taking advantage of managed services for Operations. In the near future, DevOps as we know it may become less prominent because of the rise of Serverless (read NoOps), and the related built-in automations in server patching and scaling, not to mention advancements in containerization, security and networking. Almost all domains appear to be moving toward providing this model - Imploring us to evolve the way we build teams and assign roles.
What should those whose main job role is DevOps do? In my opinion, everyone should be learning app development in a popular language to stay ahead of the curve. Languages like Python, Node.js, and Rust are probably good candidates overall. Development jobs may likely remain safe as we slowly learn to bow to the GitHub Copilot and Amazon CodeWhisperer like AI systems 😅!
Ticket Fighter II
“Never tell me the odds.”
- Han Solo, The Empire Strikes Back
It’s difficult to fight against a sea of Trouble Tickets, not even in The Millennium Falcon, and you don’t want your DevOps team to be hunted down by Developers around every turn because of slower response times. This is not productive for anyone.
I have a bottleneck destroying proposal for you to think about: Let’s allow Developers the freedom to build, within guardrails, without asking a DevOps team to do this work for them. Allowing the DevOps team the much needed time to look for impactful improvements, instead of always putting out fires. Sounds good? Let’s venture forward!
Step 1. Embed a couple of the DevOps Engineers in a Development team. Don’t have them take on any development work yet, and they must attend Dev stand-ups. This helps in acclimating both sides of the isle, and starts the process of breaking down the silo mentality.
Step 2. The Dev stand-ups will provide valuable insights which will be critical for the DevOps folks to learn from. Enable bringing this feedback to the core Cloud DevOps Team, and draft plans on what can be improved, or what needs to be invented from scratch, to make the day-to-day lives easier for Developers.
Step 3. Implement the improvements through primarily the core Cloud DevOps Team, while allowing the embedded DevOps folks to focus on enabling self-service for either the smaller, existing solutions, or the net new solutions. Pick one or the other, but not both, as it would be too much work to see meaningful gains. Lowest hanging fruit should be the first target.
Step 4: Create standardized, security approved automation. Work with the Dev team to show how self-service would work. This could be a script which is handed off, an infra-as-code Template, or both. Ideally, the solution should be usable by Dev teams without knowing the intricacies or having the in-depth knowledge about the underlying details. The embedded DevOps team stays the owners of these solutions and helps troubleshoot, and rolls out new features as needed. Overtime, rotate folks out. Rinse and repeat. This setup can energize collaboration and help with cross-training as well.
Step 5: If you think Step 4 is easier said than done from an implementation point of view, then let’s take this further and look at one of my favorite services: AWS Service Catalog!
Be Your Own Vendor
“Congratulations. You are being rescued.”
- K-2SO, Rogue One: A Star Wars Story
AWS Service Catalog has come a long way since it’s inception. We’ll look at just the core features in this post - The features which will set you on your way to providing a self-serving, vending machine in the AWS Cloud! The assumption being that this will allow Developers to provision solutions themselves, for example, deploying a standardized CICD Pipeline for their apps.
Whoa there, this sounds like CDK/CloudFormation, I already know this!
Note: There is also Terraform and ITIL tooling support to an extent. See the recap toward the end.
Well, AWS Service Catalog is essentially a governance and repository layer on top of CloudFormation:
It can host your CloudFormation Templates, and allows you to add Versioning (e.g. when releasing new Features, or fixing bugs).
You can distribute these Templates from a central DevOps Account to the whole Organization by way of sharing the Portfolio of your Products - which are just CloudFormation Templates in a logical grouping. Just register this Account as a Delegated Administrator for Service Catalog for this sharing capability. Henceforth, each Template version you decide to add or delete automatically syncs with all the shared Accounts.
You can optionally add constraints, and associate a library of mandatory Tags with drop-down values. Solving this ubiquitous missing-tags conundrum at least for new infrastructure.
The consumers just need to decide which Principals (think IAM Roles, Groups, Users) on their team are allowed to access the shared Portfolio, and then they can vend an instance of your awesome infra-as-code solution whenever they want to!
And there you have it, your very own serverless, and managed, vending machine in the AWS Cloud. How cool is that?
Let’s take a quick look at a few use-cases this can now unlock.
The Engineering Corner
A team of Developers want to provision EC2 Instances in their own Account and Region. These EC2s should have auto-scaling enabled and be behind an Application Load Balancer. There should be an option for which VPC to deploy this under.Solution: Pretty standard request. As DevOps, create a CloudFormation Template with the AMI and VPC/Subnets as Parameters. Ensure the Template works on its own and adjust as necessary. Create a Portfolio in the central DevOps Account with a meaningful name (e.g. “DevOps Portfolio”, or “EC2 AppDev”), add your Template in it as a Product. Add a small constraint on which types of EC2 instances can be provisioned in order to have some control over costs, and share this Portfolio with the Developer Account(s). In the Developer Account, this shared DevOps Portfolio appears in the “Imported” section of the Service Catalog Portfolio list. Work with the Dev team to select this Portfolio, and choose the Developer IAM Role(s) - ideally SSO Roles which are allowed to Provision this solution. Let the Developers then interact with AWS Service Catalog Provisioning section to select and launch as many instances of your solution as they would like. At least now Developers do not need to engage your DevOps team for this type of work since its readily available for self-provisioning!
Some Developers would like their app to interact with an RDS DB when they use your solution. Others would like use a NoSQL DB, specifically, a DynamoDB Table. Solution: No sweat! As DevOps, let’s update the previous CloudFormation Template to include a Parameter allowing the Users to choose between DynamoDB and a flavor of an RDS Engine. Add Conditions to the Template to update the IAM Role and which Resources to provision based on the DB selection. Add a new Product version to the Portfolio with a description showing that this version includes a Database Option. With the auto-syncing power of Service Catalog, your new Versions appear almost instantaneously across all shares. The Developers can simply head back to their AWS Service Catalog Provisioning section and update the existing Provisioned Product, select the new Version, choose which DB they would like to have, and they are off to the races!
Developers would like to have a CICD Pipeline available to deploy their web application to the previously provisioned EC2 instances, under an autoscaling group. Additionally, they would like to do Blue/Green deployments as well!Solution: Bring it on, I say! As DevOps, create a new CloudFormation Template which connects their repo, such as AWS CodeCommit, and adds a build stage using CodeBuild, uses AWS CodeDeploy for the deployment to EC2s, all orchestrated via CodePipeline. Allow the EC2 Autoscaling Group Name, and the Target Group from the Load Balancer to be defined as Template Parameters to make this solution a bit more flexible, and because CodeDeploy will need to know about it. By the way, CodeDeploy can roll-out software incrementally, supports rollbacks, and handles Blue/Green deployments out of the box! It just needs an AppSpec file alongside the application code. Once the Template is ready, distribute this solution as a new Product through the same Portfolio. Any Developer who wishes to add this capability just needs to vend this extra solution after they have deployed their load balanced, autoscaling groups, and they can fill in the Parameters which apply to their workload. CICD added, just like that!
Pro Tip: CloudFormation has a Resource Type for Service Catalog. This allows chaining existing Products: You can create smaller Templates, and then use these to create different combinations of architectures as a new Product. Furthermore, since Custom Resources are supported by CloudFormation, your Service Catalog Product can also invoke your own business logic - This really takes your infra-as-code solutions to the next level when you can utilize the power of Lambda Functions!
Final Thoughts and Takeaways
“The First Order wins by making people think they are alone. We’re not alone. Good people will fight if we lead them.”
- Poe Dameron, The Rise of Skywalker
Understand the pain-points of the consumers by participating in their stand-ups on a set cadence. This will require some Manager-to-Manager dialog, colleague to colleague discussions, and Scrum Masters may help facilitate this collaboration.
Keep self-service as a goal and encourage code contributions to all automation solutions. If you have a Cloud Center of Excellence which sets architecture standards, then pass automation solutions through it as well.
If CDK and CloudFormation is not your cup of tea, then there is also Terraform support to an extent. See this and this. Along with Service Connectors for Jira and ServiceNow integrations.
Make a serious effort in moving away from managing servers and start integrating Serverless computing, as less maintenance will save a lot of Operations headaches, and will minimize 3am wake-up calls.
Setup your DevOps team to actively dedicate time to do Lunch-and-Learns, and demos, with an open invite. As a group, explore different architecture patterns, data lake automations, machine learning setups, security policy enforcements, improving logging and monitoring, and serverless and containerization options. This could also be watching interesting topics from the AWS YouTube Channels, as a team, when seeking best-practices! Overall, this helps in continuous learning and continuous development: CLCD = )
You made it! That was a lot, and I have a good feeling that your teams are ready to take it to the next level, and get out of the day-to-day grunt work. You can do this. Start small. Experiment. See what works for your teams and your Organization, and make adjustments as needed.
Call for Action: Talk to your team and identify two to three solutions which are good candidates for being part of a self-service pattern, and use AWS Service Catalog for its implementation.
Thank you for reading, and I hope you found my opinions valuable or at the very least thought provoking. See you in the next post!