Why reinvent the wheel? Use existing tools.
Software engineers love to solve problems.
But just because we could build something ourselves, doesn’t mean that we should.
If a tool can deliver a better result at a fraction of the cost, we shouldn’t be building it ourselves.
For us, that service is the Gruntwork Reference Architecture.
Our experience is that the Gruntwork Reference Architecture is 10x faster, 10x higher quality and has been up to 1000x cheaper*.
In this article, we will discuss the advantages and disadvantages of not writing infrastructure code ourselves, what we did instead, he problem with DevOps consultants and possible security issues.
Some companies are still opting to bypass tools like Gruntwork and opting to build their own. We think this is a better way.
Find a tool that you can configure to your own standards.
We want flexibility. But we don’t want to build it ourselves.
We used to write our own infrastructure using open source cloud formation templates. But, that left much to be desired and many of the import details like user and ssh key management, secrets management and CI/CD pipelines to be solved.
The team probably should have started out with a Platform as a Service solution like Heroku or Convox. With a PaaS the team would have gotten up and running faster and avoided the technical debt and deficiencies of designing the infrastructure from scratch.
And while fully PaaS solutions are secure, they're not flexible enough.
If you’re just getting started, Platform as a Service can be a great option as they provide a simple API for deploying your code.
But, that means they can be a big bottleneck as you grow. Without more control it’s difficult to debug, scale, customize your infrastructure, and keep costs under control.
As a result, most software companies migrate to an Infrastructure as a Service (IaaS) provider such as AWS, Google Cloud, or Azure.
This gives them more control but comes with a massive increase in complexity. Few companies have the expertise, time and resources to do this well.
That brings us to the Gruntwork Reference Architecture. Something that sits between Iaas and PaaS.
With Gruntwork, we get the full control and power of something like AWS with almost the simplicity of a PaaS.
And, since all of this infrastructure is backed by code, you can easily debug, scale, and customize things as much as you want.
The question is, is the cost worth it?
The problem many teams are facing
Many teams can’t get over this hump. They’re stuck writing their own production grade infrastructure, have too much of the process in someone’s head instead of scripted and automated, and can’t solve the problem of software development.
They don’t understand what they’re working with. Without understanding it, they can’t solve it.
The process of documenting and automating the development, build and release, and infrastructure is the first step of understanding what you're doing, since you have to explicitly write down the steps and replace those manual steps with code.
Why this matters
The Gruntwork code base has been built up over ten years using code which was mostly written while the founders worked as consultants.
Their reference architecture is now 3.5 years old and in continual improvement.
Any developer who thinks they can write code of similar quality, functionality and cost is vastly underestimating how difficult it is. It's like a developer saying that they’d rather use their own frontend or backend framework.
Getting here, via Gruntwork, is an indication of the ability to efficiently run a development team that focuses on developing features. There’s no need to run a team dedicated to DevOps. The low month fee covers maintenance. Developers then have a solid foundation to start practicing DevOps.
The Gruntwork Reference Architecture and Infrastructure as Code Library is the best option for us.. But, while everyone’s situation is different, we honestly think it’s a good fit for 99% of the cases.
Unfortunately, many many companies are still opting for the more complex, less effective and less successful route.
The problem with DevOps consultants
Consultants build from scratch, maximize billable hours and launch time is highly variable (often 3 to 12 months).
Plus, as I wrote earlier - they are very expensive.
It is insanely risky to have people with the wrong incentives responsible for a major part of your product. We decided early on that we’d never take this route.
Gruntwork means that we shouldn’t have to.
Leveraging Gruntwork gave us the time to focus on solving the technical debt in our development process for just 5000 USD and an ongoing 795 USD a month. -- a bargain, when you consider that I’ve worked on teams which were building all this themselves.
Once in-house teams realise they’re in over their heads, they turn to consultants who aren’ t incentivised to solve problems quickly. So then you’ve got consultants being paid by the hour solving a problem that is already solved (by Gruntwork) and a company that is sinking deeper into the sunk cost fallacy.
I've seen teams of 5 to 10 DevOps engineers spending 1.5 years to build the same reference architecture as Gruntwork. Only worse. They’ll have 50,000 lines of messy code that fell short of providing some basic functionality.
Consultants who could get the job done in 12 months end up sticking around for 3-5 years (or leave behind a code base that companies struggle to find the resources to maintain). The documentation is also never good enough. It epitomizes a broken culture.
Just like the government software projects that rely on large consulting firms that cost tens of millions for simple stuff. This is crazy! But it happens every day.
Recommending that companies scrap their code base and use the Gruntwork Reference Architecture is almost always met with resistance.
Compare that to CreditStretcher, where Gruntwork has done the heavy lifting for us for ¼ the cost of hiring one consultant for a month and 1/30th the ongoing cost. Companies can spend up to 1000x, as shown below for less functionality, higher risk, and poorer maintainability and developer happiness.
It’s refreshing to work here.
Gruntwork has been tested by hundreds of companies in production. This poses a massive security benefit over designing the infrastructure code yourself.
Few, if any, developers have the infrastructure and security expertise to design and code something from scratch that compares to what Gruntwork have spent almost ten years working on.
They know Terraform and security better than the best full stack developers.
The resistance you often see amongst developers towards DevOps is due to their lack of operations experience. And ops people have a lack of experience in development. Very few developers are experts in both.
A large part of the security benefit for us on AWS is avoiding the bugs and design flaws in writing the infrastructure as code ourselves. This includes a well designed multi-account setup, network, groups, roles and users, and all the other small details included in the production readiness checklist. There’s also a great level of support through the free community slack channel to ask any questions you aren’t sure about. Even with all this, it’s still a lot of work.
We have strong opinions at CreditStretcher. But that doesn't mean that we are 100% correct or that this is the only way, we're not and it isn't.
But this approach is working very well for us right now.
If you have alternate approaches and strong opinions about it, we would love to hear from you. If you're interested to learn more about our approach, reach out!
It's all about finding a tool that you can configure to your own standards without spending a fortune on unneeded services.
1: Based on my past experience gruntwork is 5K usd plus 800usd per month for a total of about 15K per year with no devops team required and the devops work being spread amongst developers. The contractor option is about 20K per month per contractor and there has been 5-10 on a project that has lasted 18 months and is ongoing. If we go with the low 5 contractor number for the 18 months that’s 100K x 18 = 1.8M USD which is 1000x. As a startup we could never bear this cost.