The Top Five SRE Best Practices: Create a More Resilient Environment for Operational Maturity and a Better Customer Experience

Today’s modern enterprise is demanding more than ever, and businesses are looking to technology to help deliver results. Customer experience has become king. Logistical planning needs to be managed down to the second. Capacity must be leveraged. And, if customer behavior and needs can be predicted ahead of time to serve them better, then do it!

Of course, when it comes to delivering on the promises of technology, everyone looks in one direction to ensure that SLA targets are met and efforts are a success – the site reliability engineering (SRE) team.

Talk about pressure!

Today SREs impact all phases of the software lifecycle – from ensuring Product and Engineering teams consider requirements that support reliability, to validating automation and the architecture of massive internal platforms. In essence, SREs have become the “Guardians of Production.” That means they must fight for the user throughout the process. Does that also mean they have to be at odds with Product and Engineering organizations? Preferably not. Instead, relationships need to be carefully maintained and collaborative in order to drive the desired results.

But how do you begin to establish and nurture these relationships? It starts with a foundational site reliability engineering strategy built upon trust. Everyone throughout the organization must trust that the SRE group will consistently make the right decision to ensure the success of the product in question.

From inception to delivery, engineering and operations organizations must work closely together to build and deliver technology that works for the business and meets organizational goals. Unfortunately, that can be easier said than done, even when SREs have strived to consider every ounce of prevention – and prepared for a response in the event the inevitable happens and a production incident occurs.

So, how can SREs do better to help drive availability towards the magic number – 99.95% availability – and advance the maturity of operations and the SRE team? And how can you ensure that you’re where you want to be in 30 days, six months, or even a year from now?

The five best practices below are a great start.

1. Engage early. Too often, the SRE team is engaged too late. What’s this look like? It’s not pretty. Only minimum requirements are met. Costs escalate. Goals are deprioritized. Needs of the business are not met.
2. Monitor and observe. Ensure you have the ability to observe the inner workings of systems to be able to make effective decisions and form hypotheses that guide roadmaps and operationalization of capabilities.
3. Incident management. Teams need to be able to follow a framework and manage incidents successfully, because if incidents aren’t managed correctly, they will wreak havoc for your applications and environment.
4. Experiment. Experimentation is integral to validating hypotheses and making changes to improve application performance, cost, or reliability – unfortunately, relying on humans alone to perform experiments isn’t the solution because you can’t put enough human resources toward experimentation to make a real difference.
5. Optimize. Automation can run years’ worth of performance tests over the course of just a few hours, and tweak the configuration with each successive run to see how the application performs and what resource utilization results, while machine learning provides the ability to analyze all those results and recommend configurations.