SRE Responsibilities
In Google, SREs (Site Reliability Engineers) are not responsible for the deployment or launch of specific services. Instead, their primary focus is ensuring service reliability and performance, as well as managing data center resource allocation to reserve capacity for critical services. As mentioned earlier, the counterpart to SREs is SWEs (Software Engineers), who handle the actual development work.
For example, during my time in Google’s Internet Advertising division, our team was responsible for a server that collected user data for targeted advertising. The development, testing, and deployment of this server were entirely handled by SWEs.
SREs do not handle the specific implementation of servers. Their main role is to respond quickly to emergencies such as server outages, restore services as soon as possible, and minimize losses caused by downtime.
Note: Here, “server” refers to server-side programs, not physical servers. In Google, neither SWEs nor SREs have access to physical servers.
SRE Requirements
Since SREs are responsible for service stability and performance, there are strict requirements for any server before it is handed over to them. For instance, the frequency of alerts must not exceed a certain threshold, and CPU and memory consumption must stay within predefined limits.
Only after a server fully meets SRE requirements will they take over its management:
When issues arise, SREs perform emergency fixes to restore service first. Then, they collaborate with SWEs, who ultimately fix the bugs in the server code.
Additionally, for major updates to a server, SWEs must notify SREs in advance to prepare thoroughly before releasing the new version:
To enable SREs to take over a server, SWEs must optimize it extensively based on SRE guidelines. First, SREs provide various performance metrics, such as service response latency and resource usage. Furthermore, SREs require SWEs to submit evaluation results, including stress test outcomes and end-to-end test reports. SREs also assist SWEs in troubleshooting performance issues.
Thus, SREs hold a high status at Google. To ensure successful server launches, SWEs strive to maintain good relationships with SREs.
Let me illustrate with a specific example of how SWEs and SREs collaborate to launch a server at Google.
Our team refactored the code for our assigned server. After refactoring, we needed SRE approval before launching the updated server.
To demonstrate compliance, our SWEs had to prove that the refactored server did not increase resource overhead—meaning CPU, memory, disk, and bandwidth consumption remained unchanged—and that performance metrics such as end-to-end response latency, QPS, and stress test responses did not degrade.
After refactoring, one performance metric failed to meet SRE standards: contention, which refers to the wait time for competing resources in multi-threaded environments. Post-refactoring, contention levels rose significantly, indicating longer wait times under concurrent conditions.
We spent considerable time debugging without finding the root cause, as we hadn’t explicitly used many mutexes in the code. We then requested assistance from SREs. Using Google’s internal graphical profiling tool, cprof, they helped us identify the issues.
We identified two main causes:
- Frequent memory requests by tc_malloc in multi-threaded scenarios led to increased contention;
- Extensive use of hash maps without pre-allocating sufficient memory space resulted in excessive calls to the underlying tc_malloc.
SRE Work Content
In short, Google’s SREs do not perform low-level hardware maintenance. Instead, they ensure the performance and stability of Google’s various services. In other words, SREs guarantee software-level performance and stability, covering both software infrastructure and application services.
SREs must have a deep understanding of Google’s internal software infrastructure and typically possess strong skills in problem diagnosis, debugging, and rapid server recovery.
Here are some examples of common Google software infrastructure components:
- Borg: Distributed task management system;
- Borgmon: Powerful monitoring and alerting system;
- BigTable: Distributed Key/Value storage system;
- Google File System: Distributed file system;
- PubSub: Distributed message queue system;
- MapReduce: Distributed big data batch processing system;
- F1: Distributed database;
- ECatcher: Log collection and retrieval system;
- Stubby: Google’s RPC implementation;
- Proto Buffer: Data serialization protocol and RPC protocol;
- Chubby: Provides services similar to Zookeeper.
- Google has many more software infrastructure systems, such as Megastore, Spanner, and Mustang, which I haven’t used and am therefore less familiar with.
Future Directions for Operations
I believe that in the cloud computing era, operations engineers will gradually evolve into roles like Google’s SREs, moving away from low-level hardware maintenance and focusing more on software infrastructure to help enterprise clients build robust software foundations.
With strong software infrastructure, enterprise clients can better handle complex and changing business demands, accelerating their growth.
Another personal view: Why do Google products seem to have such high technical content?
It may not necessarily be because Google’s SWEs are significantly more skilled than engineers at Microsoft, LinkedIn, or Facebook. Rather, it is mainly due to Google’s extremely powerful software infrastructure (as detailed above). With such a strong foundation, building powerful products becomes much easier.
Google’s internal software infrastructure basically satisfies most common distributed functional requirements, greatly facilitating SWEs in business development. In other words, when developing at Google, SWEs can focus primarily on business logic, while the performance of application services is largely guaranteed by the underlying software infrastructure.