{"id":473,"date":"2021-11-05T16:12:28","date_gmt":"2021-11-05T20:12:28","guid":{"rendered":"https:\/\/freedville.com\/blog\/?p=473"},"modified":"2021-11-05T16:14:14","modified_gmt":"2021-11-05T20:14:14","slug":"book-review-site-reliability-engineering","status":"publish","type":"post","link":"https:\/\/freedville.com\/blog\/2021\/11\/05\/book-review-site-reliability-engineering\/","title":{"rendered":"Book Review: Site Reliability Engineering"},"content":{"rendered":"\n
\"Dashboard
A dashboard to monitor the health of a service. Photo by Luke Chesser<\/a> on Unsplash<\/a><\/figcaption><\/figure>\n\n\n\n

Book Review: Site Reliability Engineering<\/strong><\/p>\n\n\n\n

I\u2019ve just finished reading the 2016 book Site Reliability Engineering: How Google Runs Production Systems<\/a>.\u00a0 I\u2019ve spent 20 years as a developer and I\u2019ve appreciated the DevOps philosophy, but this is the first \u201cOps-heavy\u201d book I\u2019ve read.\u00a0 DevOps has always suggested to introduce Ops into the development cycle rather than only at deployment time.\u00a0 This book told me what\u2019s going to happen when that Ops focus shifts left.<\/p>\n\n\n\n

The book starts with the reminder that a typical application spends 20% of its life in development and 80% of its life in maintenance.\u00a0 The call to action is to act like it – focus on maintenance<\/strong>.\u00a0 Google proclaims that they view maintenance through a developer\u2019s lens.<\/p>\n\n\n\n

The central insight is that manual intervention is linear and does not scale well<\/strong>.\u00a0 There are plenty of possible exponentials possible in production: number of services, volume of traffic, or resource consumption.\u00a0 Demand can easily double; at Google scale demand might spike 1000x or more temporarily or permanently.\u00a0 Applications and processes cannot rely on manual intervention to keep them running.\u00a0 You can\u2019t hire enough engineers to manually intervene on an exponential process.\u00a0<\/p>\n\n\n\n

\"A
Are you scaling linearly or exponentially? Photo by Chris Barbalis<\/a> on Unsplash<\/a><\/figcaption><\/figure>\n\n\n\n

Most of the book describes how to use automation to keep team size stable despite increased demand.  The core method is to make sure SREs spend 50% of their time tactically and 50% strategically<\/strong>.  Tactical time is what I traditionally thought of Ops as: working tickets, administering servers, being on-call, restoring service during an outage, etc.  Strategic time is finding root causes and preventing outages.  In tactical time an SRE may resolve ten nearly identical tickets; in strategic time they will find the common root cause and resolve it, as well as review how that root problem was introduced (using Five Whys, Postmortem, etc).<\/p>\n\n\n\n

This 50\/50 time split is a valuable insight!\u00a0 I\u2019ve made a point to carve out time for strategic work and I encourage my mentees to do the same.\u00a0 It\u2019s encouraging that Google seems to explicitly<\/em> measure and require this split. \u00a0They monitor each developer\u2019s time allocation and have processes to increase or decrease the strategic time.\u00a0 Increasing the strategic time reduces burnout.\u00a0 Decreasing the strategic time keeps SREs from getting rusty on being effective while on-call. Practice makes perfect. Google wants their SREs to have enough on-call “practice” that they stay sharp, without getting burned out from too much.<\/p>\n\n\n\n

Further, the Google process is not just heavily focused on SLO\/SLA but tracks the inverse as an error budget<\/strong>.  A service with 99.5% uptime SLA has 3.6 hours of error budget per month.  (3.6 is 0.5% of the monthly hours).  Error budgets are used in interesting two ways:<\/p>\n\n\n\n