TL112 – Red Hat Transformational Learning: Introduction to Pragmatic Site Reliability Engineering — Lesson 3: Observability, Incidents, Blameless Learning, Toil, and Automation Opportunities

Learning Objectives

• Evaluate integration, security, resilience, performance, and exceptions affecting observability signals, incident response, and blameless reviews.
• Apply toil identification and automation opportunities while preserving reviewable technical evidence.
• Recognize and correct the condition where the implementation succeeds normally but fails during dependency loss, scale, policy enforcement, or recovery.

Detailed Topics

• observability signals: apply the concept to the course scenario, identify dependencies and controls, and record objective evidence of the result.
• incident response: apply the concept to the course scenario, identify dependencies and controls, and record objective evidence of the result.
• blameless reviews: apply the concept to the course scenario, identify dependencies and controls, and record objective evidence of the result.
• toil identification: apply the concept to the course scenario, identify dependencies and controls, and record objective evidence of the result.
• automation opportunities: apply the concept to the course scenario, identify dependencies and controls, and record objective evidence of the result.

Instructor Demonstration

The instructor demonstrates observability, incidents, blameless learning, toil, and automation opportunities by tracing architecture, commands, configuration, system messages, and evidence. The walkthrough tests how the implementation succeeds normally but fails during dependency loss, scale, policy enforcement, or recovery and explains how to isolate the cause without masking related issues.

Hands-On Exercise

Introduce a realistic boundary condition and test corrective behavior for Observability, Incidents, Blameless Learning, Toil, and Automation Opportunities and add the verified result to the service reliability charter, SLI and SLO register, error-budget policy, and improvement roadmap.

1. Confirm scope, versions, access, dependencies, assumptions, and acceptance criteria.
2. Perform the workflow while recording commands, configuration, output, events, and decisions.
3. Test the normal path and one course-specific exception or boundary condition.
4. Retest the correction and document rollback, ownership, and follow-up actions.

Scenario Analysis

• Identify which dependency most strongly influences observability signals in the stated scenario.
• Compare two approaches for toil identification and explain the security, reliability, performance, and support trade-offs.
• Decide what evidence must be retained in the service reliability charter, SLI and SLO register, error-budget policy, and improvement roadmap before operational acceptance.

Troubleshooting Guidance

Begin with the first observable difference from the expected result. Check integration, security, boundary conditions, resilience, and performance separately, then correlate logs, events, commands, metrics, and configuration history. Treat the assumption that the implementation succeeds normally but fails during dependency loss, scale, policy enforcement, or recovery as a hypothesis until a reproducible retest confirms the cause.

Validation Checklist

• The required integration, security, boundary conditions, resilience, and performance were checked and recorded.
• The normal path and a course-specific exception were tested.
• The service reliability charter, SLI and SLO register, error-budget policy, and improvement roadmap links conclusions to commands, configuration, logs, metrics, or other objective evidence.
• Corrective actions, rollback, ownership, and handover notes are complete.

Lesson Outcome

Learners can apply Observability, Incidents, Blameless Learning, Toil, and Automation Opportunities, explain the observed platform behavior, and support the result with a reviewable service reliability charter, SLI and SLO register, error-budget policy, and improvement roadmap.