Lorin Hochstein

The Netflix video streaming system is composed of many interacting services. in such a large system, failures in individual services are not uncommon. This paper describes the Chaos Automation Platform, a system for running failure injection experiments on the production system to verify that failures in non-critical services do not result in system outages.

Modern software-based services are implemented as distributed systems with complex behavior and failure modes. Many large tech organizations are using experimentation to verify such systems' reliability. Netflix engineers call this approach chaos engineering. They've determined several principles underlying it and have used it to run experiments.

Manually configuring servers and using multi-step checklists to deploy your applications is no fun at all. Ansible is a great tool for automating your infrastructure tasks, with a gentle learning curve.

This book covers how to use Ansible to automate your configuration management, deployment, and orchestration tasks.

In virtual organizations, such as Open Source Software (OSS) communities, we expect that the impressions members have about each other play an important role in fostering effective collaboration. However, there is little empirical evidence about how peer impressions form and change in virtual organizations. This paper reports the results from a survey designed to understand the peer impression formation process among OSS participants in terms of perceived expertise, trustworthiness… Show more

In virtual organizations, such as Open Source Software (OSS) communities, we expect that the impressions members have about each other play an important role in fostering effective collaboration. However, there is little empirical evidence about how peer impressions form and change in virtual organizations. This paper reports the results from a survey designed to understand the peer impression formation process among OSS participants in terms of perceived expertise, trustworthiness, productivity, experiences collaborating, and other factors that make collaboration easy or difficult. While the majority of survey respondents reported positive experiences, a non-trivial fraction had negative experiences. In particular, volunteer participants were more likely to report negative experiences than participants who were paid.
The results showed that factors related to a person's project contribution (e.g., quality and understandability of committed codes, important design related decisions, and critical fixes made) were more important than factors related to work style or personal traits. Although OSS participants are very task focused, the respondents believed that meeting their peers in person is beneficial for forming peer impressions. Having an appropriate impression of one's OSS peers is crucial, but the impression formation process is complicated and different from the process in traditional organizations. Show less

This book offers hard-earned experience from OpenStack operators who have run OpenStack in production for six months or longer. They've gathered their notes, shared their stories, and learned from each other in the room. We invite you to join in the quest for best practices in OpenStack cloud operations.

All compiled software systems require a build system: a set of scripts to invoke compilers and linkers to generate the final executable binaries. For scientific software, these build scripts can become extremely complex. Anecdotes suggest that scientific programmers have long been dissatisfied with the current software build tool chains. In this paper, we describe preliminary results from a case study of two projects to estimate the fraction of effort devoted to maintaining these scripts, which… Show more

All compiled software systems require a build system: a set of scripts to invoke compilers and linkers to generate the final executable binaries. For scientific software, these build scripts can become extremely complex. Anecdotes suggest that scientific programmers have long been dissatisfied with the current software build tool chains. In this paper, we describe preliminary results from a case study of two projects to estimate the fraction of effort devoted to maintaining these scripts, which we refer to as the `build tax'. While estimates based on line counts are on the order of only 5%, estimates based on activity-related metrics suggest much higher values. Show less

Current cloud computing infrastructure typically assumes a homogeneous collection of commodity hardware, with details about hardware variation intentionally hidden from users. In this paper, we present our approach for extending the traditional notions of cloud computing to provide a cloud-based access model to clusters that contain a heterogeneous architectures and accelerators.

Large-scale distributed systems must be built to anticipate and mitigate a variety of hardware and software failures. In order to build confidence that fault-tolerant systems are correctly implemented, Netflix (and similar enterprises) regularly run “failure drills” in which faults are deliberately injected in their production system. Existing failure testing approaches either explore the space of potential failures randomly or exploit the “hunches” of domain experts to guide the search. In… Show more

Large-scale distributed systems must be built to anticipate and mitigate a variety of hardware and software failures. In order to build confidence that fault-tolerant systems are correctly implemented, Netflix (and similar enterprises) regularly run “failure drills” in which faults are deliberately injected in their production system. Existing failure testing approaches either explore the space of potential failures randomly or exploit the “hunches” of domain experts to guide the search. In this paper, we describe how we adapted and implemented a research prototype called lineage-driven fault injection (LDFI) to automate failure testing at Netflix. Show less