Section 2: Operating system evolution

In this section, we’ll discuss how operating systems have changed over time. Our way in is a research paper published in 2019, which you should read:

An Analysis of Performance Evolution of Linux’s Core Operations

This paper presents an analysis of how Linux’s performance has evolved over the past seven years. Unlike recent works that focus on OS performance in terms of scalability or service of a particular workload, this study goes back to basics: the latency of core kernel operations (e.g., system calls, context switching, etc.). To our surprise, the study shows that the performance of many core operations has worsened or fluctuated significantly over the years. For example, the select system call is 100% slower than it was just two years ago. An in-depth analysis shows that over the past seven years, core kernel subsystems have been forced to accommodate an increasing number of security enhancements and new features. These additions steadily add overhead to core kernel operations but also frequently introduce extreme slowdowns of more than 100%. In addition, simple misconfigurations have also severely impacted kernel performance. Overall, we find most of the slowdowns can be attributed to 11 changes.

Ren XJ, Rodrigues K, Chen L, Vega C, Stumm M, Yuan D. “An Analysis of Performance Evolution of Linux’s Core Operations.” In Proc. SOSP 2019, pp554–569.

PDF link


Read the paper before section. As you read, think about the following topics:

  1. What are the evolutionary pressures affecting Linux? In other words, what motivates the introduction of changes into the kernel?

  2. Of the changes discussed in the paper, which required new hardware support? Could future hardware optimizations mitigate some observed performance problems?

  3. The performance heatmap figures (Fig. 1 and Fig. 8) lets us visually detect apparent phase changes in Linux’s performance. Which versions represent the biggest phase changes, and what happened in those versions?

  4. The paper is somewhat critical of the kernel development process. Do you agree with that critical stance? Do you have any ideas for improving kernel development?

  5. The main results in the paper focus on microbenchmarks: measurements of the performance of individual system calls. How do these compare with macrobenchmark results (Section 5)? The paper claims that “our evaluation results from the macrobenchmarks confirm the relevance of [the microbenchmarks]”: in what ways do you agree?

  6. Section 4 describes many root causes of performance changes, which are specific kernel features and changes. Do you understand these root causes? Come with questions!

Before your section, post a question about the paper, or a reaction to the paper, as a comment on Ed (just a couple sentences).

On reading research

Reading research papers is a skill! Papers feature dense writing and many technical terms. The peer review process can cause defensive writing, where whole sections of the paper exist to mollify specific reviewer concerns but confuse everyone else. It can take several passes to understand a paper, so don’t feel bad if you get lost.

Active curiosity will improve your reading. Don’t read purely passively start to end. Instead, do a quick first pass: read the abstract, glance through section headings and figures, then come up with specific questions to guide your focus in a later pass. For example, in this paper, here are some questions inspired by the abstract (and, of course, there are the topics above):

  1. What are the “security enhancements and new features” that kernel subsystems have been “forced” to accommodate? Were they worth it?
  2. How can a “misconfiguration” (i.e., a configuration mistake) impact kernel performance for everyone? How do they know the misconfiguration was a mistake?
  3. Are “core kernel operations” a fair way to measure kernel performance?
  4. Is Linux especially bad?

Question-focused reading may help you tolerate boring sections and zoom in on interesting details. But everyone has their own style, so do what works for you.

Many guides for paper reading are available on line. Here are three well-known ones:

  1. S. Keshav’s “How to Read a Paper” (local copy) recommends a three-pass method, where the first five-minute pass develops a mental framework for the paper, and the second pass fills in that framework. (He also recommends a third, extremely detailed pass, that’s overkill for us.)

  2. Michael Mitzenmacher’s paper-reading instructions.

  3. A good guide from SJSU.