Reproducibility

Your second full assignment, due March 26 (intermediate deadline March 18), is to attempt to reproduce one or more experiments from a systems paper that you were not involved in.

Reproduction can be uninteresting. When authors publish their full code and experimental setup scripts, reproducing an experiment can be as easy as typing make graph, and the results will vary minimally from the published version. This kind of reproduction, which is more specifically called replication (ACM description), is limited because, for example, when the authors made a mistake in experimental design, it simply repeats their mistake. (Replication is still super valuable though!) On the other hand, when authors rely on proprietary data and resources, such as a billion users, a data center, or a quantum computer, reproduction is impossible on its face, which is also uninteresting for us. Reproduction is most valuable when reproducers put in some independent effort. That makes reproduction instructive for the reproducers (who learn experimental design issues and tools). It is also revealing about the reproduced work: maybe technological evolution since original publication has changed results; maybe the original work was wrong or overlooked something important!

Detailed description

  1. Select a systems paper that contains at least one experiment you want to try to reproduce. The paper must contain data, which can be in graph form, table form, or even just references to experiments in text (e.g., “In Featherstitch asynchronous mode, after crashing, fsck nearly always reported that the file system contained…errors…. With our soft updates dependencies, the file system was always consistent” [Frost et al. 2007]). The paper must not originate from your research group, past or present. We need not have read it in class.

    Note that reproduction does not mean that you run exactly the same experiment. For example, you might reproduce Figure 2 of Liedtke 1996, but using modern hardware and operating systems. In many cases, the experimental design matters more than the specific code being evaluated.

    Figure 3 of Liedtke 1996

  2. Develop a plan for how you will reproduce this experiment. Is the code for the experiment available, or will you need to in some part write your own code? Assuming code is available, how will you ensure that you are reproducing, rather than merely replicating, the paper’s results?

    On March 18 in class, be prepared to discuss parts 1 and 2.

  3. Reproduce the experiment.

  4. Write up the reproduction effort. You will turn in both a short paper (2-5 pages in the same format as the speculative OS) and any code you wrote. Your paper should:

    • Describe original paper in your own words, including its overall goal.
    • Describe the original experiment in your own words, and explain its relationship to the paper’s goal.
    • Critique that experiment, in terms of experimental setup, result quality, and technological developments since original publication.
    • Explain your reproduction effort: What new code was needed? What experimental setup changes were required? What assumptions did you make? Did you run into any difficulties?
    • Present the reproduction and explain differences between the original version and your reproduction.
    • Discuss the original paper in light of your reproduction.

The full writeup is due Friday, March 26.

I expect the assignment to take substantial time—say a handful of 4-hour sessions. If you find yourself done in 4 hours total, reproduce more results.

You may work on this assignment in pairs but I will expect a proportionally more awesome effort.

Examples

Many of the papers we’ve read so far have reproducible components. For example:

This assignment was inspired by an assignment originally developed for Margo Seltzer’s version of CS261. Her current version of the assignment has many more paper links. Reach out if you want more guidance on experiment choice.