Our second homework assignment is to reproduce and/or replicate a graph from previous research.
Research reproduction involves repeating a published experiment with the same code and/or artifacts, but a different team. When authors publish their full code and experimental setup scripts, reproducing an experiment can be as easy as typing
make graph, and the results will vary minimally from the published version. Reproduction has limited value; for example, if the authors made a mistake in experimental design, reproduction simply repeats their mistake. On the other hand, reproduction demonstrates that an artifact has been packaged well enough to be used—reproducible artifacts are likely to be useful in other contexts.Research replication involves repeating a published experiment, but with independently-developed artifacts. This is more likely to avoid mistakes in experimental setup, and also offers satisfaction and deeper engagement with the original work.
Your job for this assignment is to:
- Pick a paper that we’ve discussed in class (and that you were not involved in).
- Select one or two graphs and/or experiments from that paper.
- Attempt to reproduce and/or replicate those experiments.
Come to class on March 12 prepared to discuss the initial outcome of your replication project. You should have something to show, but due to time constraints, that should be the equivalent of at most 2 slides.
A 4-to-6-page writeup (formatted single-spaced) of the replication project is due March 27 (Thursday after break). Your writeup should include these sections:
- Abstract.
- Background. What paper did you choose for reproduction/replication? Which experiment did you concentrate on? Why did you choose this paper and experiment (what drew to you to the work)?
- Evaluation. How did you reproduce/replicate the relevant experiments? What hardware and/or software did you use? Include a graph and/or table, and describe interesting differences from the original publication.
- Discussion. What did you learn from the project? Did the authors provide sufficient information to reproduce/replicate their research, or did you have to make assumptions about aspects of their experiments? If the project was surprisingly difficult, or your results less thorough than you hoped, this section should talk about how and why.
- Conclusion.
Your own work should be reproducible/replicatable! Provide enough detail that another student would get the same results. This may require that you provide a link to your source code.