Project – CS 260r

Your final assignment, due May 15, is a research project and paper involving computer software systems. This is the biggest work product of the course.

The project has these components:

A paper of sufficient density and polish that you would feel comfortable submitting it for scientific peer review (8–14 pages).
(For group projects only) An individual contribution document detailing your contribution to the work, and your perceptions of the contributions of others (≤1 page).
A short self-evaluation (1–2 paragraphs). You may assign yourself a grade, or you may simply discuss your self-perceived level of effort and investment over the course.

Purpose: Self-evaluations are a common feature of performance reviews in industry and academia. They have advantages and disadvantages. They are often difficult to write; they can reward self-aggrandizement and enforce existing power structures. But they also enforce valuable introspection, and help people realize just how much work they’ve actually accomplished.
Any code and data used for the project. No need for superb software engineering, but ideally the code should be accompanied by enough documentation that a motivated user could attempt to replicate your results.

All course material is due on our final deadline, Saturday May 14 at 11:59pm.

Choosing a project

Over the next few weeks, all students in the class should brainstorm project ideas and teams and settle on a project direction.

A good project in this class will involve systems ideas that are evaluated on generally-available hardware and software.

To generate an idea, ask yourself: What interests you? What papers have inspired you? For which papers have you felt that you could do better? But facility at idea generation has limited impact on project quality. Far more important than a “good idea” is the incremental process of tweaking an idea into something that can be built and evaluated.

When considering an idea, ask yourself how to evaluate it—how to define a hypothesis statement that could be tested. A good hypothesis statement can guide and simplify your implementation and your design. For instance, you can use the hypothesis statement as a razor to decide which features to skip. (“I have an idea about how to build a better file system for non-persistent data. Since I can evaluate my hypothesis statement entirely using RAM disks, I don’t need to implement any file system consistency features or the fsync system call.”)

Some papers put their hypothesis statements prominently in the abstract, the introduction, or the introduction to the evaluation section. Here’s a made-up hypothesis statement (for which paper?). Note that it has several testable parts:

“File system read workloads will increasingly fit in memory caches. Therefore, file system performance will be dominated by writes. A file system that stores all data in a circular log will offer better write performance, and therefore better performance overall.”

Your hypothesis statement may require several iterations.

Questions for project development

As you work on your project, you should know at least in outline the answers to these questions. Periodically check in with the questions to see if you’re on track.

What kind of system are you interested in?
What properties matter for that kind of system?
What is your hypothesis statement?
What is your project schedule? What do you aim to have completed each week?
What is your division of labor? Who’s doing what? How can you work in parallel?
What are the risks? What are you most worried about in your development? What simpler hypothesis statements might be easier to show?
What is your (hypothetical) future work? How could future generations build on your effort?

Paper

The final project writeup should be a PDF at least 8 pages long and no more than 14 pages long. Follow the format of a typical research paper: 11pt font, two-column, single-spaced. Do not double-space.

Writeups will be shared with the class and posted publicly unless you explicitly withhold permission to do so.

Michael Ernst has some interesting advice on how to write a technical paper (He has lots of other advice too.) Some key points:

“The goal of writing a paper is to change people’s behavior: for instance, to change the way they think about a research problem or to convince them to use a new approach. … As a general rule, your paper needs to convince the audience of three key points: that the problem is interesting, that it is hard, and that you solved it.”

“A common mistake is to focus on what you spent the most time on. Do not write your paper as a chronological narrative of all the things that you tried, and do not devote space in the paper proportionately to the amount of time you spent on each task. Most work that you do will never show up in any paper; the purpose of infrastructure-building and exploration of blind alleys is to enable you to do the small amount of work that is worth writing about. Another way of stating this is that the purpose of the paper is not to describe what you have done, but to inform readers of the successful outcome or significant results, and to convince readers of the validity of those conclusions.”

“A related work section should not only explain what research others have done, but in each case should compare and contrast that to your work and also to other related work. After reading your related work section, a reader should understand the key idea and contribution of each significant piece of related work, how they fit together (what are the common themes or approaches in the research community?), and how your work differs.”

But do not get too hung up on these points, especially in the context of a class project. There are many ways to write a good research paper, and class projects need not be as polished as research papers. Regarding the quotes above:

Negative results are important research, but negative results may not change people’s current behavior. (They can change future researchers’ behavior by stopping them from wasting time.)
“Interesting” and “hard” are subjective and depend on audience receptivity and authorial skill. A problem can be interesting for many reasons, including that the problem has intrinsically interesting features, that other people care about the problem (whether or not the authors do), or that a solution to the problem would open up a new class of problem. And a problem can be hard for many reasons, including that the problem has simply never been noticed before. A hard problem need not have a hard solution.
A chronological narrative can be interesting if done elegantly and if uninteresting details are skipped.

Your contribution is a key aspect of the project writeup. What did you discover?

A typical writeup will follow this format, which is not mandatory.

Title. Something grabby that correctly describes a part of the contribution.
Abstract. A paragraph or two that concisely describes the motivation for the work (the problem the work addresses), the contribution of the work, and a highlight of your results. The abstract tends to target an already-expert audience.
Introduction. A page or so that expands on the abstract. The introduction is written for a broader audience than the abstract; its purpose is to situate the technically adept, but not necessarily expert, reader within the domain of your work, helping them understand your research problem as you define it. Your introduction, for example, might start out with a brief explanation of distributed computation infrastructures or machine learning pipelines, followed by a description of the issue with these infrastructures that you address.

Often an introduction will end with a list of contributions, and an “outline” paragraph that says “The rest of this paper is organized as follows. Section 2, etc.”
Related work (this may also appear just before the conclusion). A description of related research, especially research closely related to your own work. The purposes of this section are citation and comparison. Foundational work like MapReduce or Spark requires citation, but not necessarily comparison. Work that is closer to your domain requires comparison. Describe for each closely-related work (1) the core idea, (2) what is complementary with your work, (3) what is more advanced than your work, and (4) what is advanced upon by your work. (2)–(4) are optional—some papers will be entirely complementary with or orthogonal to your work.
Design. The system you built, or the experiments you designed, described in enough relevant detail that a skilled system builder could replicate your work. Design is not about the names of your classes or your software engineering prowess, though you should include these details when they are relevant. Multiple pages.
Implementation. Aspects of your system that are not relevant to the design, but might be relevant to understanding your evaluation. Typically a page or less.
Evaluation. For systems work, this will often include the following subsections:
- Experimental setup. Describe how you ran your experiments. What kinds of machine? How much memory? How many trials? How did you prepare the machine before each trial? (Interesting reading: Systems Benchmarking Crimes, by Gernot Heiser.)
- The experiments themselves, grouped by purpose. Include figures.
- A summary of the experimental results.
Choosing experiments is a critical part of a systems paper, and in the kind of research my group does, we’ll often cycle through many experiments and figures before settling on the experimental designs and figures that best explain our work. So start thinking about experiments early!

I find performance hypotheses useful for organizing evaluations. A performance hypothesis is a concrete, falsifiable hypothesis about your system’s performance. One example might be “The Dask LIFO scheduler is better than FIFO on short tasks within one computer because of its interaction with processor cache memory, but worse than FIFO on longer, distributed tasks because it encourages stragglers.” This hypothesis suggests some experiments and excludes others: you would measure LIFO vs. FIFO on short tasks within one computer and on longer tasks in a distributed setting, and would also validate your hypothesis by looking at cache performance and stragglers, respectively. It’s OK if your experimental results falsify a performance hypothesis—that indicates progress!

An alternative to performance hypotheses is exploratory evaluation, where you run a bunch of experiments, collect all kinds of data, and then try to understand the results. This can be fun, but one can also drown in data and come to no conclusions.

Not all projects will have systems evaluations, but every project should be evaluated. If you are introducing a new computing model, for example, you could evaluate whether existing systems would fit the model.
Discussion. This optional section can speculate about your work. Feel free to put stuff here that is based on intuition and that you haven’t backed up with experiment. Do you think, in the end, that the project was a good idea? What would you do next? What should readers learn?
Conclusion. A paragraph or so.

Do not punt the writeup. In previous years, I have seen writeups with “lorem ipsum dolor” text in the abstract, and an introduction that said “Our verification VERIFIES SOMETHING HERE”. This leaves me annoyed.