Presenting papers

You’ll present one or more research papers during class. Critically reading research papers is a skill! Here are some tips, ideas, and techniques to help learn it.

Reading the paper

This may be the first time you’re reading research papers, or even the first time you’re reading research papers in the systems area. Welcome!

Research papers are parts of a big conversation among the researchers and practitioners that make up the “research community.” Many papers are best understood as reactions to, and participants in, that conversation, and the ongoing determination of the research community’s values. An important, and often implicit, part of paper reading is learning those values. It’s hard to pick this up in advance. Instead, you just have to dive in and read papers critically. An understanding of the community will come.

The style of most papers is dense (and often just plain bad). But you can get through the density by reading critically.

Don’t try to understand every word of the paper on first read. Instead, try to pick up the most important points by skimming. Then go back in more depth.

On the first read, focus on the paper’s overall goals and techniques. First, read to develop a paper summary in your own words. Here are four questions the summary should answer:

• What is the goal of the paper’s research area? The goal can be something nebulous, like “improving security,” or something concrete, like “improving network utilization.” But in systems the goal usually ties back to something like “spending less money” or “handling more data.” Usually this comes out in the first couple sentences of the abstract and in the introduction. Often those sentences are clichés.
• What is the technical problem the paper is trying to solve? The problem is usually an obstacle to achieving the goal, and it should be concrete; for instance, “Big Data computations in the cloud are delayed too much by stragglers (nodes that complete their portion of the computation much slower than other nodes).” This also is found in the abstract or introduction, but it may be best explained in the conclusion section.
• What is the technical contribution that the paper uses to solve this problem? Is this contribution a new idea, or a combination of old ideas? A good introduction will present the contribution(s), but sometimes you have to dig deeper—into a section called “Design” or “Architecture”—to get to the technical meat.
• What is the evidence that proves the contribution actually addresses the problem? This is presented in depth in the “Evaluation” section, and should be previewed in the abstract and introduction.

Once you have answers to those questions, you can already think critically.

• How far apart are the goal and the problem? For instance, many security papers describe a large, society-level goal, using terms like “Security breaches are estimated to cost US commerce $10B a year,” but then actually address just a tiny slice of that overall goal.
• How much of the problem does the contribution address? Could you have achieved similar improvements using a simpler technique?
• How interesting is the technical contribution on its own? Sometimes a paper is interesting because it identifies and solves a new technical problem, even though its solution techniques seem obvious in retrospect. Sometimes a paper is interesting because it uses new solution techniques, even though those techniques address a problem that’s more easily solved in another way. The best papers do both, but that’s rare. The worst papers do neither.
• How much evidence is provided that the contribution works? Are the experiments well chosen?
• Does the paper actually advance the state of the art? How does it compare to related work? This is usually addressed in a separate “Related Work” section, which might come second (I prefer it there) or right before the conclusion.

With thoughts about these questions in mind, you can now go back and read the paper in more depth.

• The paper will usually present many techniques and ideas. Which of them seem most important, and which are filler?
• Are there any cool tricks and techniques that you could use in your own systems?
• What would you do differently?

The aim of reading papers critically is not to prove the paper wrong. Always remember that the authors spent much more time working on the paper than you did, and authors rarely lie. (But it does happen!) Instead, read actively, as if you’re in dialogue with the paper. Ask the paper tough questions, and then read to get the responses. If you don’t get a response, that is a flaw in the paper; then ask, is that flaw technical or in the exposition?

Presenting the paper

First share the summary with us. Assume we’ve read the paper, but that we need to be reminded of its contents. Use slides if you need to. (In computer systems most papers are presented at conferences, and the authors’ slides from the conference are often available on the Web. Use them!) Then talk about its coolest ideas and its biggest gaps. Share with us what you might have done differently. Your critical thinking will engage the class more and help us all to better understand the work.

Links

Presenting someone else’s research paper isn’t too different from presenting your own. Check out How to give a good research talk by Simon Peyton-Jones et al.

There’s some advice online about how to present at a journal club. Much of this advice applies specifically to medical papers, but some is good general advice. Some tips from Johns Hopkins