How to Referee a Math Paper

I have refereed over 90 papers and, after submitting my report for most of them, I felt I did a bad job. After all, if I’m refereeing, the implication is that I’m enforcing the rules. I’m supposed to certify a paper as correct or find an error. But usually I can’t do that. The authors worked collectively for months to prove their theorems, and I can only spare a few days or, more realistically, hours. How can I understand a complex argument better than its own authors in that time? Unless all the ideas in the paper are ideas I’ve personally worked with already, being an effective enforcement officer is impossible.

So I come up with filler. I point out that the paper’s title is misleading. I say there should be more examples accompanying definitions and theorems. I identify notation that is confusing, overloaded, or unnecessary. I suggest that the authors give more intuition where needed. After making this list of superficial improvements, I’ve used up all the time I can afford to spend, and I send in a report. Did I fail to complete my certification duties? Yes.

It took me a while to realize that certification isn’t what authors need though. They don’t need me to carefully check their proofs; they were trained extensively for that. Reviewers of particle physics papers don’t recreate the experiments! That would be ludicrous.

What authors need is help communicating. When writing their paper, most of their focus is on correctness, so they understandably think of the paper as consisting of theorem justifications. But this is misaligned with the needs of readers, who hardly ever care about the details of proofs. Readers don’t even care about theorems necessarily. They want intuition. William Thurston recorded his realization of this 5, page 174:

When I started working on foliations, I had the conception that what people wanted was to know the answers. I thought that what they sought was a collection of powerful proven theorems that might be applied to answer further mathematical questions. But that’s only one part of the story. More than the knowledge, people want personal understanding.

There is a huge difference between making correct arguments and effectively communicating ideas to a reader. So the feedback I was giving about examples and notation isn’t second-rate filler like I thought. It’s extremely valuable, because that’s where the majority of communication happens.

The good news is that often it’s easy to help authors communicate, and you don’t need to know more than they do. In fact, it’s better if you know less. You have a perspective that the authors don’t, because they never saw their paper for the first time. As an outsider, you can tell whether the new ideas are explained well and whether enough background information is provided. You can definitely tell where the intuition is missing, and you can tell where the writing is not easy to follow. You can tell how successfully a coherent picture emerges from all the pieces. Simply pay attention to your own level of understanding. When something is unclear, don’t accept the blame; it’s not your lack of intelligence but rather an opportunity for the authors to improve the exposition. If you’re insightful enough, you can eventually articulate what you wish the authors had told you to help you see the path more clearly. Sometimes you will then find that the authors did in fact say what you needed to hear, but they didn’t say it with enough clarity or emphasis or redundancy for it to sink in. Authors have blind spots in all these areas. Unless they are exceptionally good writers, they will need your help.

So is that how to referee a paper? Well, that’s how to review a paper. We use the two terms interchangeably, but we shouldn’t, because they have completely opposite connotations! Reviewing is more useful to the authors and to future readers.

A Short Checklist

If you’re not the right person to review a paper, then decline the request and suggest someone better if you can. If you do accept, here are specific items to look for.

Are the title and abstract accurate? These will receive more traffic than any other part of the paper, so they’ve got to be right! Authors often choose titles that are overly grandiose or too vague. Titles like “Binomial coefficients modulo prime powers” and “A note on $\beta$-expansions of real numbers” can apply equally well to hundreds of different papers, so they aren’t specific enough.

Are the theorem statements correct? I have seen quite a number of false theorem statements in submitted (and published) papers. Inevitably the problem isn’t with the proof but rather that the theorem statement doesn’t actually say what the authors think it says. The fix is quite simple, but the authors didn’t notice there was a problem. Experimenting and coming up with your own examples are better ways to find mistakes in a theorem than carefully reading a proof.

On that note: The three most important factors that determine the readability of a paper are (to borrow a real estate adage) examples, examples, examples. Are there enough examples? It’s much easier to understand a theorem through an example than through its statement.

Is the notation optimal? If you’re having trouble remembering all the notation, then so will every other reader. It’s too easy for authors to introduce notation, and it’s hard for a reader to keep track. Authors need checks and balances here. Almost every paper I review has notation that should be eliminated and other notation that should be improved. Ideally, notation should be self-documenting so it doesn’t require the reader to remember anything. For the base-$b$ representation of $n$, the notation $r_b(n)$ is more suggestive than $(n)_b$, but $\operatorname {rep}_b(n)$ is even better than $r_b(n)$. Mathematicians have a lot to learn from programmers, who figured out a long time ago that single-letter function names are a horrible idea.

If it becomes clear that the authors have not put as much time into polishing the paper as they should have, stop reading and kindly recommend rejection, no matter how good the results are. Yes, you’re allowed to do this, even as a new researcher in your field. Your time is valuable. You are providing highly skilled work that you are not getting paid for. If the authors are wasting your time, don’t let them waste even more of it.

Read Freely

Rather than reading a paper from start to finish, you should absolutely feel free to jump around, because other readers will jump around too, in search of what’s interesting to them. As you do this, you may find yourself frustratedly hunting for definitions or notation you “missed” by not reading linearly. In that case, the definitions and notation should be easier to find, either with more prominent placement or with explicit references to them (or by making the notation self-documenting!).

A paper should also be readable on multiple levels. Initially, skip the proofs. The proof details make up the bottom level, and hardly anyone will read at that level. A convincing and coherent argument should be clear at higher levels, and anyway you need to understand the outline and major steps before you can understand why any details are relevant. After writing their proofs, authors sometimes forget to go through the paper and provide all the connective tissue, in which case you’ll find that information is too fragmented to tell how all the parts relate to the whole without first putting in a ton of work to reconstruct the missing connections. It isn’t your job to do this work; request that the authors provide it.

Your report will typically begin with a few sentences summarizing the paper and explaining its contribution in the context of other results. If this is difficult to write, it means the authors neglected to include this in their paper! It should also be easy to write a few paragraphs summarizing the paper (for example, if you are reviewing for Mathematical Reviews). On smaller scales, it should be easy to summarize each section and determine its role relative to the others. If not, the authors forgot to summarize.

The Ideas Behind Proofs

Given that you’re already reviewing a paper, why not also check the details of the proofs? If you have the time and inclination for this, by all means. But as your career progresses and your responsibilities accumulate, you will have less time while simultaneously finding yourself qualified to review more papers. At some point, certifying correctness will become infeasible. Sure, authors will be grateful if you notice a mistake in a proof, just like they will be grateful if you point out a typo, but this isn’t your job.

Much more important than checking proof details is checking that the ideas behind proofs are stated and stated clearly. A long proof stripped of its ideas isn’t a vehicle for communication — it’s a magic trick. But this is an easy mistake to make! Often the few missing sentences are so fundamental to the proof that the authors took them for granted by the time they started writing. Their absence is conspicuous to a fresh pair of eyes.

Mistakes in a proof are much easier to detect when authors explain what they’re doing in addition to doing it. To quote Thurston again 5, page 169:

People are usually not very good in checking formal correctness of proofs, but they are quite good at detecting potential weaknesses or flaws in proofs.

Without a high-level explanation of a proof, readers who do want to follow the proof must resort to checking that one line follows from the previous, and they will likely make the same subtle mistakes as the authors.

Here is some advice to authors: Actually write down your hard-earned intuition. You know which theorems were difficult to prove and why; readers do not. Convey the significance of your results, and say where the novel ideas are. Do a little marketing! Advocate for your theorems, so that when a journal’s guidelines direct the reviewer to only recommend acceptance of important, not just correct, results, they have some material to work with.

Benefits of Reviewing

By reviewing rather than refereeing, you can approach a paper as an ally of the authors rather than as a gatekeeper. This is more satisfying for you and more healthy for the profession. Your job is to make the paper more digestible for the people who will read it after publication, whether it appears in the journal where it’s currently submitted or elsewhere. If you do that, you have made a positive contribution.

When you’re not holding yourself to the task of certifying correctness, there is no need to procrastinate for months as an avoidance coping strategy. I don’t know whether this is the primary cause of the excessive turnaround times in mathematical publishing, but it certainly doesn’t help.

Reviewing is still a job, and it still takes time, but maybe it can become something you look forward to. Personally I’m still working on that. But without a doubt, improving the writing of others makes you a better writer, so in addition to making one paper better you’re also improving all the future papers you have yet to write.

Other Hot Takes

As a community, we should discuss refereeing/reviewing more than we currently do. However, additional thoughts can be found in several previous pieces in the Notices 1234.