Think before you shrink: a story on battling with reviewers

My PhD-students are often disappointed when the peer reviews on their latest manuscript have entered their email boxes. They raise question like: “Why don’t they understand the point?”; “Did they read the manuscript properly?”; “What, is this not novel. What is then?”; “How many more comparisons/simulations should I do?”. For me the task to say some softening words. “It’s not that bad really”; “Please, bear in mind that most reports are 10% positive and 90% negative”; “the reviewer has a point, but maybe we can address this such and such”. But I have to admit: it is often a really tough process that, at times, may feel rather unfair.

Like many other senior scientists I have experience with both sides of the peer review process. I do understand that the power balance between authors and reviewers needs to be tilted towards the reviewers, as one may expect them to be more unbiased than the authors. The situation becomes more difficult when the manuscript is rather critical on other people’s work. Recently, I experienced this, ending up in a long-lasting, but polite battle between us – the authors – and them, the reviewers. All ends well, but let me take you through the journey as it was a learningful experience for me.

Some technicalities first
Shrinkage is a technique that stabilizes parameter estimates in models. For good old regression models, popular shrinkage techniques like lasso and ridge have proven their value in high-dimensional settings, implying more parameters to estimate than samples. Unfortunately, these techniques are often also blindly used in traditional, low-dimensional settings. Mostly in cases when the number of parameters is not huge, but still fairly large. For example, one has p = 30 parameters for predicting survival with n = 200 samples. Not unusual in medical prediction problems. In such a setting, plain regression without shrinkage can be applied, technically, but this may not be optimal in terms of prediction accuracy. Moreover, it likely renders the parameter estimates to be very imprecise.

So what do many people do: they use plain ridge or lasso shrinkage, which shrinks all parameters – the irrelevant and relevant ones – in the same way. Much like using one heavy hammer to smash all nails into the wood. Then, what do other people (our reviewers?) do: advocate that one should not use shrinkage, because it biases results and leads to bad calibration, as predictions are too compressed compared to the true values. They are right, but they don’t tell the whole story. Plain lasso and ridge are not the best way to do this. Because, what the story really should be about – in our opinion – is: you need to think before you shrink.

Our mistakes
So there we have the (initial) title of our manuscript. I will get back to this. But first: what did we write about? For once, we were not striving for novelty, but ‘just’ selling other people’s good ideas: grouped and local shrinkage. Without going into too much detail*: we showed that these solutions can strongly alleviate some of the criticisms on global shrinkage methods like lasso and ridge. We did this by redoing some of the simulations of the shrinkage criticasters. These showed that issues like miscalibration were, at least in part, resolved. Case closed, we thought. We forgot one big issue: communication. In hindsight almost all our mistakes that prolonged the review process can be deduced to that. Let me discuss those.

Mistake 1: Mathematical arrogance. These alternative shrinkage methods are somewhat more ‘mathematical’ than lasso and ridge. We just gave the equations and assumed that would do. Wrong. We are trying to convince a rather applied audience for whom ridge and lasso are quite advanced techniques already. So, we should illustrate what these other shrinkage techniques can bring. That’s what we did later on and now I’m quite proud of that illustration, in fact!

Mistake 2: Too many battles. Humans don’t like to be criticized, and will do as much as they can to avoid the criticism to spread. Or to downtune the message. We knew we would step on some toes. Yet, we initially chose to fight at least five battles, hurting even more toes. Only later we realized we’d better pick one or two.

Mistake 3: Ignorance about the criticasters. They consist of a fairly large community, making it quite likely that our manuscript will end up there for review. Moreover, they have some good reasons to bash shrinkage techniques like ridge and lasso. Most prominently: these techniques are misused by many practitioners as a panacee for having a too low sample size. Fair enough that they plea for a larger sample size; any statistician will agree with that. So, we had to frame our solution better: the best thing is to increase sample size, but if you can’t, here are some solutions that are better than the default ones, lasso and ridge.

The process
I don’t want to bore you too much with the details of the reviewing process. It took almost two years from initial submission to acceptance. This included one resubmission and three revisions. Initially, the manuscript was rejected. Of course, we didn’t agree. Usually, I’d say: take your loss and go to another journal. There is one exception. If you can prove that some of the statements by the reviewers are wrong, you have a chance when appealing to the decision. We did this successfully and got the chance to resubmit.

During the revision process it became clear that this manuscript sparked more discussion than usual, leading to three rounds of revisions before acceptance. I think it is very important for the field of biostatistics to have those discussions. Unfortunately, the format of this and most other journals is quite static, leaving little room for such discussions. Why not give one or two referee the floor – say one page linked to the paper – to reflect on it when the associate editor indeed senses that such a discussion may be appreciated by the community? The advantage for the authors would be that their standpoint would require less attenuation during the revision process, possibly making that process more efficient as well. In the role of referee I would have been prepared to do this for some papers.

The title
Finally, what about the title, ‘Think before you shrink…’? We lost that battle: one reviewer and the editor found it too provocative and we had to change it too something more neutral: ‘Alternatives to default shrinkage methods can improve …’. ^$. Correct, but somewhat boring. Fortunately, we also have other media nowadays to express our thoughts!

Notes:

*See here for a preprint with the original title.

^$See here for the paper as published