I know we all like to complain about the academic review process every now and then, but this time, I’d like to take a slightly different angle – from a reviewer’s perspective: Why not forget about our huge, competitive “flagship” conferences or organize them differently?
For instance, is IJCAI (probably not least because of the recent “AI” hype) becoming just a huge mass paper munging machine flooded with more scientific paper submissions than the PC can seriously cope with or is it only me who is getting too dumb over time to do proper reviews for such an event?
I mean: (i) I had in my review stack mostly ML papers although this is not my expertise and despite the topic preference choices that could be done in the system, then (ii) two almost identical papers that looked like a mix between self-plagiarism and auto-generation, and (iii) also when I look at the other reviews I get in general the impression – no offense – that many papers are being reviewed by non-experts giving high scores not even understanding the contribution in detail (how should you, if you tick “low confidence” on a compressed to 8 pages technically dense paper?). So, acceptance is probably very arbitrary and results can hardly be checked thoroughly… at least I am not sure how this can be coped with, and we as academic scholars have become deadline-surfers for those conferences to publish and as reviewers have become deadline-surfers for submitting the reviews in time… this both affects quality negatively.
I don’t have a solution, but I have the strong impression that these huge very competitive conferences turn into a huge waste of reviewers’ and authors’ resources…Don’t get me wrong, I am sure the PC chairs and area chairs will do the best possible to get the best papers in, but it is just getting more difficult by sheer scale, I am talking from my own experiences as PC chair (at admittedly smaller conferences than the likes of IJCAI, but I still plead guilty of not being able to assess overall whether the set of papers we accepted was the right one in each case).
It makes me think why to submit/review for those conferences at all and not just concentrate on workshops (where a narrow audience is at least really interested to discuss a concrete topic) and journals (anyways, many of my non-CS colleagues look down on conference publications not knowing how crazily tough they have become partially to get accepted). Even for good works where almost all reviewers admit they have learned something from, they “have to be reject due to the competitive nature of the conference” (haven’t we all used this “excuse” to reject borderline papers? – that’s against the sense of a conference: meet to discuss research and what you can learn from each other!
I for my part at least try to follow the rule not to reject any paper that I have learned something from, but it is very tough in the one-shot nature of conferences, where you cannot demand a revision most of the time and you feel torn between some factual error forcing rejection in such a setting vs. actual good ideas you have learned something from.
I know I am talking from the luxury position of someone who has “ticked the box” of having published in competitive conferences and that many younger colleagues struggle because they have to.
But I think this is just a huge problem of CS that is becoming bigger every year. So, why not let us resort to the model of journal publications only and extended abstracts (e.g. of those journals) to conferences… Some CS conferences, like VLDB and ICLP have already changed the system to have rolling calls on a journal and meet once a year to discuss the batch of papers accepted that are published in an associated journal, or combining the conference acceptance with a revision cycle for an associated journal while streamlining the review process for these journals.
And BTW, I don’t buy the excuse that CS is developing too fast to afford the long turn-around of journals. CS journals have (as I heard in a panel at VSL by Sweitze Roffel from Elsevier a while ago, if I remember correctly) *longer* journal review cycles than most other disciplines: no surprise, if we are all caught in the review deadline-surfing cycle for conferences, we need to postpone those journal reviews on our stack.
Opinions welcome!
Doing things with data 
We need more journal style conference submissions like VLDB!
yes, that’s a good summary… of course, then there are other topics and issues down the road, like whether our way of scientific publishing is still adequate, cf. e.g. some people believe
– that the blockchain might be the solution https://www.blockchainforscience.com/2018/02/09/sponbc2018/ (via @sherminvo) … I am not convinced (yet), but let’s see what the next years bring.
– that we can do publication and reviewing all on HTML on the Web https://linkedresearch.org/ (via @csarven) … I like the idea and cause a lot, but not sure we’re there yet.
anyway, both these topics are IMHO orthogonal, separate discussions.
p.s.: by orthogonal I meant that this blog was NOT about the way we publish, but about that *conferences* loose their sense of meeting and discussing research, if they are only based on a publish-or-perish principle.