What your references say about you
I’m reviewing papers this week and next. I suspect there will be several posts letting off steam or musing about meta-research.
Do other reviewers find themselves looking closely at the bibliography? I find my impression of the scholarship is influenced by it—I like it when the references show breadth in time and authorship, and I look to it to support or counter my impression of the work itself. Good, clear presentation of the problems and approach are usually accompanied by good, broad references covering (at a minimum) most of what I know of related work and (usually) more.
NIPS uses number citations (boo hiss), so on first read I don’t match up in-paper cites to the bibliography. But you can usually get a pretty good sense of who they’re referencing from the text (and I love it when people give context despite the style guide).
One stunning negative example I just ran into cited textbooks almost exclusively. Interestingly enough, I had been wondering as I read through the paper if the authors had read certain books/papers. Then, there they were. Alongside the textbooks, there were only three papers, none of which were about the algorithms being explored. You know when you’re reading along and just waiting for the a-ha moment when the authors point out that their problem is like this known task, where there is this family of approaches, and here’s the new thing they’re doing? Yeah. I hit the bibliography before that moment came. And then the textbook-heavy citation page reinforced the impression that they didn’t know what was actually going on in the relevant fields.
I suspect this distribution of citations is a very, very bad sign in general. It’s not that textbooks are bad as such, but it is difficult to believe someone has carefully read almost a dozen dense, long books and *none* of the original related work.
I’ve definitely cited textbooks myself, but I try to restrict it to the obvious and general things—like a well-known algorithm or “for a comprehensive overview.” I get nervous when my only reference for rather specific information is a textbook.
Anyone out there have similar reactions? Or am I alone in this?