Similarly to what I did for the Journal of Open Research Software, I keep following some proposals to write consumer reports of academic journals. It is the turn of PeerJ.
PeerJ is an open access peer-reviewed journal in the areas of Biological, Medical and Health sciences. While it appears to be out of scope for a researcher in software engineering as I am, it accepts submissions from other fields if there is applicability in their subject areas. The journal was founded something more than one year and a half ago, and it started publishing one year ago. It was started by the former PLOS publisher Peter Binfield and the former VP of Mendeley R&D Jason Hoyt. The journal has been backed by Tim O’Reilly. Yes, that O’Reilly.
There is lot to be said about PeerJ, but here are what in my opinion are its three strengths:
It is fully (gold) open access, with the articles released under a CC-BY 3.0 license, yet it has a ridiculously convenient 99$ pay once, publish for life business model.
Its peer review criteria are only about the soundness of the science, not about how much “impact” or “novelty” articles carry. The review process is an opt-in open peer review model that spawns at two levels. Reviewers choose to reveal their identity to authors, and authors choose to make the review comments and article revisions public when the article is published. The editorial process is driven by independent academic editors only. That is, only the authors, one editor, and some reviewers are involved in the process. Additionally, post-publication peer review can happen through a Q/A mode.
Published articles look cool and modern. The PDF versions of articles are polished, and the typographic choices make them easy to read and pleasant for the eyes. The HTML versions look fine as well and offer interesting technological improvements like
As a bonus, the journal is well-indexed even for those researching in Computer Science and, for those who like impact factors, it is going to receive its first IF in about two years. I have to give another bonus: PeerJ claims a First Decision median time of 24 days. That is impressive.
Enough free advertisement. Time to tell my story.
My experience as an author
PeerJ has recently published the following article:
Graziotin D, Wang X, Abrahamsson P. (2014) Happy software developers solve problems better: psychological measurements in empirical software engineering. PeerJ 2:e289 http://dx.doi.org/10.7717/peerj.289
What the article is about will be the subject of another blog post. Strictly speaking, it is an empirical investigation on how the affects (emotions and moods) of software developers can be indicators of creative or analytic performance.
Why not a software engineering venue?
The reader might be curious. Why was this not submitted to a software engineering venue? Well, it was. This article was rejected before. I am sick of seeing researchers neglecting that articles are more often rejected than accepted. Let us be honest: everybody’s articles are rejected during peer review. Some of them are not. Whatever scientific article you are reading right now, there is a fair change that it was rejected elsewhere before. Reasons for rejections have been vague, e.g. “This manuscript does not carry enough value nor novelty“. One reviewer, however, suggested that “The paper seems more suitable for a psychology venue“. Now, that looked interesting. Maybe it was a provocation, maybe a suggestion. We followed it, and wanted to submit to a respectable open access venue. We did not have the money for PLOS One, and traditional publishers have exorbitant article processing charges for going open access. While lurking on Twitter (#openscience should be checked out daily), I discovered PeerJ, and how people (mainly from Biology) where excited about it.
We were curious about this journal, so I did what any curious researcher would do. I sent an e-mail to one PeerJ co-founder, Peter Binfield. His friendly reply arrived in 24 hours, and it encouraged submission. We decided to give it a try. Registration to the website is easy and resembles joining a social network.
We submitted on
2013-07-25. The submission process has been straightforward and currently the most pleasant in my whole life. PeerJ wants almost plain documents as .doc(x), .odt, .txt, .rtf, and .TeX. Tables and figures are submitted separately from the manuscript, which seems time-consuming, but it makes sense when submitting revisions. At the end of the submission process, a PDF is built, for author approval. An e-mail acknowledged the submission was immediately received. Four hours later, a second e-mail told us that the manuscript passed internal checks by PeerJ editorial staff. They were looking for an editor.
The review process
On 2013-07-29, we were informed that an academic editor was assigned to the article. Nice! In the submission system, the article was shown as “being peer-reviewed”.
However, about 20 days later, the article switched back to the status “awaiting for an academic editor”. I contacted Peter Binfield, who immediately replied (he was already writing me an e-mail, I discovered). Unfortunately, the assigned academic editor had to withdraw from the commitment. On the other hand, they could find another academic editor, who sent out several review invitations already. Unfortunate event but no issues at all. We were happy that they informed us. On 2013-08-17, we received the official e-mail “An Academic Editor has been assigned to your PeerJ submission”.
On 2013-09-08, we received the first editorial decision. It was major revisions. The assigned academic editor, Shane T. Mueller, could spot that the reviewers were quite unhappy with the manuscript. One of them even suggested rejection. However–and I think this was because he is a psychologist with a strong Computer Science background–he could grasp that the dissatisfaction with the manuscript was more because of how it was written than with its basic science. Indeed, the manuscript was written by a novice researcher in software engineering who structured it as a research article for software engineering. Yet, it was sent to a now psychology venue. Instead of rejecting the paper, the editor decided to give us another possibility and wrote detailed instructions on what he would expect as a revision. We happily followed the requests of the editor and the reviewers. As it turned out, I needed to carefully read the Publication Manual of the American Psychological Association which, despite of some criticisms, is a nice publication manual. Other rewriting was needed because we missed some important articles in the literature review. I sat down and began reading further papers from psychology research. A heavy learning process, still interesting and pleasant. We submitted our revised, re-structured article with its related rebuttal letter. We could address all the reviewers concerns, otherwise we commented them.
On 2013-11-2013, we received a second editorial decision. It was major revisions again. But, it was a happily unusual one. The academic editor was still not satisfied with the structure and wording of the papers, and with some missing citations. He did not even send it out for review. However, instead of just trashing it, he reviewed the article. Actually, he wrote a nearly 4000 words review providing enhancements for the text. I have never seen anything like that before. He also addressed some concerns as well. We submitted a revised version, taking into account all of his suggestions and concerns.
The peer review process actually felt like the famous cartoon by by Nick D Kim (http://strange-matter.net) but in a constructive way.
On 2014-01-17, we received a third editorial decision. Minor revisions this time! The manuscript was sent out to one of the first two reviewers, who gave a thumbs up for it. Happy news! Still, the reviewer addressed a minor concern regarding the discussion of related work. We revised the text again, and submitted. At this point, the manuscript was dramatically improved by the editorial process.
We decided it was time to pay our 99$/author fee. Unfortunately, the receipt had to look in a certain way in order for our budgeting system to reimburse the money. No issues at all. Jason Hoyt was happy to perform the necessary modifications for us. All in a matter of hours.
Finally, on 2014-02-04 we received a thumbs up from the editor. Still, in the editorial decisions, he suggested some wording enhancements. This paper required quite some work, but it has been a gratifying experience. We are thankful to the academic editor, who put a lot of effort and was able to understand that some misunderstanding with the reviewers was born because we come from another domain. I have never had such a constructive review process.
Our reviews are public
If the reader wants to know more about the review process, here is the review history of our paper. I am not afraid to share it. People are often academically scolded when they submit a paper, and papers do dramatically change during the revision process. Good reviewers and editors deserve credit, when they perform a sane peer review process. As much as it happens with hidden rejections, researchers should just stop hiding this stuff. It just happens to everybody.
Apparently, PhD students around the world are also happy to see public peer reviews.
It was time to focus on the production of the article. On 2014-02-05 I was contacted by Jacqueline Thai, the head of PeerJ Publishing Operations. We needed to obtain permission to reproduce Figure 1. It was not that easy, as several authors of the photographs are afraid to have their picture posted in a journal article. Jackie has been very supportive and helpful. Once we obtained permission (thank you again, i.witness), it was time to correct the proof. PeerJ sends out the most beautiful proofs ever. The reason? They look like the published PDF. In addition, Jackie performed her own review of the language, and let me perform minor corrections.
After that, all was done. On 2014-03-11, our paper went live.
Article dissemination
Immediatelly upon publication, the article received a lot of support from PeerJ social network accounts. Especially on Twitter, where it was tweeted several times
Do Happy software developers solve problems better? New research in PeerJ #Human–Computer-Interaction https://t.co/XE3qVFMUjk
— PeerJ (@thePeerJ) March 11, 2014
Additionally, Jason Hoyt took the time to personally post the link on Hacker News as well. That was beneficial for the visibility of the paper. In a matter of minutes, the paper appeared on SoylentNews and discussions began on Twitter, especially about our decision to make the reviews public. The buzz expanded, and ITworld decided to give it visibility. After all, the paper is open access. All these people could just read it.
Would all of this be possible without the article being open access? I do not think so.
The ITworld article significantly added visibility to the article. And then, 48 hours after the publication, the big news: we have been slashdotted! As a computer scientist, it has been a great honor. I honestly do not think that this could have happened in such a short period, if the paper was not open access.
Summing up
I wholeheartedly recommend submitting to PeerJ. Although it might be difficult to fit a software engineering paper there–unless it is about human aspects in software development–but other Computer Science research areas fit well there, e.g., HCI, Computational Science, and Computational Biology.
What I liked most about PeerJ, besides the peer review process that was more tied to the editor and the reviewers, is how PeerJ staff has treated me in a friendly, human way. They strive to support you, and they do their best to disseminate your work. As a plus, I have never seen articles looking that good graphically. PeerJ business model is sustainable for authors (hopefully for the journal as well) and its way of encouraging open, public, pre- and post-publication peer review is, in my opinion, the best approach ever. PeerJ encourages you to be as much open as possible. But it never forces you.
The first editorial decision did not arrive 24 days after we submitted. Do we care? No. Despite the initial difficulties in finding the editor and the reviewers for this manuscript, everything went smooth and fast. Still way faster than our experience with traditional venues. Faster than academic conferences, too.
PeerJ people made me happy to submit there, and I hope to conduct more studies in the future that will fit the journal scope. Otherwise, I will do my best to convince them to open a section regarding software engineering, or at least about human aspects in software development.
Featured image credits: PeerJ, from https://peerj.com/about/press