Many questions arise while browsing an unknown, yet promising journal website. How will the editorial process work? Will the submission be acknowledged as received? How long will peer review be? Will the process be beneficial for the manuscript? Are reviews fair?
I think all of us could hear terrifying publication-related experiences from colleagues. What if we could have instant access to reviews of journals written by authors?
Indeed, there are proposals to write consumer reports of academic journals.
While people discuss on the tools to be employed for journal reviews, I decided to contribute with a review of a (for now) little known, yet promising and interesting publication venue: the Journal of Open Research Software (JORS).
This post is divided into two parts. In the first part, I briefly describe what is JORS and what it deals about. In the second part I will provide my experience as an author of a JORS article.
The Journal of Open Research Software
The Journal of Open Research Software (JORS) is a recent proposal in the Metajournals series by Ubiquity Press. It is an Open Access journal. This is how JORS defines itself:
The Journal of Open Research Software (JORS) features peer reviewed software papers describing research software with high reuse potential.
A software paper is a journal article written by following certain strict guidelines and it describes an open-source software written for research purposes. The description is written in terms of the software application, its implementation and architecture, the quality control in the development process, its availability (where to get it, how to install it), and its reuse potential within and outside the field of research of the authors.
Arguably, an immediate benefit when publishing a software paper is the gain in the visibility of the produced software and the resulting reuse by other researchers.
Often researchers make their software available on Github. Unfortunately, this is how the software often ends its scope. Usage of the software is ensured by some media coverage by the authors.
A published software paper enhances the visibility of a software written for research. Figure 1 is JORS description of the benefits of publishing software papers.
However, the most important benefit is that the software becomes citable. In many fields, especially those falling under the general subject of Computer Science, writing software is an activity that almost every researcher has to take. Sometimes the implementation of a software system is the most time-spending operation of a research activity. I do not see any reason why a software should not become part of the publication CV of researchers.
Figure 1: Journal of Open Research Software benefits for authors.
Picture licensed under a Creative Commons Attribution 3.0 License. Copyright Ubiquity Press.
It should be kept in mind that a software paper is a journal article and JORS is a journal, not a repository. All software papers are peer-reviewed and must satisfy the reviewers and the associated editor in order to be published. The review process is transparent, and the reviewer guidelines are publicly available.
Each published article receives a DOI, in order to be uniquely identified and properly referenced. The contents of the journal are also archived in the LOCKSS digital archiving system and the CLOCKSS distributed archive. The articles are safe and can be retrieved even if JORS suddenly disappears.
My Experience as an Author
Hello JORS
Near the end of March 2013, I was finishing the first public release of the Semat Accelerator, which is a Web-based modeling tool for SEMAT Essence theory of Software Engineering.
I developed the tool together with my supervisor Pekka Abrahamsson. The idea was to create a prototype as awareness creator for the theory. A tool would help practitioners and researchers to understand Essence, in order to evaluate its effectiveness in case studies and controlled experiments. The prototype was well received from SEMAT community, so we decided to open-source it with the hope that research could be built on top of it and to find collaboration.
As researchers, we discussed the possibility to publish the tool in a scientific venue. It turned out it is difficult to publish a software in our field, Software Engineering. Pity, isn’t it? Apparently, the only possibilities are to write a Workshop paper or a research paper which employs the tool. If you are fortunate, people will cite it when employing the tool for their own research purposes. There had to be a way to cite the software in a way that rewards the researchers.
On 2013-03-29, the following was tweeted from the @openscience Twitter account:
The Journal of Open Research Software from @UbiquityPress is accepting papers http://t.co/qfaffaR8WH #opensource #openaccess #openscience
— Open Science Fedn (@openscience) March 29, 2013
(I would like to thank all the people behind @openscience. You guys are handling the most informative Twitter account I am following.).
This is how we stumbled upon JORS.
The Submission Process
On the same day we discovered JORS, we contacted the Editor-in-Chief, Neil Chue Hong, who kindly answered to all our questions and solved all our doubts. We decided to adhere to JORS cause and to submit our work there.
We wrote the paper by employing the supplied templates (available as .doc and .odt). Online writing and submission is also possible. The major difficulty we encountered was that, despite the templates, we had no clue on what JORS would expect as the content in the software paper. There were no published software papers in JORS website. What an adventure for us! Fortunately, the journal describes the review process and the reviewing criteria carefully, in Figure 2.
Figure 2: Journal of Open Research Software submission process.
Picture licensed under a Creative Commons Attribution 3.0 License. Copyright Ubiquity Press.
We did our best and submitted the paper on 2013-04-05 (a Friday, late afternoon). The paper length was 7 pages (3 figures).
The editor acknowledged our submission on 2013-04-08 (Monday morning). We were informed that after an initial screening, the paper would have been peer-reviewed. Very well.
Meanwhile, the first two JORS articles had been published. Their content is not relevant to our field, but we thought that our submission was of a fair quality in comparison.
On 2013-05-06 we contacted Neil Chue Hong, asking what was typical review length. He promptly informed us that the manuscript was under review and that they expected them to arrive in a couple of weeks. We were also informed that JORS was assigned a Managing Editor, Samuel Moore from Ubiquity Press and that the editorial process would have gained benefit. This was already an interesting experience to perceive the difficulties when setting up a scientific journal.
We were contacted by the Managing Editor on 2013-06-18. Reviews arrived. We were positively surprised: around 1300 lines of feedback from two reviewers! This was near the amount of the text written in the paper (about 1900 words).
The feedback was a little bit harsh sometimes, especially because one reviewer was clearly not a fan of SEMAT Essence. However, it was very constructive. The article was accepted with major revisions. We were provided the reviews and a brief summary of the mandatory points to be clarified with the new submission.
The reviewers demanded both a revision of the paper and of the software. This was brilliant. They actually helped us to improve the software, not only the paper. We went back to work.
On 2013-07-26, we submitted the revised version and the updated software. The resulting work from peer-review was impressive. The revised article doubled in its size: 12 pages, 5 figures and 3850 words. The software drastically changed, too: 22 commits on Github, 124 changed files with 23,955 additions and 11,544 deletions. Both artifacts significantly improved. We submitted the revised paper with pride and satisfaction.
On 2013-08-12, the managing editor came back to us with a notification of acceptance. However, this was not the end of it. He kindly offered a slightly edited version of the manuscript (with track mode on), in which he corrected minor mistakes of our non-native English. He also highlighted those sentences requiring rewording, and he provided suggestions as Word comments. This was surely not expected and it was very welcome.
Our article is scheduled to be published tomorrow, 2013-09-02. I will update this post with a link to it.
The process sums up to 2 months, 13 days from the submission to the editorial decision. It should be noted that our submission occurred at the very beginning days of JORS. They were without a Managing Editor most of this time. Given that the second round of reviews arrived in 17 days, I expect JORS to be even quicker with future submissions.
Summing Up
My co-author Pekka Abrahamsson and I are very satisfied with the Journal of Open Research Software. I firmly believe that JORS has potential value to the scientific community and a significant reward in terms of personal satisfaction to have a software as a publication.
Especially in Computer Science-related fields, the time spent on building software for research activities is huge and seldom rewarded. JORS rewards researchers who write and share their software for research activities, and permits the possibility to third party researchers to reward the software developers, with citations.
Each published article is Open Access and released under the Creative Commons CC-BY license. This means that everybody is free to access, read and read the article. Additionally, everybody is free to redistribute the article, to modify it and even making commercial use of it given that the original work is properly attributed and cited.
Figure 3: Open Access Cookie.
Picture licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.0 License. Copyright biblioteekje.
Being Open Access, the article processing charges for an accepted paper is 25.00 GBP (roughly 30,00 EUR). However, the fees are fully waived if authors can not personally afford them. This is great.
The published articles look good, with a polished layout and a graphics pleasant for the eyes.
The people behind JORS handle a very transparent process. As author, I felt respected and treated as a human being in a friendly, constructive environment.
Feedback from authors is welcome at JORS. Indeed, Samuel Moore already received some of my suggestions to improve the submission process and the overall perception of the journal from its website. I think my involvement will not end tomorrow after the publication of the article, because I believe in the cause of existence of this journal and I will support it.
I recommend researchers in the fields of Software Engineering, Information Systems, and the whole Computer Science area to submit a software paper to the Journal of Open Research Software. We build software for research continuously. Let’s share it and be rewarded for this.