Double Blind Review Process FAQ

1. Why Double­Blind?
2. How to prepare your ICST2018 submission for double-­blind reviewing?
3. I previously published an earlier version of this work in a venue that doesn’t have double-blind. What should I do about acknowledging that previous work?
4. Our submission makes use of work from a PhD/master’s thesis dissertation/report which has been published. Citing the dissertation might compromise anonymity. What should we do?
5. I’m submitting to the industry track. Should i double-blind my submission?
6. I want to include a link to an online appendix in my submission. How should I do this?

Q: Why Double­Blind?

There are many reasons for this decision but in short we think that current evidence points to it being the most ethical and professional thing to do: it can reduce bias against gender, less well-known individuals, groups, universities, etc and the “costs” for its implementation are relatively low. For those who are interested in further motivations for double blind reviewing, a very well­ argued, referenced and evidenced article in favour of double blind review processes for Software Engineering conferences can be found in the blog post by Claire Le Goues: https://www.cs.cmu.edu/~clegoues/double-blind.html. And here is a list of double-blind resources from Robert Feldt: http://www.robertfeldt.net/advice/double_blind_reviewing/. And a more formal study of the subject by Moritz Beller​​ and Alberto Bacchelli​​: https://peerj.com/preprints/1757/

Q: How to prepare your ICST2018 submission for double-­blind reviewing?

In order to comply, you do not have to make your identity undiscoverable; the double­-blind aspect of the review process is not an adversarial identity discovery process. Essentially, the guiding principle should be to maximize the number of people who could plausibly be authors, subject to the constraint that no change is made to any technical details of the work. Therefore, you should ensure that the reviewers are able to read and review your paper without having to know who any of the authors are. Specifically, this involves at least the following four points:

1. Omitting all authors’ names, affiliations, emails and related information from the title page as well as in the paper itself.
2. Referring to your own work in the third person. You should not change the names of your own tools, approaches or systems, since this would clearly compromise the review process. It breaks the constraint that “no change is made to any technical details of the work”. However, you should always refer to the authorship/provenance of tools, approaches or systems in the third person, so that it is credible that another author could have written your paper.
3. Not relying on supplementary material (your web site, GitHub repository, YouTube channel) in the paper. Supplementary information might result in revealing author identities.
4. Anonymize project and grant names and numbers or those of funding agencies or countries as well as any acknolwedgements of support to the work you report on.

We further expect you to follow the excellent advice on anonymization from ACM: https://icer.hosting.acm.org/icer-2017/additional-advice-on-anonymization/

When anonymizing your email, affiliations, name etc. try to refrain from being overly creative or “funny” by coming up with your own, anonymized versions. For emails preferably use author1@anon.org, author2@anon.org, etc. since initial DBR screening will be done by an automated tool.

Q: I previously published an earlier version of this work in a venue that doesn’t have double-blind. What should I do about acknowledging that previous work?

Double-blind does not and cannot mean that it’s impossible for the referees to discover the identity of the author. However, we require authors to help make it easy for author identity to not play a role in the reviewing process. Therefore, we ask that in the materials you submit to us to be reviewed author identity is not revealed.

If the work you are submitting for review has previously been published in a non-peer-reviewed venue (e.g., arXiv departmental tech report), there is no need to cite it, because unrefereed work is not truly part of the scientific literature. If the previous work is published in a peer-reviewed venue, then it should be cited, but in the third person so that it is not clear whether or not this work was done by the author of the submitted paper or some other set of authors unknown. However, if citing in the third person would still risk that it is easy to identify the authors please err on the side of caution by also anonymising the papers being extended (both when cited and in the reference list).

Q: Our submission makes use of work from a PhD/master’s thesis dissertation/report which has been published. Citing the dissertation might compromise anonymity. What should we do?

It’s perfectly OK to publish work from a PhD/master’s thesis, and there’s no need to cite it in the version submitted for review because prior dissertation publication does not compromise novelty. In the final (post-review, camera ready) version of the paper, please do cite the dissertation to acknowledge its contribution, but in the refereed version of the paper that you submit, please refrain from citing the dissertation.

However, you need not worry whether or not the dissertation has appeared, since your job is only to help the committee review your work without awareness of author identity, but not to make it impossible for them to discover the identity of authors. The referees will be trying hard not to discover the authors’ identity, so they will likely not be searching the web to check whether there is a dissertation related to this work.

Q: I’m submitting to the industry track. Should i double-blind my submission?

No you should not. Since industry papers typically relies heavily on the industrial/practical context in which the work was carried out it would be too much to ask to require this context to be anonymized.

Q: I want to include a link to an online appendix in my submission. How should I do this?

Ideally the information in the appendix should be anonymous and it should be uploaded to an anonymous service such as for example figshare. These sites will give you a link that is anonymous. Later, if the paper is accepted you can turn that link into a non-anonymized link or just put the appendix on your own site and change the link in the camera ready version of the paper. An alternative solution is to not include the link in the submission; normally papers should be possible to review based on only the material of the paper itself.

To upload material on figshare please create an account there, then add a new item, use the keywords “Supplemental Materials” and add the other item-specific data and then select “Make file(s) confidential” and select “Generate private link”. Copy the url generated there and then “Save changes”. Your file(s) can now be accessed anonymously at the given url so you can put it in your ICST submission.