Double-Blind Reviewing FAQ
The goal of the double-blind reviewing process is to help members make a judgement about the paper without bias, not to make it impossible for them to discover the authors if they were to try. Nothing should be done for the sake of anonymity that potentially could weaken the submission or makes the job of reviewing the paper more difficult (e.g., important background references should not be omitted or anonymized).
For Additional Information
A: Our goal is to give each a reviewer an unbiased "first look" at each paper. Studies have shown that a reviewer's attitude toward a submission may be affected, even unconsciously, by the identity of the author (see links at http://www.cs.princeton.edu/~dpw/popl/15/dbr-faq.html#studies more details). We want reviewers to be able to approach each submission without such involuntary reactions as "Barnaby; he writes a good paper" or "Who are these people? I have never heard of them." For this reason, we ask that authors to omit their names from their submissions, and that they avoid revealing their identity through citation. Note that many systems and security conferences use double-blind reviewing and have done so for years (e.g., SIGCOMM, OSDI, IEEE Security and Privacy, SIGMOD). POPL and PLDI have done it for the last several years.
A key principle to keep in mind is that we intend this process to be cooperative, not adversarial. If a reviewer does discover an author's identity through a subtle clue or oversight, the author will not be penalized.
A: Studies of blinding with the flavor we are using show that author identities remain unknown 53% to 79% of the time (see Snodgrass for details). Moreover, about 5-10% of the time (again, see Snodgrass), a reviewer is certain of the authors, but then turns out to be at least partially mistaken. So, while sometimes authorship can be guessed correctly, the question is, is imperfect blinding better than no blinding at all? If author names are not explicitly in front of the reviewer on the front page, does that help at all even for the remaining submissions where it would be possible to guess? Our conjecture is that on balance the answer is "yes".
A: We have heard of this happening, and this is indeed a serious issue. In the approach we are taking for ICSME, author names are revealed to reviewers after they have submitted their review. Therefore, a reviewer can correct their review if they indeed have penalized the authors inappropriately. Unblinding prior to the PC discussions also avoids abuses in which committee members end up advancing the cause of a paper with which they have a conflict.
A: Your job is not to make your identity undiscoverable but simply to make it possible for our reviewers to evaluate your submission without having to know who you are. The specific guidelines stated in the call for papers are simple: omit authors' names from your title page (or list them as "omitted for submission"), and when you cite your own work, refer to it in the third person. For example, if your name is Smith and you have worked on amphibious type systems, instead of saying "We extend our earlier work on statically typed toads (Smith 2004)," you might say "We extend Smith's (2004) earlier work on statically typed toads." Also, be sure not to include any acknowledgements that would give away your identity. If you have any questions, feel free to ask the Track Co-Chairs.
A: You can submit anonymized supplemental material through EasyChair. However, for materials that cannot, for some reason, be uploaded to the official site (e.g., a live demo), you can use anonymous hosting service and include a URL to that material in the paper. We emphasize that authors should strive to make their paper as convincing as possible within the submission page limit, in case reviewers choose not to access supplemental material. Also, see the next question.
A: In general, we discourage authors from providing supplementary materials via links to external web sites. It is possible to change the linked items after the submission deadline has passed, and, to be fair to all authors, we would like to be sure reviewers evaluate materials that have been completed prior to the submission deadline. Having said that, it is appropriate to link to items, such as an online demo, that can't easily be submitted. Needless to say, attempting to discover the reviewers for your paper by tracking visitors to such a demo site would be a breach of academic integrity. Supplementary items such as PDFs should always be uploaded to EasyChair.
A: No. The relationship between systems and authors changes over time, so there will be at least some doubt about authorship. Increasing this doubt by changing the system name would help with anonymity, but it would compromise the research process. In particular, changing the name requires explaining a lot about the system again because you can't just refer to the existing papers, which use the proper name. Not citing these papers runs the risk of the reviewers who know about the existing system thinking you are replicating earlier work. It is also confusing for the reviewers to read about the paper under Name X and then have the name be changed to Name Y. Will all the reviewers go and re-read the final version with the correct name? If not, they have the wrong name in their heads, which could be harmful in the long run.
A: No. But we recommend you do not use the same title for your ICSME submission, so that it is clearly distinguished from the prior paper. In general there is rarely a good reason to anonymize a citation. One possibility is for work that is tightly related to the present submission and is also under review. But such works may often be non-anonymous. When in doubt, contact the Track Co-Chairs.
A: As far as the authors' publicity actions are concerned, a paper under double-blind review is largely the same as a paper under regular (single-blind) review. Double-blind reviewing should not hinder the usual communication of results.
That said, we do ask that you not attempt to deliberately subvert the double-blind reviewing process by announcing the names of the authors of your paper to the potential reviewers of your paper. It is difficult to define exactly what counts as "subversion" here, but a blatant example might include sending individual e-mail to members of the Program Committee about your work (unless they are conflicted out anyway), or broadcasting the work on social media during the review period. If you're not sure about what constitutes "subversion", please consult directly with the Track Co-Chairs. If possible, we recommend that you avoid publishing the preprint of your work (e.g., on arXiv or on your website) until it is accepted for publication.
A: Using Double-blind reviewing does not change the principle that reviewers should not review papers with which they have a conflict of interest, even if they do not immediately know who the authors are. As an author, you should list PC members (and any others, since others may be asked for outside reviewers) whom you believe have a conflict with you. While particular criteria for making this determination may vary, please apply the ACM SIGSOFT guidelines on conflicts of interest, identifying a potential reviewer Bob as conflicted if:
- Bob is/was an advisor or advisee of yours. Note that this conflict never expires.
- Bob was affiliated with the same institution within the past 3 years, including anywhere in your organization or parent company. Note that this includes your previous affiliations if you have changed jobs within the past 3 years, and your future institutions if you are in the process of changing jobs.
- Bob was your co-author, pending co-author, or submitted co-author of a publication within the past 3 years, in any technical area, including patents and technical reports.
- Bob has a non-trivial financial stake in your work (e.g., invested in your startup company).
- Bob has or had had a financial relationship with you (e.g., grants, contracts, consultancies, equity investments, stock options, etc.) within the past 3 years.
- Bob’s personal goals would be furthered by acceptance or rejection (e.g., a competitor).
- Bob is a family member or close friend.
- Bob is you.
Also please identify institutions with which you are affiliated; all employees or affiliates of these institutions will also be considered conflicted.
If a possible reviewer does not meet the above criteria, please do not identify them as conflicted. Doing so could be viewed as an attempt to prevent a qualified, but possibly skeptical reviewer from reviewing your paper. If you nevertheless believe that a reviewer who does not meet the above criteria is conflicted, you may identify the person and send a note to the Track Co-Chairs.
A: If at any point you feel that the authors' actions are largely aimed at ensuring that potential reviewers know their identity, you should contact the Track Co-Chairs. Otherwise you should not treat double-blind reviewing differently from regular blind reviewing. In particular, you should refrain from seeking out information on the authors' identity, but if you discover it accidentally this will not automatically disqualify you as a reviewer. Use your best judgment.
A: Contact the Track Co-Chairs, who will download the material on your behalf and make it available to you.
A: The conference review system will ask that you identify conflicts of interest when you get an account on the submission system. Please see the related question applied to authors to decide how to identify conflicts. Feel free to also identify additional authors whose papers you feel you could not review fairly for reasons other than those given (e.g., strong personal friendship).
A: PC members are allowed to submit papers, which will be treated the same as other submitted papers.
Additional Information about Bias in Merit Reviewing