How to Write a DAC Paper Review

• GOALS OF THE DAC EXTERNAL REVIEW PROCESS
  • Draw on your knowledge, insight, and experience in the field
  • Help the technical program committee (TPC) make a good decision
  • Help the authors improve their paper, whether for DAC or for another future conference
  
  
  
• EVALUATION CRITERIA
  • What are distinctions between "design tools/automation", "methodology", and "design" papers?
    • Design tools and design automation papers have at their core novel algorithms (occasionally, algorithm implementations) for important problems facing leading-edge IC designers.
    • Methodology papers: Is this really a "methodology"? A methodology has at its core a high-level recipe for composing and applying point-tool, optimization and analysis building blocks to achieve a particular design goal. A methodology may be very specific, both in scope ("floorplan-stage substrate coupling prediction...") and application domain ("... for design of wireless telephony chipsets"). However, any worthwhile methodology innovation will be scalable, migratable, and generalizable.
  • No matter what type of paper you're reviewing, several universal criteria apply. The most important is validation. Authors must show that they went the distance to validate their claims, and to fully elucidate the limitations of what they propose.
    • Comparing with others' techniques is very useful, and should be done whenever possible. Avoiding obvious comparisons is typically not a promising sign.
    • It is not always possible to evaluate ideas within a design flow, but this should occur whenever possible.
    • Costs (runtime and memory, on a well-specified platform) should be clearly reported.
    • Did the authors provide enough details for readers to replicate the experiments?
  • Over-incrementality should be avoided. If it's "same idea, slight tweak", then perhaps it should not be given 1 percent of the DAC program space for the year (N.B.: DAC has around 120 papers per year).
  
  
  
• GENERAL PRECEPTS
  • WRITE COMMENTS, NOT JUST NUMBERS. Your job is NOT done when you've entered a bunch of numbers ranging from 1 to 5. These numbers have limited value, mostly to the authors - and are worthless if they aren't supported by written comments. (The main number that the TPC will use, along with your Comments, is the "Overall Recommendation" number.)
  • BE CONSTRUCTIVE. We're all here to advance the design automation field. Help authors improve their work. Back up what you say. Strong criticisms or negative recommendations, especially, should be accompanied by concrete suggestions for improvement.
    • Don't just say "this is well-known" - give instances of the ideas that are replicated.
    • Don't just say "inadequate experiments" - sketch an experiment or a table/plot - and the desired conclusion - that would have swung your verdict the other way.
    • Don't just say "this could never work" - explain why you believe such to be the case.
  • BE FAIR-MINDED. It is good to list both positives and negatives of the paper to provide a balanced picture.
  • (HELP) MAKE A CALL, IF YOU CAN. There are 780+ submissions to DAC-2004, and typically around 120 of these make it into the technical program. If your recommendations are all "3"s then you aren't really helping the TPC come to a decision. And, if you find yourself recommending acceptance for every paper you see, perhaps higher standards are warranted.
  
  
  
• RECOMMENDED STRUCTURE OF YOUR "COMMENTS" SECTION Your comments are the most important part of your review. Be as specific as you can - it is details that matter. Give a sense of the conceptual and technical contributions of the paper, as part of a bigger picture.
  • State opinion and major concerns/benefits in the first paragraph. What's your main point? Should the paper be in or out? Why? Then, the next paragraphs of the review can discuss evidence.
  • Thus, Paragraph 1 can say things such as "experimental results are flawed", "writing is too poor", "contribution is incremental", etc. - or the main highlights - "results are very strong", "theory is solid", "well-written", "new idea that hasn't been seen before", etc.
  • Paragraphs 2, 3, 4 can then give evidence supporting the main point. If the paper is incremental, give the other paper reference, and explain why what's proposed isn't different. By this time, the committee should have all it needs to know to make a decision.
  • You can then put in a line of dashes or some white space, and continue with "Additional Comments". Start bulleting different points that you'd like the authors to address, or give more examples, with more detail.
  • While TPC members may spend hundreds of hours on their DAC assignments, the following comment from a TPC member is noteworthy: "We have to look over hundreds of reviews (5 per paper, 40 papers = 200 reviews). I want to know in 10 seconds what the reviewer thinks the main reason for acceptance or rejection is. And within 60 seconds I should be able to know the evidence behind the recommendation."
  
  
  
• OTHER
  • On References. It is a good idea to read (at least the abstracts of) the main references cited, to evaluate contributions relative to previous work.
  • On the "Author Comments" vs. "Committee Comments". The rule of thumb is to word Author comments as you would speak - helpfully - to the authors' faces. And, if your review is matter-of-fact and unemotional, there is little need to have separate comments to the committee - unless there is a true need for confidentiality.
  • On "Novelty". While "Reject - Not Novel" is often an easy way out for the reviewer, please keep in mind the following.
    • A well-argumented negative result may be as valuable as a positive result if it resolves contradictions in existing literature, prevents waste of resources, etc. However, strong negative results are rare because "we tried it, and it did not work" may be due to poor implementation.
    • Reported re-implementations and evaluations of published algorithms and methodologies, with positive or negative results, are of interest. However, they must be described in sufficient detail if they are to be a real contribution.
    • Drastic simplifications/speed-ups of well-known algorithms and methodologies may be of interest to DAC attendees.
  • On "Anonymity". Submissions are required to be "anonymous", with no obvious indicators (URLs, email addresses, etc.) of the authors' identities. Anonymity helps preserve the blind nature of the review process, and violations of this policy should be flagged for the TPC members.
  • On Double-Submission. DAC does not permit concurrent submission to other conferences, nor does it permit submission of previously published works. Instances of double-submission should be flagged for the TPC members.