How to Write a DAC Paper Review
Andrew B. Kahng, January 2004
(Thanks to Igor Markov, Sachin Sapatnekar, Ion Mandoiu, Chuck Alpert and
Soha Hassoun for their inputs. Send your comments by email to firstname.lastname@example.org )
- GOALS OF THE DAC EXTERNAL REVIEW PROCESS
- Draw on your knowledge, insight, and experience in the field
- Help the technical program committee (TPC) make a good decision
- Help the authors improve their paper, whether for DAC or for another
- EVALUATION CRITERIA
- What are distinctions between "design tools/automation",
"methodology", and "design" papers?
- Design tools and design automation papers have at their core
novel algorithms (occasionally, algorithm implementations) for
important problems facing leading-edge IC designers.
- Methodology papers: Is this really a "methodology"? A methodology
has at its core a high-level recipe for composing and applying
point-tool, optimization and analysis building blocks to achieve
a particular design goal. A methodology may be very specific, both
in scope ("floorplan-stage substrate coupling prediction...") and
application domain ("... for design of wireless telephony
chipsets"). However, any worthwhile methodology innovation will
be scalable, migratable, and generalizable.
- No matter what type of paper you're reviewing, several universal
criteria apply. The most important is validation. Authors
must show that they went the distance to validate their claims,
and to fully elucidate the limitations of what they propose.
- Comparing with others' techniques is very useful, and should
be done whenever possible. Avoiding obvious
comparisons is typically not a promising sign.
- It is not always possible to evaluate ideas within a
design flow, but this should occur whenever possible.
- Costs (runtime and memory, on a well-specified platform)
should be clearly reported.
- Did the authors provide enough details for readers
to replicate the experiments?
- Over-incrementality should be avoided. If it's
"same idea, slight tweak", then perhaps it should not be given
1 percent of the DAC program space for the year (N.B.: DAC has
around 120 papers per year).
- GENERAL PRECEPTS
- WRITE COMMENTS, NOT JUST NUMBERS. Your job is NOT
done when you've entered a bunch of numbers ranging from 1 to 5.
These numbers have limited value, mostly to the authors - and are
worthless if they aren't supported by written comments.
(The main number that the TPC will use, along with your
Comments, is the "Overall Recommendation" number.)
- BE CONSTRUCTIVE. We're all here to advance the design
automation field. Help authors improve their work. Back up
what you say. Strong criticisms or negative recommendations,
especially, should be accompanied by concrete suggestions for
- Don't just say "this is well-known" - give instances of the ideas
that are replicated.
- Don't just say "inadequate experiments" - sketch an experiment
or a table/plot - and the desired conclusion - that would have
swung your verdict the other way.
- Don't just say "this could never work" - explain why you believe
such to be the case.
- BE FAIR-MINDED. It is good to list both positives and
negatives of the paper to provide a balanced picture.
- (HELP) MAKE A CALL, IF YOU CAN. There are 780+ submissions to
DAC-2004, and typically around 120 of these make it into the
technical program. If your recommendations are all "3"s then you
aren't really helping the TPC come to a decision. And, if you find
yourself recommending acceptance for every paper you see, perhaps
higher standards are warranted.
- RECOMMENDED STRUCTURE OF YOUR "COMMENTS" SECTION
Your comments are the most important part of your review.
Be as specific as you can - it is details that matter.
Give a sense of the conceptual and
technical contributions of the paper, as part of a bigger picture.
- State opinion and major concerns/benefits in the first paragraph.
What's your main point? Should the paper be in or out? Why?
Then, the next paragraphs of the review can discuss evidence.
- Thus, Paragraph 1 can say things such as "experimental
results are flawed", "writing is too poor", "contribution is
incremental", etc. - or the main highlights - "results are
very strong", "theory is solid", "well-written", "new idea
that hasn't been seen before", etc.
- Paragraphs 2, 3, 4 can then give evidence supporting the main
point. If the paper is incremental, give the other paper
reference, and explain why what's proposed isn't different.
By this time, the committee should have all it needs
to know to make a decision.
- You can then put in a line of dashes or some white
space, and continue with "Additional Comments". Start
bulleting different points that you'd like the authors to
address, or give more examples, with more detail.
- While TPC members may spend hundreds of hours on their
DAC assignments, the following comment from a TPC member is
noteworthy: "We have to look over hundreds of reviews
(5 per paper, 40 papers = 200 reviews). I want to know
in 10 seconds what the reviewer thinks the main reason for
acceptance or rejection is. And within 60 seconds I should
be able to know the evidence behind the recommendation."
- On References. It is a good idea to read (at least the abstracts of) the main
references cited, to evaluate contributions relative to previous work.
- On the "Author Comments" vs. "Committee Comments". The rule of thumb
is to word Author comments as you would speak - helpfully -
to the authors' faces.
And, if your review is matter-of-fact and unemotional, there is
little need to have separate comments to the committee -
unless there is a true need for confidentiality.
- On "Novelty". While "Reject - Not Novel" is often
an easy way out for the reviewer, please keep in mind the following.
- A well-argumented negative result may be as valuable as a
positive result if it resolves contradictions in existing
literature, prevents waste of resources, etc. However,
strong negative results are rare because "we tried it, and it
did not work" may be due to poor implementation.
- Reported re-implementations and evaluations of published
algorithms and methodologies, with positive or negative results,
are of interest. However, they must be described in sufficient
detail if they are to be a real contribution.
- Drastic simplifications/speed-ups of well-known algorithms
and methodologies may be of interest to DAC attendees.
- On "Anonymity". Submissions are required to be "anonymous",
with no obvious indicators (URLs, email addresses, etc.) of the
authors' identities. Anonymity helps preserve the blind nature
of the review process, and violations of this policy should be
flagged for the TPC members.
- On Double-Submission. DAC does not permit concurrent
submission to other conferences, nor does it permit submission of
previously published works. Instances of double-submission
should be flagged for the TPC members.