Reviews that Work

Posted in NEWS on November 16, 2005

By Wizards of the Coast

It’s the end of the day at a premiere event. You are tired, but also (hopefully) elated with another smooth day. At this point the head judge reminds the crew: “Don’t forget to perform judge reviews before you leave today.” Most judges get that “Oh darn” look in their eyes.

The review process is a necessary and (dare I say) interesting part of our job as judges. This simple process gives us an opportunity to assist our colleagues in improving themselves and, by extension, the DCI judge program as a whole.

What is the review process?

The review process is a formal means for judges to provide feedback to each other. From the philosophy section of the review form:

Judge review forms are a means to provide lasting reminders of constructive feedback to your peers. The judge program is a peer-review community based on mentorship between its members. The form should be the end-result of your mentorship with each other after an event, not the first feedback a judge hears from you.

The judge review process is an opportunity for the other judges at an event to provide you with some insight into their perception of your activities as a judge. Since everyone perceives things in a different way, getting another’s views of your actions can provide insights that might not have occurred to you. If you are willing to listen to another’s perception of your actions, you can possibly learn of areas in which you can further grow.

The review process provides an opportunity to provide constructive feedback. The reviewer will want to provide information on how they perceived the reviewed person’s actions and interactions with others (both positive and negative). Team leaders are expected to review their team members, and they are expected to review the leader. Knowing who you are expected to review is important, as the review process is a lot easier and more effective if you collect data during the day instead of waiting until the end of the day to try to remember what you observed.

Mentorship and feedback should occur at the end of the day of an event. Judges should sit with each other and provide each-other with feedback on their strengths and weaknesses face-to-face whenever possible. At this time the review form should be used to take _notes_ on the feedback you are presenting in person. After you return home use these notes to fill out a complete form electronically and email this form to the judge you reviewed and a copy to myself (please include the name of the judge reviewed in the subject line for archiving purposes).

  • Provide constructive feedback.
    It is important to concentrate on how you perceived the reviewed judge’s actions. Focus on presenting your observations and not drawing conclusions from them. You want to provide facts to the reviewed judge, not pass judgment on them. For example, you should say "You seemed abrupt with player ABC" (just an observation) instead of "You are a rude person since you seemed abrupt with player ABC" (a personal comment on the reviewed judge's character). Constructive feedback can be difficult to formulate. A future article will elaborate on different ways to express feedback.
    You want to point out two things that the person did well. These could be specific situations that the judge handled well. They could be techniques that they used to ensure that the event went well. They could be the ways that the judge interacts with players, other judges, staff and spectators. The review form provides a list of thirteen areas that you can provide feedback on. This list is intended as a starting point; you should feel free to comment on other areas where appropriate. There is no need to provide input on all 13 areas; simply pick two areas where you observed something about the person being reviewed and provide feedback.
    You also want to point out two or three things that the person might want to improve. These are things that you believe the judge could do, or work on, to improve themselves as a judge. These do not have to be negative (although the often are). You could provide different approaches to resolve situations the judge faced during the event. They could be procedures that could make the judge more efficient. The objective is not to tear the person apart, but rather to provide information that the person being reviewed might not be aware of about themselves or their procedures.
  • Provide face-to-face feedback.
    This is a relatively new requirement, but it is critical to the success of the process. Reviewers are expected to sit down with the person being reviewed and discuss the comments that will be on the review. For many people, this is a difficult process. It’s easy to provide praise, but often difficult to provide negative feedback. One useful approach is to have a hard copy of the review form in hand while you provide the face-to-face feedback. This will provide you with a convenient place to take notes to complete the review form with later.
    One of the major advantages to providing direct feedback is that it provides a more detailed discussion of the feedback being provided. This dialogue typically leads to a more thorough understanding of the feedback as it allows the judge to ask questions, discuss alternatives and identify specific opportunities for improvement.
    Another advantage of the face-to-face discussion is the opportunity for higher-level judges being reviewed to mentor the reviewer. A reviewer will sometimes identify a deliberate behavior on the part of a judge as an issue. The face-to-face discussion allows an opportunity to explain why the behavior might be acceptable (or even desirable) for the specific situation.
    The face-to-face discussion is also a training opportunity for the reviewer. As judges advance through the program, the ability to provide feedback to other judges becomes more critical. Mentoring and training judges cannot occur without providing feedback, so reviewers should see this as an opportunity to develop these skills.
  • Complete the review form.
    After the face-to-face, judges are expected to complete the actual review form. Starting now (2005/10/25), judges are able to complete the form online within the new DCI Exam website (DCIX), located at Log in using your DCI number and your Personal Information password. Click on the review link in the top menu bar, read the instructions, and use the online form to complete the review. This review form will be stored online and made available to the person reviewed, the reviewer, level 4 and above judges (for advancement purposes), and certain WotC personnel (Andy Heckt).

But why do we review?

  • To provide positive feedback for the things done well.
    There are dozens (if not hundreds) of decisions and actions a judge must make during a day. Most are simple, others are complex, and many of them are unconscious and automatic. Each of these gives us an opportunity to do something right whether we know it or not. When a colleague identifies things we’ve done well, it reinforces the actions that work well for us. If it was a behavior that we did not consciously choose, it makes it more likely that we might repeat it on a more conscious level the next time the same situation occurs. Now, being conscious of the behavior, we are also more likely to remember and include it in future mentoring opportunities.
  • To provide information on challenges that could use some improvement.
    Despite our best intentions, plans, and training, there may be situations we handle less than optimally. Receiving feedback from our colleagues is a good way of being made aware of these challenges. Once we are aware of things we might need to work on, we can actively act to improve ourselves. We can research what we might need to learn, or we can return to our home areas and discuss these areas with our mentors/peers/friends.
    The key is to provide food for thought to the reviewed judge. You want to provide them with some areas they can concentrate on in the future.
  • To allow for tracking improvement over time.
    Over a number of events, the review process allows the DCI, and us, to track improvement over time. This is an important part of our ongoing development (or lack thereof) as judges. If we keep seeing the same feedback over multiple events, then it remains a point of concern and we need to work harder at it. If we see that a prior negative feedback has now become positive feedback, it validates our efforts to improve.
  • Evaluating someone else forces us to evaluate our own work.
    A side effect of reviewing others is that it forces us to review our own efforts during an event.
    When we are reviewing a higher-level judge, we naturally compare ourselves to them…especially if they are at a level we aspire to. It points out things that we could do better. It can often point out methods and techniques that we have not experienced before, with our own mentors, and that we can bring back with us to our own parts of the world.
    When we are reviewing a lower level judge, we tend to identify actions we could have taken to assist them in performing better. Personally, if I notice a behavior that needs improvement, I have to evaluate what I could have done to assist the judge in improving it. As a part of providing the feedback, it is important that I also consider how the judge can improve on any challenges. This is probably the most critical component of the feedback…not only must I identify opportunities for improvement; I must also suggest methods for improving.
    When reviewing judges at our own level, we tend to compare them to ourselves. We see the things that they do better than us, as well as things that we might do differently. Being made aware of differences is a great way to improve.


For many judges, the review process is an uncomfortable part of working premiere events. You never know what to include as feedback, you don’t feel comfortable critiquing a higher-level judge, you don’t want to hurt someone’s feelings, you don’t want to interfere with someone’s advancement, etc.

Don’t look at the review as a confrontational process. It is intended as a learning process for both the reviewer and the person being reviewed. Look at it as an opportunity to strengthen yourself, your colleagues, and the DCI program as a whole.

Mike King, Level 3