Submitting Content for the Judge Center

Posted in NEWS on July 3, 2007

By Wizards of the Coast

"Harry plays Goblin Grenade, targeting Peter, who is at 3 life. What happens?"

Most of us take for granted that the above question is so simple, that it's not worth testing a candidate with. However, there are at least three things the above question can possibly test. It's that we sometimes take knowledge of the rules for granted that we overlook how they can be tested. It is the intention of this article to shed light on what kinds of questions should be asked, as well as help anyone who wants to submit a question to the Judge Center help better their submission.

By the end of this article, I'll show you how the above question could be tested.

Research and Benchmarking

Before any content should be submitted to the Judge Center, you need to do a little research. Specifically on content that may already exist in the Judge Center and is in use on a live exam. Using the Orb of Insight in the Judge Center, you should look up not only the rules that your potential submission would test, but also the cards involved. While it may not tell you exactly how the rule or card is tested (since the actual content is hidden), it at least shows how many questions test knowledge of that rule or card. That is a good gauge of whether or not your submission may be a viable question for a live test. Only so many questions testing the same rules can be in the same test pool, especially when we want tests to be evenly balanced for the candidate. Tests that only consist of questions on how the various layers of continuous effects do not help establish how well a candidate may understand the Comprehensive Rules as a whole, which is more the need of any judge test.

Consider who the possible candidate for this question may be. While there may be no set standard on what rules should be known by a Level 1 candidate, there are certain rules that do lend themselves to testing a candidate on knowledge necessary to answer a question that a player may have. Good examples are knowledge of state-based effects, the steps to playing a spell, or the various steps and phases in a turn. This is what most people are expected to know when playing, but a judge would have to be even more knowledgeable about to answer a question. So, your submission could be written in order to test details of these rules that may be overlooked by most people, but a judge would have to know correctly in order to issue a ruling to someone playing in a Grand Prix Trial. A Level 2 candidate, who is expected to head judges events, would need to know the very same rule in more detail as they likely to have to handle an appeal.

For a Level 3 candidate, knowledge of the rules and how they are applied approaches an even more detailed level. This is especially true when considering that a Level 3 judge may have to explain these same rules to candidates they may be testing for Level 1 and Level 2. The same scenario posed to the Level 1 candidate could also be posed to the Level 3 candidate, but in a much more detailed and precise manner. The question posed to the Level 3 candidate is not only about knowing the steps to playing the spell, but specific interactions that may come into play as a result of this process. Good examples are knowing precisely when certain triggered abilities may trigger on a spell being played and how this interacts with certain cards, where the ability may or may not actually trigger.

I would certainly recommend utilizing the various practice exams in the Judge Center to see what content is being tested, and how it is being tested. Between the Easy Practice, Hard Practice, and the new L3 Practice tests, there are many examples of how knowledge of the rules and cards are tested. Doing this can also give you a feel for how detailed a question may need to be, and what is or is not appropriate to test. If you don't see a question involving multiple replacement effects on an Easy Practice test, that's because it's too difficult a question for the individual expected to take that kind of test. But, it is likely to appear in some form on the Hard Practice test, which likely identifies it as something that may be expected for a Level 2 or Level 3 candidate to know. The same is true depending on how many cards may be involved in a question: more cards usually means that more detailed knowledge may be expected from the candidate. Whereas two or three cards is better for establishing a baseline understanding of the rules.

Simply put, it is not only the rules tested that matter in the question, but also the level of detail expected for the candidate to know and identify from reading the question. Even how the question is presented and the format used (single answer versus multiple answer) can be used to differentiate what knowledge is expected for the candidate to know and how they are expected to apply that knowledge.

Writing and Formatting

There are a few conventions that the Judge Center uses for content, as well as a few expectations that we have for submissions. Most of this is detailed on the "Instructions" page of the Content section of the Judge Center. For example, the correct answers should be listed first in the answer key, player tags (for the active player and non-active player) should be used exclusively for names and pronouns, and card tags only need to be used for the first instance that a card is mentioned in a question. This is just standard formatting used for questions, and submitting questions that use this format can help ease the review and editing process.

Another important style point is the use of number words versus numerals in questions. While not specifically noted in the "Instructions", the usage of number words and numerals is standardized. Specifically, numbers are used for any time a player's life total is specified, when an amount of damage is referred to, or when life is gained or lost. So, a player isn't dealt two damage, a player is dealt 2 damage. Number words instead are used for quantities of things, such as the number of cards drawn, the number of counters on a creature, or how many copies of a spell may be created. A creature doesn't have 5 +1/+1 counters on it, it has five +1/+1 counters. It's also good to keep the quantities down in a question, as this makes it more manageable. Dealing 1,000 damage isn't usually necessary.

Lastly, as a general rule of thumb, all card names need to be spelled out completely. Whenever referring to a card, it's best to write out the name of the card specifically. If the active player plays Shock and targets the non-active players Goblin Cadet, don't then write "The Cadet is dealt 2 damage." As innocuous a problem this may be, when questions are translated into other languages, the names of the cards can be different than they appear in English. So, "Goblin Cadet" may be different enough in another language that "Cadet" isn't appropriate. We do make some exceptions for the names of certain legendary creatures (especially those from the Kamigawa block), but it's best to use the full card name when submitted then question, then the reviewers and editors can work with it.

Corner Cases and "Guru" Questions

Everyone likes to ask a question that stumps nearly every judge who hears it. After all, what is more fun than seeing just how much a person really knows about the rules. However, is that really a fair question to be asking a judge candidate? Does it test knowledge of the rule in question? Or does it really just test knowledge of that corner case?

Generalized questions are considerably more valuable as test questions, because they establish a working knowledge of the Comprehensive Rules. Which is what is expected that judge candidates should know. Any questions a player asks at a sanctioned event is often a good example of something that involves working knowledge of the Comprehensive Rules. This represents the kind of generalized knowledge that is expected of the judge by the very individuals we are supposed to be helping, which is a good benchmark for whether or not that kind of knowledge should be tested. This does not preclude an interaction that came up in a free-for-all game between three players, as a good question for knowledge can come from any source or situation, only that you have to be careful that the question isn't too "unique" a scenario.

Likewise, if it is difficult to cite an appropriate section of the Comprehensive Rules to explain the answer to a question, it may be that is knowledge of rulings associated with the card that aren't fully covered. While a ruling may have been issued by one of the Netreps, that doesn't make the question viable for testing. A candidate may not have access to that forum to begin with. Instead, a candidate is expected to have read through the Comprehensive Rules, so knowing what is in that document is more critical for testing purposes than knowing what rulings may exist for a card.

Returned Submissions

Sometimes a submission is returned to a submitter, even if that person has gone through the above process, identifying a question good for testing, isn't a corner case and doesn't look like it duplicates existing content. The reviewer will spell out the exact reasons why this is the case, for two reasons. Firstly, it helps the submitter to identify things that may have been overlooked in the original submission. There may have been a formatting error in the appearance of the submission. Or, the explanation of the correct answer is too long and detailed, and needs to be shortened to focus only on the critical details necessary to explain the correct answer. Secondly, another question could have been identified that tests the very same knowledge. Even with searches on references to the Comprehensive Rules and specific card names could miss that another similar enough question already exists.

The reviewers comments are designed to help better the submission, as well as get it into shape for reviewing and editing. Even in ideal circumstances, a submission can go through a number of revisions in order to make it suitable for use on a live test. Easing this process from the beginning can ensure that it goes smoothly for everyone involved. Likewise, perhaps there is an unclear spot in the scenario presented in the submission, or a misunderstanding involving a rule that should be cleared up by the submitter. As the person who submitted the question, you should make certain that it's clear to the next person who reads the question what is being asked.

In some questions, a returned question may be subsequently archived. This does happen, but the reviewers comments will explain why the question was ultimately archived. Most reasons are either because the question involved a corner case, or duplicated existing content. Archived questions are still in the Judge Center, and could be used at a later date. While unfortunate, we still want new submissions to the Judge Center, and you should still keep submitting questions as you think them up.


Now, back to the original question that I posed at the beginning of this article. How and what exactly do we want test knowledge of with this question? As I said, you can test at least three things from this question.

Firstly, you can test knowledge of the process by which the spell resolves. Specifically, when is Goblin Grenade's damage dealt versus when Goblin Grenade is put into the graveyard. If each of the answers somehow involves the order that this process occurs, then you are testing knowledge of 413.2b versus 413.2i. The candidate should know that the damage is dealt first, then Goblin Grenade is put into the graveyard. A candidate who incorrectly identifies that Goblin Grenade is put into the graveyard as the first step or resolving the spell could actually give an incorrect answer to a player in a situation where this order matters (for example, how Misdirection can change Counterspell's target to Misdirection itself).

Secondly, you can test knowledge of when state-based effects are checked. Specifically, the answers could involve identifying when they are checked versus when damage is dealt and when the spell is put into the graveyard. That adds knowledge of 408.1b, 408.1c and 420.3 to this mix. A candidate should be expected to know that the state-based effects are checked after Goblin Grenade is put into the graveyard. A candidate who identifies an answer where the damage is dealt first, then state-based effects are checked could tell a player in a real situation that Reborn Hero can't be returned to play after that player played Pyroclasm and had six other cards in the graveyard.

Lastly, you can test knowledge of knowing how the damage is dealt. There is sometimes a common misconception that the damage from Shock is assigned, then dealt separately, to the target. While it may not be something that is specifically tested for in a question, it can be used as a good way to rephrase an otherwise correct answer incorrectly. A candidate that chooses this answer may require a bit more work in reading and understanding the Comprehensive Rules.

Therefore, an answer to this question could possibly read:

"Goblin Grenade resolves, assigning 5 damage to Peter, then Goblin Grenade is put into Harry's graveyard. The 5 damage assigned to Peter resolves, and he loses the game."

While this may not be the most difficult question to pose someone, it is a example of a very real situation that does test knowledge of the rules. The same is true for asking a candidate to identify what card has a static ability from a list of five cards, or when a triggered ability may be put on the stack. While not as "sexy" as a question involving Humility, Opalescence, Pandemonium and Replenish, it is a better gauge of knowing the rules of the game. Which, ultimately, helps the DCI identify people who have the necessary knowledge to be certified judges.

Brian Schenck, L3