Nuts & Bolts: Evaluation

Posted in Making Magic on February 20, 2017

By Mark Rosewater

Working in R&D since '95, Mark became Magic head designer in '03. His hobbies: spending time with family, writing about Magic in all mediums, and creating short bios.

Welcome to "Nuts & Bolts," my annual series dedicated to players interested in designing their own personal Magic sets. It's a little more technical than most of my columns, but a good insight into, well, the nuts and bolts of what it takes to design a Magic set.

As always, let me quickly recap the previous Nuts & Bolts articles:

Nuts & Bolts #1: Card Codes

The first column was the most technical, explaining how we use a system to make sure everyone is always talking about the same card.

Nuts & Bolts #2: Design Skeleton

The second column introduces the most important tool in designing a set, something called a design skeleton. (It makes use of card codes, which is why that article came first.)

Nuts & Bolts #3: Filling in the Design Skeleton

The third column talks about how you begin filling in your design skeleton, starting with the common cards.

Nuts & Bolts #4: Higher Rarities

The fourth column talks about filling in all the other rarities.

Nuts & Bolts #5: Initial Playtesting

The fifth column discusses how to best use playtesting to gather feedback and make improvements on your set.

Nuts & Bolts #6: Iteration

The sixth column talks about the concept of iteration and how you can incrementally improve your set.

Nuts & Bolts #7: Three Stages of Design

The seventh column explains the three different stages of design, walking you through how your priorities shift as the set evolves.

Nuts & Bolts #8: Troubleshooting

The eighth column answered a number of questions about common problems that can happen in early-to-mid design.

Which brings us to today's column, number nine in the series. You've made your design skeleton, filled it in, playtested it, iterated it, evolved it, troubleshot it. Now we've gotten to the point where we need to put our pencils down, step back from the set, and evaluate it with a fresh eye.

Seeing the Bigger Picture

One of the most difficult things to do as a designer is learning to see your own work with an unbiased eye. Along the way, you've gotten close to the project. You've made so many decisions and have spent giant amounts of time thinking about it. It's hard to distance yourself from the emotional bond you've formed with your set, but part of designing is being able to answer some tough questions—and that means you have to get to a point where you can recognize ugly attributes of your baby.

The first trick is to get a little bit of distance. Before an evaluation, I like to give myself some time away from the set. I let my brain move off it and onto something else so that when I come back to look at it, I have a fresher perspective. One of the best times to do a review is after a vacation or following a period of time highly focusing on another task.

The second trick is to ask a series of questions so that you're focused on answering the questions rather than trying to subjectively judge your set. Today I will be walking you through the seven major questions I ask myself when I'm evaluating one of my own sets.

The third trick is internalizing the idea that by being harsh, by asking tough questions of your set, that you are helping to make it better. There's a time where you need to dote on your set and give it all the love and attention you can, but there's also a time where you need to be able to step back and look objectively to make sure that you haven't drifted, that intimacy with your set hasn't made you blind to changes that have happened.

The fourth trick is not doing this alone. If you are working with a team, this is something all of you can be involved in. If you're working by yourself, pull in someone you trust who can give you their take on these questions. Having other opinions to bounce off of makes this process much easier.

With all that said, let's get onto the seven questions.

Question #1: Does Your Set Have a Unified Vision? (Is It About Something?)

You have up to seven words. Describe your set. Can you do it?

Kaladesh: Feel like an optimistic aetherpunk inventor

Shadows of Innistrad: Insanity engulfs a Gothic horror plane

Battle for Zendikar: Stop the alien Eldrazi from destroying Zendikar

The first question asks if you've created a bunch of individual cards or a cohesive set. A Magic set is not a random selection of cards thrown together; it's a carefully crafted experience. All the components of your set have to be working together to tell one consistent story. I don't mean the literal story (although that can help) but rather what the player is doing when they encounter your mechanics and cards. What experience do you mean for them to have? What does it mean to play your set?

One of the biggest problems I see with novice designs is that there's a lack of focus. The designer collects a bunch of different things that they like in isolation and then puts them together. A set requires more than just "enough cards." It requires that all the cards work together to create something greater than the sum of its parts.

The metaphor I'll use for this question is to think of your set like a recipe. You don't make a good recipe by just throwing together various tasty flavors. A good recipe is finding a group of ingredients that taste good in conjunction with one another. No one eats a cookie and compliments you on the butter or the flour. They compliment you on the cookie.

Your set is no different than the cookie. You don't want your audience focused on what's in the cookie; you want them focused on how great the cookie, as a single unified thing, is. Success at set design is not the audience liking a piece of it, but rather them enjoying the entirety of what your set is doing.

The reason I ask this first is because if you don't pass this test, nothing else is going to matter. If your set, at its core, isn't about something, it's never going to satisfy the questions that come next. And be aware that it's possible that when your set started, it was about something, but as you made changes to make it work mechanically, it may have lost that focus.

Question #2: Does Everything in Your Set Serve That Unified Vision?

Once it's clear your set is about one specific vision, the next question is whether all the various pieces tie into that vision. That's not to say your set can't have numerous functions, but does each serve the larger goal? Kaladesh, for instance, was an aetherpunk world that made the players feel like inventors. Different parts of the set captured different aspects of that vision. Some of the set was creating the atmosphere of the world, some building the structure that allowed different pieces to be clicked together, some working to make the mechanics interact and the gameplay flow smoothly. Your set gets to have lots of different elements. This question asks if everything in the set is working in the same direction toward your vision.

Another common mistake of novice designs is to include things not because they serve the set, but because you like them in a vacuum. It's such a cool card, who cares if it doesn't fit in with what the set's doing? The answer is you have to care because in a trading card game, you don't control the order in which your audience is going to see your set. Any card could be the introduction. And every card has the potential to play with each other card in your set, so any card in your set could shape how the other cards are viewed.

For this question, I'll use a movie metaphor. Pick your favorite humorous movie scene of all time. The one that just makes you laugh every time you see it. I'm going to take that scene and put it in a different movie of my choosing. That movie is going to be Schindler's List, an Academy Award–winning film about one man's mission to save Jewish lives during the Holocaust. If you've never seen it, it's an emotional, gut-wrenching drama. What would adding your comedic movie scene do to the film? It would feel grossly out of place and probably be a lot less funny, possibly even offensive. Context matters. Things are not judged in isolation, but through their relationship with everything around them.

That card that's just so awesome in a vacuum might be horribly unfun in the context of your set. Or it might lead players to create expectations that your set is never going to deliver on. A player drafting your set might be getting signals that aren't true. A Magic set is judged by its interactions, which means that every component of your set has to be working in the same direction. This question forces you to look at every card and ask yourself, "Is my set better because this card is here?" If the answer to any card is no, the card has to change.

Question #3: Do the Pieces Work Together?

Every element of your set follows your unified vision. That means that all of it should work together wonderfully. Uh, no. There are many ways to follow a theme, and some of the ways can contradict. I'll give an example. I've decided to make Mons happy by making a set I'll call Goblin Explosion. Goblin Explosion is a tribal set dedicated to Goblins. Every creature in it is a Goblin (Goblin Dwarves, Goblin Merfolk, Goblin Zombies, Goblin Elves, you name it). Every non-creature spell references Goblins in some way.

Let's imagine I decided to make a bunch of the creatures have static abilities that grant things to Goblins. "All Goblins gain Haste and Menace." That kind of thing. Then also imagine I made a mechanic for instants and sorceries that allowed you to fork the spell if you sacrifice a Goblin. Each of those pieces is in theme, but they don't necessarily work with one another. The static abilities want there to be a lot of Goblins on the battlefield. The spell mechanic wants you to sacrifice Goblins. Those two things don't normally go together.

This question is asking whether the different elements are working in conjunction with one another. Another novice mistake is overvaluing tension. "Oh, you'll have difficult decisions to make, that'll make things fun." Most often, that isn't true. The best sets give clear messages of what the players are supposed to want. The fun comes in figuring out how to achieve those goals in the most efficient and powerful way, not in trying to figure out what the goal is supposed to be.

To use a real-life example, in early Scars of Mirrodin, we messed around with having cards that rewarded you for being poisoned. Wouldn't it be cool if sometimes it was beneficial to have poison counters? What ended up happening though was players weren't sure if they were supposed to attack or not with poison creatures, and if they were wrong, they felt as if they'd made a mistake even if there was no information at the time to tell them which was the right call. We took that stuff out and players had a lot better time because they then knew what to do with their poison creatures—attack!

When fleshing out your theme, it's fine to examine a lot of different avenues to figure out what you like best. But at some point, you have to pick your direction and then focus on that. This question makes sure that you've picked a direction within your vision so that various parts of your design aren't working at odds with one another.

Question 4: Is It Fun to Play?

Your set is about something. All of it. And the pieces are working in conjunction rather than at odds with one another. The next question is a very basic one: is the set enjoyable to play?

This might seem like a silly question, but it's actually quite a serious one. A common mistake I see with novice designs is they get so focused on all the minutiae of trying to make the set work, they take their eye off the big picture. A game, or in this case a set, lives or dies on one basic element: do people want to play it? Is the act of doing what your set is asking the players to do fun?

Of all the questions I'm asking of you, this one is the easiest to get blind to because the act of making something, of dwelling weeks upon weeks and months upon months on the same designs, dulls you to the connection between the designer, the player, and the cards. It's similar to what happens to a film editor after they've watched the same scene hundreds of times. It becomes difficult to form an untainted opinion when you're so intimately involved with the thing in question.

This is partly why you have playtests and involve other people. It's important though to get a little distance personally so that you can reapproach your own design. Part of leading something is being the one to sign off on what your design has become. You have to be able to pull back and see the forest for the trees. This requires playing with your set and being honest with yourself about whether you enjoy what's being asked of you as a player.

The interesting thing about this process is that when you change your vantage point, when you get your head out of the details, you'll at times find a clarity that you didn't previously have. I often discover during evaluation time that I can start to think of my design in a different way, which allows me the freedom to analyze it in a new light and start to see if I drifted to a place that isn't living up to the vision's potential.

Note that this question isn't asking you to evaluate all the components, but rather check in on whether not the larger focus you've chosen is inherently going to lead to a fun Magic set.

Question 5: Is There Depth? (Is There Room for Discovery?)

This next question wants to evaluate your set on a different vector, so let me talk about something I call layering. This is a writing technique I learned from my favorite college writing professor. One day she said to us, "Whenever you write, you want to make sure that each time someone reads your work, they're getting something different out of it—that the first reading is a different experience than the second, which is a different experience from the third, and so on."

A lot of us liked the idea, but didn't understand how you would do that. She used as an example a famous Japanese movie called Rashomon. While most of you have probably not seen the original movie, the type of story it created has become a staple of modern story telling. A "Rashomon" story is one where you keep hearing parts of it from different people's perspective, and with new perspectives you start to reevaluate previous information you had.

My professor explained that each reading of your story should allow the reader to have new perspective such that, like a "Rashomon" story, the reader reevaluates what they already know. Just think about any story where there was a twist. On the second reading, you know the twist and can start seeing where the author began setting it up, dropping clues that you previously missed.

Layering is when you stick things in your story that will not be fully understood without information gathered elsewhere, usually occurring later in the story. For a trading card game, layering is adding in things players can learn, but not until they understand other aspects of the set. The idea is you build in parts of your gameplay that have to be "unlocked" in a video game sense. A good example of this in Magic is adding in synergies that might not be obvious at first glance. It's not until the player happens to have a card from each part that they learn they can be combined. The player will then start looking for new ways to combine these effects, which allows them to discover other new things that have been layered in.

This question is basically asking if you've made sure that your set has things players will discover as more and more parts of your set begin interacting.

Question #6: Have You Added Something New? (Is There Something to Surprise Your Audience?)

The very first thing players ask when a set is previewed is "What's new?" Now I believe innovation can be overvalued in design, but you always need to have some. Part of the experience of playing a new Magic set is viewing the game through a slightly different lens than you're used to seeing it. I often explain how I know a set has an identity when I find myself caring about some aspect of gameplay I don't normally care about (land drops in Zendikar is a famous example).

It's common in Magic design to start in a brand-new place and begin creeping closer and closer to the way things normally operate as we evolve the set. I can't count the number of times we have created a brand-new mechanic and then, as we playtest and iterate it, seen the mechanic start turning into one that already exists. Magic turns 24 years old this year, which means we've tread a lot of ground. It's important to look at your set and make sure that you're actually doing something new, rather than just redressing something the players have seen before. (By the way, I'm not against redressing old things to match the new flavor of the set; just don't put too much weight of innovation on it.)

Question 7: Does It Feel Like Magic? (Will Players Have Things They Could Add to Their Already Existing Decks?)

This question is the opposite end of the spectrum from the previous question. Yes, you want each set to have some new elements to it, but you also want it to have enough familiarity that it's still the game players know and love. It's possible to create a Magic set that technically works in the rules, but that doesn't feel like Magic. The goal in any design is to create something that adds to the overall experience, not distances itself from everything that came before.

The biggest issue here is what R&D refers to as parasitism. That is, have you made things within your set that plays nicely with one other but don't connect with the rest of Magic? You do want some parasitism. It's nice if your set has new themes that require you to look within it, but you want to make sure those themes still allow intermixing with other sets.

This final question is trying to make sure that you've reached a good balance. You want your set to work with itself, but not to the exclusion of working with other sets. You want your set to feel fresh and new, but not so much that it doesn't feel like Magic. You want your set to have an identity, but one that feels part of the larger identity of the game. You've done everything that your set is supposed to do; let's just double-check that you've made a Magic set.

Seven Up

And those, my fellow designers, are the seven questions to ask when you evaluate your nearly complete set. (Only nine years in, we're getting close.) The key with each of these questions is allowing yourself the ability to be brutally honest. Good design requires emotionally bonding to the thing you're creating, so taking the time to step back and critically evaluate your set is a hard thing to do—a very hard thing to do—but a critical one. This is the last big gauntlet before your set starts the final process of completion. Take the time and energy to do it right.

That's all the time I have for today. This column has given you a lot to chew on and think about. If you have any general questions about the process I've outlined today (I can't answer specifics especially about new cards or new mechanics), please send them my way through either my email or any of my social media accounts (Twitter, Tumblr, Google+, and Instagram). Also, if you have any feedback on the column itself, I'd also love to hear that as well.

Join me next week for Modern Masters 2017 Edition previews.

Until then, keep asking yourself the hard questions.


 
#410: Ravnica Cards, Part 2
#410: Ravnica Cards, Part 2

36:50

This is the second part of a five-part series on the cards of original Ravnica.

 
#411: Ravnica Cards, Part 3
#411: Ravnica Cards, Part 3

40:36

This is the third part of a five-part series on the cards of original Ravnica.

Latest Making Magic Articles

MAKING MAGIC

November 12, 2018

The Rabiah Scale, Part 1 by, Mark Rosewater

Many years ago, on my blog (Blogatog on Tumblr), I created a scale, dubbed the Storm Scale, to indicate how likely I thought a mechanic was to return to a Standard-legal set, from 1 being...

Learn More

MAKING MAGIC

November 5, 2018

More Odds & Ends: Guilds of Ravnica by, Mark Rosewater

Last week, I started answering your questions about Guilds of Ravnica. This week, I continue because you all asked so many interesting things. Hybrid is at the exact same level it's al...

Learn More

Articles

Articles

Making Magic Archive

Consult the archives for more articles!

See All

We use cookies on this site to personalize content and ads, provide social media features and analyze web traffic. By clicking YES, you are consenting for us to set cookies. (Learn more about cookies)

No, I want to find out more