It's Top 8 Week, and this week I want to talk about lessons I've learned from leading a development team. What team is that, you ask? Well, you've probably never heard of it. Which is actually true, at least if you're reading this when my article goes up. Eventually you will, but you'll have to be patient. The time-travel effect of working so far in advance can be frustrating, especially when I am excited about the fruits of my labor, but you won't see it for some time. Still, what I can talk about are the lessons I learned from it that I can take to working on future sets, and maybe you can learn from in your own walks of life.
1. Nothing Beats Good Process
Many of the people who start working in R&D are used to being the smartest person in the room in many of their school or social situations. That probably isn't true once you start working in R&D, which is full of very intelligent people, many of whom have spent a lot of time thinking about Magic. If you come into R&D thinking you can be fully effective from Day One, you are probably kidding yourself. There is a lot to learn, and that means spending the first few months listening a lot more than speaking.
What becomes obvious over time is that simply being in a room full of intelligent people doesn't automatically mean that everything we do is going to work. It doesn't mean that we can eyeball things and suddenly come up with solutions. It's humbling at times, but there are a lot of problems that creating Magic sets introduce, and the solutions aren't easy. You don't generally solve these with raw brainpower. You also need good process.
The great thing about good process is that you get to stand on the shoulders of giants. Magic is a game we have produced for the last 21 years, and we've learned a lot over that time period. It might be possible for someone to come in and succeed without all of this previous knowledge, but I doubt it. It would be almost impossible to know all of the pitfalls that come up while making a set from scratch because of how many interlocking parts there are, not just within the set itself, but while dealing with the realities of how other people do their jobs.
One of the most valuable tools that anyone can add to the Magic R&D team is to figure out a way to do something better—not just because it improves one product for one time, but because it improves every product we produce from that point on. The more we improve our processes, the better our end products are.
2. Iterate, Iterate, Iterate
And when you're done with that iterate some more. The great thing about having good processes in place is that they give us the time to iterate our products and constantly improve them. Instead of spending months trying to get the basics of a Limited environment to work, we can usually get the format to be pretty reasonable within just a few weeks—which gives us a lot of time to keep improving it. It gives us enough time to try out new things (for good or bad), and to really figure out the kinds of cards we can add to the environment to give it character.
The thing about making a Magic set—or anything really—is that you aren't going to get it right on your first draft. You probably aren't getting it right on your six or seventh draft, either. You may not ever get it perfect, but the more time you have to go through things again and again, the better things will end up in the end. Ultimately, we produce Magic sets on a schedule—we have the dates planned out for our main set releases for quite a while in the future. We can't miss those dates, so there is a "pencils down" moment—but we are usually very proud of the set we hand off at that point.
3. Math is More Reliable Than Intuition
I wouldn't describe myself as a math-y person. In fact, I think of all the developers, I am the only one without a background in a math- or science-based discipline. For that reason, when I started, it took me a while to realize how much of my job would just come down to math. And just how much my own intuition could lie to me. Intuition helps in getting things right, for sure, but you can't trust it when the numbers disagree.
A lesson I learned very early on was that numbers don't lie. When we run playtests for Limited, one of the things we record is how many cards people played of each color. We add those up, and put them in a table form. Often, what happens in a playtest is that things will feel about right. But when you look at the numbers, you'll notice that they aren't even closed to balanced. The thing is, each individual person only saw a small snapshot of the playtest as a whole, and so the individual's green-blue deck played against a green-red deck, a black-white deck, and a green-black deck. But looking at the numbers, it's easy to notice that six of the eight players played green and only one ended up in white—a surefire sign that something is wrong with the color balance of the set. It's also possible that the number of players in a color is about right, but nobody actually played a heavily red-focused deck, instead everyone who did play the color decided to just splash it for removal. Again, this is something that numbers tell us quickly. It might take dozens of individual playtests before people notice how out of whack things are, where the math of adding up the colors in people's decks tells us that right away.
As I mentioned in the previous lesson, a lot of this comes down to standing on the shoulders of giants, and learning from people and processes that came before I even started.
4. Don't Try to Fix Everything at Once
We have a schedule for how long it takes to develop a set. We tend to do about one to two playtests a week of each set in development, and we have around twenty total weeks to get everything done. That's not a lot of time, but it's more than enough to pace yourself and experiment. Perhaps most importantly, that time is there so there's enough time to try things that aren't right and get to the correct answers. The important thing is that playtests, especially early ones, are there to try out solutions for problems, not to present a finalized set.
For myself, there was often a desire to get my set to the point, with each playtest, that everything was perfect. The problem is that, no matter what happens, there's always something that could be improved. If I changed too much of my set from one playtest to the next, and things still didn't work, it wouldn't be clear how to actually progress for the next one. Maybe half of the changes worked and half didn't. If I made a reasonable number, it would be obvious on how to progress for my next meeting. If I made too many, I would almost certainly need to backtrack a bit before I could go forward. That would end up taking more time in total than if I had just made a reasonable number of changes to begin with.
By limiting the rate of changes, it's easier to tell what is improving and what isn't. It can be pretty terrifying to go to the table with things that you suspect aren't right. I've found it important to trust in the process, though, and that making improvements at a steady and continuous pace is generally better than constantly swinging for the fences and hoping everything is correct.
5. Try the Wrong Way First
It may seem strange, but this is tied to the idea of iteration. Many times, when working on a problem, you are presented with two options: one looks like it will work about 75% of the time, and other has maybe a 25% chance of working. The problem with always trying the most likely one first is that you will often find out that it works, but you won't find out if the other option could've worked better. You can be safe and always go for the option that looks the most likely, but you'll leave a lot of interesting things on the table and never know if things could've been better if you'd tried out something riskier.
Now, trying the wrong way first does lead to occasionally blowing up a playtest or two, so you can't do this for every playtest. But we generally have ample time to take the risk of blowing up a playtest now and then. The bigger risk would be if we didn't take any risks and always played it safe. Many of the most popular mechanics and individual cards from the last two decades wouldn't exist if we'd gone that route.
6. Change is Good… and Frightening
From the time I worked on my first development team, the thing that most blew my mind was how much a set could change from the first day of development to the last. For some reason, I pictured most of development's work to be taking the card set that design hands off, adjusting the prices of a few cards, playtesting, and then adjusting a few costs again. Hardly. It's not unusual for a huge number of the cards in the design file to get cut or changed significantly during the development process.
The goal for development is to keep the spirit of the set intact while changing many of the individual components. While we try to keep the cards the design team feels are important for the structure of the set, we don't try and keep too much sacred. That would keep us from developing the best possible set.
It's very easy to get attached to things that are working. Often, the best part about an early playtest of a set will end up being the worst thing in a month—that's the beauty of constantly improving things. It often takes you to unexpected places and sometimes leads to that awkward meeting where it becomes obvious that the thing holding the set back is probably the thing that you loved about it only a month ago.
Ultimately, it's our goal to produce high-quality sets that people will enjoy playing, not to show how clever we are with awesome solutions to the problems that came up during playtesting. If we do our job correctly, the end players won't ever know just how smooth or rocky the development process was. They won't have any idea what the original file looked like—instead, they get the idealized final product.
7. Listen to Dissenting Voices
This was one of the hardest lessons for me to learn. Things can get pretty rocky early on in a set's development until all of the iteration I mentioned earlier starts to really reap rewards. Once things start working, it can be easy to listen to the people who are telling you they had a good time, and to not listen to someone who didn't. Because it's more immediately satisfying to hear that what you've done is good than to keep being critical.
The thing is that people are not usually wrong when they say they didn't enjoy something. You can't just sit someone down, tell the person everyone else liked it so he or she is wrong. Every person who comes and plays in one of our playtests represents some portion of our players. And if one person didn't have fun, there's a good chance that someone in the real world would have a similar experience. That doesn't mean that you have to overhaul the entire experience to please that one person. But it's often possible to try and find out what wasn't working and figure out something else to put into the set that would've made that player happy.
Ultimately, we try to make each of our sets for as wide an audience as possible, but we know that it isn't possible to please everyone all of the time. Looking at our best-received sets in the last few years, it's easy for me to find people who have posted that they are their least favorite in a decade or more. It's also easy to find sets that I don't believe are particularly good, and see people hold them up as their favorites. People's tastes differ, and there isn't a surefire solution to solve everything. But the more we work together within R&D to make sure there is something for everyone to enjoy, the more you all will probably enjoy our sets as a whole.
8. Making Magic Sets is Really Hard
With more than ten years of columns of both Making Magic and Latest Developments, it should go without saying that there is a lot that goes into making a Magic set. We make what I believe to be the best game in the world, but that doesn't mean that Magic is inherently fun—it takes a lot of work to bring out the fun elements of Magic sets. If it weren't hard, we wouldn't go through dozens of mechanics (or more) when working on a design file. And we wouldn't need to spend months trying to get Limited to the point where the colors are basically balanced, and that there is enough for players to find on their second, third, or thirtieth draft.
It's very easy to get attached to the things you create, and we create so much over the course of taking a Magic set from pre-design to the actual booster packs, that things can get pretty frustrating when time and time again you create a card or mechanic, only to watch it go up in flames. Sometimes it's a quick death. Other times it looks like your card will make it out the door, only to get killed at the last second and replaced by something intelligent that we need to balance Standard. It just happens. The thing is, everyone who works in R&D cares deeply about Magic. And each and every person wants what is best for the game as a whole, even if that means their own creations never see the light of day. I think we'd all rather have them go straight from our brains to the booster, but that isn't really an option.
At the same time, it's pretty easy to get too focused on getting everything balanced, instead of valuing how to make the flaws of one set into its strengths. Everyone has different preferences about what they like in Magic, and different tolerances for the things they don't. The fact is that there is no perfect Magic set, and there never will be. What we try to do is figure out what is the thing in our newest set that will frustrate players and make that into a strength. Take Khans of Tarkir, for example—players tend to not like getting color screwed, and this is a set where that will happen on a higher-than-average basis. What we've done is to make it so there are the tools to play around color screw, with two color decks, or you can play into it by taking dual lands and playing all five colors.
Next year's set won't be a gold set. People will go back to getting color screwed less often, and there will be something else that excites some player and frustrates others. And we will put it front and center, and use it as the selling point of the set. It means that year after year (or soon, simply block after block), things feel different, at the cost of some people having less fun than they would with another set.
Internally, that means that things can get pretty contentious, as people argue over what makes the best possible version of a set, since we too have different tolerances for a set's elements. We all work together to produce our sets, but each individual set lead has to be the vision holder for his or her own set and make the hard decisions for where the goal posts are.
That's it for this week. Join me next week for World's Week.