Daniel Cook, Lost Garden

Introduction
To build a successful game, developers and publishers must master both the art of game creation and the science of selling games at a profit. These two are like the heart and the lungs of our industry; both are critical to survival.

On the art side, the last two decades have seen great strides in the craft of building immersive, imaginative games. Art is more expressive, programs are more powerful, and games are ever more sophisticated. On the business side, the game industry continues to grow annually, many years ago surpassing even motion picture box office revenues. However, the techniques and processes for creating diverse, profitable products has actually decreased in the past two decades. Less and less of the game-playing public is interested in playing major, hard-core epics, yet industry practices and business models support making only those types of games.

As development costs escalate and household penetration stagnates, it becomes much more important to create both titles with a high chance of success and that appeal to untapped audiences. The focus switches from getting a game out the door to building the right game, one that appeals to a receptive audience and has a high probability of contributing positive income to the company's game portfolio.

The current situation
Most publishers and developers are trapped in an archaic pitching and greenlighting system that punishes rational risk taking, does little to predict the market success of titles, and is poor at tapping into new market segments. No one really knows what makes a hit, so those in charge use mystical heuristics, gut checks and unreliable expert opinions. The result is a large numbers of questionable titles are developed at considerable cost, only to fail financially in the market place.

This practice has both financial and human costs. On the money side, resources are wasted, publishers face instability, and accordingly, value is not efficiently passed along to the customer. On the human side, teams get burnt out, and job insecurity and team churn are commonplace. And, on the product side, innovative games as a category go unexplored. In the face of great uncertainty, the industry uses the crudest of measures to predict success – all future games are required to look exactly like recent hits.

Solutions abound
The irony of all this is that many of these problems have been studied for decades in other industries to improve their new product success rates. The New Product Development (NPD) process has been studied extensively for at least fifty years, and these practices have made companies like 3M, Corning Glass, IBM, Visa, Proctor and Gamble and Guinness into blue-chip market leaders.

This article will:

• Highlight current poor business practices that contribute to the glut of poorly aimed titles on the market.
• Provide an overview of techniques such as the Stage Gate process that can improve success rates.

Are we any good at making profitable games?
A publisher considers themselves lucky if 20% of their launched titles turn a profit. 80% will break even or lose money and perhaps 3% will be hits that make enough money to keep the publisher afloat.

Depending on how these numbers are interpreted, they are either very reasonable – or quite poor.

• These numbers are far below the 67% average success rate for products tracked across multiple industries. The best companies consistently reach 80% success rates, four times the rate of the games industry.
• It is exactly in line with average 20% success rate for the software industry, which is expected for such a similar discipline.
• Games are immensely more successful than the single-digit success rates for commodity media products such as books or CDs.

Retail games are not commodity media
While the industry can boast better overall financial successes than books and audio CDs, on closer examination, games are a completely different class of product, requiring different metrics of success.

Music, novels and comics possess low costs of production and minimal entry barriers. Anyone with a few spare moments can write a novel or record a CD. Most do it solely for the pleasure of the experience and are hurt little if their product fails in the market.

Each year, hundreds of thousands of new media products flood the market. Customers cherry-pick the few that are most accessible and pertinent to their entertainment needs. Publishers exist primarily as a marketing and distribution mechanism for titles with proven appeal. They take on very little product development risk and are willing to launch hundreds of titles in the hope of gaining one hit. This hit can provide a constant stream of revenue for decades.

Some indie game efforts fall into this category, but most commercial games do not. Modern retail games are released in limited numbers and are the efforts of large, well-organized teams. They require vast human and monetary resources to build and the cost of failure is high. By almost every economic and business measure, games do not look like commodity media.

Historically games, being entertainment products, have been misclassified as commodity media. Much of our industry's cultural value system is focused on releasing numerous, artist-driven products in the hopes that one will be a hit. Unfortunately, due the core business differences, applying either the development or publishing lessons from these industries ends up being a rather bloody and expensive mistake.

A better benchmark
A better benchmark for measuring success are similar products that require high development and deployment costs. These operate under a different set of constraints and prefer very different development strategies than commodity media that better reflect the realities of the game industry.

If you consider the business factors, retail games match up most closely with traditional products like consumer electronics, Post-it notes, and boxed cereal, not music, comics or books. This is not to say games lack creativity, far from it: movies fall into this category as well.

By the standard of traditional New Product Development, games are doing quite poorly. 80% of our resources are at the best being underutilized, under-appreciated, or at worst, wasted. On the positive side, the NPD perspective provides analytic tools that help us understand what is doing poorly and how it can be improved.

Factors that feed failure
A tremendous opportunity exists to improve overall financial return in the games industry by comparing it to other, more profitable industries. First, we need to examine the root causes that limit success rates. A few factors are external, and therefore out of the direct control of developers and publishers; but far more are internal.

External factors
Much has been made of the external factors that affect the profitability of games:

• No long tail: Games often have a very small window, a few years at most, before they are technologically obsolete. Slow burn, niche products often stop working before they turn a profit.
• Limited distribution channels: Limited diversification of distribution channels means a game has a finite number of chances to find a prospective audience.

Internal factors dominate
The primary factors that influence success are internal and controllable by both developers and publishers. The game industry has immense control over the games it releases. They choose the theme, the target audience, the budget, the marketing, the timing, and of course, the game design.

Internal factors are the strongest predictor of market failure.

• Lack of customer focus
• Lack of reliable shared success criteria
• Lack of business expertise
• Outdated Practices

Lack of customer focus
Generally, companies fail to make games that appeal to their target customers. This is, in many ways, the cardinal sin of commercial product development.

• Cowboy culture: Designers, business development and producers will often use themselves as the ultimate judge of a products appeal or quality. They rely on inaccurate ‘gut checks' that typically do not reflect the real needs of their audience. Very often they choose products and designs that they personally desire, even when this conflicts with existing data. In this swaggering, opinionated stew of ill-informed decision making, reputation is the single largest influence on which games get made, not actual customer value.
• Testing is focused only on functional issues: Teams rarely test the appeal of new designs with external groups, especially at early stages. Formal human factors engineering and usability testing is uncommon. Testing programs that do exist tend to look for functional flaws instead of problems with the product's underlying value proposition.
• Too little information, too late: What market testing does occur often happens at the very end of a game's production, either in the form of betas or actual product launches. At this point there is little opportunity for making changes based off the inevitable customer feedback. Radical changes to setting and core mechanics are nearly impossible in beta. Yet typical preproduction is a completely inward looking activity that provides no opportunity for feedback from target customers.
• The wrong people are doing early evaluations: Often, business people who know little about game design or technology will be asked to evaluate a pure gameplay prototype. They lack the knowledge to give the early stage product a fair and objective evaluation. Similarly, game designers will often be asked to make critical decisions about their target audience when they lack basic marketing training.

Lack of reliable, shared success criteria
The lack of customer focus is allowed to persist because very few groups in the industry actually know what makes a successful game. We have little ability to predict if a game will succeed or fail in the market. In the absence of science, superstition reigns unchecked.

• Testing the appeal of new game designs is hard: Nontraditional game designs are expensive to prove. Typical techniques such as paper documentation and videos are at best ‘hand waving'. Prototyping shows great promise, but it takes a long time to bring many prototypes to a testable stage. Productive prototyping techniques are just starting to be established at forward thinking developers, but have yet to be widely adopted or accepted as necessary. Without evidence that these methods are cost-effective method, publishers are reluctant to ‘foot the bill'. The overall result is that funding is scarce.
• Poor data collection methods: Games are rarely evaluated for their market appeal except during a brief green lighting stage. After this point, it is assumed that they will be a success. What data exists is rarely in any format that is useful for deriving success criteria. Even retrospective evaluation of products is difficult as teams tend to dissipate rapidly due to the high turn over rate
• Over-sensitivity to perceived trends: Without useful data, developers and publishers magnify and misinterpret even the slightest market indicator and incorporate it into their decision-making heuristics. For example, if Gears of War had chainsaw guns, then future hit titles should also have chainsaw guns. After all, it worked for the shotgun in Doom; so this heuristic of selecting games based off their most popular weapon must be pure gold. Such heuristics fail to predict market success.

Lack of business expertise
Often, publishers and developers lack the business training necessary to fix the situation in which they find themselves. Some people realize that there is a problem, but they don't know the terminology or resources available that may provide a solution.

• Designers lack basic financial fundamentals: Only rarely do you see an informed business plan referenced by a game design. In this age of brands, sequels, expansion packs, subscription models and item driven economies, most design decisions have a critical impact on the economic success of their product.
  If game designers fail to master the tools of business at a relatively deep level, they are unable to design and build games that maximize profit. It simply doesn't occur to them that they could spend $100,000 to A) add an item-based system that drives another $2 million in lifetime revenue, instead of B) adding another expert level at the 35 hour mark that will be viewed by 5% of the purchasers.
• Publishers lack basic business fundamentals: Common concepts taught in almost every business school in the nation such as product portfolio management, use of cross-functional teams and metrics-based management are rarely practiced or discussed. The number of MBA in game publishing is surprisingly low, in part due to the early cowboy nature of the industry. Our heroes are people who never went to school or read even the slimmest book on the dirty concept of business. The preference is to hire interns in the mold of Horatio Alger, not people with degrees. The thought is that ‘book learning' never released a hit game. It is possible to learn these concepts without getting a formal degree, the important factor is to make use of the basic business fundamentals and resources available, which are better utilized in other industries.

Outdated practices
A final symptom is that the practices in the game industry are outdated and resistant to change. Much of what we do has been passed down in a fragmented manner from one mayfly-like developer to the next. The collected superstitions are practiced as a form of mysterious craft by both publishers and developers.

• Reliance on waterfall methodologies: Most games have a rigidly defined preproduction, production, testing and deployment phase. Upon green lighting the team is launched towards a target date and scope with little opportunity for feedback. More agile and iterative techniques are still only practiced occasionally.
• Siloing: Games live and die by the intricately connected contribution of many different disciplines. Yet companies insist on placing test, engineers, artists and marketing in different organizations, buildings and even countries. These groups lack shared goals and communicate by tossing limited artifacts over a very high wall. It is easy to imagine each silo as an ancient isolated guild, blindly following their handed down rituals while fighting off the heathens from other silos.
• Primitive green lighting process: The poor customer focus, lack of success criteria, and lack of business expertise culminate in the dreadful green lighting process.

Improving how we select successful new products

Systemic problems require systemic solutions
This litany of ills may seem overwhelming. However, our goal is to demonstrate that there is no single person or group that is the root cause of the low success rate facing our products. It is not just about the publishers or the developers. We have systemic issues that are deeply rooted in the history and culture of game development.

The good news is that the majority of our difficulties are completely under our control. By adopting a systemic solution that alters how we evaluate and choose products, we have a clear opportunity to drastically improve the success rate of the titles we release onto the market. Any solution will require both developers and publishers working together, but the rewards of higher profitability and greater stability are deeply valued by all parties.

Game productions have long pre-production cycles that can be better used to gather high quality of data about potential customers. In some cases, the actual hardware that the games are played on can be controlled. The keys to success are imminently achievable.

The tools exist
Very few of these difficulties that cause so many products to fail and force a lack of design diversity are unique to the games industry. Most manufacturing companies ran into the same issues two decades ago when switching from traditional engineering practices to lean manufacturing. The past twenty years has also seen the rise of a field called NPD, or New Product Development, that focuses on solving what they call the "fuzzy front end" of generating, choosing and implementing new product ideas.

The result is a vast body of knowledge of best practices that have been successfully adapted to a wide range of markets ranging from consumer products to financial services and pretty much anything in between. As long as there exists measurable customer value, the techniques of NPD can help increase a company's product success rate.

Many proven tools are available:

• Value stream analysis: How do we identify and remove unnecessary processes that do not contribute customer value?
• Ideation techniques: How do we generate strong new ideas that capture the customer's unspoken needs?
• Constraints-based product requirements: How do we keep our options open to encourage necessary learning and exploration?
• Onsite Customers and Quality scorecards: How do we establish rapid feedback cycles and communicate the results with the team so that we know exactly how close our product is to success at any point?
• Small Cross-functional teams: How do we build efficient problem solving teams that solve production problems immediately instead of letting errors persist and snowball?

These techniques and many others have been proven to slash time to market, increase product quality and improve success rates.

Stage Gate process
One widely adopted technique called the Stage Gate process can act as a framework for the other tools listed above. It provides a master unifying system that encourages companies to steadily and consistently improve both their portfolio of products and the techniques by which they choose them. The Stage Gate process can be used to increase customer focus, define success criteria, create a common business language across disciplines and eliminate outdated practices. (In fact, the Stage Gate process can be applied not only to evaluating game concepts, it can be applied to evaluating game features and mechanics as well!)

Current greenlighting process
To understand the power of the stage gate process, it is interesting to compare it with the existing greenlighting process used by most companies in the game industry.

Most publishers receive product pitches in a wide variety of different forms. Some are simple ideas from a reputable developer. Others are pre-rendered presentations that capture the feel or theme of a game. Sometimes, publishers receive complete design documents, and on occasion, a working demo. Recently, more and more have licenses attached. And all too often, they're in the form of a previously-released game with an ever-iterating Roman numeral appended.

The game ideas are then evaluated by the ubiquitous green light committee. Typically, this involves an in-person pitch and an evaluation of key criteria, such as:

• Does the team have the ability to execute on the design?
• Does the publisher have the money to fund production, and is its preliminary return on investment robust?
• Does the game meet the "gut check" of the expert evaluators?
• Does the game meet the current hot-list of fashionable heuristics that the publisher is currently pursuing? (For example, does it have a chainsaw gun.)

Overall, the focus has a lot to do with execution and has very little thought put into whether customers would actually want such a title. In fact, original titles without a narrowly-defined, already-proven market segment have little chance of advancing beyond this point. Occasionally, if the publisher doubts the reputation of the team, they will request the development of a demo or vertical slice. This happens rarely because neither party really wishes to pay for such "extra" activities. In the end, if the simplistic greenlighting criteria is met, a handful of products end up being fully funded. The rest are rejected and their assets and ideas are typically discarded.

But – once something passes this stage – a fascinating thing occurs: the greenlighted products are now assumed to be future successes! Otherwise, the logic goes, they wouldn't have been greenlighted. Development teams make plans to fully staff for a 12 to 18 month production cycle. Some publishers go so far as to put the revenue from the greenlighted games in their financial plans with the assumption of 100% success rate.

Even when strong evidence emerges that the product will fail in the market place, decision makers still gamble that releasing the products may manage to "recoup development costs." In a market where launch costs can equal development costs, it typically results in tossing money down the drain. Though the occasional producer who pushes a bad product onto the market is venerated for his gutsy move, in general this practice reflects poorly on the overall success rate of the portfolio.

When the games reach market, the customers end up deciding that only 1 out of 5 is actually worth their money, meaning 80% of the resources that the publisher and developers have expended is at best under-appreciated, underutilized, or at worst, lost.

A stage gate comparison
The stage gate process uses a different strategy. The company continually measures the product against ever-stricter sets of release criteria and kills those projects that will probably not succeed. Instead of having only one greenlighting stage, there are many.

The stage gate process is straightforward.

• A product starts out as a simple idea and moves through several production stages before release.
• Each stage has clearly-defined, objective exit criteria, spelled out well in advance, in the form of a kill gate. At the kill gate only the products that meet the success criteria are allowed to move on to the next development stage. Other products are killed at this point and receive no further investment.
• Products that move on to more advanced stages receive both increased funding and stricter success criteria.
• The process continues until a few, highly vetted products are released into the market.

This is an options-based decision model where options are kept open as long as possible, at as low a cost as possible until good information is brought to bear that it is wise to proceed. Compared to the greenlighting process, the Stage Gate model offers some great advantages.

Statistically validated success criteria
Each gate uses strongly validated success criteria. The company studies hundreds of games and identifies potential success criteria. As they release more titles, they are constantly tuning their success metrics. The success criteria for each stage are written down in advance and are made available to both developers and gate committees from the very start of the project.

The good news is that a lot of work has already been done in this area. Numerous studies across multiple industries have identified many areas that should be invested in as a matter of course in order to increase the likelihood of success.

For example, Robert G. Cooper's latest studies list the following as important factors:

Each one of these items is readily quantifiable and measurable. Cooper breaks out the first criteria in the following check list:

• Offers unique features not available in competitive products
• Meets customer needs better than competitive products
• Has a higher relative product quality
• Solves a problem the customer has had with a competitive product
• Reduces a customer's total costs
• Innovative: First of their kind in the market.

Even at the concept stage, it is possible to ask some of these questions and receive clear and concise answers. During later stages, the specificity around items such as quality and competitive comparisons is increased.

Killing products is critical
Unlike the traditional greenlighting process, the stage gate process is very good at killing projects. This is important because resources that are spent on failing late stage projects could have funded the exploration of dozens of seed concept, which in turn might yield a successful title. The opportunity cost of not killing projects is huge.

Also, a well-organized stage gate process allows for a concept bank to store old projects components for reuse by other teams. The cross-fertilization that occurs when a team has access to hundreds of older projects can help accelerate new product development substantially.

Evolving the process
When an organization starts using stage gates, the initial kill gate definitions may be uncertain, and will certainly be untested. But by measuring which projects made it through all of the gates, observing different aspects of their market success, and attributing those successes back to the gates through which they passed, the gate definitions can be continuously refined as best meets the need of the organization.

Other Benefits
There are numerous other benefits that the stage gate process yields.

• Higher profitability: By spending more money vetting a wide variety of options in a low cost manner, companies end up producing successful products for less overall investment.
• Predictable revenue: Success becomes more predictable for later stage projects.
• More diverse designs: Designs thought to be too experimental may pass early gates and receive important consumer feedback required to make them qualify through later gates.
• Shorter time to market: Efficiencies that come from understanding the target customer clearly allow targeted products to be updated more quickly. The short development cycle of a product like Brain Age is a good example of how this might happen in the game industry.
• Aura of success: The projects that clear the gates and are released into the market are generally more successful than those that did not benefit from such a process. The brand trust therefore increases, as well as customer loyalty.
• Pursue innovative products as a portfolio strategy: Once the appropriate success criteria is in place, more flexibility is gained in allocating a percentage of a product portfolio to innovative new-to-the-world products. Best-in-class companies typically generate 30% of their revenue from product lines created in the last five years.
• Increased flexibility: Since more ideas are in production at any one time compared to the traditional game development model, it is much easier to accelerate (or delay) certain ones as market conditions shift.

Problems
Implementing a successful stage gate process requires widespread cultural change, and as such, is rife with situations where the spirit of the process can be corrupted by old thinking.

• Failure to kill projects: There may be a strong tendency for the old greenlighting teams become the new gate teams. Politics and relationships can take precedence over data-driven decision-making. The result is that weak products still make it out onto the market.
  
  Encouraging the use of public and transparent checklists as a decision making tool can help alleviate some of this issue. Everyone from the top of the company to the lowest production worker should know what ideas in different states of production are being scored on, and how the results are interpreted.

• Use as a command and control method: The stage gate process aligns goals and helps all parties succeed. However, the extensive data gathering and rigid gates also can be used as an oppressive reporting and control system.
  
  Often this requires a shift in the priorities of top management such that they empower their production teams to make decisions. Training on the use and application of the gate criteria before the projects hit the gate is of critical importance.

• Bureaucracy: All the checklists and data collection can rapidly build up to a crippling amount of paper work.
  
  Regular postmortems and the use of techniques like value stream analysis can help streamline the process. New adopters often implement a more cumbersome system with fixed gates. However, as they gain experience, the process rapidly evolves into a more agile system with increased parallelism.

Conclusion
Everybody – designers, developers, and publishers alike – want to release profitable games. Money, for all its ability to corrupt, also is the lifeblood that allows the people fortunate enough to be in the industry to spend their lives creating this amazing new form of art. We've looked at four key issues that prevent us from creating successful games. We have also identified a proven framework in the form of the stage gate process that yields solutions to these issues. Stage Gates can be widely adapted and can be used in weekly, monthly, and annual measures.

References
Winning at New Products: Accelerating the Process From Idea To Launch, 3rd edition Robert G. Cooper, copyright 2001.

Success of software 20%: Page 93; Failure rate of industrial product 67%: Page 11, Winning at New Products, survey by Kleinschmidt

section 5

next section

select a section:
1. Introduction  2. Speakers  3. Executive Summary  
4. Online People's Revolutionary Army of Horseshoe
5. Building Innovative Games That Sell
6. The Legitimacy of Games
7. The Creation of Radically New Game Experiences
8. Schedule & Sponsors