Industrial companies evaluation criteria in New Product Development

Industrial companies evaluation criteria in New Product Development - Transcript

J PROD INNOV MANAG 2003 20 22 36 r 2003 Product Development Management Association

Industrial Companies Evaluation Criteria in New Product Development Gates
Susan Hart Erik Jan Hultink Nikolaos Tzokas and Harry R Commandeur

This article presents the results of a study on the evaluation criteria that companies use at several gates in the NPD process The ndings from 166 managers suggest that companies use different criteria at different NPD evaluation gates While such criteria as technical feasibility intuition and market potential are stressed in the early screening gates of the NPD process a focus on product performance quality and staying within the development budget are considered of paramount importance after the product has been developed During and after commercialization customer acceptance and satisfaction and unit sales are primary considerations In addition based on the performance dimensions developed by Grif n and Page 1993 we derive patterns of use of various evaluative dimensions at the NPD gates Our results show that while the market acceptance dimension permeates evaluation at all the gates in the NPD process the nancial dimension is especially important during the business analysis gate and after market launch The product performance dimension gures strongly in the product and market testing gates The importance of our additional set of criteria i e product uniqueness market potential market chance technical feasibility and intuition decreases as the NPD process unfolds Overall the above pattern of dimensions usage holds true for both countries in which we collected our data and across rms of different sizes holding different market share positions with different NPD drivers following different innovation strategies and developing different types of new products The results also are stable for respondents that differ in terms of expertise and functional background The results of this study provide useful guidelines for project selection and evaluation purposes and therefore can be helpful for effective investment decisionmaking at gate meetings and for project portfolio management We elaborate on these guidelines for product developers and marketers wishing to employ evaluation criteria in their NPD gates and we discuss directions for further research

Introduction

R

ecent investigations of new product developments NPD have reinforced the concept of a process based on a series of develop

Address correspondence to Susan Hart Department of Marketing University of Strathclyde Stenhouse Building 173 Cathedral Street Glasgow G4 0RQ Scotland UK Phone 44 141 548 4927 E mail susan hart strath ac uk

ment stages interpolated by a series of evaluative stages 9 31 37 Preceded by the formulation of new product strategy also coined Protocol 7 and Product Innovation Charter 12 the development stages include idea generation concept development building the business case product development market testing and market launch 7 13 After each of these stages an evaluation takes place essentially to determine whether the new product should advance

INDUSTRIAL COMPANIES EVALUATION CRITERIA IN NEW PRODUCT DEVELOPMENT GATES

J PROD INNOV MANAG 2003 20 22 36

23

BIOGRAPHICAL SKETCH Susan Hart is vice dean of research at Strathclyde Business School In addition Professor Hart has worked for a variety of private sector companies ranging from multinational to small manufacturers in consumer and industrial enterprises She sits on the executive committee of the Academy of Marketing and recently was elected to the Senate of the Chartered Institute of Marketing Furthermore she edits the Journal of Marketing Management and is a member of the Journal of Product Innovation Management She also acts as reviewer for seven other academic journals and is a proposal reviewer for four grant awarding bodies Erik Jan Hultink is an Associate Professor of Marketing at the Faculty of Design Engineering and Production of Delft University of Technology where he received his PhD He received his MSc in Economics from the University of Amsterdam His research interest concentrates on launch strategies and new product development methods and measurement He has published on these topics in journals such as the International Journal of Research in Marketing the Journal of Product Innovation Management and the Journal of High Technology Management Research Nikos Tzokas is Professor of Marketing at the University of East Anglia He graduated and received an MBA from the Athens University of Economics in Greece He was awarded a PhD in 1993 from the School of Management at the University of Bath Formerly at the University of Strathclyde and the University of Stirling he joined the School of Management in November 2000 His research areas of interest include Relationship Marketing Customer Relationship Management New Product Development and Sales Management He has published widely on these issues in journals such as The Journal of Selling and Major Account Management Industrial Marketing Management International Business Review and the Journal of Product Innovation Management and the Journal of Marketing Management Harry R Commandeur holds the Dr F J D Goldschmeding Chair of Economics for Increasing Returns at Nyenrode University Breukelen The Netherlands He also teaches at the Department of Marketing and Organization Faculty of Economics Erasmus University Rotterdam The Netherlands

extent that these criteria are derived from the corporate and new product strategy of the rm and are focused to the speci c requirements of each stage of the NPD process they can help reduce managerial uncertainty and can identify areas where additional attention and resources are needed Furthermore they can strengthen the strategic decision making process of the rm by helping management develop and can deploy the right competencies and resources across the NPD effort As Hauser and Zettelmeyer 23 maintain Select the right metric for each activity and you can encourage the right decisions and actions by scientists engineers and managers p 32 That said key writers observe that the go kill decision points often are missing from representations and studies of the NPD process 7 34 Motivated by the important role of criteria throughout the NPD effort we attempt here to shed additional light on the nature and utilization of these criteria in contemporary industrial rms The aim of this article is to examine how the use of different criteria varies at different gates of the NPD process The remainder of this article is divided into four sections First we review the NPD literature in order to de ne our conceptual framework for study and to specify our research questions based on this conceptualization Second we describe a survey of the procedures of 166 industrial rms for evaluating new product projects as they evolve from incomplete ideas to nascent products and thus become increasingly resource hungry 9 30 36 Then we present results of the study and last we discuss their research and managerial implications

further or should be terminated These evaluation stages have been termed gates 7 or convergent points 21 Such an evaluation requires the collection and consideration of information and the application of criteria against which this information can be assessed Much research into NPD has dealt with the techniques used for the collection of relevant information 30 31 yet more articles have been published regarding the intricacies of disseminating and using such information 28 In contrast the subject of criteria to be used in evaluating the information inputs to NPD is scant and where it does exist it tends to be con ned to articles dealing with the idea and concept screening gates 8 14 The criteria act as guideposts against which the performance of the NPD effort can be evaluated and adjustments can be made if necessary 34 To the

Background and Research Questions
Despite the changes in the conceptualization of NPD projects that have taken place over the past 25 years evaluation of the development project as it journeys from intangible idea to actual customer offering remains central 16 27 35 Indeed much research demonstrates that a phased review process commonly is used 6 7 17 Even research disputing the ef cacy of the traditional linear sequential representation of the NPD process tends to recognize the central role of evaluation and the reevaluation of the development project as it progresses 3 20 21 Common to all phased review representations of the NPD process is the notion that it is made up of development phases

24

J PROD INNOV MANAG 2003 20 22 36

S HART ET AL

i e development stages and evaluation phases i e corresponding evaluation gates Evaluation within a gate includes techniques that may be repeated as alternative or re ned design concepts of product con gurations are developed Different techniques are used to assess the commercial and technical feasibility of the developing product The commercial set of techniques includes beta testing perceptual mapping conjoint analysis Quality Function Deployment QFD A T R models break even analysis and sensitivity testing In addition to these commercial evaluation techniques technical evaluation sets include the assessment of design production and functional feasibility and speci cations In short as well as commercial evaluation and reevaluation it is of critical importance to evaluate the function form and production of the new product project Each gate is a different combination of technical and commercial evaluation sets Idea screening is the rst of a series of evaluations beginning when the collection of new product ideas is complete It is an initial assessment to weed out impractical ideas This initial evaluation cannot be very sophisticated as it is concerned with identifying ideas that can be developed into concepts and can be evaluated for their technical feasibility and market potential As the development project proceeds the information collected regarding both technical and commercial feasibility becomes more copious and because it relates less to something vague and abstract and more to something tangible and complete the information has greater potential for being reliable and valid 7 Thus when a project reaches market testing the information yielded will be more complete and will encompass customer opinions buying behavior operation of the product in use production and delivery and communication to its target market Yet despite the recognition that the amount and quality of information increases incrementally through each stage of the NPD process the criteria used to evaluate the information remain unclear It has been 15 years since Ronkainen s 33 observation that a major issue that has been overlooked is whether or not the same set of criteria is used at every decision making point or whether the weights of individual criteria vary from one point to another pp 171 72 Apart from this one study however there has been little further light shed on this matter Ronkainen 32 found that in four industrial companies decision

makers used three sets of criteria for evaluative purposes product market and nance with the last category being of greatest importance Further he found that market criteria were determinants of the go no go decision at the concept screening gate and that during the product testing gate product related criteria took over as being more important Finally he suggested that nancial criteria were stressed more in the last two gates of the NPD process which he de ned as scale up and standardization Despite the important contribution of these ndings to our understanding of how NPD processes are applied within rms they are limited in some respects First the small scale of the study coupled with its age and restriction to large American rms make it unwise to assume general applicability of the ndings to NPD evaluation Second the conceptualization of the process used while relevant for the four rms investigated makes no reference to common frameworks used to depict the work of new product developers and it is therefore dif cult to slot the implications into the mainstream of NPD research and theory A further knowledge gap regarding evaluation criteria relates to the evaluative dimensions to be used during the evaluation gates With the exception of those articles dealing with idea and concept screening or post launch evaluation the literature as yet has not made a consistent attempt to provide insights into how the evaluative dimensions are deployed throughout the NPD process 8 18 24 Below we argue that criteria related to new product performance measures can become the basis for examining the evaluative dimensions for NPD gates The measurement of new product performance has been the subject of signi cant research studies within the past 10 years 18 19 20 24 Such research has af rmed that there are several dimensions of NPD performance including technical nancial and market based performance 18 20 The ndings of these studies can be linked to the criteria used in the monitoring of an NPD project In other words if there are different kinds of new product performance outcomes to be achieved then the use of evaluative criteria related to the performance dimensions would be an appropriate means of steering the NPD process through the go no go evaluation gates More speci cally as new product performance on one dimension may be achieved at the expense of performance on another it is crucial that the evaluative criteria at the gates mirror those used to

INDUSTRIAL COMPANIES EVALUATION CRITERIA IN NEW PRODUCT DEVELOPMENT GATES

J PROD INNOV MANAG 2003 20 22 36

25

assess the performance of the product once launched In our research we sought explicitly to incorporate the dimensions of new product performance detailed in the literature in investigating how NPD projects are evaluated throughout their development In short we have synthesized the concepts of evaluative criteria and performance indicators The work of Hauser 22 and Hauser and Zettelmeyer 23 supports our approach Using a three tier metaphor to categorize different types of research development and engineering RD E activities i e projects programs and explorations in 10 research intensive organizations Hauser and Zettelmeyer 23 found that the appropriateness of different metrics varies for different types of RD E projects They concluded that metrics that are best for one type of activity may be counter productive for another type p 38 and warned that if a rm applies the same metrics throughout the RD E process it does not get the most out of its technological efforts p 33 The use of the threetier metaphor by the researchers above highlights the fact that metrics or evaluative criteria should be aligned to the different objectives and therefore to the different requirements of each type However Hauser and Zettelmeyer 23 examined only the use of metrics for different types of RD E processes and not at different gates of the process Nevertheless most rms had a concept that the management of technology varied depending on the stage of the process p 33 Our study has been developed to address this level of detail Drawing on the foregoing review of the relevant literature our speci c research questions that guided our study were as follows 1 Which criteria are used most frequently at the NPD evaluation gates 2 Which evaluative dimensions are used most frequently at the NPD evaluation gates We assessed the results for these two research questions in two separate countries to examine the stability of the ndings Moreover we were interested in exploring how a rm s pro le was related to the nature and number of criteria and dimensions used at the NPD gates Speci cally we examined rm level factors rm size market share position and NPD driver NPD program level factors innovation strategy and product newness and respondent factors experience in the rm and functional background

Research Method
Our research ndings are based on a survey of 166 managers from Dutch and UK companies that are developing and manufacturing industrial goods This research instrument was used to collect information on criteria that companies use at various evaluation gates in the NPD process In addition the survey collected background and demographic data on the respondent and the rm The survey was pretested in two rounds with four companies in each round Interviews with six managers after the second round indicated that the meanings of the questions and answer categories were clear and that the survey could be completed without dif culties The questionnaire originally was developed in Dutch for data collection in The Netherlands and subsequently was translated by a native speaker in English and was back translated by a different translator in Dutch as a check for data collection in the UK The procedure of collecting the data was identical in both countries All potential respondents were prenoti ed by phone by one of the project members They introduced themselves as contributors to an international study on NPD evaluation criteria Preliminary noti cation by phone was used to solicit cooperation to check whether the company had developed and launched any new products recently to identify the respondent who had been responsible for the NPD effort and to increase the response rate 39 Accordingly questionnaires were mailed out to those identi ed as having primary responsibility for NPD in each rm by name More details on the data collection in the two countries are provided below The Netherlands We used a CD ROM Business Directory by Generator BV to develop a sample frame of 1 927 manufacturing companies with more than 20 employees in some selected industries chemicals medical equipment electronics and computers From a total of 509 randomly selected companies contacted by phone 228 companies agreed to participate in the research and received the mail questionnaire Thirty four companies had not developed any new products in the last ve years 134 companies only assembled or sold new products but did not develop them 52 companies were not interested in participating in the research and 61 of the companies could not be reached or had gone bankrupt This procedure and an additional reminder postcard led to 134 usable questionnaires an effective response rate of 59 percent of those who agreed to

26

J PROD INNOV MANAG 2003 20 22 36

S HART ET AL

Table 1 Pro le of Dutch and UK Samples
The Netherlands Number Firm level factors Number of Employees Less than 50 51 to 100 101 to 1000 More than 1000 Market Share Position 1 2 4 5 or Lower NPD Driver Market driven Technology driven NPD Program level factors Innovation Strategy Technological innovator Fast imitator Cost reducer NP Newness Line addition Improvement Completely new Respondent factors Duration of Employment Less than 1 year 1 to 3 years 3 to 5 years More than 5 years Number of NPD Projects Involved 1 new product 2 4 new products 5 9 new products More than 10 Functional Area of Respondent Marketing sales R D development General management Other 16 57 19 4 17 59 20 4 43 11 12 3 62 16 17 4 59 68 31 7 36 41 19 4 10 43 19 25 10 44 20 26 6 30 19 14 9 43 28 20 16 73 38 39 10 44 23 23 8 11 17 61 8 11 18 63 5 12 13 39 7 17 19 57 13 23 30 100 8 14 18 60 16 65 16 17 67 17 23 42 4 33 61 6 39 107 20 24 64 12 47 38 12 49 39 12 39 16 14 57 23 20 86 54 26 52 33 16 80 15 84 16 50 19 72 28 130 34 79 21 26 50 19 27 53 20 22 35 12 32 51 17 48 85 31 29 52 19 32 29 34 2 33 30 35 2 4 8 43 14 6 12 62 20 36 37 77 16 22 22 46 10 Percent United Kingdom Number Percent Total Sample Number Percent

INDUSTRIAL COMPANIES EVALUATION CRITERIA IN NEW PRODUCT DEVELOPMENT GATES

J PROD INNOV MANAG 2003 20 22 36

27

participate We checked the responses from early and late respondents to assess nonresponse bias 1 No signi cant differences were found between both sets of respondents thus minimizing concerns for nonresponse bias To obtain a homogeneous sample of respondents we will use only the responses from the companies who developed and launched new products for industrial customers N597 Table 1 contains the pro le of the Dutch sample United Kingdom We used the FAME CD ROM Business Directory by Bureau van Dyke to develop a sample frame of 500 manufacturing companies with more than 20 employees in the same selected industries chemicals medical equipment electronics and computers From this original sample frame of 500 rms after initial contact by phone 48 were excluded because of a policy of company con dentiality A further 14 had not been engaged in any NPD efforts and therefore were excluded Of the remaining 438 respondents that agreed to participate 100 returned the questionnaire thus providing a response rate of 23 percent Again nonresponse bias was examined and was not found to be a major problem 1 To also obtain a homogeneous sample of UK respondents only the responses from industrial companies were included N569 Table 1 also provides an overview of the UK sample Questionnaire The questionnaire focused on the 15 core project level indicators of new product performance 18 19 Since Grif n and Page 18 19 were mainly interested in measuring new product performance after launch we also scanned the literature for criteria that are used in earlier gates of the NPD process Five additional criteria were identi ed product uniqueness market potential marketing chance technical feasibility and intuition 2 11 15 26 32 By comparing seven models of the NPD process 4 6 9 13 25 30 36 six distinct evaluation gates were selected idea screening concept screening business analysis product testing analyzing test market results and after launch assessment Hultink and Robben 24 made a distinction between measuring new product performance in the short term and in the long term after launch They found that the importance attached by managers to the indicators of new product performance depended strongly on this time perspective Therefore we decided to include the short term as well as the long term performance assessment Hence for each of the seven evaluation gates respondents assessed which of the 20 evaluation

criteria were used Respondents were asked to answer the questions in relation to representative new products that they had developed and had launched in the previous ve years The ability of the respondents to provide well informed answers to these questions was checked by means of the number of NPD projects they were involved in the previous ve years as well as by the duration of their employment with the rm It was found that 90 percent of the respondents were involved with the development of more than two new products during the previous ve years and that 78 percent of them had been with the rm for more than three years see Table 1 These ndings increased our con dence in the ability of the respondents to provide wellinformed answers regarding their rms and associated NPD practices Respondents received along with the questionnaire detailed de nitions and a diagrammatic representation of the NPD stages and evaluation gates These de nitions and diagram were pretested with six respondents in four different companies increasing the face validity of our instrument as it communicated a consistent picture of NPD stages and gates across the respondents The pretest showed that it was not necessary to provide de nitions for the 20 evaluation criteria as the respondents felt comfortable with the criteria as stated and considered that de nitions for the criteria were super uous and even distracting

Analysis and Results
This section consists of three parts First the most frequently used evaluation criteria at the seven NPD gates will be presented Then we will focus on the patterns of use of various evaluative dimensions Finally we will test whether any differences exist with regard to the evaluative dimensions used per NPD gate for different subgroups of respondents

Evaluation Criteria at NPD Gates
For those respondents who said they evaluated the new product in a certain gate we calculated the percentage of those rms that used any of the 20 evaluation criteria Table 2 shows the results from this analysis shadowed boxes include the percentage of rms that exceeds our cut off point of 50 percent indicating what we regard as the most

28

J PROD INNOV MANAG 2003 20 22 36

S HART ET AL

Table 2 Percentage of Companies Using Evaluation Criteria at NPD Gates
The Netherlands

Customer Satisfaction

Customer Acceptance

Product Performance

IRR ROI

Idea Screening Concept Screening Business Analysis Product Testing Test Market Post Launch S T Post Launch L T

46 48 31 36 68 54 31

31 37 22 37 66 59 52

24 11 51 6 9 41 38

27 1 38 2 7 28 42

31 4 37 5 12 33 36

38 14 62 22 24 59 51

11 6 27 11 8 15 7

24 10 46 9 12 41 38

5 9 26 6 8 19 17

27 8 46 16 16 44 40

8 9 16 39 17 12 10

19 14 22 34 29 32 9

51 48 25 65 63 45 31

36 36 18 67 56 35 23

17 9 20 21 18 14 4

51 18 20 28 23 26 14

52 22 50 10 19 35 27

28 14 39 6 17 22 13

70 47 27 36 43 8 3

54 20 20 20 19 16 13

United Kingdom

Customer Satisfaction

Customer Acceptance

Product Performance

IRR ROI

Idea Screening Concept Screening Business Analysis Product Testing Test Market Post Launch S T Post Launch L T

54 60 31 45 77 77 57

38 45 25 39 66 75 76

46 20 60 20 27 70 68

27 10 52 14 16 64 72

27 10 55 13 16 52 74

27 23 70 16 20 71 68

11 10 42 13 16 32 28

32 15 67 20 20 57 72

16 8 67 16 17 29 47

35 23 73 27 25 71 79

19 25 31 59 33 16 6

14 22 31 45 41 43 13

33 45 27 73 83 54 55

22 32 22 69 79 61 64

35 23 46 47 36 32 0

75 42 36 42 36 32 30

73 53 69 30 38 36 32

40 35 48 17 19 23 17

75 70 33 59 48 9 8

62 28 19 19 11 14 8

frequently used NPD evaluation criteria As expected some criteria were used more often in the early gates of the NPD process for example technical feasibility and intuition while other criteria were more often used in later gates of the process for example sales in units meeting pro t objectives and margin Some criteria were frequently used in only one or two of the evaluation gates for example intuition and product uniqueness while other

criteria were frequently used in at least three of the seven evaluation gates for example customer satisfaction sales in units and product performance Below we will discuss the three most frequently used criteria at the different NPD evaluation gates see Table 3 Idea screening Technical feasibility is the most frequently used criterion for idea screening purposes Intuition also plays a major role see Table 2 This is

Intuition

Quality

Margin

NPD Evaluation Gates

Technical Feasibility

Evaluation Criteria

Stays within Budget

Product Uniqueness

Introduced in Time

Marketing Chance

Profit Objectives

Break Even time

Market Potential

Sales Objectives

Time to Market

Sales in Units

Sales Growth

Market Share

Intuition

Quality

Margin

NPD Evaluation Gates

Technical Feasibility

Evaluation Criteria

Stays within Budget

Product Uniqueness

Introduced in Time

Marketing Chance

Profit Objectives

Break Even time

Market Potential

Sales Objectives

Time to Market

Sales in Units

Sales Growth

Market Share

INDUSTRIAL COMPANIES EVALUATION CRITERIA IN NEW PRODUCT DEVELOPMENT GATES

J PROD INNOV MANAG 2003 20 22 36

29

Table 3 Most Frequently Used Evaluation Criteria at NPD Gates
The Netherlands Indicator Percent of use Indicator Idea Screening Technical feasibility Intuition Market potential Customer acceptance Product performance Technical feasibility Sales in units Sales objectives Market potential Quality Product performance Stays within budget Customer acceptance Customer satisfaction Product performance Customer satisfaction Sales in units Customer acceptance Customer satisfaction Sales in units Sales growth 70 54 52 48 48 47 62 51 50 67 65 39 67 65 63 59 59 54 52 51 42 Technical feasibility Product uniqueness Market potential Concept Screening Technical feasibility Customer acceptance Market potential Business Analysis Margin Sales in units Market potential Product Testing Product performance Quality Stays within budget Product performance Quality Customer acceptance Customer acceptance Customer satisfaction Sales in units Margin Customer satisfaction Market share 73 69 59 73 69 59 77 75 71 79 76 74 73 70 69 70 60 53 75 75 73 United Kingdom Percent of use

Analyze Test Market Results

Post Launch Evaluation Short Term

Post Launch Evaluation Long Term

not surprising given the large uncertainties and the lack of relevant valid information at this stage In addition market potential and product uniqueness are assessed It seems that rms in our sample follow a balanced approach at this gate by evaluating both technical and market aspects of new ideas Concept screening Table 3 shows that rms most frequently used customer acceptance product performance and technical feasibility at this gate UK companies also test the market potential of the new product concept Of interest is the lack of use of any nancial criteria in the rst two evaluation gates of the NPD process Business analysis After the business analysis is performed large investments are needed to proceed

to the next stage of the NPD process Hence it is critical to forecast the sales and pro t levels for the proposed new product Accordingly in the business analysis gate companies tend to use sales criteria such as sales in units instead of product level criteria Market potential also gures heavily as there is a clear association between sales and market potential UK companies also evaluate the product s margin extensively Product testing After the product development stage it is of paramount importance to check whether the new product has met its objectives from a technical point of view To prevent technical dogs from being developed 5 not surprisingly the most frequently used criteria in this stage have to do with

30

J PROD INNOV MANAG 2003 20 22 36

S HART ET AL

product performance and quality Firms also are keen to measure whether they stay within the development budget at this gate Analyze test market results After the market testing stage in the NPD process companies measure the performance and quality of the new product again In addition customer reactions to the new product are collected By evaluating technical aspects as well as customer acceptance and satisfaction companies apparently try to prevent launching better mousetraps that nobody wants 5 Post launch evaluation short term This gate is the rst real test for the new product in the market Now it becomes clear whether the product will be accepted in the marketplace and whether any pro t will be made on the new product investments In addition it is the market launch stage where positive or negative word of mouth owing to customer satisfaction can build or break the new product s success Hence the most frequently used evaluation criteria measure the customer acceptance of the new product customer satisfaction and sales levels Post launch evaluation long term At this time the new product should be well established in the marketplace Initial problems to the new product have been sorted out Performance now is assessed mainly in terms of sales and market share However it is interesting to note that customer satisfaction also is being assessed in this gate This we argue is important since according to Woodruff 38 changes in customer expectations may in uence the level of satisfaction they get from a product Firms therefore need to be alert to detect such changes which if surpassed may undermine their competitive position Firms in the UK also frequently measure the new product s margin at this gate

Evaluative Dimensions at NPD Gates
To address research question 2 our objective was to move away from single criteria and examine what is happening in a broader context for example do companies show a usage preference of some evaluative dimensions over others at different gates in the NPD process As mentioned earlier the 20 criteria used in our study correspond to the 15 project level criteria identi ed by Grif n and Page 18 19 and include ve more because our study examined the entire range of the NPD process and not just after launch performance In the present step of our analysis we used

Grif n and Page s 18 three dimensions of performance market acceptance nancial performance and product performance as well as the additional indicators we introduced in a category of their own additional indicators Table 4 presents the results of this analysis Each cell in Table 4 depicts the number of evaluation criteria from each of the four dimensions used per NPD gate The total average number of criteria for each dimension used throughout the entire process is shown column totals as is the total average use of criteria for each dimension per stage row totals Furthermore all are broken down for the Dutch and UK sample and statistical signi cance of differences tested using t tests Table 4 shows that the market acceptance dimension permeates all the evaluation gates in the NPD process indicating the market orientation of these rms The nancial dimension emerges prominently only during the business analysis evaluation and after market launch in the short and long term The product performance dimension despite guring in almost every evaluation gate becomes prominent during the product testing and test market evaluation gates Finally our dimension containing additional criteria shows a fair use over the entire range of the NPD evaluation gates yet with prominence during the idea screening gate These results are consistent for both countries in which we collected our data In fact only six out of 29 tests between the two countries appeared to be statistically signi cant see Table 4 We proceeded to examine the emerging patterns in evaluative dimensions over the NPD evaluation gates Figure 1 presents four graphical representations of the number of criteria used per dimension over the NPD evaluation gates Examining this gure it becomes apparent again that the two country samples show a similar pattern of usage although rms in the UK generally use more criteria an observation strengthened by the fact that one of the six statistically signi cant differences between the two countries was the total average number of criteria used for each stage throughout over the entirety of the evaluative gates Finally in order to compare the relative usage of the four dimensions over the NPD evaluation gates we calculated the standardized number of criteria per dimension used in each evaluation gate i e number of criteria per dimension used divided by the number of criteria in each dimension Figure 2 provides a graphical representation of the results of this analysis

INDUSTRIAL COMPANIES EVALUATION CRITERIA IN NEW PRODUCT DEVELOPMENT GATES

J PROD INNOV MANAG 2003 20 22 36

31

Table 4 Number of Criteria per Evaluative Dimension at the NPD Gates

Evaluation Dimensions Financial Performance Financial Performance Additional Indicators Product Performance Additional Indicators Product Performance Market Acceptance

Idea Screening Concept Screening Business Analysis Product Testing Test Market Post Launch S T Post Launch L T Total Average

2 0 1 2 2 4 1 1 1 8 2 7 2 5 13 71

0 7 0 3 1 5 0 4 0 4 1 2 1 0 5 58

1 3 1 2 1 0 2 3 1 8 1 4 0 8 9 8

2 5 1 2 1 6 1 3 1 2 1 1 0 7 9 62

6 5 3 9 6 5 5 1 5 3 6 4 5 0 38 71 5 53

Total

2 2 1 7 2 9 1 5 2 2 4 1 4 1 18 7

0 9 0 6 2 5 0 8 0 8 1 9 2 3 9 8

1 2 1 5 1 6 2 9 2 7 2 1 1 4 13 4

3 2 2 3 2 1 1 7 1 5 1 1 0 9 12 8

7 6 6 1 9 0 6 9 7 2 9 2 8 7 54 7 7 8

Note Entries in bold indicate a signi cant difference in the number of criteria used per dimension between The Netherlands and the UK We adjusted the a level for the number of tests For all signi cant differences means were higher for the UK

The pattern of relative usage of the evaluative dimensions over the NPD gates was identical in both countries Figure 2 therefore shows the relative usage of these dimensions for the total sample Our set of additional indicators which included criteria such as product uniqueness technical feasibility and intuition is used most frequently at the idea and conceptscreening gates Then the nancial performance and market acceptance dimensions take over at the business analysis gate Assessing product performance dominates the product testing and test market gates during which the nancial performance dimension is largely ignored The fact that nancial performance hardly is assessed at the test market gate is somewhat surprising as it suggests that only the functional performance and market acceptance are assessed without their attendant nancial implications Thereafter however the market acceptance and nancial performance dimensions resume their primacy over the other dimensions as the NPD process draws nearer to its conclusion Patterns of usage in Figure 2 corroborate our earlier ndings and provide support to the work of Ronkainen 32 and Hultink and Robben 24 Indeed the latter authors found that the importance of nancial criteria is increased in the long term

assessment of NPD success The usage pattern of nancial criteria in Figure 2 supports this variation that also is evident in Ronkainen s 32 work However whereas Ronkainen 32 found that market acceptance criteria scored lower in usage than nancial performance criteria in the concept screening gate our ndings suggest a different pattern Figure 2 shows that market acceptance criteria score consistently is higher than nancial criteria in every evaluation gate apart from the business analysis evaluation As explained earlier this may be an indication of the higher market orientation of rms in our sample as compared to those studied by Ronkainen 32 in 1985 This would be consistent with the development in the last decade of the concept of market orientation which might be expected to lter into numerous management practices including the evaluative dimensions of the NPD process producing a stronger emphasis on market reaction as a primary indication of the green light throughout the development process Given the paucity of previous empirical research on the evaluation gates in NPD looking at what might account for different usage of evaluative dimensions may lead to greater insights as to when and why some dimensions should be deployed We

Total

NPD Evaluation Gates

Market Acceptance

32

J PROD INNOV MANAG 2003 20 22 36

S HART ET AL

Country Variations in Market Acceptance
Number of Criteria

6 0 5 0 4 0 3 0 2 0 1 0 0 0
Idea Screening Concept Screening Business Analysis Product Testing Test Market Post Launch S T Post Launch L T

Netherlands U K

NPD Evaluation Gates

Country Variations in Financial Performance
Number of Criteria
4 0 3 0 2 0 1 0 0 0

Netherlands U K

Idea Screening

Concept Screening

Business Analysis

Product Testing

Test Market

Post Launch S T

Post Launch L T

NPD Evaluation Gates

Country Variations in Product Performance
Number of Criteria
5 0 4 0 3 0 2 0 1 0 0 0

Netherlands U K

Idea Screening

Concept Screening

Business Analysis

Product Testing

Test Market

Post Launch S T

Post Launch L T

NPD Evaluation Gates

Country Variations in Additional indicators
Number of Criteria 5 0 4 0 3 0 2 0 1 0 0 0 Idea Screening Concept Screening Business Analysis Product Testing Test Market Post Launch S T Post Launch L T Netherlands U K

NPD Evaluation Gates

Figure 1 Number of Evaluation Criteria Used per Dimension over the NPD Gates

therefore decided to investigate the impact of the sample pro le variables to examine the stability of the patterns of evaluation

The Impact of Pro le Variables on the Evaluative Dimensions Used
Three sets of pro le variables were analyzed for evidence of changing patterns of evaluation These pro le variables included rm level factors size market share position and NPD driver NPD program level factors innovation strategy and product newness and respondent factors experience

and functional background Table 5 shows the number of signi cant differences out of 28 tests for each of these sets of factors The tests used were either t tests or one way analysis of variance depending upon the number of groups used to describe the pro le variables see Table 1 Of the rm level factors only one difference among rms of different sizes was found to be signi cant statistically At the business analysis stage larger rms used more product performance criteria than the smallest companies in the sample This however may be a chance nding In addition rms with different market share positions tended to use the same

INDUSTRIAL COMPANIES EVALUATION CRITERIA IN NEW PRODUCT DEVELOPMENT GATES

J PROD INNOV MANAG 2003 20 22 36

33

Variations in Performance Dimensions over NPD Evaluation Gates
Standardized Number of Criteria per Dimension
0 6 0 5 0 4 0 3 0 2 0 1 0 0
Idea Concept Screening Screening Business Analysis Product Test Market Post Launch Post Launch Testing S T L T

Market Acceptance Financial Performance Product Performance Additional Indicators

NPD Evaluation Gates

Figure 2 Relative Use of Evaluative Dimensions over NPD Gates for total sample

number of criteria per dimension and when the sample was subdivided into those rms that were technologically driven and those that were market driven again the number of criteria used for each evaluative dimension showed no change Similarly when we examined the NPD program level factors no differences in the number of evaluation criteria per dimension were found

Table 5 Differences in Number of Criteria Used per Dimension across Subgroups
Number of Differences Profile Variables Out of 28 tests Firm Level Factors Number of employees Market share position NPD driver NPD Program Level Factors Innovation strategy Product newness 0 0 1 0 0

Finally when looking at the respondent factors the number of evaluation criteria used per dimension did not uctuate with different levels of respondent expertise A slight difference was found in four out of 28 tests with respect to the functional area of the respondent Speci cally at the idea screening gate marketing sales respondents used more market acceptance criteria than research and development R D respondents At the concept screening gate marketing sales respondents used more criteria for the product performance dimension while at the product testing and long term post launch evaluations marketing sales respondents used more of the additional criteria The number of differences however is so low that we cannot be sure that they are not merely chance results Overall given the vast number of tests that were insigni cant we have no strong evidence in support of rm size market position NPD program type or allegiances of the respondents being signi cant moderators of any variations in the evaluative dimensions used at different gates of the NPD process That said the results hint that there may be a preference for using evaluative dimensions consistent with the traditions and training of one s professional functional area and overall measurement is more intense in the UK than in the Netherlands

Respondent Factors Duration of employment Experience in NPD Functional area 0 0 4

Discussion Conclusions and Recommendations
It is agreed widely that the notion of complexity is inherent in the NPD efforts of industrial companies However it is agreed equally that this complexity can and should be managed for the successful development of new products One means of doing so is by

34

J PROD INNOV MANAG 2003 20 22 36

S HART ET AL

establishing guideposts against which management activity can be evaluated controlled and modi ed if needed throughout the NPD process The signi cance of the latter becomes apparent if one considers the ever increasing costs of NPD and the detrimental effects to the rms resources if they commit to the wrong project Research in the past has outlined a number of criteria used by rms to evaluate new product projects Research also has demonstrated that as these projects progress they impose different requirements to the rm s management and its operational resources Therefore it is logical to assume that the evaluative criteria used per gate of the NPD process should re ect the different tasks and competencies required at the corresponding stages Yet evidence in support of this assumption is only scant thus producing an incomplete picture of the NPD measuring processes in industrial rms The present study aimed to ll a part of this gap in extant knowledge Our ndings rst show that companies use different criteria at different NPD evaluation gates While such criteria as technical feasibility intuition and market potential are stressed in the early screening gates of the NPD process a focus on product performance quality and staying within the development budget are considered of paramount importance after the product has been developed Customer acceptance and satisfaction and unit sales are primary considerations during and after market launch Our study also produced interesting patterns of usage of evaluative dimensions over the NPD process The market acceptance dimension permeates the entire NPD process and gains in prominence during the short and long term performance evaluation after launch As we argued earlier this may be attributed to the continuous attention to customer acceptance and satisfaction paid by the rms in our sample From a theoretical point of view this is noteworthy as marketing theory always has advocated in favor of a continuous orientation to the needs of the customer 29 Furthermore the fact that this orientation permeates the whole NPD process is a rst indication that despite speci c considerations at each stage e g conceptual technical nancial and market rms attempt to impose an overall layer of customer and marketing orientation We speculate that this may enable them to listen and react to the voice of the customer in a systematic and coherent way alongside the whole spectrum of their NPD efforts

The nancial dimension emerges prominently in the business analysis gate and gains importance in the short and long term performance evaluation after launch Again use of this dimension corresponds to the requirements of the speci c stage Evaluation criteria of a nancial nature may assist management to assess the ef ciency and effectiveness of their efforts and to identify whether their products need additional support a rejuvenating injection of capital or a strategy of deletion to give way to other products or release resources for other NPD projects As expected the product performance evaluative dimension gures strongly in the product and market testing gates This we suggest re ects management efforts to avoid go errors that have to do with the launch of technical dogs or better mousetraps no one wants 5 Our set of additional criteria proved to be especially relevant in the idea screening gate Consideration of product uniqueness market potential and technical feasibility at the early stages of the NPD project re ects we argue a tendency of our sampled rms to follow a balanced yet holistic approach in their NPD efforts From a managerial point of view the above patterns may provide useful guidelines for focusing attention and expending resources during the entire NPD process We argue that the informed use of evaluation criteria as guideposts for increased managerial attention and identi cation of problems may help management to prevent drop and go errors in their innovation efforts Managers may compare and contrast ndings from this study with their own NPD practices and by doing so may enrich the knowledge pool upon which they draw to make well informed decisions We suggest that among other things three key issues emerged for managers First we recommend that managers should strive to develop and to implement evaluative criteria targeted to the speci c requirements and expectations from each stage of their NPD project This we argue will allow them to detect problems and to identify opportunities as they occur Moreover although not considered in this article carefully focused evaluative criteria allow management to allocate responsibility and to exercise control that re ects managerial fairness and justice Second we recommend that managers should listen to the voice of the customer throughout their NPD process This can be accomplished by evaluating each stage of their NPD process among other things with customer acceptance and customer

INDUSTRIAL COMPANIES EVALUATION CRITERIA IN NEW PRODUCT DEVELOPMENT GATES

J PROD INNOV MANAG 2003 20 22 36

35

satisfaction in mind Of course as suggested earlier management should consider evaluative criteria that are aligned to the requirement and expectations of each stage yet customer acceptance and satisfaction emerged as evaluative criteria with high usage along the whole spectrum of NPD activities in our sampled rms Third it was interesting to nd that the pattern of evaluative dimensions used over the NPD gates was similar across subgroups of respondents Although the present study did not include all possible relevant rm level NPD program level and respondent factors those mentioned frequently in the literature were investigated These ndings suggest that it is not important how large the company is what their market share position is what the driver of NPD is what innovation strategy they follow and what kind of new products they develop All rms likely should measure their NPD processes in terms of the evaluative dimensions in a pattern consistent with the one found for our samples to assess the attractiveness of new product opportunities at different stages of development

Limitations and Further Research Directions
This research project has helped de ne better the complexity and structure of NPD measuring processes in industrial rms However as with all research the methods employed have inherent limitations which lead to opportunities to improve future research in this area First we asked respondents to answer the questions in relation to representative new products they had developed and had launched in the previous ve years This retrospective methodology has several limitations For example halo bias effects may be present because the performance of the new products chosen was known prior to completing the survey There also may be differences between respondents recalled and actual measurements For example selectivity of recall rationality bias and reconstruction bias may cause respondents to bias upwardly their responses in order to make their rms look good Second this research investigated the use of criteria across several NPD evaluation gates An obvious next step would be to measure the perceived importance of the criteria at the different evaluation gates Although it may be suggested that the frequency of use of a

certain criterion re ects or is related to the importance of measuring this criterion additional research is necessary to substantiate this A third limitation is that this research is based on what managers reported that they have done Thus the research is descriptive providing insight only into the number and nature of criteria used at the evaluation gates While this is useful it would be even more helpful if we could tell them what to do For example by contrasting successful with unsuccessful rms or successful with failed new products differences may appear in the criteria that are used at the evaluation gates The striking similarities that were identi ed in our results across subgroups of respondents may indicate that heterogeneity of samples in research investigating NPD evaluation criteria need not bias the results However we also suggest using alternative lenses for detecting any differences or similarities in the use of NPD evaluative criteria For example an analysis comparing emerging new economy with traditional sectors old economy may produce additional insights to this subject Also we suggest that in line with our earlier discussion one would expect differences in the use of evaluative criteria on the basis of differences on strategic objectives set or other situational and environmental conditions surrounding particular NPD projects For example researchers should examine whether time pressures and hostile competitive environments moderate both the number and relative importance of evaluative criteria used per stage of the NPD process The type of the NPD project may call for the use of additional evaluative criteria as in the work of Hauser 22 Also collaborative NPD projects may require development and implementation of different evaluation criteria Furthermore differences in the number of criteria used by Dutch and UK companies deserve further attention Country variations in the average size of rms and the functional background of the respondents in our sample suggest that there may be value in research seeking to assess if such variations may moderate the number of evaluation criteria used at different gates of the NPD process Finally as evaluative criteria may re ect areas where managerial attention is directed one can expect a link between evaluative criteria used and strategic and operational capabilities of the rm Researchers examining this link may nd an excellent opportunity for integrating strategic concepts into the theory and practice of NPD Overall results of this project

36

J PROD INNOV MANAG 2003 20 22 36

S HART ET AL

increase the current state of knowledge with regard to the pool of criteria used by management to safeguard the success of their NPD efforts The signi cance of the latter can be outlined by referring to Cooper and Kleinschmidt 10 who stated If businesses are to survive and prosper managers must become astute at selecting new product winners and at effectively managing the process from idea to launch

19 Grif n A and Page A L PDMA s success measurement project recommended measures by project and strategy type Journal of Product Innovation Management 13 478 496 1996 20 Grif n A The PDMA Best Practice Report PDMA 1997 21 Hart S Dimensions of success in new product development an exploratory investigation Journal of Marketing Management 9 23 41 1993 22 Hart S and Baker M Learning from success multiple convergent processing in new product development International Marketing Review 11 1 77 92 1994 23 Hauser J R Research development and engineering metrics Management Science 44 1670 1689 1998 24 Hauser J R and Zettelmeyer F Metrics to evaluate RD E Research Technology Management 40 32 38 1997 25 Hultink E J and Robben H S J Measuring new product success the difference that time perspective makes Journal of Product Innovation Management 12 392 405 1995 26 Kotler P Marketing Management Analysis Planning Implementation and Control 7th edition Englewood Cliffs NJ Prentice Hall 1991 27 Lipovetsky S Tishler A Dvir D and Shenhar A The relative importance of project success dimensions R D Management 27 2 97 106 1997 28 Mahajan V and Wind J New product models practice shortcomings and desired improvements Journal of Product Innovation Management 9 2 128 139 1992 29 Moenaert R K and Souder W E An information transfer model for integrating marketing and R D personnel in NPD projects Journal of Product Innovation Management 7 2 91 107 1990 30 Narver J C and Slater S F The effect of marketing orientation on business performance Journal of Marketing 54 October 20 35 1990 31 Page A L Assessing new product development practices and performance establishing crucial norms Journal of Product Innovation Management 8 18 31 1993 32 Rochford L and Rudelius W How involving more functional areas within a rm affects the new product process Journal of Product Innovation Management 9 4 287 299 1992 33 Ronkainen I A Criteria changes across product development stages Industrial Marketing Management 14 171 178 1985 34 Saren M S Reframing the process of new product development from stages models to a blocks Journal of Marketing Management 10 7 633 644 1994 35 Schmidt J B and Calantone R J Are really new product development projects harder to shut down Journal of Product Innovation Management 15 March 111 123 1998 36 Takeuchi H and Nonaka I The new product development game Harvard Business Review Jan Feb 137 146 1986 37 Urban G L and Hauser J R Design and Marketing of New Products Englewood Cliffs NJ Prentice Hall 1993 38 Wind J and Mahajan V Issues and opportunities in new product development Journal of Marketing Research 34 1 12 1997 39 Woodruff R B Customer value the next source for competitive advantage Journal of the Academy of Marketing Science 25 2 139 153 1997

References
1 Armstrong J S and Overton T S Estimating non response bias in mail surveys Journal of Marketing Research 14 August 396 402 1977 2 Balachandra R Critical signals for making go no go decisions in new product development Journal of Product Innovation Management 1 2 92 100 1984 3 Biemans W G Managing Innovation within Networks London and New York Routledge 1992 4 Booz Allen and Hamilton New Products Management for the 1980s New York NY Booz Allen Hamilton 1982 5 Calantone R J and Cooper R G A discriminant model for identifying scenarios of industrial new product failure Journal of the Academy of Marketing Science 7 3 163 183 1979 6 Cooper R G Stage gate systems a new tool for managing new products Business Horizons May June 44 54 1990 7 Cooper R G Winning at New Products Accelerating the Process from Idea to Launch Reading MA Addison Wesley 1993 8 Cooper R G and de Brentani U Criteria for screening new industrial products Industrial Marketing Management 13 149 156 1984 9 Cooper R G and Kleinschmidt E J An investigation into the new product process steps de ciencies and impact Journal of Product Innovation Management 3 71 85 1986 10 Cooper R G and Kleinschmidt E J Success factors in product innovation Industrial Marketing Management 16 215 223 1987 11 Craig A and Hart S Where to now in new product development research European Journal of Marketing 26 1 49 1992 12 Crawford C Merle De ning the charter for product innovation Sloan Management Review Fall 3 12 1980 13 Crawford C M New Product Management Homewood IL Richard D Irwin 1994 14 De Brentani U Do rms need a custom designed new productscreening model Journal of Product Innovation Management 3 108 119 1986 15 Feldman L P and Page A L Principles versus practice in new product planning Journal of Product Innovation Management 1 43 55 1984 16 Gehani R R Concurrent product development for fast track corporations Long Range Planning 25 6 40 47 1992 17 Grif n A Metrics for measuring product development cycle time Journal of Product Innovation Management 10 2 112 125 1993 18 Grif n A and Page A L An interim report on measuring product development success and failure Journal of Product Innovation Management 10 291 308 1993