REQUIREMENTS-DRIVEN RISK

 

CONTENTS

 

                                                                                                                                                                                                          

1       INTRODUCTION

 

2      THE NATURE OF REQUIREMENTS-DRIVEN RISK

                                             2.1        Causes of Requirements-Driven Risk

                                             2.2        Consequences of Requirements-Driven Risk

                                             2.3        The Cost of Requirements Driven Risk

 

3      MINIMISING REQUIREMENTS-DRIVEN RISK

                                             3.1        The Problem

                                             3.2        The Solution – Requirements Engineering

                                             3.3        Key Features of Requirements Engineering

                                             3.4        The Cost of Requirements Engineering

                                             3.5        Applying RE to Existing projects

                       3.6        Getting Started

                                             3.7        How RE Reduces Risk Throughout the Project Life Cycle

 

4      SUMMARY

 

5      CONCLUSION

 

6       RECOMMENDATION

 

Table

Principal Benefits to Specific Project Phases of Applying RE.

 

Annex

Requirements Catastrophies

-------------||-------------

 

 

 


1               INTRODUCTION

 

Requirements-Driven Risk (RDR) is a problem, shared across all government, industry and commercial sectors, that is probably the single biggest cause of failure to meet time, cost and performance targets for major enterprises - if you do not know if you are acquiring, assessing or supporting the right thing, what greater risk could you have?  In order to minimize the impact of RDR, we must:

             Understand the nature of the risk;

             Minimize it by:

-                             Using proven procedures;

-                             Measuring it to allow its management.

 

This paper presents an overview of the nature of Requirements-Driven Risk and summarises some of the more important aspects of ISA’s approach based on more than 40 elapsed years of in-depth experience developing requirements specifications for clients across all market sectors.

 

 

 

2        THE NATURE OF REQUIREMENTS-DRIVEN RISK

2.1    Causes of Requirements-Driven Risk

a    Misunderstanding

We all experience difficulty when expressing requirements, without exception and whether or not we realize it.  Natural language does not lend itself directly to expressing requirements in a complete, coherent, consistent and correct (C4) form[1] - and formal languages have not yet evolved sufficiently for this purpose.  In fact there is significant evidence to support the view that they never will.  If the information that a Statement of Requirements (SOR) contains is not well understood, it will be impossible to realize to time and cost as a system which works as intended by the specifier – even in the best and most well run organizations.

 

 

b    Subjectivity

The International Standards Organization (ISO) insists that a requirement is not correctly specified unless it contains an objective test by which it can be demonstrated to have been satisfied.  Failure to do so at the outset results in misinterpretation and problems for both specifier and deliverer in order to truly understand what is being asked for (even if they think they do at the time), with concomitant problems at acceptance.

 

 

c   Validity

SORs for complex systems are necessarily large and complicated.  Such SORs invariably contain conflicting requirements that are hard to spot until it’s too late (usually in the design, manufacture or even the in-service phase).  Put another way: we do not know if it will work as expected, even if it is built to specification!

 

 

d   Incompleteness

SORs for complex systems contain statements at various levels of detail, from several viewpoints and covering several layers of concern.  Requirements at intermediate levels or layers are often missed by the specifier and have to be guessed at during system realization – often with disastrous implications for the system in question and other systems with which it is to interact.

 

e   Complexity

Every system exists as part of, and will interact with, other elements in a wider system.  The consequences of changes in the SORs for related systems are rarely identified and taken into account, resulting in conflicts and gaps in capability between such systems.

 

 

2.2    Consequences of Requirements-Driven Risk

Failure to get the requirement right can cause a variety of problems with timeliness, cost and performance.  Examples are:

 

             Failure to gain project approvals (weak business case, based on poor requirement)

              Contractual repercussions (poor understanding of requirement by one, or both parties)

              Cost overruns (re-work; “hidden” requirements)

             Time overruns (re-work; adding in missing detail)

             Performance failures (failure to understand real operational requirement)

             Nugatory work or cancelled projects (failure to filter out spurious requirements)

             Poor initial support arrangements (poorly understood support viewpoint )

             Poor in-use decisions (poorly understood operational baseline).

 

 

2.3    The Cost of Requirements-Driven Risk

It is now becoming widely accepted, from research in the USA and in Europe, that if a fault that would have cost 1000 to correct during requirement specification is left until the design phase has been initiated and it is left to the designers to detect and correct, then the cost of correction will increase by a factor of 10, to 10,000.  Similarly, if the fault is not detected during subsequent design (which is often the case when parts of complex systems are designed separately) and the fault is left until the system is being manufactured before it is detected and corrected, the cost of doing so will increase by a factor of 100 over that during specification, to 100,000.

 

If the detection and correction of the fault is not carried out until after the system has gone into service, the cost of correction will increase by a factor of 1000 over that during specification - to 1M!  The cost of “working around” such a fault in service can be many times higher again for systems with long life-cycles, such as warships, railway networks and international financial systems.  The only certainty is that the fault will be revealed eventually – the longer it takes, the more painful and costly the consequence.

 

 

The above costs are not exaggerations.  Hardly a week goes by without a system catastrophe being reported in the technical (and, increasingly, national) press as being due to inadequate attention having been devoted to the specification phase of the project.  The cost, in Europe alone, of inadequate attention to specifications, is estimated to be running at between 27 billion and 30 billion per year and rising.  Such estimates do not include the additional costs due to the inadequacy of the resulting systems and all their knock-on consequences.  A report produced by the Standish Group (see Annex below) suggests that the overall knock-on costs of poorly produced specifications could well be $Trillions!

 

 

 

3        MINIMIZING REQUIREMENTS-DRIVEN RISK

3.1    The Problem

The fundamental problem is the need to identify the specific requirement for the enterprise and express it in a way that maximizes the probability that it will be satisfied as cheaply and quickly as possible without compromising the performance of the delivered system(s).

 

3.2    The Solution – Requirements Engineering

There is now a proven, systematic approach known as Requirements Engineering, which will achieve this crucial goal if applied methodically.  Before describing this approach we need to make distinctions between Requirements Capture (RC) - “getting it in”, Requirements Engineering (RE) - “getting it right” and Requirements Management (RM) - “getting it done”[2].  All three are equally important but sadly, whilst RC and RM are usually addressed, RE is at best given lip service and, most often, is completely ignored – particularly by those organizations that believe that RE and RM are the same thing.  Failure to get the requirement right means that all other efforts on behalf of the enterprise must be called into question.  To quote a well known truism:

 

Rubbish in = rubbish out, no matter how well the rubbish is managed”.

 

 

3.3    Key Features of Requirements Engineering

To be worthy of the “Engineering” label, RE must have an associated and formally defined methodology.   Any such methodology must feature a logically derived procedure to develop the requirement, plus comprehensive and objective metrics that are able to demonstrate improvements in the stated requirement as it is developed:

a.                   The need for procedural guidance in developing requirements is inescapable if we are to have any confidence that we are going to produce output of consistently high quality across all systems, learning from previous mistakes and inventing wheels only once.  

b.                  The value of objective and meaningful requirements metrics to senior and project management cannot be overstated, since the degree of requirements-driven risk inherent in a particular SOR, at a particular time, must be made explicit in order to allow informed decision-making at all levels, including reassurance of stakeholders.

  

A good RE methodology must be able to handle complexity - not only within a specific SOR, but also across many related SORs (i.e. “system of systems”).  This is variously referred to as “Capability Management” or “Knowledge Management” and it demands robust structural guidance from the methodology.

 

 

3.4            The Cost of RE

 

The cost of not getting the requirement right is always too high – usually catastrophically so (see Annex to this note and, in particular, the CHAOS report in Section A1).  The cost of ensuring the quality of the requirement by means of a genuine RE approach is disproportionately low, compared to the benefits it confers.  Typically, it is less than 1% of the project whole life cost[3] – and this is able to be reduced even further (by at least an order of magnitude) if RE is employed to create generic requirements for use across system domains (e.g. Warships, Aircraft, Information Systems, etc) and if it is employed sufficiently early in the life of the project.

 

 

3.5    Applying RE to Existing Projects

The earlier RE is applied, the lower will be the overall cost of getting the requirement right and the greater will be the cumulative risk reduction throughout the project lifecycle.  That said, RE could be introduced at any stage into an existing project in order to identify and remove the un-quantified requirements-driven risk that are invariably present in all “non-engineered” project specifications.  Any resources consumed in doing this will be justified, by several orders of magnitude, as a result of the consequential performance improvements and time & cost savings.

 

 

 

3.6    Getting Started

 

A minimum risk starting point for anyone contemplating the use of Requirements Engineering, but who are nevertheless nervous about adopting what might be regarded as a new approach, would be to have a formal Subjectivity Audit.  There is always delivered value from such an audit and there is invariably a revelation in appreciation of the actual quality of the subject specification compared with the previously perceived quality.

 

From a client organisation point of view a formal audit of a preferred supplier’s tender can deliver more value than going out to competitive tender to a number of companies.  For one thing the process is considerably less expensive and, for another, the problem of trying to compare disparate approaches does not arise.  Formal subjectivity audits also benefit the manufacturer as well as the client.  For example when a manufacturer receives a specification, from a client for a proposed system, its arrival invariably triggers a rash of clarification questions.  A formal audit produces a complete set of such questions, in addition to its other benefits, and prepares the way for further development in a manner that is not possible by any means based on visual inspection of documents.

 

 

3.7    How RE Reduces Risk Throughout the Project Life Cycle

Table 1 shows the principal benefits to be gained at various stages of the project life cycle. Although the table follows the UK MoD’s “CADMID” cycle, the benefits can be read across to other cycles.

 


 

 

4        SUMMARY

The main points of this paper can be summarized as:

             The requirement provides the greatest source of project risk.

             The consequences of requirements-driven risk are unacceptable.  

             Procedural guidance, metrics and complexity management are necessary to minimize the risk.

             A good Requirements Engineering methodology will address all three of the above issues.  

             The cost of minimizing requirements-driven risk by means of a genuine RE methodology is disproportionately low, compared to the benefits it confers.

             It is never too late to start.

 

 

 

5        CONCLUSION

If major projects are ever to be delivered to time, cost and performance, a RE methodology, with the attributes described in Section 3.3 above, must be applied to all projects, regardless of their current stage, to obviate requirements-driven risk.

 

 

 

6        RECOMMENDATION

For those who have little or no experience of producing high quality specifications it needs to be recognized that, just as for any other engineering discipline, the development of a sound RE methodology is a non-trivial exercise.  It demands considerable and dedicated effort.  Such development cannot be expected to be undertaken as a part-time activity – even by the experienced.  It should not be expected that an effective methodology can be produced on the back of any single project.  In order to ensure that it will provide adequate guidance a RE methodology needs to have been developed by widely experienced, professional, requirements engineers and scientists - and it needs to have been well tried and tested before it is permitted to be used in anger.

 

It is strongly recommended that a common and proven RE methodology, with the attributes described in Section 3.3 above[4], together with appropriate consultancy and software support, is applied to all major enterprises, and that the metrics so derived be used as standard performance indicators in risk reduction.

 

-------------||-------------

 


 

 

Table:      PRINCIPAL BENEFITS OF APPLYING RE

 

 

Project Phase

 

Benefits

 

A good RE methodology assists the user to:

Concept

       Reduction in nugatory research & development.

         Early identification of capability gaps.

         Progressive reduction in cost, time & effort necessary to develop successive user requirements documents in application areas.

         Flexibility to adapt user requirements to new operational scenarios or technologies across linked systems.

         Early metrics for risk management across capability areas.

    manage complexity by structuring and linking information at all levels and across all systems;

    acquire and manage knowledge;

    identify generic, re-usable requirements within application domains;

    measure requirements quality.

Assessment

       Identification and reduction of requirements-driven risk in project approval submissions.

         Objective, defensible requirements for business cases.

         Progressive elimination of requirements-driven risk from the Through-Life Management Plan.

         Objective assurance for all stakeholders of the degree to which their concerns have been addressed.

         Assistance with the management of technology risk by avoidance of early solution lock-in.

    demonstrate objectively the correctness, completeness, coherence and consistency of the user requirement at any time & from any viewpoint;

    identify and manage constraints imposed by existing systems and infrastructure, the environment, legislation etc;

    derive and link system requirements from the user requirement to provide a solution-independent framework for the assessment of technical proposals.

Demonstration

       Optimization of performance, time and cost requirements.

         Reduced contracting risk for customer and supplier.

         System will work if built to specification.

    monitor and demonstrate the impact across all linked requirements of proposed changes;

    identify and remove ambiguity, incompleteness and inconsistency from the SOR;

    dynamically model the complete requirement against required scenarios before design commences.

Manufacture

       Right first time

         System architecture confidence

         Product data management & enterprise integration

         Minimize acceptance risks

    design the system against a SOR which has been demonstrated to be correct, complete, coherent & consistent;

    model the impact of proposed and existing equipment on the system before design commitment;

     use the solution-independent, structured requirement as a central reference for distributed design development activities;

    ensure the existence of  objective acceptance tests against every requirement, regardless of level (also essential for removing ambiguity);

In-Service

       Optimized availability within resource constraints.

                                 Reduced contracting risk for support.

                                 Identification and management of in-service capability shortfalls.

    identify the impact of support decisions on other aspects of the requirement such as availability, and vice-versa;

    maintain a current operational baseline against which in-service decisions and contracts should be based;

    model the impact of changes to the operational base-line requirement (eg change of use, new scenarios/threats) or system parameters (eg capability updates, changes to associated systems)

Disposal

       Identification and management of disposal risks

    Ensure that stakeholder viewpoints are engineered into the overall SOR in such a way that the impact on each viewpoint of changes to user or system requirements and constraints will be identified

 

 

-------------||-------------

 


ANNEX
REQUIREMENTS CATASTROPHIES

 

A1.             The Standish Group Report - CHAOS

 

This report is based on a study of US IT projects.  The US spends about $250B/year on IT application development for approximately 175,000 projects.

The report states that 31% of all projects are cancelled and 53% cost 189% of their original estimate.

Only 9% of projects are completed on time and to budget and many of these are a mere shadow of their original specification.

Projects completed by the largest American companies have only 42% of the originally proposed features and functions.  Smaller companies do much better.  Large companies overrun their time budget by an average of 230% and their cost budget by an average of 178%.  

The three main reasons that a project will succeed are given in the report as: User involvement, executive management support and a clear statement of requirements.

An opinion poll identified incomplete requirements as the main reason why projects are cancelled.

“The lost opportunity costs are not measurable, but could easily be in the trillions of dollars”!

 

AND THINGS ARE NOT GETTING BETTER:  A high percentage of executive managers (48%) were prepared to admit ‘there are more project failures now than five and ten years ago’.

 
A2. IEEE Study of 9 major US Government Software Contracts (Quoted in “Scientific Data Management”, Mar 98)

-       43% never delivered;

-       47% never used.

-       Of those delivered, 66% required significant modification.

 

Problems were caused by:

-       Poor/misunderstood User Requirements (i.e. building the wrong system);

-       Inability to cope with impact of change/new requirements;

-       Poor, or no user documentation, deriving from poorly expressed requirements.

 

A3.   UK Government IT Survey 1997 (Price Waterhouse)

“Projects go over budget and more money is simply poured in.  There is little thought about why this is happening, or whether more money is the correct thing.

Poor communications, vague specifications and inadequate project management were named as the causes, with 35% of projects being at serious risk.”

 

A4.   Ariane Flight 501 Failure – Board of Enquiry Report

“The failure of Ariane 501 was caused by the complete loss of guidance and attitude information 37 seconds after start of the main engine ignition system (30 seconds after lift-off).  This loss of information was due to specification and design errors in the software of the inertial reference system”.

 

A5.             “The Killing of VIXEN (50m)” –  Report from “The Engineer” 20 Feb 97

“VIXEN, an intelligence gathering system … was cancelled.  Siemens was paid the 50m, but the MOD wanted its money back.  Siemens refused because it considered the equipment completed to order.  The project appeared to have foundered on the rocks of rapidly changing international politics.  The UK’s Defence needed look different at the end of the project than in 1987 when the contract was awarded.  Much of the disagreement rested on how well Siemens was able to react to the army’s changing needs.

 

…..Siemens claimed it built in extra flexibility, and tests were approved in 1995 and 1996.  But the MOD disputed this, claiming: ‘tests conducted in 1995 and 1996 demonstrated that VIXEN had operational shortcomings….”

ISA comment:  If the VIXEN requirement had been properly engineered, then changes to the SOR to reflect “the Army’s changing needs” would have been recorded unambiguously, together with objective tests and acceptance criteria against each requirement.  This would have left no room for argument over the conclusions.  Instead, the only beneficiary of the dispute was the legal profession!

 

A6.  EDS and Airtours Head for Court (Reported  in “Computing” 25 January 2001)

“Services giant EDS is heading for a legal showdown with the holiday firm Airtours following the collapse of their 100 million outsourcing deal….”

 

‘Only a tiny proportion of these project disputes reach the courts,’ said Stuart Chapman, partner at technology law firm Pinset Curtis…… ‘The usual problem is that the customer and supplier have different perspectives at the procurement stage [of what is required].  Once they get down to the project, they try to reconcile these different views and that’s where the pain starts…….’

 

 

A7.   NAO Report “Accepting [Military] Equipment Off Contract And Into Service” 11 February 2000

“…in one third of the projects surveyed, the contract acceptance criteria did not fully reflect the staff requirement”.  

ISA Comment:  The International Standards Organization (ISO) defines a requirement as a statement of need, together with the tests(s) and criteria by which it will be demonstrated to have been met.  Without an objective acceptance test and criteria, no single requirement is complete.  Any RE methodology must ensure that such tests exist for every requirement in the SOR, regardless of level.

 

A8.   Home Office’s Case Record Management System (CRAMS) - 100m+ Project Cancelled due to:

-       Poor overall Leadership;

-       Inadequately identifies needs of individual probation services.

(Home Office Report quoted in Times, 02 Nov 00).

 

A9.   Future Surface Escort – Failed Initial Gate Sep 99

-       Caused by “Poor business case, derived from unclear requirement”.

(Quote from IPT Requirements Management Team, 21 Sep 00).

 

A10.           Railtrack Network Management Centre (NMC) – Report by Project Manager (Brown & Root)

In 1996 Railtrack commissioned a consulting systems house to undertake the requirements capture for the project and produce a User Requirements Specification.  The usual steps were gone through; interviews of users, workshops, feedback, peer reviews and finally user review of the specification.  The consultant was duly paid and we planned the phase which would translate the User Requirements into a specification of system requirements against which we could invite industry to tender.

 

That was when we discovered that no-one in the company really understood the User Requirements Specification inasmuch as it might be used to define a set of system requirements.  The consultant had gathered a lot of information, much of it necessarily anecdotal, analysed it, divided it up into a number of academically pure and unrecognisable processes and written it down in a notation that was entirely unintelligible to the people who, alone, could have validated it.  Six months of project programme had been used up in the process and this put a number of constraints on the ensuing phases.

 

What the client needs from the consultant who is going to undertake the requirements capture process, especially from end-users, is a demonstration of how simple it can be, not how clever and complex it all is.”

 

ISA Comment:  To ensure the correctness, completeness, coherence and consistency of a SOR, each requirement must be stripped to its “atomic” form and captured and manipulated by the requirements engineer in a structured database that is capable of managing high degrees of complexity in many dimensions.  Obviously, presenting the end-user with “screen-shots” from such a database will lose the contextual information and natural language presentation necessary for the end-user to understand the requirements statements. This is equally true of the less capable databases found in requirements management tools.  The G-MARC approach avoids this problem by making its requirements engineering database transparent to the end user.  At the stroke of a single key, the G-MARC operator can display the natural language passage(s) within the original (source) document(s), or original statements by contributors, which gave rise to the atomic requirement.  Hence, the end-user is presented with the atomic requirement together with all the associated contextual information in a language that he can readily understand and relate to.  

Furthermore, the structure revealed by the engineering of the User Requirement is used by G-MARC to capture System Requirements as they are identified (many are identified naturally during the development of the User Requirement – it would be foolish to throw them away, only to have to re-discover them later).  This means that, by the time the project is ready to “translate the User Requirement into a specification of System Requirements” the structural links between the various layers of concern (user, facilities, infrastructure, equipment etc) are already known.  This both speeds up the development of the SRD and obviates the risk that the System Requirements, which are often developed by different people,  will not derive directly and explicitly from the User Requirement (see examples 6. & 7. above).

 

 

A11.  NAO Major Projects Report 2000

The following is an extract from the National Audit Office’s Major Projects Report dated 22 November 2000:

“2.5.  The Defence Procurement Agency has also developed and approved a performance indicator that is the percentage of the total procurement cost falling before Main Gate.  In the longer term, the Defence Procurement Agency plans to monitor the trend in the performance indicator over time to place emphasis on spending more in the earlier stages of projects …….”

“2.17.   The Strategic Defence Review suggested that up to 15 per cent of the total procurement cost of a project be spent before Main Gate”.  Paragraph 2.17 then goes on to say that this is not a hard and fast target, owing to variations in procurement approach.  Nevertheless, paragraph 2.18 states:   

“2.18.  ..…The ten pre-Main Gate projects in Major Projects Report 2000 included five collaborative programmes and one off-the-shelf buy.  Approved assessment phase cost was never more than six per cent on any of the ten projects, although two of the projects, BOWMAN and MLS were forecast at 31 March 2000 to spend 15 per cent and 19 per cent respectively of their total procurement costs on assessment work.  Nor is there any indication that more recently approved projects which have yet to pass Main Gate are forecast to spend a higher proportion during the assessment phase.”

 

ISA Comment: applying RE to projects as early as possible will shift the balance of expenditure forward by significantly de-risking later phases and greatly reduce the overall project cost.  Since it rarely represents more than 1% of the overall project cost, the arguments for applying RE are overwhelming.

 

-------------||-------------

 

   

 



[1] complete = no missing requirements.  Coherent = statements grouped by level of detail.  Consistent = no contradictions.  Correct = accurate, atomic, objective statements.  

[2] These definitions are consistent with those in the MOD’s Smart Procurement/Acquisition Management System Document “Guidance to the Writing of User Requirements Documents”.

[3] The Strategic Defence Review recommended that  up to 15 per cent of the total procurement cost of a project be spent before Main Gate - MOD currently averages 6% (NAO Major Projects Report 2000 refers – see Annex A).

[4] There is now a MOD-proven and commercially available methodology & tool set (G-MARC) that meets these criteria.  G-MARC is supported by highly qualified and experienced requirements engineers at ISA.  By addressing the quality of the SOR in a logical and repeatable way and providing a full range of supporting metrics, the G-MARC methodology facilitates the minimization of requirements-driven risk to any enterprise.