The Rise and Fall of the Chaos Report Figures

This weekend I got an email from X (Chris Verhoef). He noticed I used the research of the Chaos Report and his research in a blog post. Chris and J. Laurenz Eveleens wrote an article in IEEE Software (January and February issue) on the Chaos Report. They stated; Although the Standish Group‚The Chaos reports are often used to indicate problems in application software development project management, the reports contain major flaws. They concluded:

1. Misleading definitions: They‚use misleading because they are onlybased on estimation accuracy of cost, time, and functionality. But Standish labels projects as successful or challenged, suggesting much more than deviations from their original estimates.
2. Unrealistic Rates: The next issue is whether the Standish estimation accuracy definitions are sound. They are not. The Standish Group‚use measures are one-sided because they neglect underruns for cost and time and overruns for the amount of functionality.
3. Perverting Accuracy: The third problem is that steering on the Standish definitions causes large cost and time overestimations (and large functionality underestimations), which perverts rather than improves estimation accuracy.

Chris applied the definitions to data from projects from four organizations. Read the article for the stunning results. From an Agile perspective it is difficult to define the success of a project on the original estimation of the functionality, time and cost. The scope of the functionality will change. That’s the point of Agile development. So are all Agile projects challenged? Isn’t the real measure: the business value? Not long ago a witnessed the celebration of finishing of release. A video was shown of the landing on the moon. This project was a success. A few weeks later I got a demo of the application…… Success is in the eyes off the beholder.”