Show simple item record

dc.contributor.authorKitchenham, Barbaraen_NZ
dc.contributor.authorMacDonell, Stephenen_NZ
dc.contributor.authorPickard, Lesleyen_NZ
dc.contributor.authorShepperd, Martinen_NZ
dc.date.available2011-04-07T03:06:06Z
dc.date.copyright1999-06en_NZ
dc.identifier.citationKitchenham, B., MacDonell, S., Pickard, L., & Shepperd, M. (1999). Assessing prediction systems (Information Science Discussion Papers Series No. 99/14). University of Otago. Retrieved from http://hdl.handle.net/10523/1015en
dc.identifier.urihttp://hdl.handle.net/10523/1015
dc.description.abstractFor some years software engineers have been attempting to develop useful prediction systems to estimate such attributes as the effort to develop a piece of software and the likely number of defects. Typically, prediction systems are proposed and then subjected to empirical evaluation. Claims are then made with regard to the quality of the prediction systems. A wide variety of prediction quality indicators have been suggested in the literature. Unfortunately, we believe that a somewhat confusing state of affairs prevails and that this impedes research progress. This paper aims to provide the research community with a better understanding of the meaning of, and relationship between, these indicators. We critically review twelve different approaches by considering them as descriptors of the residual variable. We demonstrate that the two most popular indicators MMRE and pred(25) are in fact indicators of the spread and shape respectively of prediction accuracy where prediction accuracy is the ratio of estimate to actual (or actual to estimate). Next we highlight the impact of the choice of indicator by comparing three prediction systems derived using four different simulated datasets. We demonstrate that the results of such a comparison depend upon the choice of indicator, the analysis technique, and the nature of the dataset used to derive the predictive model. We conclude that prediction systems cannot be characterised by a single summary statistic. We suggest that we need indicators of the central tendency and spread of accuracy as well as indicators of shape and bias. For this reason, boxplots of relative error or residuals are useful alternatives to simple summary metrics.en_NZ
dc.format.mimetypeapplication/pdf
dc.publisherUniversity of Otagoen_NZ
dc.relation.ispartofseriesInformation Science Discussion Papers Seriesen_NZ
dc.subjectprediction systemsen_NZ
dc.subjectempirical analysisen_NZ
dc.subjectmetricsen_NZ
dc.subjectgoodness-of-fit statisticsen_NZ
dc.subjectEstimationen_NZ
dc.subject.lcshQA76 Computer softwareen_NZ
dc.titleAssessing prediction systemsen_NZ
dc.typeDiscussion Paperen_NZ
dc.description.versionUnpublisheden_NZ
otago.bitstream.pages25en_NZ
otago.date.accession2010-11-10 20:30:21en_NZ
otago.schoolInformation Scienceen_NZ
otago.openaccessOpen
otago.place.publicationDunedin, New Zealanden_NZ
dc.identifier.eprints998en_NZ
otago.school.eprintsSoftware Metrics Research Laboratoryen_NZ
otago.school.eprintsInformation Scienceen_NZ
dc.description.referencesBelady, L.A. and Lehman, M.M. (1979) The characteristics of large systems. In Research Directions in Software Technology. P.Wegner (ed..), Cambridge , MA, MIT Press, p106-138. Chulani, S, Clark, B. Boehm, B.W. (1998), "Calibrating the COCOMO II Post Architecture Model," 20th International Conference on Software Engineering, Conte, S.D., Dunsmore, H.E., and Shen, V.Y.(1986). Software Engineering Metrics and Models, Benjamin/Cummings, Menlo Park CA. Iglewicz, B. (1983) Robust scale estimators and confidence intervals for location. In Understanding Robust and exploratory data analysis. Hoaglin, D.C., Mosteller, F. and Tuket, J.W. (eds), John Wiley & sons Inc. Jorgensen, M., (1995). Experience with the accuracy of software maintenance task effort prediction models, IEEE Trans. Soft. Eng. 21, 674-681. Kemerer, C.F., (1987). An empirical validation of software cost estimation models, CACM, 30(5), pp416-429. Kitchenham, B.A., (1992). Empirical studies of assumptions that underlie software cost estimation models, Information & Softw. Technol., 34(4), pp211-218. Kitchenham, B. A., and Linkman, S. G. (1997). Estimates, uncertainty and risk, IEEE Software, 14(3), 69-74. Kok, P., B.A. Kitchenham, and J. Kirakowski. (1990). The MERMAID approach to software cost estimation, in Proc. Esprit Technical Week, 1990. Lo, B.W.N., and Gao, X. (1997). Assessing software cost estimation models: criteria for accuracy, consistency and regression. Australian J. of Information Systems, 5(1), 30-44. MacDonell, S.G., Shepperd, M.J., and Sallis, P.J. (1997). Metrics for Database Systems: An Empirical Study. In 4th IEEE Intl. Metrics Symp., Alberqueque, NM Miyazaki, Y., Takanou, A., Nozaki, H., Nakagawa, N., and Okada, K, (1991). Method to estimate parameter values in software prediction models, Information & Softw. Technol., 33(3), 239-243 Miyazaki, Y., et al., (1994). Robust Regression for Developing Software Estimation Models, J. of Systems & Software, pp3-16,. Pickard, L.M., Kitchenham, B.A. and Linkman, S.J. (1999) Investigation of Analysis Techniques for Software datasets. Dept. Computer Science. Keele University, TR99-05.. Rousseeuw, P.J., and Leroy, A.M., (1987). Robust Regression and Outlier Detection, John Wiley & Sons, New York. Stensrud, E. and I. Myrveit. (1998) “Human performance estimating with analogy and regression models: An empirical validation, Proceedings of the Fifth International Software Metrics Symposium IEEEComputerSociety Press. Shepperd, M.J. and C. Schofield, (1997). Estimating software project effort using analogies, IEEE Trans. on Softw. Eng., 23(11), pp736-743, 1997.en_NZ
otago.relation.number99/14en_NZ
 Find in your library

Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record