|||"Empirical Studies on Software Product Maintainability Prediction: A Systematic Mapping and Review", In e-Informatica Software Engineering Journal, vol. 13, no. 1, pp. 141–202, 2019.
DOI: , 10.5277/e-Inf190105.|
Get article (PDF)View article entry (BibTeX)
Sara Elmidaoui, Laila Cheikhi, Ali Idri, Alain Abran
Background: Software product maintainability prediction (SPMP) is an important task to control software maintenance activity, and many SPMP techniques for improving software maintainability have been proposed. In this study, we performed a systematic mapping and review on SPMP studies to analyze and summarize the empirical evidence on the prediction accuracy of SPMP techniques in current research.
Objective: The objective of this study is twofold: (1) to classify SPMP studies reported in the literature using the following criteria: publication year, publication source, research type, empirical approach, software application type, datasets, independent variables used as predictors, dependent variables (e.g. how maintainability is expressed in terms of the variable to be predicted), tools used to gather the predictors, the successful predictors and SPMP techniques, (2) to analyze these studies from three perspectives: prediction accuracy, techniques reported to be superior in comparative studies and accuracy comparison of these techniques.
Methodology: We performed a systematic mapping and review of the SPMP empirical studies published from 2000 up to 2018 based on an automated search of nine electronic databases.
Results: We identified 82 primary studies and classified them according to the above criteria. The mapping study revealed that most studies were solution proposals using a history-based empirical evaluation approach, the datasets most used were historical using object-oriented software applications, maintainability in terms of the independent variable to be predicted was most frequently expressed in terms of the number of changes made to the source code, maintainability predictors most used were those provided by Chidamber and Kemerer (C&K), Li and Henry (L&H) and source code size measures, while the most used techniques were ML techniques, in particular artificial neural networks. Detailed analysis revealed that fuzzy & neuro fuzzy (FNF), artificial neural network (ANN) showed good prediction for the change topic, while multilayer perceptron (MLP), support vector machine (SVM), and group method of data handling (GMDH) techniques presented greater accuracy prediction in comparative studies. Based on our findings SPMP is still limited. Developing more accurate techniques may facilitate their use in industry and well-formed, generalizable results be obtained. We also provide guidelines for improving the maintainability of software.
systematic mapping study, systematic literature review, software product maintainability, empirical studies
 A. Kaur and K. Kaur, “Statistical comparison of modelling methods for software maintainability prediction,” International Journal of Software Engineering and Knowledge Engineering, Vol. 23, No. 06, 2013, pp. 743–774.
 M. Riaz, E. Mendes, and E. Tempero, “A systematic review of software maintainability prediction and metrics,” in Proceedings of the 3rd International Symposium on Empirical Software Engineering and Measurement. IEEE Computer Society, 2009, pp. 367–377.
 M. Riaz, “Maintainability prediction of relational database-driven applications: a systematic review,” in 16th International Conference on Evaluation & Assessment in Software Engineering. IET, 2012, pp. 263–272.
 B.A. Orenyi, S. Basri, and L.T. Jung, “Object-oriented software maintainability measurement in the past decade,” in International Conference on Advanced Computer Science Applications and Technologies (ACSAT). IEEE, 2012, pp. 257–262.
 A.M. Fernández-Sáez, M. Genero, and M.R. Chaudron, “Empirical studies concerning the maintenance of UML diagrams and their use in the maintenance of code: A systematic mapping study,” Information and Software Technology, Vol. 55, No. 7, 2013, pp. 1119–1142.
 A.M. Fernández-Sáez, D. Caivano, M. Genero, and M.R. Chaudron, “On the use of UML documentation in software maintenance: Results from a survey in industry,” in 18th International Conference on Model Driven Engineering Languages and Systems (MODELS). IEEE, 2015, pp. 292–301.
 K. Petersen, S. Vakkalanka, and L. Kuzniarz, “Guidelines for conducting systematic mapping studies in software engineering: An update,” Information and Software Technology, Vol. 64, 2015, pp. 1–18.
 B.A. Kitchenham, D. Budgen, and O.P. Brereton, “Using mapping studies as the basis for further research – A participant-observer case study,” Information and Software Technology, Vol. 53, No. 6, 2011, pp. 638–651.
 A. Kaur, K. Kaur, and K. Pathak, “Software maintainability prediction by data mining of software code metrics,” in International Conference on Data Mining and Intelligent Computing (ICDMIC). IEEE, 2014, pp. 1–6.
 P. Omidyar, eBay web services. [Online]. http://developer.ebay.com [accessed: 2018-10-26].
 A. Jain, S. Tarwani, and A. Chug, “An empirical investigation of evolutionary algorithm for software maintainability prediction,” in Students’ Conference on Electrical, Electronics and Computer Science (SCEECS), 2016, pp. 1–6.
 X. Jin, Y. Liu, J. Ren, A. Xu, and R. Bie, “Locality preserving projection on source code metrics for improved software maintainability,” in Australasian Joint Conference on Artificial Intelligence. Springer, 2006, pp. 877–886.
 K. Beck and E. Gamma, JUnit. [Online]. https://junit.org [accessed: 2018-10-26].
 R. Malhotra and A. Chug, “Application of group method of data handling model for software maintainability prediction using object oriented systems,” International Journal of System Assurance Engineering and Management, Vol. 5, No. 2, 2014, pp. 165–173.
 S.J. Sayyad and T. Menzies, The PROMISE Repository of Software Engineering Databases, School of Information Technology and Engineering, University of Ottawa, Canada. [Online]. http://promise.site.uottawa.ca/SERepository Accessed: 2017-09-11.
 D. Spinellis, Chidamber and Kemerer Java Metrics (CKJM). [Online]. http://www.spinellis.gr/sw/ckjm/ [accessed: 2017-11-21].
 Jetbrains Homepage. [Online]. http://www.jetbrains.com/idea/ [accessed: 2017-11-19].
 Krakatau Professional Homepage. [Online]. http://www.powersoftware.com/kp/ [accessed: 2017-11-19].
 R. Ferenc, A. Beszedes, M. Tarkiainen, and T. Gyimothy, “Columbus – reverse engineering tool and schema for C++,” in Proceedings of the International Conference on Software Maintenance (ICSM’02), ICSM ’02. Washington, DC, USA: IEEE Computer Society, 2002, pp. 172–181.
 G.A. Di Lucca, A.R. Fasolino, F. Pace, P. Tramontana, and U. De Carlini, “WARE: a tool for the reverse engineering of web applications,” in Proceedings of the Sixth European Conference on Software Maintenance and Reengineering, 2002, pp. 241–250.
 B.R. Reddy, S. Khurana, and A. Ojha, “Software maintainability estimation made easy: A comprehensive tool coin,” in Proceedings of the Sixth International Conference on Computer and Communication Technology 2015. New York: ACM, 2015, pp. 68–72.
 Analyst4j standard tool. [Online]. https://codeswat.com/ [accessed: 2018-01-01].
 J. Wen, S. Li, Z. Lin, Y. Hu, and C. Huang, “Systematic literature review of machine learning based software development effort estimation models,” Information and Software Technology, Vol. 54, No. 1, 2012, pp. 41–59.
 P. Brereton, B.A. Kitchenham, D. Budgen, M. Turner, and M. Khalil, “Lessons from applying the systematic literature review process within the software engineering domain,” Journal of Systems and Software, Vol. 80, No. 4, 2007, pp. 571–583.
 S. Muthanna, K. Kontogiannis, K. Ponnambalam, and B. Stacey, “A maintainability model for industrial software systems using design level metrics,” in Proceedings Seventh Working Conference on Reverse Engineering. IEEE, 2000, pp. 248–256.
 M. Thwin and T. Quah, “Application of neural networks for estimating software maintainability using object-oriented metrics,” in International Conference on Software Engineering and Knowledge Engineering, 2003, pp. 69–73.
 M. Genero, M. Piattini, E. Manso, and G. Cantone, “Building UML class diagram maintainability prediction models based on early metrics,” in 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry. IEEE, 2003, pp. 263–275.
 M. Kiewkanya, N. Jindasawat, and P. Muenchaisri, “A methodology for constructing maintainability model of object-oriented design,” in Fourth International Conference on Quality Software. IEEE, 2004, pp. 206–213.
 G.A. Di Lucca, A.R. Fasolino, P. Tramontana, and C.A. Visaggio, “Towards the definition of a maintainability model for web applications,” in Eighth European Conference on Software Maintenance and Reengineering. IEEE, 2004, pp. 279–287.
 J.H. Hayes and L. Zhao, “Maintainability prediction: A regression analysis of measures of evolving systems,” in 21st International Conference on Software Maintenance (ICSM’05). IEEE, 2005, pp. 601–604.
 S.S. Dahiya, J.K. Chhabra, and S. Kumar, “Use of genetic algorithm for software maintainability metrics’ conditioning,” in 15th International Conference on Advanced Computing and Communications. IEEE, 2007, pp. 87–92.
 M. Genero, E. Manso, A. Visaggio, G. Canfora, and M. Piattini, “Building measure-based prediction models for UML class diagram maintainability,” Empirical Software Engineering, Vol. 12, No. 5, 2007, pp. 517–549.
 K. Shibata, K. Rinsaka, T. Dohi, and H. Okamura, “Quantifying software maintainability based on a fault-detection/correction model,” in 13th Pacific Rim International Symposium on Dependable Computing. IEEE, 2007, pp. 35–42.
 K. Aggarwal, Y. Singh, A. Kaur, and R. Malhotra, “Application of artificial neural network for predicting maintainability using object-oriented metrics,” World Academy of Science, Engineering and Technology, International Journal of Computer, Electrical, Automation, Control and Information Engineering, Vol. 2, No. 10, 2008, pp. 3552–3556.
 Y. Tian, C. Chen, and C. Zhang, “AODE for source code metrics for improved software maintainability,” in Fourth International Conference on Semantics, Knowledge and Grid. IEEE, 2008, pp. 330–335.
 H. Yu, G. Peng, and W. Liu, “An application of case based reasoning to predict structure maintainability,” in International Conference on Computational Intelligence and Software Engineering. IEEE, 2009, pp. 1–5.
 A. Sharma, P. Grover, and R. Kumar, “Predicting maintainability of component-based systems by using fuzzy logic,” in International Conference on Contemporary Computing. Springer, 2009, pp. 581–591.
 L. Wang, X. Hu, Z. Ning, and W. Ke, “Predicting object-oriented software maintainability using projection pursuit regression,” in First International Conference on Information Science and Engineering. IEEE, 2009, pp. 3827–3830.
 M.O. Elish and K.O. Elish, “Application of treenet in predicting object-oriented software maintainability: A comparative study,” in 13th European Conference on Software Maintenance and Reengineering. IEEE, 2009, pp. 69–78.
 C. Jin and J.A. Liu, “Applications of support vector mathine and unsupervised learning for predicting maintainability using object-oriented metrics,” in Second International Conference on Multimedia and Information Technology, Vol. 1. IEEE, 2010, pp. 24–27.
 L. Cai, Z. Liu, J. Zhang, W. Tong, and G. Yang, “Evaluating software maintainability using fuzzy entropy theory,” in 9th International Conference on Computer and Information Science. IEEE, 2010, pp. 737–742.
 S.O. Olatunji, Z. Rasheed, K. Sattar, A. Al-Mana, M. Alshayeb, and E. El-Sebakhy, “Extreme learning machine as maintainability prediction model for object-oriented software systems,” Journal of Computing, Vol. 2, No. 8, 2010, pp. 49–56.
 S.K. Dubey, A. Rana, and Y. Dash, “Maintainability prediction of object-oriented software system by multilayer perceptron model,” ACM SIGSOFT Software Engineering Notes, Vol. 37, No. 5, 2012, pp. 1–4.
 T. Bakota, P. Hegedűs, G. Ladányi, P. Körtvélyesi, R. Ferenc, and T. Gyimóthy, “A cost model based on software maintainability,” in 28th International Conference on Software Maintenance (ICSM). IEEE, 2012, pp. 316–325.
 Y. Dash, S.K. Dubey, and A. Rana, “Maintainability prediction of object oriented software system by using artificial neural network approach,” International Journal of Soft Computing and Engineering (IJSCE), Vol. 2, No. 2, 2012, pp. 420–423.
 P. Hegedűs, G. Ladányi, I. Siket, and R. Ferenc, “Towards building method level maintainability models based on expert evaluations,” in Computer Applications for Software Engineering, Disaster Recovery, and Business Continuity. Springer, 2012, pp. 146–154.
 D. Chandra, “Support vector approach by using radial kernel function for prediction of software maintenance effort on the basis of multivariate approach,” International Journal of Computer Applications, Vol. 51, No. 4, 2012.
 H. Aljamaan, M.O. Elish, and I. Ahmad, “An ensemble of computational intelligence models for software maintenance effort prediction,” in International Work-Conference on Artificial Neural Networks. Springer, 2013, pp. 592–603.
 P. Hegedűs, T. Bakota, G. Ladányi, C. Faragó, and R. Ferenc, “A drill-down approach for measuring maintainability at source code element level,” Electronic Communications of the EASST, Vol. 60, 2013.
 X.L. Hao, X.D. Zhu, and L. Liu, “Research on software maintainability evaluation based on fuzzy integral,” in International Conference on Quality, Reliability, Risk, Maintenance, and Safety Engineering (QR2MSE). IEEE, 2013, pp. 1279–1282.
 F. Ye, X. Zhu, and Y. Wang, “A new software maintainability evaluation model based on multiple classifiers combination,” in International Conference on Quality, Reliability, Risk, Maintenance, and Safety Engineering (QR2MSE). IEEE, 2013, pp. 1588–1591.
 S.O. Olatunji and A. Ajasin, “Sensitivity-based linear learning method and extreme learning machines compared for software maintainability prediction of object-oriented software systems,” ICTACT Journal On Soft Computing, Vol. 3, No. 03, 2013.
 A. Mehra and S.K. Dubey, “Maintainability evaluation of object-oriented software system using clustering techniques,” Internationa Journal of Computers and Technology, Vol. 5, No. 02, 2013, pp. 136–143.
 A. Kaur, K. Kaur, and K. Pathak, “A proposed new model for maintainability index of open source software,” in Proceedings of 3rd International Conference on Reliability, Infocom Technologies and Optimization. IEEE, 2014, pp. 1–6.
 A. Pratap, R. Chaudhary, and K. Yadav, “Estimation of software maintainability using fuzzy logic technique,” in International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT). IEEE, 2014, pp. 486–492.
 L. Geeta, A. Kavita, and B. Rizwan, “Maintainability measurement model for object oriented design,” International Journal of Advanced Research in Computer Science and Software Engineering, Vol. 4, No. 11, 2014, pp. 945–956.
 R. Malhotra and A. Chug, “A metric suite for predicting software maintainability in data intensive applications,” in Transactions on Engineering Technologies. Springer Netherlands, 2014, pp. 161–175.
 S. Misra and F. Egoeze, “Framework for maintainability measurement of web application for efficient knowledge-sharing on campus intranet,” in Computational Science and Its Applications – ICCSA 2014. Cham: Springer International Publishing, 2014, pp. 649–662.
 L. Kumar and S.K. Rath, “Neuro-genetic approach for predicting maintainability using Chidamber and Kemerer software metrics suite,” in Recent Advances in Information and Communication Technology 2015. Cham: Springer International Publishing, 2015, pp. 31–40.
 S.O. Olatunji and A. Selamat, “Type-2 fuzzy logic based prediction model of object oriented software maintainability,” in Intelligent Software Methodologies, Tools and Techniques. Cham: Springer International Publishing, 2015, pp. 329–342.
 L. Kumar and S.K. Rath, “Hybrid functional link artificial neural network approach for predicting maintainability of object-oriented software,” Journal of Systems and Software, Vol. 121, No. C, 2016, pp. 170–190.
 A. Chug and R. Malhotra, “Benchmarking framework for maintainability prediction of open source software using object oriented metrics,” International Journal of Innovative Computing, Information and Control, Vol. 12, No. 2, 2016, pp. 615–634.
 L. Kumar, K. Mukesh, and K.R. Santanu, “Maintainability prediction of web service using support vector machine with various kernel methods,” International Journal of System Assurance Engineering and Management, Vol. 8, No. 2, 2017, pp. 205–6222.
 S. Tarwani and A. Chug, “Predicting maintainability of open source software using gene expression programming and bad smells,” in 5th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), 2016, pp. 452–459.
 S. Tarwani and A. Chug, “Sequencing of refactoring techniques by greedy algorithm for maximizing maintainability,” in International Conference on Advances in Computing, Communications and Informatics (ICACCI), 2016, pp. 1397–1403.
 L. Kumar, S.K. Rath, and A. Sureka, “Empirical analysis on effectiveness of source code metrics for predicting change-proneness,” in Proceedings of the 10th Innovations in Software Engineering Conference, ISEC ’17. New York, NY, USA: ACM, 2017, pp. 4–14.
 G. Kanika and C. Anuradha, “Evaluation of instance-based feature subset selection algorithm for maintainability prediction,” in International Conference on Advances in Computing, Communications and Informatics (ICACCI), 2017, pp. 1482–1487.
 K. Lov and K.R. Santanu, “Software maintainability prediction using hybrid neural network and fuzzy logic approach with parallel computing concept,” International Journal of System Assurance Engineering and Management, Vol. 8, No. S2, 2017, pp. 1487–1502.
 L. Kumar, A. Krishna, and S.K. Rath, “The impact of feature selection on maintainability prediction of service-oriented applications,” Service Oriented Computing and Applications, Vol. 11, No. 2, 2017, pp. 137–161.
 L. Kumar, S.K. Rath, and A. Sureka, “Using source code metrics and multivariate adaptive regression splines to predict maintainability of service oriented software,” in 18th International Symposium on High Assurance Systems Engineering (HASE), 2017, pp. 88–95.
 L. Kumar, S.K. Rath, and A. Sureka, “Using source code metrics to predict change-prone web services: A case-study on ebay services,” in IEEE Workshop on Machine Learning Techniques for Software Quality Evaluation – MaLTeSQuE. IEEE, 2017, pp. 1–7.
 R. Malhotra and R. Jangra, “Prediction and assessment of change prone classes using statistical and machine learning techniques,” Journal of Information Processing Systems, Vol. 13, No. 4, 2017, pp. 778–804.
 G. Szőke, G. Antal, C. Nagy, R. Ferenc, and T. Gyimóthy, “Empirical study on refactoring large-scale industrial systems and its effects on maintainability,” Journal of Systems and Software, Vol. 129, 2017, pp. 107–126.
 Y. Gokul and M. Gopal, “An authoritative method using fuzzy logic to evaluate maintainability index and utilizability of software,” Advances in Modelling and Analysis B, Vol. 60, No. 3, 2017, pp. 566–580.
 P. Hegedűs, I. Kádár, R. Ferenc, and T. Gyimóthy, “Empirical evaluation of software maintainability based on a manually validated refactoring dataset,” Information and Software Technology, Vol. 95, No. 1, 2018, pp. 313–327.
 L. Kumar and S. Ashish, “A comparative study of different source code metrics and machine learning algorithms for predicting change proneness of object oriented systems,” arXiv preprint arXiv:1712.07944, 2018.
 G. Scanniello, C. Gravino, M. Genero, J.A. Cruz-Lemus, and G. Tortora, “On the impact of UML analysis models on source-code comprehensibility and modifiability,” ACM Trans. Softw. Eng. Methodol., Vol. 23, No. 2, 2014, pp. 13:1–13:26.
 A.M. Fernández-Sáez, M.R.V. Chaudron, M. Genero, and I. Ramos, “Are forward designed or reverse-engineered UML diagrams more helpful for code maintenance?: A controlled experiment,” in Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering. New York, NY, USA: ACM, 2013, pp. 60–71.
 G. Scanniello, C. Gravino, G. Tortora, M. Genero, M. Risi, J.A. Cruz-Lemus, and G. Dodero, “Studying the effect of UML-based models on source-code comprehensibility: Results from a long-term investigation,” in Proceedings of the 16th International Conference on Product-Focused Software Process Improvement, Vol. 9459, New York, 2015, pp. 311–327.
 A.M. Fernández-Sáez, M. Genero, D. Caivano, and M.R. Chaudron, “Does the level of detail of UML diagrams affect the maintainability of source code?: A family of experiments,” Empirical Software Engineering, Vol. 21, No. 1, 2016, pp. 212–259.
 G. Scanniello, C. Gravino, M. Genero, J.A. Cruz-Lemus, G. Tortora, M. Risi, and G. Dodero, “Do software models based on the UML aid in source-code comprehensibility? Aggregating evidence from 12 controlled experiments,” Empirical Software Engineering, Vol. 23, No. 5, 2018, pp. 2695–2733.