| 2026 | |
| [1] | , "From Draft to Standard: An Expert-Validated Catalog of API Gateway Software Metrics", In e-Informatica Software Engineering Journal, vol. 20, no. 1, pp. 260106, 2026.
DOI: 10.37190/e-Inf260106. Download article (PDF)Get article BibTeX file |
Authors
Eder dos Santos, Sandra Casas
Abstract
Context: In the fast-growing API Economy scenario, API Gateways became an essential tool for microservice reliability and scalability, yet lack standardized quality models and measures to assess these tools in an objective fashion.
Objective : This study quantitatively and qualitatively assessed the accuracy and relevance of a set of 59 metrics, gathered from prior analysis of 68 mainstream API Gateway software offerings.
Method: An expert judgment evaluation study was conducted with seven domain specialists (N=7) from both academia and industry, who assessed each metric’s accuracy and relevance using 5-point Likert scales. A multi-statistical method was performed to assess ratings, inter-observer consistency and reliability.
Results: Although ratings were generally neutral to positive, statistical analysis revealed low consensus across experts. A subset of 24 core metrics with strong agreement and high scores was identified, alongside moderate- and low-agreement sets. These items cover key API management concerns, including traffic monitoring, error handling, latency, and resource usage.
Conclusions: By combining practical definitions, expert judgment, and a tiered classification mechanism, this work established a broad and comprehensive set of metrics that will help to bridge the gap between metric adoption in real-world platforms and their theoretical grounding in software quality models.
Keywords
Software quality, Software measurement and metrics, Internet software systems development, Empirical and experimental studies in software engineering
References
1. S. Preibisch, API development: a practical guide for business implementation success . New York, NY: Springer Science+Business Media, 2018.
2. M. Waseem, P. Liang, M. Shahin, A. Di Salle, and G. Márquez, “Design, monitoring, and testing of microservices systems: The practitioners’ perspective,” arXiv preprint arXiv:2108.03384 , 2021.
3. E. dos Santos and S. Casas, “API management and SQuaRE: A comprehensive overview from the practitioners’ standpoint,” in Argentine Congress of Computer Science . Springer, 2023, pp. 137–150.
4. D. Bermbach and E. Wittern, “Benchmarking web API quality – revisited,” Journal of Web Engineering , Vol. 19, No. 5-6, 2020, pp. 603–646.
5. B. De, API Management: An Architect’s Guide to Developing and Managing APIs for Your Organization . Apress Berkeley, CA, 2023.
6. E. dos Santos and S. Casas, “Unveiling quality in API management: A systematic mapping study,” in 2024 L Latin American Computer Conference (CLEI) . IEEE, 2024, pp. 1–10.
7. B. Iyer and M. Subramaniam, “The strategic value of APIs,” Harvard Business Review , Vol. 1, No. 7, 2015, p. 2015.
8. ISO/IEC 25010:2011 – Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — System and software quality models , International Organization for Standardization; International Electrotechnical Commission Std., Mar. 2011, withdrawn Mar 4 2024; superseded by ISO/IEC 25010:2023, ISO/IEC 25002:2024, ISO/IEC 25019:2023.
9. E. dos Santos and S. Casas, “A catalog of API gateway metrics and its quantitative evaluation,” DYNA , Vol. 92, No. 237, 2025, pp. 106–114.
10. K.J. Stol and B. Fitzgerald, “The abc of software engineering research,” ACM Trans. Softw. Eng. Methodol. , Vol. 27, No. 3, Sep. 2018. [Online]. https://doi.org/10.1145/3241743
11. S.L. Pfleeger and B.A. Kitchenham, “Principles of survey research: part 1: turning lemons into lemonade,” ACM SIGSOFT Software Engineering Notes , Vol. 26, No. 6, 2001, pp. 16–18.
12. B. Kitchenham and S.L. Pfleeger, “Principles of survey research part 4: questionnaire evaluation,” ACM SIGSOFT Software Engineering Notes , Vol. 27, No. 3, 2002, pp. 20–23.
13. ISO/IEC 25010:2023 – Systems and Software Engineering – Systems and Software Quality Requirements and Evaluation (SQuaRE) – Product Quality Model , International Organization for Standardization; International Electrotechnical Commission Std., 2023, iSO/IEC 25010:2023.
14. Y. Duan, G. Fu, N. Zhou, X. Sun, N.C. Narendra et al., “Everything as a service (XaaS) on the cloud: Origins, current and future trends,” in 2015 IEEE 8th International Conference on Cloud Computing . IEEE, pp. 621–628. [Online]. http://ieeexplore.ieee.org/document/7214098/
15. C. Fehling, F. Leymann, R. Retter, W. Schupeck, and P. Arbitter, Cloud Computing Patterns: Fundamentals to Design, Build, and Manage Cloud Applications . Springer Vienna. [Online]. https://link.springer.com/10.1007/978-3-7091-1568-8
16. R.T. Fielding, “Architectural styles and the design of network-based software architectures.”
17. S. Andreo and J. Bosch, “API management challenges in ecosystems,” in Software Business , S. Hyrynsalmi, M. Suoranta, A. Nguyen-Duc, P. Tyrväinen, and P. Abrahamsson, Eds. Springer International Publishing, Vol. 370, pp. 86–93, series Title: Lecture Notes in Business Information Processing. [Online]. http://link.springer.com/10.1007/978-3-030-33742-1 _8
18. A. Brown, J. Fishenden, and M. Thompson, API Economy, Ecosystems and Engagement Models . Palgrave Macmillan UK, pp. 225–236. [Online]. http://link.springer.com/10.1057/9781137443649 _13
19. J. Bloch, “How to design a good API and why it matters,” in Companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applications . ACM, pp. 506–507. [Online]. https://dl.acm.org/doi/10.1145/1176617.1176622
20. A. Gamez-Diaz, P. Fernandez, and A. Ruiz-Cortés, “Governify for APIs: SLA-driven ecosystem for API governance,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ESEC/FSE 2019. Association for Computing Machinery, pp. 1120–1123, event-place: Tallinn, Estonia. [Online]. https://doi.org/10.1145/3338906.3341176
21. M. Medjaoui, E. Wilde, R. Mitra, and M. Amundsen, Continuous API management . ” O’Reilly Media, Inc.”, 2021.
22. M. Mathijssen, M. Overeem, and S. Jansen, “Identification of practices and capabilities in API management: a systematic literature review,” arXiv preprint arXiv:2006.10481 , 2020.
23. ISO 9001:2015 – Quality Management Systems – Requirements , International Organization for Standardization Std., 2015, iSO 9001:2015.
24. ISO/IEC/IEEE 24765:2017 – Systems and Software Engineering – Vocabulary , International Organization for Standardization; International Electrotechnical Commission; Institute of Electrical and Electronics Engineers Std., 2017, iSO/IEC/IEEE 24765:2017.
25. P. Nistala, K.V. Nori, and R. Reddy, “Software quality models: A systematic mapping study,” in 2019 IEEE/ACM International Conference on Software and System Processes (ICSSP) . IEEE, 2019, pp. 125–134.
26. ISO/IEC 9126:1991 – Software Engineering—Product Quality , International Organization for Standardization; International Electrotechnical Commission Std., 1991, iSO/IEC 9126:1991.
27. ISO/IEC 9126:2001 – Software Engineering—Product Quality , International Organization for Standardization; International Electrotechnical Commission Std., 2001, iSO/IEC 9126:2001.
28. ISO/IEC 14598:1999 – Software Engineering—Product Evaluation , International Organization for Standardization; International Electrotechnical Commission Std., 1999, iSO/IEC 14598:1999.
29. ISO/IEC 25000:2014 – Systems and Software Engineering—SQuaRE—Guide to SQuaRE , International Organization for Standardization; International Electrotechnical Commission Std., 2014, iSO/IEC 25000:2014.
30. ISO/IEC 25023:2016 – Systems and Software Engineering—Measurement of System and Software Product Quality , International Organization for Standardization; International Electrotechnical Commission Std., 2016, iSO/IEC 25023:2016.
31. ISO/IEC 25040:2024 – Systems and Software Engineering—Quality Evaluation Framework , International Organization for Standardization; International Electrotechnical Commission Std., 2024, iSO/IEC 25040:2024.
32. S.S. Stevens, “On the theory of scales of measurement,” Science , Vol. 103, No. 2684, 1946, pp. 677–680.
33. N. Fenton and J. Bieman, Software metrics: a rigorous and practical approach . CRC press, 2014.
34. G. Poels and G. Dedene, “Distance: A framework for software measure construction,” DTEW Research Report 9937 , 1999, pp. 1–47.
35. N. Dalkey and O. Helmer, “An experimental application of the delphi method to the use of experts,” Management science , Vol. 9, No. 3, 1963, pp. 458–467.
36. J. Kontio, J. Bragge, and L. Lehtola, “The focus group method as an empirical tool in software engineering,” in Guide to advanced empirical software engineering . Springer, 2008, pp. 93–116.
37. M. Shaw, “Writing good software engineering research papers,” in 25th International Conference on Software Engineering, 2003. Proceedings. Portland, OR, USA: IEEE, 2003, pp. 726–736.
38. M. Overeem, M. Mathijssen, and S. Jansen, “API-m-FAMM: A focus area maturity model for API management,” Information and Software Technology , Vol. 147, 2022, p. 106890.
39. R. Yamamoto, K. Ohashi, M. Fukuyori, K. Kimura, A. Sekiguchi et al., “A quality model and its quantitative evaluation method for web APIs,” in 2018 25th Asia-Pacific Software Engineering Conference (APSEC) , 2018, pp. 598–607.
40. A. Machini and S. Casas, “A preliminary gqm model to evaluate web API usability,” Memorias de las JAIIO , Vol. 10, No. 2, 2024, pp. 1–13.
41. V.R. Basili, G. Caldiera, and H.D. Rombach, “The goal question metric approach,” in Encyclopedia of software engineering , 1994.
42. M. Constanzo, S.I. Casas, G.B. Vidal, and D. Cruz, “Usos y problemas de las APIs web en la república argentina,” No. 44, pp. 79–97. [Online]. https://rtyc.utn.edu.ar/index.php/rtyc/article/view/913
43. A. Machini and S. Casas, “An empirical study on web API usability: The consumer-developer perspective,” Brazilian Journal of Technology , Vol. 7, No. 4, 2024, pp. e74 474–e74 474.
44. K. Peffers, T. Tuunanen, M.A. Rothenberger, and S. Chatterjee, “A design science research methodology for information systems research,” Journal of management information systems , Vol. 24, No. 3, 2007, pp. 45–77.
45. B. Kitchenham and S.L. Pfleeger, “Principles of survey research part 6: data analysis,” ACM SIGSOFT Software Engineering Notes , Vol. 28, No. 2, 2003, pp. 24–27.
46. C. Wohlin, P. Runeson, M. Höst, M.C. Ohlsson, B. Regnell et al., Experimentation in Software Engineering . Springer Berlin Heidelberg. [Online]. http://link.springer.com/10.1007/978-3-642-29044-2
47. F.J. Fowler Jr, Survey research methods . Sage publications, 2013.
48. M. Kasunic, “Designing an effective survey,” Carnegie Mellon University , 2005.
49. RStudio, PBC, RStudio: Integrated Development Environment for R , 2025, version 2025.05.1+513. [Online]. https://posit.co/download/rstudio-desktop/
50. R Core Team, R: A Language and Environment for Statistical Computing , R Foundation for Statistical Computing, Vienna, Austria, 2024, version 4.5.0. [Online]. https://www.R-project.org/
51. H. Wickham, ggplot2: Elegant Graphics for Data Analysis , 2023, version 3.5.2. [Online]. https://ggplot2.tidyverse.org
52. R.C. Schmidt, “Managing delphi surveys using nonparametric statistical techniques,” Decision Sciences , Vol. 28, No. 3, 1997, pp. 763–774. [Online]. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1540-5915.1997.tb01330.x
53. R.A. Fisher, “Statistical methods for research workers,” in Breakthroughs in statistics: Methodology and distribution . Springer, 1970, pp. 66–70.
54. J.W. Tukey, “Comparing individual means in the analysis of variance,” Biometrics , 1949, pp. 99–114.
55. M.G. Kendall and B.B. Smith, “The problem of m rankings,” The annals of mathematical statistics , Vol. 10, No. 3, 1939, pp. 275–287.
56. L.J. Cronbach, “Coefficient alpha and the internal structure of tests,” Psychometrika , Vol. 16, No. 3, 1951, pp. 297–334.
57. J.L. Fleiss, “Measuring nominal scale agreement among many raters,” Psychological Bulletin , Vol. 76, No. 5, 1971, pp. 378–382.
58. P. Becker, L. Olsina, and M.F. Papa, “Especificación de métricas de mantenibilidad para mejorar código java: Caso aplicado en un curso avanzado de ingeniería en sistemas,” in Actas del XXI Congreso Nacional de Ingeniería Informática/Sistemas de Información (CoNaIISI), Tucumán, Argentina , 2023, pp. 1–15.