| 2026 | |
| [1] | , "Bridging Humans and LLMs: Investigating Human-AI Collaboration in Multi-agent Requirements Analysis for Organizational AI Adoption", In e-Informatica Software Engineering Journal, vol. 20, no. 1, pp. 260103, 2026.
DOI: 10.37190/e-Inf260103. Download article (PDF)Get article BibTeX file |
Authors
Malik Abdul Sami, Zheying Zhang, Muhammad Waseem, Kai-Kristian Kemell, Zeeshan Rasheed, Tomas Herda, Pekka Abrahamsson
Abstract
Context: Organizations adopting Artificial Intelligence (AI) face challenges in eliciting and analyzing requirements that align with strategic objectives, especially when human oversight and iterative refinement are needed. Large Language Models (LLMs)-based Multi-agent systems provide a~potential solution by supporting structured and collaborative Requirements Engineering (RE) processes for AI adoption planning.
Objective: The objective of this study is to investigate whether a multi-agent system, built on LLMs and supported by human input, can assist in requirements analysis for AI adoption.
Method: We used a mixed-method approach: (i) designed and developed a~multi-agent system to support the generation and prioritization of requirements for AI adoption, (ii) conducted multiple case studies with four companies to evaluate the system, and (iii) collected data through post-session questionnaires from nine participants and follow-up interviews, one per company.
Results: Questionnaire and interview findings together indicate that the system may assist in identifying relevant and goal-aligned requirements. Seven participants considered the generated requirements relevant, and six found them aligned with organizational goals. Participants noted that iterative feedback improved completeness and feasibility, often within two feedback rounds. Both data sources show that human input was essential to clarify technical details, ensure contextual accuracy, and validate prioritization results. Participants from all companies also identified usability, transparency, and scalability as areas requiring further refinement for broader organizational use.
Conclusions: LLM-based multi-agent systems can support strategic AI planning by enabling iterative refinement with human experts. Future work will include more interviews with stakeholders and adjustments to system features to improve transparency, usability, and scalability.
Keywords
Large Language Model (LLM), Multi-agent Systems, Requirements Analysis, Organizational AI Adoption, Strategic Planning, Case Study, Human-AI Collaboration, Human-in-the-Loop
References
1. A. Dasgupta and S. Wendler, “Ai adoption strategies,” Working Paper Series No. 9, Centre for Technology and Global Affairs, University of Oxford, March 2019. [Online]. https://www.politics.ox.ac.uk/sites/default/files/2022-03/201903-CTGA-Dasgupta%20A-Wendler%20S-aiadoptionstrategies.pdf
2. S. Kaggwa, T.F. Eleogu, F. Okonkwo, O.A. Farayola, P.U. Uwaoma et al., “Ai in decision making: transforming business strategies,” International Journal of Research and Scientific Innovation, Vol. 10, No. 12, 2024, pp. 423–444.
3. Gartner, “Gartner survey finds generative ai is now the most frequently deployed ai solution in organizations,” https://www.gartner.com/en/newsroom/press-releases/2024-05-07-gartner-survey-finds-generative-ai-is-now-the-most-frequently-deployed-ai-solution-in-organizations, 2024, accessed: 2025-04-06.
4. Gartner, “Gartner survey finds 79% of corporate strategists see ai and analytics as critical to their success over the next two years,” 2023. [Online]. https://www.gartner.com/en/newsroom/press-releases/2023-07-05-gartner-survey-finds-79-percent-of-corporate-strategists-see-ai-and-analytics-as-critical-to-their-success-over-the-next-two-years
5. K.K. Kemell, M. Saarikallio, A. Nguyen-Duc, and P. Abrahamsson, “Still just personal assistants? a multiple case study of generative ai adoption in software organizations,” Information and Software Technology, Vol. 186, 2025, p. 107805.
6. D. Russo, “Navigating the complexity of generative ai adoption in software engineering,” ACM Trans. Softw. Eng. Methodol., Vol. 33, No. 5, Jun. 2024. [Online]. https://doi.org/10.1145/3652154
7. D. Herremans, “aistrom–a roadmap for developing a successful ai strategy,” IEEE access, Vol. 9, 2021, pp. 155 826–155 838.
8. K. Ahmad, M. Abdelrazek, C. Arora, M. Bano, and J. Grundy, “Requirements practices and gaps when engineering human-centered artificial intelligence systems,” Applied Soft Computing, Vol. 143, 2023, p. 110421.
9. A. Choudhury and H. Shamszare, “Investigating the impact of user trust on the adoption and use of chatgpt: survey analysis,” Journal of Medical Internet Research, Vol. 25, 2023, p. e47184.
10. F. Zambonelli, N.R. Jennings, and M. Wooldridge, “Developing multiagent systems: The gaia methodology,” ACM Transactions on Software Engineering and Methodology (TOSEM), Vol. 12, No. 3, 2003, pp. 317–370.
11. U.e. Habiba, M. Haug, J. Bogner, and S. Wagner, “How mature is requirements engineering for ai-based systems? a systematic mapping study on practices, challenges, and future research directions,” Requirements Engineering, 2024, pp. 1–34.
12. X. Franch, A. Jedlitschka, and S. Martínez-Fernández, “A requirements engineering perspective to ai-based systems development: A vision paper,” in international working conference on requirements engineering: foundation for software quality. Springer, 2023, pp. 223–232.
13. A. Mehraj, Z. Zhang, and K. Systä, “A tertiary study on ai for requirements engineering,” in International Working Conference on Requirements Engineering: Foundation for Software Quality. Springer, 2024, pp. 159–177.
14. Z. Zhang, M. Rayhan, T. Herda, M. Goisauf, and P. Abrahamsson, “Llm-based agents for automating the enhancement of user story quality: An early report,” in International Conference on Agile Software Development. Springer Nature Switzerland Cham, 2024, pp. 117–126.
15. D. Jin, Z. Jin, X. Chen, and C. Wang, “Mare: Multi-agents collaboration framework for requirements engineering,” arXiv preprint arXiv:2405.03256, 2024.
16. P. Bornet, J. Wirtz, T.H. Davenport, D.D. Cremer, B. Evergreen et al., Agentic Artificial Intelligence: Harnessing AI Agents to Reinvent Business, Work, and Life, kindle edition ed. Singapore: World Scientific Publishing, March 2025, available on Amazon Kindle, 572 pages. [Online]. https://www.amazon.com/dp/B0CQYZN7QM
17. M.A. Sami, Z. Zhang, M. Waseem, K.K. Kemell, Z. Rasheed et al., “A multi-agent llm system for automated requirements analysis: A study on user story generation and prioritization,” in Euromicro Conference on Software Engineering and Advanced Applications. Springer, 2025, pp. 178–187.
18. M.C. Lee, H. Scheepers, A.K. Lui, and E.W. Ngai, “The implementation of artificial intelligence in organizations: A systematic literature review,” Information & Management, Vol. 60, No. 5, 2023, p. 103816.
19. P. Hamm and M. Klesel, “Success factors for the adoption of artificial intelligence in organizations: A literature review,” in 27th Americas Conference on Information Systems, AMCIS 2021: Digital Innovation and Entrepreneurship. Association for Information Systems, 2021.
20. D. Smit, S. Eybers, and A. van der Merwe, “Towards human-ai symbiosis: Designing an artificial intelligence adoption framework,” South African Computer Journal, Vol. 36, No. 1, 2024, pp. 76–104.
21. A. Tursunbayeva and H.C.B. Gal, “Adoption of artificial intelligence: A top framework-based checklist for digital leaders,” Business Horizons, Vol. 67, No. 4, 2024, pp. 357–368.
22. J. Yang, Y. Blount, and A. Amrollahi, “Artificial intelligence adoption in a professional service industry: A multiple case study,” Technological Forecasting and Social Change, Vol. 201, 2024, p. 123251.
23. F. Selten and B. Klievink, “Organizing public sector ai adoption: Navigating between separation and integration,” Government Information Quarterly, Vol. 41, No. 1, 2024, p. 101885.
24. G. Vaz Pereira, V. Jackson, R. Prikladnicki, A. van der Hoek, L. Fortes et al., “Exploring genai in software development: Insights from a case study in a large brazilian company,” in IEEE International Conference on Software Engineering (ICSE), 2025.
25. J. Flyckt, T. Gorschek, D. Mendez, and N. Lavesson, “Identifying key ai challenges in make-to-order manufacturing organisations: A multiple case study,” Journal of Systems and Software, Vol. 230, 2025, p. 112559.
26. Y. Feng, S. Vanam, M. Cherukupally, W. Zheng, M. Qiu et al., “Investigating code generation performance of chat-gpt with crowdsourcing social data,” in Proceedings of the 47th IEEE Computer Software and Applications Conference, 2023, pp. 1–10.
27. F.A. Shah, A. Sabir, R. Sharma, and D. Pfahl, “How effectively do llms extract feature-sentiment pairs from app reviews?” in International Working Conference on Requirements Engineering: Foundation for Software Quality. Springer, 2025, pp. 123–138.
28. A. Ferrari and P. Spoletini, “Formal requirements engineering and large language models: A two-way roadmap,” Information and Software Technology, Vol. 181, 2025, p. 107697. [Online]. https://www.sciencedirect.com/science/article/pii/S0950584925000369
29. O. Abed, K. Nebe, and A.B. Abdellatif, “Ai-generated user stories supporting human-centred development: an investigation on quality,” in International Conference on Human-Computer Interaction. Springer, 2024, pp. 3–13.
30. L. Zhong, Z. Wang, and J. Shang, “Ldb: A large language model debugger via verifying runtime execution step-by-step,” arXiv preprint arXiv:2402.16906, 2024.
31. B. Wei, “Requirements are all you need: From requirements to code with llms,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE). IEEE, 2024, pp. 416–422.
32. S. Manish, “An autonomous multi-agent llm framework for agile software development,” International Journal of Trend in Scientific Research and Development, Vol. 8, No. 5, 2024, pp. 892–898.
33. C. Hymel and H. Johnson, “Analysis of llms vs human experts in requirements engineering,” arXiv preprint arXiv:2501.19297, 2025.
34. A. Ferrari and P. Spoletini, “Formal requirements engineering and large language models: A two-way roadmap,” Information and Software Technology, Vol. 181, 2025, p. 107697.
35. C. Zhang, K. Yang, S. Hu, Z. Wang, G. Li et al., “Proagent: building proactive cooperative agents with large language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, No. 16, 2024, pp. 17 591–17 599.
36. X. Zou, Y. Liu, X. Shi, and C. Yang, “Goal2story: A multi-agent fleet based on privately enabled sllms for impacting mapping on requirements elicitation,” arXiv preprint arXiv:2503.13279, 2025.
37. W. Tao, Y. Zhou, Y. Wang, W. Zhang, H. Zhang et al., “Magis: Llm-based multi-agent framework for github issue resolution,” Advances in Neural Information Processing Systems, Vol. 37, 2024, pp. 51 963–51 993.
38. J. Liu, G. Wang, R. Yang, J. Zeng, M. Zhao et al., “Rtadev: Intention aligned multi-agent framework for software development,” in Findings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 1548–1581.
39. H. Takagi, S. Moriya, T. Sato, M. Nagao, and K. Higuchi, “A framework for efficient development and debugging of role-playing agents with large language models,” in Proceedings of the 30th International Conference on Intelligent User Interfaces, 2025, pp. 70–88.
40. K.T. Tran, D. Dao, M.D. Nguyen, Q.V. Pham, B. O’Sullivan et al., “Multi-agent collaboration mechanisms: A survey of llms,” arXiv preprint arXiv:2501.06322, 2025.
41. A. Singh, A. Ehtesham, S. Kumar, and T.T. Khoei, “Enhancing ai systems with agentic workflows patterns in large language model,” in 2024 IEEE World AI IoT Congress (AIIoT). IEEE, 2024, pp. 527–532.
42. Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li et al., “Autogen: Enabling next-gen llm applications via multi-agent conversation,” arXiv preprint arXiv:2308.08155, 2023.
43. N.A. Parikh, “Managing ai-first products: Roles, skills, challenges, and strategies of ai product managers,” IEEE Engineering Management Review, 2025, pp. 1–11.
44. N. Tasneem, H.B. Zulzalil, and S. Hassan, “Enhancing agile software development: A systematic literature review of requirement prioritization and reprioritization techniques,” IEEE Access, 2025.
45. F. Essam, H. El, and S.R.H. Ali, “A comparison of the pearson, spearman rank and kendall tau correlation coefficients using quantitative variables,” Asian J. Probab. Stat, 2022, pp. 36–48.
46. M.A. Sami, M. Waseem, Z. Zhang, Z. Rasheed, K. Systä et al., “Early results of an ai multiagent system for requirements elicitation and analysis,” in Product-Focused Software Process Improvement, D. Pfahl, J. Gonzalez Huerta, J. Klünder, and H. Anwar, Eds. Cham: Springer Nature Switzerland, 2025, pp. 307–316.
47. P. Murley, Z. Ma, J. Mason, M. Bailey, and A. Kharraz, “Websocket adoption and the landscape of the real-time web,” in Proceedings of the Web Conference 2021, 2021, pp. 1192–1203.
48. B. Lubanovic, FastAPI. ” O’Reilly Media, Inc.”, 2023.
49. A. Hurst, A. Lerer, A.P. Goucher, A. Perelman, A. Ramesh et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276, 2024.
50. P. Runeson and M. Höst, “Guidelines for conducting and reporting case study research in software engineering,” Empirical software engineering, Vol. 14, 2009, pp. 131–164.
51. V. Braun and V. Clarke, “Using thematic analysis in psychology,” Qualitative research in psychology, Vol. 3, No. 2, 2006, pp. 77–101.
52. R.K. Yin, “Case study research and applications: Design and methods,” SAGE Publications, 2018.
53. I. Etikan, S.A. Musa, R.S. Alkassim et al., “Comparison of convenience sampling and purposive sampling,” American journal of theoretical and applied statistics, Vol. 5, No. 1, 2016, pp. 1–4.
54. I. Garcia, C. Pacheco, A. León, and J.A. Calvo-Manzano, “A serious game for teaching the fundamentals of iso/iec/ieee 29148 systems and software engineering–lifecycle processes–requirements engineering at undergraduate level,” Computer Standards & Interfaces, Vol. 67, 2020, p. 103377.
55. H. Zhao, H. Chen, F. Yang, N. Liu, H. Deng et al., “Explainability for large language models: A survey,” ACM Transactions on Intelligent Systems and Technology, Vol. 15, No. 2, 2024, pp. 1–38.
56. D. Russo, “Navigating the complexity of generative ai adoption in software engineering,” ACM Transactions on Software Engineering and Methodology, Vol. 33, No. 5, 2024, pp. 1–50.
57. McKinsey & Company, “One year of agentic ai: Six lessons from the people doing the work,” https://www.mckinsey.com/capabilities/quantumblack/our-insights/one-year-of-agentic-ai-six-lessons-from-the-people-doing-the-work, 2025, accessed: 2025-10-13.
58. J. He, C. Treude, and D. Lo, “Llm-based multi-agent systems for software engineering: Literature review, vision and the road ahead,” ACM Transactions on Software Engineering and Methodology, 2024.
59. S. Amershi, A. Begel, C. Bird, R. DeLine, G. Gall et al., “Software engineering for machine learning: A case study,” International Conference on Software Engineering (ICSE), 2019, pp. 291–300. [Online]. https://doi.org/10.1109/ICSE.2019.00042
60. C.O. Retzlaff, S. Das, C. Wayllace, P. Mousavi, M. Afshari et al., “Human-in-the-loop reinforcement learning: A survey and position on requirements, challenges, and opportunities,” Journal of Artificial Intelligence Research, Vol. 79, 2024, pp. 359–415.
61. R.K. Yin, Case Study Research: Design and Methods, 5th ed. Thousand Oaks, CA: Sage Publications, 2014.