{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T19:06:44Z","timestamp":1772219204082,"version":"3.50.1"},"reference-count":61,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2023,9,23]],"date-time":"2023-09-23T00:00:00Z","timestamp":1695427200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"MITRE Independent Research and Development Program\u2019s Technology Futures Innovation Area","award":["23-1929"],"award-info":[{"award-number":["23-1929"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Systems"],"abstract":"<jats:p>Despite noble intentions, new technologies may have adverse effects on the resilience of the sociotechnical systems into which they are integrated. Our objective was to develop a lightweight method to elicit requirements that, if implemented, would support sociotechnical system resilience. We developed and piloted the Resilience-Aware Development Exercise Protocol (RAD-XP), a method to generate tabletop exercises (TTXs) to elicit resilience requirements. In the pilot study, this approach generated 15 requirements from a one-hour TTX, where the majority of requirements were found to support resilience. Participants indicated via survey that RAD-XP was effective and efficient, and that they would want to use RAD-XP regularly throughout the agile development process. We discuss future research and development to refine this approach to eliciting resilience requirements.<\/jats:p>","DOI":"10.3390\/systems11100487","type":"journal-article","created":{"date-parts":[[2023,9,24]],"date-time":"2023-09-24T10:48:31Z","timestamp":1695552511000},"page":"487","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["RAD-XP: Tabletop Exercises for Eliciting Resilience Requirements for Sociotechnical Systems"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1872-853X","authenticated-orcid":false,"given":"Stephen L.","family":"Dorton","sequence":"first","affiliation":[{"name":"The MITRE Corporation, McLean, VA 22102, USA"}]},{"given":"Emily","family":"Barrett","sequence":"additional","affiliation":[{"name":"The MITRE Corporation, McLean, VA 22102, USA"}]},{"given":"Theresa","family":"Fersch","sequence":"additional","affiliation":[{"name":"The MITRE Corporation, McLean, VA 22102, USA"}]},{"given":"Andrew","family":"Langone","sequence":"additional","affiliation":[{"name":"The MITRE Corporation, McLean, VA 22102, USA"}]},{"given":"Kelly J.","family":"Neville","sequence":"additional","affiliation":[{"name":"The MITRE Corporation, McLean, VA 22102, USA"}]}],"member":"1968","published-online":{"date-parts":[[2023,9,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Chopra, A.K., Dalpiaz, F., Aydemir, F.B., Giorgini, P., Mylopoulos, J., and Singh, M.P. (2014, January 25\u201329). Protos: Foundations for Engineering Innovative Sociotechnical Systems. Proceedings of the 2014 IEEE 22nd International Requirements Engineering Conference (RE), Karlskrona, Sweden.","DOI":"10.1109\/RE.2014.6912247"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Aydemir, F.B., Giorgini, P., Mylopoulos, J., and Dalpiaz, F. (2014, January 28\u201330). Exploring Alternative Designs for Sociotechnical Systems. Proceedings of the 2014 IEEE Eighth International Conference on Research Challenges in Information Science, Merrakech, Morocco.","DOI":"10.1109\/RCIS.2014.6861037"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1016\/S0003-6870(00)00009-0","article-title":"Sociotechnical Principles for System Design","volume":"31","author":"Clegg","year":"2000","journal-title":"Appl. Ergon."},{"key":"ref_4","first-page":"163","article-title":"Cognitive Engineering: Issues in User-Centered System Design","volume":"1","author":"Roth","year":"2002","journal-title":"Encycl. Softw. Eng."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1007\/s10669-018-9708-3","article-title":"The Theory of Graceful Extensibility: Basic Rules that Govern Adaptive Systems","volume":"38","author":"Woods","year":"2018","journal-title":"Environ. Syst. Decis."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"222","DOI":"10.1177\/15553434221103718","article-title":"A Naturalistic Investigation of Trust, AI, and Intelligence Work","volume":"16","author":"Dorton","year":"2022","journal-title":"J. Cogn. Eng. Decis. Mak."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1177\/1071181322661146","article-title":"Adaptations to Trust Incidents with Artificial Intelligence","volume":"66","author":"Dorton","year":"2022","journal-title":"Proc. Hum. Factors Ergon. Soc. Annu. Meet."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1143907","DOI":"10.3389\/frai.2023.1143907","article-title":"Foresight for Ethical AI","volume":"6","author":"Dorton","year":"2023","journal-title":"Front. Artif. Intell."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2067","DOI":"10.1177\/1071181322661177","article-title":"The TRUSTS Work System Resilience Framework: A Foundation for Resilience-Aware Development and Transition","volume":"66","author":"Neville","year":"2022","journal-title":"Proc. Hum. Factors Ergon. Soc. Annu. Meet."},{"key":"ref_10","unstructured":"Nemeth, C., Wears, R., Woods, D., Hollnagel, E., and Cook, R. (2008). Advances in Patient Safety: New Directions and Alternative Approaches, Agency for Healthcare Research and Quality (US)."},{"key":"ref_11","first-page":"1578","article-title":"Unintended Effects of a Computerized Physician Order Entry Nearly Hard-Stop Alert to Prevent a Drug Interaction: A Randomized Controlled Trial","volume":"170","author":"Storm","year":"2010","journal-title":"Arch. Intern. Med."},{"key":"ref_12","unstructured":"Mirsa, S., Kumar, V., and Kumar, U. (2005, January 1\u20134). Goal-Oriented or Scenario-Based Requirements Engineering Technique\u2013What Should a Practitioner Select?. Proceedings of the Canadian Conference on Electrical and Computer Engineering, Saskatoon, SK, Canada."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Raunak, M.S., and Binkley, D. (2017, January 25\u201328). Agile and Other Trends in Software Engineering. Proceedings of the 2017 IEEE 28th Annual Software Technology Conference (STC), Gaithersburg, Maryland, USA.","DOI":"10.1109\/STC.2017.8234457"},{"key":"ref_14","unstructured":"Fowler, M., and Highsmith, J. (2023, July 01). The Manifesto for Agile Software Development. Available online: agilemanifesto.org."},{"key":"ref_15","first-page":"1214","article-title":"A Research Study on Critical Challenges in Agile Requirements Engineering","volume":"6","author":"Batra","year":"2019","journal-title":"Int. Res. J. Eng. Technol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1579","DOI":"10.1109\/TSE.2020.3025317","article-title":"Real World Scrum: A Grounded Theory of Variations in Practice","volume":"48","author":"Masood","year":"2020","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"110785","DOI":"10.1016\/j.jss.2020.110785","article-title":"Integrating UX Work with Agile Development Through User Stories: An Action Research Study in a Small Software Company","volume":"170","author":"Ananjeva","year":"2020","journal-title":"J. Syst. Softw."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Kelly, S. (2010, January 19\u201323). Towards an Evolutionary Framework for Agile Requirements Elicitation. Proceedings of the 2nd ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Berlin, Germany.","DOI":"10.1145\/1822018.1822078"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Rodeghero, P., Jiang, S., Armaly, A., and McMillan, C. (2017, January 20\u201328). Detecting User Story Information in Developer-Client Conversations to Generate Extractive Summaries. Proceedings of the 2017 IEEE\/ACM 39th International Conference on Software Engineering (ICSE), Buenos Aires, Argentina.","DOI":"10.1109\/ICSE.2017.13"},{"key":"ref_20","first-page":"49","article-title":"Non-Functional Requirements Elicitation Guideline for Agile Methods","volume":"9","author":"Younas","year":"2017","journal-title":"JTEC"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Dorton, S.L., Fersch, T., Barrett, E., Langone, A., Seip, M., Bilsborough, S., Hudson, C.B., Ward, P., and Neville, K.J. (2023, January 23\u201327). The Value of Wargames and Tabletop Exercises as Naturalistic Tools. Proceedings of the HFES 67th International Annual Meeting, Washington DC, USA. article in press.","DOI":"10.1177\/21695067231192617"},{"key":"ref_22","first-page":"33","article-title":"Toward a History-Based Doctrine for Wargaming","volume":"14","author":"Caffrey","year":"2000","journal-title":"Aerosp. Power J."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2037","DOI":"10.1177\/1541931213601988","article-title":"Going Digital: Consequences of Increasing Resolution of a Wargaming Tool for Knowledge Elicitation","volume":"61","author":"Dorton","year":"2017","journal-title":"Proc. Hum. Factors Ergon. Soc. Annu. Meet."},{"key":"ref_24","first-page":"9","article-title":"Exploring Trust in Unmanned Systems with the Maritime Unmanned System Trust Game","volume":"52","author":"Strode","year":"2022","journal-title":"Hum. Factors Syst. Interact."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"93","DOI":"10.17083\/ijsg.v9i2.492","article-title":"Games to Support Disruptive Technology Adoption: The MUST Game Use Case","volume":"9","author":"Strode","year":"2022","journal-title":"IJSG"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Dorton, S.L., Maryeski, L.R., Ogren, L., Dykens, I.T., and Main, A. (2020). A Wargame-Augmented Knowledge Elicitation Method for the Agile Development of Novel Systems. Systems, 8.","DOI":"10.3390\/systems8030027"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Smith, S., Raio, S., Erbacher, R., Weisman, M., Parker, T., and Ellis, J. (2022). Quantitative Measurement of Cyber Resilience: A Tabletop Exercise, DEVCOM Army Research Laboratory (US).","DOI":"10.21236\/AD1158532"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/MIS.2010.54","article-title":"The Practitioner\u2019s Cycles, Part 1: Actual World Problems","volume":"25","author":"Deal","year":"2010","journal-title":"IEEE Intell. Syst."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1109\/MIS.2010.89","article-title":"The Practitioner\u2019s Cycles, Part 2: Solving Envisioned World Problems","volume":"25","author":"Hoffman","year":"2010","journal-title":"IEEE Intell. Syst."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"434","DOI":"10.1518\/155534307X264898","article-title":"Development and Evaluation of an Intuitive Operational Planning Process","volume":"1","author":"Bryant","year":"2007","journal-title":"J. Cogn. Eng. Decis. Mak."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Lombriser, P., Dalpiaz, F., Lucassen, G., and Brinkkemper, S. (2016, January 14\u201317). Gamified Requirements Engineering: Model and Experimentation. Proceedings of the Requirements Engineering: Foundation for Software Quality, Gothenburg, Sweden.","DOI":"10.1007\/978-3-319-30282-9_12"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1002\/j.2334-5837.2018.00474.x","article-title":"What Can You Learn About Systems Engineering by Building a LegoTM Car?","volume":"28","author":"Hahn","year":"2018","journal-title":"INCOSE Int. Symp."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Dykens, I.T., Wetzel, A., Dorton, S.L., and Batchelor, E. (2021, January 24\u201329). Towards a Unified Model of Gamification and Motivation. Proceedings of the Adaptive Instructional Systems. Design and Evaluation, Virtual Event.","DOI":"10.1007\/978-3-030-77857-6_4"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Sutcliffe, A. (2003). Scenario-Based Requirements Engineering. J. Light. Technol.","DOI":"10.1007\/978-1-4471-0217-5_6"},{"key":"ref_35","unstructured":"Sutcliffe, R.B., and Ryan, M. (1998, January 10). Experience with SCRAM, a Scenario Requirements Analysis Method. Proceedings of the IEEE International Symposium on Requirements Engineering: RE \u201998, Colorado Springs, CO, USA."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"978","DOI":"10.1109\/32.879820","article-title":"Handling Obstacles in Goal-Oriented Requirements Engineering","volume":"26","author":"Letier","year":"2000","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1007\/s00766-006-0039-4","article-title":"Using Obstacle Analysis to Identify Contingency Requirements on an Unpiloted Aerial Vehicle","volume":"12","author":"Lutz","year":"2007","journal-title":"Requir. Eng."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Ponsard, C., and Darimont, R. (September, January 31). Formalizing Security and Safety Requirements by Mapping Attack-Fault Trees on Obstacle Models with Constraint Programming Semantics. Proceedings of the 2020 IEEE FORMREQ, Zurich, Switzerland.","DOI":"10.1109\/FORMREQ51202.2020.00009"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"107264","DOI":"10.1016\/j.infsof.2023.107264","article-title":"Use of Personas in Requirements Engineering: A Systematic Mapping Study","volume":"162","author":"Karolita","year":"2023","journal-title":"Inf. Softw. Technol."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1007\/s10207-019-00438-x","article-title":"TrUStAPIS: A Trust Requirements Elicitation Method for IoT","volume":"19","author":"Ferraris","year":"2020","journal-title":"Int. J. Inf. Secur."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s42979-020-00416-4","article-title":"Data-Driven Requirements Elicitation: A Systematic Literature Review","volume":"2","author":"Lim","year":"2021","journal-title":"SN Comput. Sci."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1145\/1164394.1164418","article-title":"Can Distributed Software Development be Agile?","volume":"49","author":"Ramesh","year":"2006","journal-title":"Commun. ACM"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Tanzman, E.A., and Nieves, L.A. (2008). Analysis of the Argonne Distance Tabletop Exercise Method., Argonne National Lab. (ANL). No. ANL\/DIS\/RP-60983.","DOI":"10.2172\/924390"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"137","DOI":"10.69554\/WHGP7589","article-title":"Beyond the Hit-and-Run Tabletop Exercise","volume":"3","author":"Hansen","year":"2009","journal-title":"J. Bus. Contin. Emerg. Plan."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Raharjana, I.K., Siahaan, D., and Fatichah, C. (2019, January 10\u201312). User Story Extraction from Online News for Software Requirements Elicitation: A Conceptual Model. Proceedings of the 2019 16th International Joint Conference on Computer Science and Software Engineering (JCSSE), Chonburi, Thailand.","DOI":"10.1109\/JCSSE.2019.8864199"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Gall, M., and Berenbach, B. (2006, January 12). Towards a Framework for Real Time Requirements Elicitation. Proceedings of the 2006 First International Workshop on Multimedia Requirements Engineering (MERE\u201906\u2013RE\u201906 Workshop), Minnneapolis, MN, USA.","DOI":"10.1109\/MERE.2006.6"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Rietz, T., and Maedche, A. (2019, January 23\u201327). LadderBot: A Requirements Self-Elicitation System. Proceedings of the 2019 IEEE 27th International Requirements Engineering Conference (RE), Jeju, Korea.","DOI":"10.1109\/RE.2019.00045"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"6696695","DOI":"10.1155\/2021\/6696695","article-title":"Requirement Engineering Challenges in Agile Software Development","volume":"2021","author":"Rasheed","year":"2021","journal-title":"Math. Probl. Eng."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Heyn, H.-M., Knauss, E., Muhammad, A.P., Eriksson, O., Linder, J., Subbiah, P., Pradhan, S.K., and Tungal, S. (2021, January 30\u201331). Requirement Engineering Challenges for AI-Intense Systems Development. Proceedings of the 2021 IEEE\/ACM 1st Workshop on AI Engineering\u2013Software Engineering for AI (WAIN), Madrid, Spain.","DOI":"10.1109\/WAIN52551.2021.00020"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Hosseini, M., Shahri, A., Phalp, K., Taylor, J., Ali, R., and Dalpiaz, F. (2015, January 13\u201315). Configuring Crowdsourcing for Requirements Elicitation. Proceedings of the 2015 IEEE 9th International Conference on Research Challenges in Information Science (RCIS), Athens, Greece.","DOI":"10.1109\/RCIS.2015.7128873"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Dorton, S.L., Maryeski, L.M., Costello, R.P., and Abrecht, B.R. (2021). A Case for User Centered Design Satellite Command and Control. Aerospace, 8.","DOI":"10.3390\/aerospace8100303"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Dorton, S.L., Ganey, H.C.N., Mintman, E., Mittu, R., Smith, M.A.B., and Winters, J. (2021, January 4\u20137). Human-Centered Alphabet Soup: Approaches to Systems Development from Related Disciplines. Proceedings of the 2021 International Annual Meeting of the Human Factors and Ergonomics Society, Baltimore, MD, USA.","DOI":"10.1177\/1071181321651011"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Klein, G. (2022). Snapshots of the Mind, MIT Press.","DOI":"10.7551\/mitpress\/14342.001.0001"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Dalpiaz, F., and Brinkkemper, S. (2021, January 20\u201324). Agile Requirements Engineering: From User Stories to Software Architectures. Proceedings of the 2021 IEEE 29th International Requirements Engineering Conference (RE), Notre Dame, IN, USA.","DOI":"10.1109\/RE51729.2021.00076"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Lucassen, G., Dalpiaz, F., van der Werf, J.M.E.M., and Brinkkemper, S. (2015, January 24\u201328). Forging High-Quality User Stories: Towards a Discipline for Agile Requirements. Proceedings of the 2015 IEEE 23rd International Requirements Engineering Conference (RE), Ottawa, ON, Canada.","DOI":"10.1109\/RE.2015.7320415"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Lucassen, G., Dalpiaz, F., van der Werf, J.M.E.M., and Brinkkemper, S. (2016, January 14\u201317). The Use and Effectiveness of User Stories in Practice. Proceedings of the Requirements Engineering: Foundation for Software Quality, Gothenburg, Sweden.","DOI":"10.1007\/978-3-319-30282-9_14"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1080\/01446190701882390","article-title":"Influence, Stakeholder Mapping and Visualization","volume":"26","author":"Walker","year":"2008","journal-title":"Constr. Manag. Econ."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1080\/10875301.2015.1107000","article-title":"Walking a Mile in the User\u2019s Shoes: Customer Journey Mapping as a Method to Understanding the User Experience","volume":"20","author":"Marquez","year":"2015","journal-title":"Internet Ref. Serv. Q."},{"key":"ref_59","first-page":"162","article-title":"Making Usability Recommendations Useful and Usable","volume":"2","author":"Molich","year":"2007","journal-title":"J. Usability Stud."},{"key":"ref_60","first-page":"85","article-title":"Mining Non-Functional Requirements Using Machine Learning Techniques","volume":"15","author":"Jindal","year":"2021","journal-title":"e-Informatical Softw. Eng. J."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Franch, X., G\u00f3mez, C., Jedlitschka, A., L\u00f3pez, L., Mart\u00ednez-Fern\u00e1ndez, S., Oriol, M., and Partanen, J. (2018, January 11\u201315). Data-Driven Elicitation, Assessment and Documentation of Quality Requirements in Agile Software Development. Proceedings of the Advanced Information Systems Engineering, Tallinn, Estonia.","DOI":"10.1007\/978-3-319-91563-0_36"}],"container-title":["Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-8954\/11\/10\/487\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:56:38Z","timestamp":1760129798000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-8954\/11\/10\/487"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,23]]},"references-count":61,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,10]]}},"alternative-id":["systems11100487"],"URL":"https:\/\/doi.org\/10.3390\/systems11100487","relation":{},"ISSN":["2079-8954"],"issn-type":[{"value":"2079-8954","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,23]]}}}