{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,21]],"date-time":"2026-04-21T14:29:13Z","timestamp":1776781753473,"version":"3.51.2"},"reference-count":19,"publisher":"World Scientific Pub Co Pte Lt","issue":"11n12","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Soft. Eng. Knowl. Eng."],"published-print":{"date-parts":[[2020,11]]},"abstract":"<jats:p> Significant effort is being put into developing industrial applications for artificial intelligence (AI), especially those using machine learning (ML) techniques. Despite the intensive support for building ML applications, there are still challenges when it comes to evaluating, assuring, and improving the quality or dependability. The difficulty stems from the unique nature of ML, namely, system behavior is derived from training data not from logical design by human engineers. This leads to black-box and intrinsically imperfect implementations that invalidate many principles and techniques in traditional software engineering. In light of this situation, the Japanese industry has jointly worked on a set of guidelines for the quality assurance of AI systems (in the Consortium of Quality Assurance for AI-based Products and Services) from the viewpoint of traditional quality-assurance engineers and test engineers. We report on the second version of these guidelines, which cover a list of quality evaluation aspects, catalogue of current state-of-the-art techniques, and domain-specific discussions in five representative domains. The guidelines provide significant insights for engineers in terms of methodologies and designs for tests driven by application-specific requirements. <\/jats:p>","DOI":"10.1142\/s0218194020400227","type":"journal-article","created":{"date-parts":[[2020,10,31]],"date-time":"2020-10-31T05:08:25Z","timestamp":1604120905000},"page":"1589-1606","source":"Crossref","is-referenced-by-count":23,"title":["Guidelines for Quality Assurance of Machine Learning-Based Artificial Intelligence"],"prefix":"10.1142","volume":"30","author":[{"given":"Gaku","family":"Fujii","sequence":"first","affiliation":[{"name":"Cinnamon, Inc., Tokyo, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Koichi","family":"Hamada","sequence":"additional","affiliation":[{"name":"DeNA Co., Ltd., Tokyo, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fuyuki","family":"Ishikawa","sequence":"additional","affiliation":[{"name":"National Institute of Informatics Tokyo, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Satoshi","family":"Masuda","sequence":"additional","affiliation":[{"name":"IBM Research, Tokyo, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mineo","family":"Matsuya","sequence":"additional","affiliation":[{"name":"LIFULL Co., Ltd. Tokyo, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tomoyuki","family":"Myojin","sequence":"additional","affiliation":[{"name":"Hitachi, Ltd., Yokohama, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yasuharu","family":"Nishi","sequence":"additional","affiliation":[{"name":"University of Electro-Communications Tokyo, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hideto","family":"Ogawa","sequence":"additional","affiliation":[{"name":"Hitachi, Ltd., Yokohama, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Takahiro","family":"Toku","sequence":"additional","affiliation":[{"name":"OMRON Corporation, Kyoto, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Susumu","family":"Tokumoto","sequence":"additional","affiliation":[{"name":"FUJITSU LABORATORIES LTD. Kawasaki, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kazunori","family":"Tsuchiya","sequence":"additional","affiliation":[{"name":"FUJITSU LTD. Kawasaki, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yasuhiro","family":"Ujita","sequence":"additional","affiliation":[{"name":"OMRON Corporation Kyoto, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"219","published-online":{"date-parts":[[2021,1,21]]},"reference":[{"key":"S0218194020400227BIB003","first-page":"335","volume-title":"The 32nd Int. Conf. Software Engineering & Knowledge Engineering","author":"Hamada K.","year":"2020"},{"key":"S0218194020400227BIB004","first-page":"1","volume-title":"The 26th Symp. Operating Systems Principles","author":"Pei K.","year":"2017"},{"key":"S0218194020400227BIB005","first-page":"303","volume-title":"The 40th Int. Conf. Software Engineering","author":"Tian Y.","year":"2018"},{"key":"S0218194020400227BIB006","first-page":"118","volume-title":"The 27th ACM SIGSOFT Int. Symp. Software Testing and Analysis","author":"Dwarakanath A.","year":"2018"},{"key":"S0218194020400227BIB007","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/978-3-319-63387-9_1","volume-title":"The 29th Int. Conf. Computer Aided Verification","author":"Huang X.","year":"2017"},{"key":"S0218194020400227BIB008","first-page":"120","volume-title":"The 33rd ACM\/IEEE Int. Conf. Automated Software Engineering","author":"Ma L.","year":"2018"},{"key":"S0218194020400227BIB009","first-page":"1135","volume-title":"The 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining","author":"Ribeiro M. T.","year":"2016"},{"key":"S0218194020400227BIB010","first-page":"1885","volume-title":"The 34th Int. Conf. Machine Learning","author":"Koh P. W.","year":"2018"},{"key":"S0218194020400227BIB011","first-page":"35","volume-title":"The 23rd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining","author":"Angelino E.","year":"2017"},{"key":"S0218194020400227BIB012","first-page":"2672","volume-title":"Advances in Neural Information Processing Systems 27","author":"Goodfellow I.","year":"2014"},{"key":"S0218194020400227BIB013","first-page":"67","volume-title":"The 1st Workshop on Computer Vision for Fashion, Art and Design","author":"Hamada K.","year":"2018"},{"key":"S0218194020400227BIB014","first-page":"2234","volume-title":"The 29th Int. Conf. Neural Information Processing Systems","author":"Salimans T.","year":"2016"},{"key":"S0218194020400227BIB015","first-page":"6626","volume-title":"The 30th Int. Conf. Neural Information Processing Systems","author":"Heusel M.","year":"2017"},{"key":"S0218194020400227BIB016","first-page":"2672","volume-title":"The IEEE Conf. Computer Vision and Pattern Recognition","author":"Sun K.","year":"2019"},{"key":"S0218194020400227BIB019","first-page":"439","volume-title":"SAFECOMP 2018 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Proc.","author":"Czarnecki K.","year":"2018"},{"key":"S0218194020400227BIB020","first-page":"45","volume-title":"Proc. 37th Int. Conf., Computer Safety, Reliability, and Security","author":"Gauerhof L.","year":"2018"},{"key":"S0218194020400227BIB021","volume-title":"NIPS 2016 Workshop on Reliable Machine Learning in the Wild","author":"Zinkevich M.","year":"2017"},{"key":"S0218194020400227BIB022","first-page":"291","volume-title":"The 41st Int. Conf. Software Engineering: Software Engineering in Practice","author":"Amershi S.","year":"2019"},{"key":"S0218194020400227BIB023","first-page":"1123","volume-title":"NIPS 2016 Workshop on Reliable Machine Learning in the Wild","author":"Breck E.","year":"2017"}],"container-title":["International Journal of Software Engineering and Knowledge Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S0218194020400227","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,1,22]],"date-time":"2021-01-22T10:46:43Z","timestamp":1611312403000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/abs\/10.1142\/S0218194020400227"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11]]},"references-count":19,"journal-issue":{"issue":"11n12","published-print":{"date-parts":[[2020,11]]}},"alternative-id":["10.1142\/S0218194020400227"],"URL":"https:\/\/doi.org\/10.1142\/s0218194020400227","relation":{},"ISSN":["0218-1940","1793-6403"],"issn-type":[{"value":"0218-1940","type":"print"},{"value":"1793-6403","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11]]}}}