{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T15:16:01Z","timestamp":1778080561923,"version":"3.51.4"},"reference-count":33,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2020,5,1]],"date-time":"2020-05-01T00:00:00Z","timestamp":1588291200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Mathematics of Information Technology and Complex Systems"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,11,17]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Semi-automated segmentation algorithms hold promise for improving extraction and identification of objects in images such as tumors in medical images of human tissue, counting plants or flowers for crop yield prediction or other tasks where object numbers and appearance vary from image to image. By blending markup from human annotators to algorithmic classifiers, the accuracy and reproducability of image segmentation can be raised to very high levels. At least, that is the promise of this approach, but the reality is less than clear. In this paper, we review the state-of-the-art in semi-automated image segmentation performance assessment and demonstrate it to be lacking the level of experimental rigour needed to ensure that claims about algorithm accuracy and reproducability can be considered valid. We follow this review with two experiments that vary the type of markup that annotators make on images, either points or strokes, in tightly controlled experimental conditions in order to investigate the effect that this one particular source of variation has on the accuracy of these types of systems. In both experiments, we found that accuracy substantially increases when participants use a stroke-based interaction. In light of these results, the validity of claims about algorithm performance are brought into sharp focus, and we reflect on the need for a far more control on variables for benchmarking the impact of annotators and their context on these types of systems.<\/jats:p>","DOI":"10.1093\/iwcomp\/iwaa017","type":"journal-article","created":{"date-parts":[[2020,8,11]],"date-time":"2020-08-11T23:56:42Z","timestamp":1597190202000},"page":"233-245","source":"Crossref","is-referenced-by-count":3,"title":["Benchmarking Human Performance in Semi-Automated Image Segmentation"],"prefix":"10.1093","volume":"32","author":[{"given":"Mark","family":"Eramian","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Saskatchewan, Saskatoon, S7N5C9 Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christopher","family":"Power","sequence":"additional","affiliation":[{"name":"School of Mathematical and Computational Sciences, University of Prince Edward Island, Charlottetown, PE, C1A 4P3 Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stephen","family":"Rau","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Saskatchewan, Saskatoon, S7N5C9 Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pulkit","family":"Khandelwal","sequence":"additional","affiliation":[{"name":"Department of Bioengineering, University of Pennsylvania, Philadelphia, PA 19104-6321, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2020,8,11]]},"reference":[{"key":"2020121100533374700_ref1","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1145\/258549.258760","article-title":"Beyond Fitts\u2019 law: models for trajectory-based HCI tasks","volume-title":"Proc. of the ACM SIGCHI conf. on human factors in computing systems","author":"Accot","year":"1997"},{"key":"2020121100533374700_ref2","first-page":"9","article-title":"Fast random walker with priors using precomputation for interactive medical image segmentation","volume-title":"International conf. on medical image computing and computer-assisted intervention","author":"Andrews","year":"2010"},{"key":"2020121100533374700_ref3","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1007\/s11263-006-7934-5","article-title":"Graph cuts and efficient N-D image segmentation","volume":"70","author":"Boykov","year":"2006","journal-title":"Int. J. Comput. Vis."},{"key":"2020121100533374700_ref4","first-page":"1","article-title":"A systematic review of image segmentation methodology, used in the additive manufacture of patient-specific 3d printed models of the cardiovascular system","volume":"5","author":"Bryne","year":"2016","journal-title":"J. R. Soc. Med. Cardiovasc. Dis."},{"key":"2020121100533374700_ref5","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1016\/0925-4927(96)02790-4","article-title":"Accuracy and reproducibility of brain and tissue volumes using a magnetic resonance segmentation method","volume":"67","author":"Byrum","year":"1996","journal-title":"Psychiatry Res. Neuroimaging"},{"key":"2020121100533374700_ref6","doi-asserted-by":"crossref","first-page":"1451","DOI":"10.1109\/TMI.2006.880587","article-title":"Generalized overlap measures for evaluation and validation in medical image analysis","volume":"25","author":"Crum","year":"2006","journal-title":"IEEE Trans. Med. Imaging"},{"key":"2020121100533374700_ref7","article-title":"Evaluation of an interactive cell segmentation for fluorescence microscopy based on the graph cut algorithm","volume-title":"Proceedings of 7th workshop microscopic image analysis with applications in biology MIAAB","author":"Dach","year":"2011"},{"key":"2020121100533374700_ref8","doi-asserted-by":"crossref","first-page":"614441","DOI":"10.1117\/12.653099","article-title":"Semi-automatic knee cartilage segmentation","volume-title":"Medical Imaging 2006: Image Processing","author":"Dam","year":"2006"},{"key":"2020121100533374700_ref9","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1016\/j.imavis.2008.02.006","article-title":"Semiautomatic segmentation with compact shape prior","volume":"27,","author":"Das","year":"2009","journal-title":"Image Vision Comput."},{"key":"2020121100533374700_ref10","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1109\/CRV.2013.20","article-title":"Worst-case local boundary precision in global measures of segmentation reproducibility","volume-title":"2013 int. conf. on computer and robot vision","author":"Eramian","year":"2013"},{"key":"2020121100533374700_ref11","doi-asserted-by":"crossref","first-page":"1768","DOI":"10.1109\/TPAMI.2006.233","article-title":"Random walks for image segmentation","volume":"28","author":"Grady","year":"2006","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"2020121100533374700_ref12","doi-asserted-by":"crossref","first-page":"53","DOI":"10.2466\/pms.2001.92.1.53","article-title":"Psychosocial factors, workload, and human error in a simulated pharmacy dispensing task","volume":"92","author":"Grasha","year":"2001","journal-title":"Percept. Mot. Skills"},{"key":"2020121100533374700_ref13","article-title":"Methodology for extensive evaluation of semi-automatic and interactive segmentation algorithms using simulated interaction models","author":"Haque","year":"2016"},{"key":"2020121100533374700_ref14","first-page":"904","article-title":"NASA-task load index (NASA-TLX); 20 years later","volume-title":"Proceedings of the human factors and ergonomics society annual meeting","author":"Hart","year":"2006"},{"key":"2020121100533374700_ref15","first-page":"139","article-title":"Development of NASA-TLX (task load index): results of empirical and theoretical research","volume-title":"Adv. Psych.","author":"Hart","year":"1988"},{"key":"2020121100533374700_ref16","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1016\/j.cag.2010.12.006","article-title":"Interactive 3d medical image segmentation with energy-minimizing implicit functions","volume":"35","author":"Heckel","year":"2011","journal-title":"Comput. Graph."},{"key":"2020121100533374700_ref17","doi-asserted-by":"crossref","first-page":"1505","DOI":"10.1109\/TVCG.2009.178","article-title":"Scalable and interactive segmentation and visualization of neural processes in em datasets","volume":"15","author":"Jeong","year":"2009","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"2020121100533374700_ref18","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1007\/s00138-008-0171-x","article-title":"Interactive color image segmentation with linear programming","volume":"21","author":"Li","year":"2010","journal-title":"Mach. Vis. Appl."},{"key":"2020121100533374700_ref19","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1007\/11569541_34","article-title":"Fast 3d brain segmentation using dual-front active contours with optional user-interaction","volume-title":"Int. workshop on computer vision for biomedical image applications","author":"Li","year":"2005"},{"key":"2020121100533374700_ref20","first-page":"416","article-title":"A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics","volume-title":"Proc. 8th int. conf. computer vision","author":"Martin","year":"2001"},{"key":"2020121100533374700_ref21","doi-asserted-by":"crossref","first-page":"434","DOI":"10.1016\/j.patcog.2009.03.008","article-title":"A comparative evaluation of interactive segmentation algorithms","volume":"43","author":"McGuinness","year":"2010","journal-title":"Pattern Recognit."},{"key":"2020121100533374700_ref22","doi-asserted-by":"crossref","first-page":"3083","DOI":"10.1109\/IEMBS.2008.4649855","article-title":"Improved interactive medical image segmentation using enhanced intelligent scissors (eis)","volume-title":"Engineering in medicine and biology society, 2008. EMBS 2008. 30th annual int. conf. of the IEEE","author":"Mishra","year":"2008"},{"key":"2020121100533374700_ref23","first-page":"209","article-title":"Simulation of user interaction for performance evaluation of interactive image segmentation methods","author":"Moschidis","year":"2009","journal-title":"Medical image understanding and analysis"},{"key":"2020121100533374700_ref24","doi-asserted-by":"crossref","first-page":"928","DOI":"10.1109\/ISBI.2010.5490139","article-title":"A systematic performance evaluation of interactive image segmentation methods based on simulated user interaction","volume-title":"2010 IEEE int. symposium on biomedical imaging: from nano to macro","author":"Moschidis","year":"2010"},{"key":"2020121100533374700_ref25","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1016\/j.patcog.2009.03.004","article-title":"Interactive image segmentation by maximal similarity based region merging","volume":"43","author":"Ning","year":"2010","journal-title":"Pattern Recognit."},{"key":"2020121100533374700_ref26","doi-asserted-by":"crossref","first-page":"1159","DOI":"10.1016\/j.patcog.2011.08.017","article-title":"Interactive image segmentation by matching attributed relational graphs","volume":"45","author":"Noma","year":"2012","journal-title":"Pattern Recognit."},{"key":"2020121100533374700_ref27","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1109\/MCSE.2010.97","article-title":"Web2py for scientific applications","volume":"13","author":"Pierro","year":"2011","journal-title":"Comput. Sci. Eng."},{"key":"2020121100533374700_ref28","doi-asserted-by":"crossref","first-page":"751","DOI":"10.1016\/j.ijhcs.2004.09.001","article-title":"Towards a standard for pointing device evaluation, perspectives on 27 years of Fitts\u2019 law research in hci","volume":"61","author":"Soukoreff","year":"2004","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"2020121100533374700_ref29","volume-title":"Evaluation methods of accuracy and reproducibility for image segmentation algorithms","author":"Sun","year":"2018"},{"key":"2020121100533374700_ref30","first-page":"101","article-title":"A critique and improvement of the \u2018CL\u2019 common language effect size statistics of McGraw and Wong","volume":"25","author":"Vargha","year":"2000","journal-title":"J. Educ. Behav. Stat."},{"key":"2020121100533374700_ref31","first-page":"790","article-title":"Active learning for interactive segmentation with expected confidence change","volume-title":"Asian conf. on computer vision","author":"Wang","year":"2012"},{"key":"2020121100533374700_ref32","doi-asserted-by":"crossref","first-page":"998","DOI":"10.1007\/s00330-011-2336-7","article-title":"Novel whole brain segmentation and volume estimation using quantitative MRI","volume":"22","author":"West","year":"2012","journal-title":"Eur. Radiol."},{"key":"2020121100533374700_ref33","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1007\/s12021-018-9385-x","article-title":"User-guided segmentation of multi-modality medical imaging datasets with itk-snap","volume":"17","author":"Yushkevich","year":"2018","journal-title":"Neuroinformatics"}],"container-title":["Interacting with Computers"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/iwc\/article-pdf\/32\/3\/233\/34858456\/iwaa017.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/iwc\/article-pdf\/32\/3\/233\/34858456\/iwaa017.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,28]],"date-time":"2021-03-28T05:58:34Z","timestamp":1616911114000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/iwc\/article\/32\/3\/233\/5890683"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5]]},"references-count":33,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2020,8,11]]},"published-print":{"date-parts":[[2020,11,17]]}},"URL":"https:\/\/doi.org\/10.1093\/iwcomp\/iwaa017","relation":{},"ISSN":["0953-5438","1873-7951"],"issn-type":[{"value":"0953-5438","type":"print"},{"value":"1873-7951","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,5]]},"published":{"date-parts":[[2020,5]]}}}