{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T20:13:18Z","timestamp":1784837598664,"version":"3.55.0"},"reference-count":43,"publisher":"SAGE Publications","issue":"9","license":[{"start":{"date-parts":[[2023,12,17]],"date-time":"2023-12-17T00:00:00Z","timestamp":1702771200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"name":"NASA University Leadership Initiative","award":["80NSSC20M016"],"award-info":[{"award-number":["80NSSC20M016"]}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of Robotics Research"],"published-print":{"date-parts":[[2024,8]]},"abstract":"<jats:p>\n                    When deploying machine learning models in high-stakes robotics applications, the ability to detect unsafe situations is crucial. Early warning systems can provide alerts when an unsafe situation is imminent (in the absence of corrective action). To reliably improve safety, these warning systems should have a\n                    <jats:italic>provable<\/jats:italic>\n                    false negative rate; that is, of the situations that are unsafe, fewer than\n                    <jats:italic>\u03f5<\/jats:italic>\n                    will occur without an alert. In this work, we present a framework that combines a statistical inference technique known as conformal prediction with a simulator of robot\/environment dynamics, in order to tune warning systems to provably achieve an\n                    <jats:italic>\u03f5<\/jats:italic>\n                    false negative rate using as few as 1\/\n                    <jats:italic>\u03f5<\/jats:italic>\n                    data points. We apply our framework to a driver warning system and a robotic grasping application, and empirically demonstrate the guaranteed false negative rate while also observing a low false detection (positive) rate.\n                  <\/jats:p>","DOI":"10.1177\/02783649231221580","type":"journal-article","created":{"date-parts":[[2023,12,17]],"date-time":"2023-12-17T22:57:10Z","timestamp":1702853830000},"page":"1409-1424","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":14,"title":["Sample-efficient safety assurances using conformal prediction"],"prefix":"10.1177","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8296-4994","authenticated-orcid":false,"given":"Rachel","family":"Luo","sequence":"first","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shengjia","family":"Zhao","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan","family":"Kuck","sequence":"additional","affiliation":[{"name":"Dexterity, Inc., Redwood City, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Boris","family":"Ivanovic","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Silvio","family":"Savarese","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Edward","family":"Schmerling","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marco","family":"Pavone","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2023,12,17]]},"reference":[{"key":"e_1_3_5_2_1","unstructured":"Angelopoulos AN Bates S Malik J et al. (2020) Uncertainty sets for image classifiers using conformal prediction. International Conference on Learning Representations Vienna Austria May 7th 2024 to May 11th 2024."},{"key":"e_1_3_5_3_1","volume-title":"Recommendation systems with distribution-free reliability guarantees","author":"Angelopoulos AN","year":"2022","unstructured":"Angelopoulos AN, Krauth K, Bates S, et al. (2022) Recommendation systems with distribution-free reliability guarantees. Ithaca, NY: ArXiv preprint."},{"key":"e_1_3_5_4_1","doi-asserted-by":"publisher","DOI":"10.1214\/23-AOS2276"},{"key":"e_1_3_5_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01164"},{"key":"e_1_3_5_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCPS48487.2020.00024"},{"key":"e_1_3_5_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAC.2006.875041"},{"key":"e_1_3_5_8_1","unstructured":"Cauchois M Gupta S Ali A et al. (2020) Robust validation: confident predictions even when distributions shift. Ithaca NY: ArXiv. https:\/\/arxiv.org\/pdf\/2008.04267.pdf."},{"key":"e_1_3_5_9_1","unstructured":"Chen Y Rosolia U Fan C et al. (2020) Reactive motion planning with probabilistic safety guarantees. In: Conference on robot learning. Ithaca NY: ArXiv."},{"key":"e_1_3_5_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASE.2016.2600527"},{"key":"e_1_3_5_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2014.12.015"},{"key":"e_1_3_5_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4471-4799-2_1"},{"key":"e_1_3_5_13_1","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2016.XII.036"},{"key":"e_1_3_5_14_1","unstructured":"Feldman S Bates S Romano Y (2021) Improving conditional coverage via orthogonal quantile regression. In: Conference on Neural Information Processing Systems Canada 6\u201314 December 2021."},{"key":"e_1_3_5_15_1","doi-asserted-by":"publisher","DOI":"10.1080\/01431160903130937"},{"key":"e_1_3_5_16_1","doi-asserted-by":"publisher","DOI":"10.2202\/1544-6115.1385"},{"key":"e_1_3_5_17_1","unstructured":"Ghosh S Belkhouja T Yan Y et al. (2023) Improving uncertainty quantification of deep classifiers via neighborhood conformal prediction: novel algorithm and theoretical analysis. In: AAAI Conference on Artificial Intelligence Vancouver British Columbia 20\u201327 February 2024."},{"key":"e_1_3_5_18_1","unstructured":"Gibbs I Cand\u00e8s EJ (2021) Conformal inference for online prediction with arbitrary distribution shifts. In: Conference on Neural Information Processing Systems New Orleans Louisiana Dec 10\u201316. 2021."},{"key":"e_1_3_5_19_1","unstructured":"Gupta V Jung C Noarov G et al. (2022) Online multivalid learning: means moments and prediction intervals. In: Innovations in Theoretical Computer Science Conference California January 30 to Friday February 2 2024."},{"issue":"27","key":"e_1_3_5_20_1","first-page":"260","article-title":"Model invalidation for switched affine systems with applications to fault and anomaly detection","volume":"48","author":"Harirchi F","year":"2015","unstructured":"Harirchi F, Ozay N (2015) Model invalidation for switched affine systems with applications to fault and anomaly detection. Analysis and Design of Hybrid Systems 48(27): 260\u2013266.","journal-title":"Analysis and Design of Hybrid Systems"},{"key":"e_1_3_5_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.automatica.2018.03.040"},{"key":"e_1_3_5_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-68792-6_51"},{"key":"e_1_3_5_23_1","volume-title":"Lyft motion prediction for autonomous vehicles","author":"Jang A","year":"2020","unstructured":"Jang A, Christy, Bergamini L, et al. (2020) Lyft motion prediction for autonomous vehicles. San Francisco: Kaggle. https:\/\/kaggle.com\/competitions\/lyft-motion-prediction-autonomous-vehicles."},{"key":"e_1_3_5_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3146389"},{"key":"e_1_3_5_25_1","volume-title":"Workshop on the algorithmic foundations of robotics","author":"Luo R","year":"2022","unstructured":"Luo R, Zhao S, Kuck J, et al. (2022) Sample-efficient safety assurances using conformal prediction. In: Workshop on the algorithmic foundations of robotics. London: Springer Nature."},{"key":"e_1_3_5_26_1","doi-asserted-by":"crossref","unstructured":"Mahler J Liang J Niyaz S et al. (2017) Dex-net 2.0: deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. In: Robotics: Science and Systems (RSS) Delft Netherlands Jul 15 \u2013 Jul 19 2024.","DOI":"10.15607\/RSS.2017.XIII.058"},{"key":"e_1_3_5_27_1","doi-asserted-by":"crossref","unstructured":"Mahler J Matl M Liu X et al. (2018) Dex-net 3.0: computing robust robot suction grasp targets in point clouds using a new analytic model and deep learning. In: IEEE International Conference on Robotics and Automation Yokohama Japan 13 May\u201317 May 2024.","DOI":"10.1109\/ICRA.2018.8460887"},{"key":"e_1_3_5_28_1","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aau4984"},{"key":"e_1_3_5_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIE.2011.2167110"},{"key":"e_1_3_5_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2010.05.023"},{"key":"e_1_3_5_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0967-0661(97)00049-X"},{"key":"e_1_3_5_32_1","unstructured":"Perdomo J Zrnic T Mendler-D\u00fcnner C et al. (2020) Performative prediction. In: International Conference on Machine Learning Honolulu Hawaii Jul 23\u2013Jul 29."},{"key":"e_1_3_5_33_1","doi-asserted-by":"crossref","unstructured":"Salzmann T Ivanovic B Chakravarty P et al. (2020) Trajectron++: dynamically-feasible trajectory forecasting with heterogeneous data. In: European Conference on Computer Vision Tel Aviv Israel October 23\u201327 2020.","DOI":"10.1007\/978-3-030-58523-5_40"},{"key":"e_1_3_5_34_1","first-page":"371","article-title":"A tutorial on conformal prediction","volume":"9","author":"Shafer G","year":"2008","unstructured":"Shafer G, Vovk V (2008) A tutorial on conformal prediction. Journal of Machine Learning Research 9: 371\u2013421.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_5_35_1","unstructured":"Tibshirani RJ Barber RF Cand\u00e8s EJ et al. (2019) Conformal prediction under covariate shift. In: Conference on Neural Information Processing Systems New Orleans Louisiana December 10 to 16 2019."},{"key":"e_1_3_5_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/70.681254"},{"key":"e_1_3_5_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/0045-7906(94)90035-3"},{"key":"e_1_3_5_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/0951-8320(94)90132-5"},{"key":"e_1_3_5_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/70.406930"},{"key":"e_1_3_5_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-444-52936-7.50016-1"},{"key":"e_1_3_5_41_1","volume-title":"Mondrian confidence machine","author":"Vovk V","year":"2003","unstructured":"Vovk V, Lindsay D, Nouretdinov I, et al. (2003) Mondrian confidence machine. Available at: https:\/\/alrw.net\/old\/04.pdf."},{"key":"e_1_3_5_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/b106715"},{"key":"e_1_3_5_43_1","volume-title":"A summary of team MIT\u2019s approach to the amazon picking challenge 2015","author":"Yu KT","year":"2016","unstructured":"Yu KT, Fazeli N, Chavan-Dafle N, et al. (2016) A summary of team MIT\u2019s approach to the amazon picking challenge 2015. Arxiv. https:\/\/arxiv.org\/abs\/1604.03639."},{"key":"e_1_3_5_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2017.7989165"}],"container-title":["The International Journal of Robotics Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02783649231221580","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/02783649231221580","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02783649231221580","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T10:17:20Z","timestamp":1777457840000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/02783649231221580"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,17]]},"references-count":43,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2024,8]]}},"alternative-id":["10.1177\/02783649231221580"],"URL":"https:\/\/doi.org\/10.1177\/02783649231221580","relation":{},"ISSN":["0278-3649","1741-3176"],"issn-type":[{"value":"0278-3649","type":"print"},{"value":"1741-3176","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,17]]}}}