{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T18:26:56Z","timestamp":1776104816031,"version":"3.50.1"},"publisher-location":"Cham","reference-count":33,"publisher":"Springer International Publishing","isbn-type":[{"value":"9783031159107","type":"print"},{"value":"9783031159114","type":"electronic"}],"license":[{"start":{"date-parts":[[2022,1,1]],"date-time":"2022-01-01T00:00:00Z","timestamp":1640995200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,9,3]],"date-time":"2022-09-03T00:00:00Z","timestamp":1662163200000},"content-version":"vor","delay-in-days":245,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Bugs, misconfiguration, and malware can cause ballot-marking devices (BMDs) to print incorrect votes. Several approaches to testing BMDs have been proposed. In <jats:italic>logic and accuracy testing<\/jats:italic> (LAT) and <jats:italic>parallel<\/jats:italic> or <jats:italic>live<\/jats:italic> testing, auditors input known test votes into the BMD and check whether the printout matches. <jats:italic>Passive<\/jats:italic> testing monitors the rate at which voters \u201cspoil\u201d BMD printout, on the theory that if BMDs malfunction, the rate will increase noticeably. We provide lower bounds that show that these approaches cannot reliably detect outcome-altering problems, because: (i)\u00a0The number of possible voter interactions with BMDs is enormous, so testing interactions uniformly at random is hopeless. (ii)\u00a0To probe the space of interactions intelligently requires an accurate model of voter behavior, but because the space of interactions is so large, building a sufficiently accurate model requires observing an enormous number of voters in every jurisdiction in every election\u2014more voters than there are in most U.S. jurisdictions. (iii)\u00a0Even with a perfect model of voter behavior, the required number of tests exceeds the number of voters in most U.S. jurisdictions. (iv)\u00a0An attacker can target interactions that are intrinsically expensive to test, e.g., because they involve voting slowly; or interactions for which tampering is less likely to be noticed, e.g., because the voter uses the audio interface. (v)\u00a0Whether BMDs misbehave or not, the distribution of spoiled ballots is unknown and varies by election and possibly by ballot style: historical data do not help much. Hence, there is no way to calibrate a threshold for passive testing, e.g., to guarantee at least a 95% chance of noticing that 5% of the votes were altered, with at most a 5% false alarm rate. (vi)\u00a0Even if the distribution of spoiled ballots were known to be Poisson, the vast majority of jurisdictions do not have enough voters for passive testing to have a large chance of detecting problems but only a small chance of false alarms.<\/jats:p>","DOI":"10.1007\/978-3-031-15911-4_8","type":"book-chapter","created":{"date-parts":[[2022,9,2]],"date-time":"2022-09-02T14:04:29Z","timestamp":1662127469000},"page":"122-138","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["They May Look and\u00a0Look, Yet\u00a0Not See: BMDs Cannot be Tested Adequately"],"prefix":"10.1007","author":[{"given":"Philip B.","family":"Stark","sequence":"first","affiliation":[]},{"given":"Ran","family":"Xie","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,9,3]]},"reference":[{"key":"8_CR1","doi-asserted-by":"publisher","first-page":"432","DOI":"10.1089\/elj.2019.0619","volume":"19","author":"A Appel","year":"2020","unstructured":"Appel, A., DeMillo, R., Stark, P.: Ballot-marking devices (BMDs) cannot assure the will of the voters. Election Law J. 19, 432\u2013450 (2020). https:\/\/doi.org\/10.1089\/elj.2019.0619","journal-title":"Election Law J."},{"key":"8_CR2","unstructured":"Appel, A., Stark, P.: Evidence-based elections: create a meaningful paper trail, then audit. Georgetown Law Technol. J. 4(2), 523\u2013541 (2020). https:\/\/georgetownlawtechreview.org\/wp-content\/uploads\/2020\/07\/4.2-p523-541-Appel-Stark.pdf"},{"key":"8_CR3","doi-asserted-by":"publisher","unstructured":"Bernhard, M., et al.: Can voters detect malicious manipulation of ballot marking devices? In: 41st IEEE Symposium on Security and Privacy, pp. 679\u2013694. IEEE (2020). https:\/\/doi.org\/10.1109\/SP40000.2020.00118","DOI":"10.1109\/SP40000.2020.00118"},{"key":"8_CR4","unstructured":"Cillizza, C.: How did Georgia get it so wrong (again)? (2020). https:\/\/www.cnn.com\/2020\/06\/10\/politics\/georgia-primary-vote-brian-kemp\/index.html"},{"key":"8_CR5","doi-asserted-by":"publisher","unstructured":"DeMillo, R., Kadel, R., Marks, M.: What voters are asked to verify affects ballot verification: a quantitative analysis of voters\u2019 memories of their ballots (2018). https:\/\/doi.org\/10.2139\/ssrn.3292208","DOI":"10.2139\/ssrn.3292208"},{"key":"8_CR6","unstructured":"Fowler, S.: State outlines fix for error that halted election testing. Georgia Public Broadcasting (2020). https:\/\/www.gpb.org\/news\/2020\/09\/29\/state-outlines-fix-for-error-halted-election-testing"},{"key":"8_CR7","unstructured":"Gilbert, J.: Ballot marking verification protocol. http:\/\/www.juangilbert.com\/BallotMarkingVerificationProtocol.pdf (2019)"},{"key":"8_CR8","doi-asserted-by":"crossref","unstructured":"Han, Y., Jiao, J., Weissman, T.: Minimax estimation of discrete distributions. In: 2015 IEEE International Symposium on Information Theory (ISIT), pp. 2291\u20132295. IEEE (2015)","DOI":"10.1109\/ISIT.2015.7282864"},{"key":"8_CR9","unstructured":"Harte, J.: Exclusive: Philadelphia\u2019s new voting machines under scrutiny in Tuesday\u2019s elections. Reuters (2020). https:\/\/in.reuters.com\/article\/usa-election-pennsylvania-machines\/exclusive-philadelphias-new-voting-machines-under-scrutiny-in-tuesdays-elections-idINKBN2382D2"},{"key":"8_CR10","unstructured":"Haynes, A., III, M.H.: Georgia voter verification study. https:\/\/s3.documentcloud.org\/documents\/21017815\/gvvs-report-11.pdf (2021). Accessed 31 Oct 2021"},{"key":"8_CR11","unstructured":"Hotten, R.: Volkswagen: the scandal explained. https:\/\/www.bbc.com\/news\/business-34324772 (2015)"},{"key":"8_CR12","unstructured":"Kortum, P., Byrne, M., Azubike, C., Roty, L.: Can voters detect errors on their printed ballots? Absolutely. https:\/\/arxiv.org\/abs\/2204.09780 (2022)"},{"key":"8_CR13","doi-asserted-by":"publisher","unstructured":"Kortum, P., Byrne, M., Whitmore, J.: Voter verification of ballot marking device ballots is a two-part question: can they? mostly, they can. do they? mostly, they don\u2019t. Election Law J. Rules Polit. Policy 243\u2013253 (2021). https:\/\/doi.org\/10.1089\/elj.2020.0632","DOI":"10.1089\/elj.2020.0632"},{"key":"8_CR14","unstructured":"Los Angeles county clerk: multilingual services program (2020). https:\/\/www.lavote.net\/home\/voting-elections\/voter-education\/multilingual-services-program\/multilingual-services-program"},{"key":"8_CR15","unstructured":"Mehrotra, K., Newkirk, M.: Expensive, glitchy voting machines expose 2020 hacking risks (2019). https:\/\/www.bloomberg.com\/news\/articles\/2019-11-08\/expensive-glitchy-voting-machines-expose-2020-hacking-risks"},{"key":"8_CR16","unstructured":"National federation of the blind: blind statistics (2019). https:\/\/www.nfb.org\/resources\/blindness-statistics"},{"key":"8_CR17","unstructured":"Perez, E., Miller, G.: Georgia state election technology acquisition: a reality check. https:\/\/trustthevote.org\/wp-content\/uploads\/2019\/03\/06Mar19-OSETBriefing_GeorgiaSystemsCostAnalysis.pdf (2019)"},{"key":"8_CR18","unstructured":"Previti, E.: Northampton officials unanimously vote \u2018no confidence\u2019 in ExpressVote XL voting machine (2019). https:\/\/papost.org\/2019\/12\/20\/northampton-officials-unanimously-vote-no-confidence-in-expressvote-xl-voting-machine\/"},{"key":"8_CR19","unstructured":"Quesenbery, W.: Why not just use pens to mark a ballot? (2018). https:\/\/civicdesign.org\/why-not-just-use-pens-to-mark-a-ballot\/"},{"key":"8_CR20","unstructured":"Riggall, H.: Up to 157 incorrect ballots cast on first day of early voting, cobb elections director says. Marietta Daily Journal, Ga (2022). https:\/\/news.yahoo.com\/157-incorrect-ballots-cast-first-092000088.html"},{"issue":"1881","key":"8_CR21","doi-asserted-by":"publisher","first-page":"3759","DOI":"10.1098\/rsta.2008.0149","volume":"366","author":"R Rivest","year":"2008","unstructured":"Rivest, R.: On the notion of \u2018software independence\u2019 in voting systems. Phil. Trans. R. Soc. A 366(1881), 3759\u20133767 (2008)","journal-title":"Phil. Trans. R. Soc. A"},{"key":"8_CR22","unstructured":"Secretary of the commonwealth of Pennsylvania: report concerning the examination results of election systems and software EVS 6012 with DS200 precinct scanner, DS450 and DS850 central scanners, ExpressVote HW 2.1 marker and tabulator, ExpressVote XL tabulator and electionware EMS. https:\/\/www.dos.pa.gov\/VotingElections\/Documents\/Voting%20Systems\/ESS%20EVS%206021\/EVS%206021%20Secretary%27s%20Report%20Signed%20-%20Including%20Attachments.pdf"},{"key":"8_CR23","unstructured":"Shortell, T., Tatu, C.: Here\u2019s why northampton county\u2019s voting machines went wrong, county executive says. The Morning Call 12 December 2019 (2019)"},{"key":"8_CR24","unstructured":"Sneed, T.: Will L.A.\u2019s voting overhaul be an industry disrupter or the next election debacle? (2020). https:\/\/talkingpointsmemo.com\/news\/will-l-a-s-voting-overhaul-be-an-industry-disrupter-or-the-next-election-debacle"},{"key":"8_CR25","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1109\/MSP.2012.62","volume":"10","author":"P Stark","year":"2012","unstructured":"Stark, P., Wagner, D.: Evidence-based elections. IEEE Secur. Priv. 10, 33\u201341 (2012)","journal-title":"IEEE Secur. Priv."},{"key":"8_CR26","unstructured":"U.S. Census Bureau: city and town population totals: 2010\u20132020 (2020). https:\/\/www2.census.gov\/programs-surveys\/popest\/datasets\/2010-2020\/cities\/"},{"key":"8_CR27","unstructured":"U.S. Election Assistance Commission: voluntary voting systems guidelines 1.0, December 2005. https:\/\/www.eac.gov\/sites\/default\/files\/eac_assets\/1\/28\/VVSG.1.0_Volume_1.PDF"},{"key":"8_CR28","unstructured":"U.S. Election Assistance Commission: voluntary voting system guidlines version 1.1. U.S. Election Assistance Commission (2015). https:\/\/www.eac.gov\/sites\/default\/files\/eac_assets\/1\/28\/VVSG.1.1.VOL.1.FINAL1.pdf"},{"key":"8_CR29","unstructured":"U.S. Election Assistance Commission: The U.S. election assistance commission\u2019s 2020 election administration and voting survey (EAVS) (2020). https:\/\/eavsportal.com\/"},{"key":"8_CR30","unstructured":"U.S. Election Assistance Commission: report of investigation, dominion voting systems D-suite 5.5-B. Williamson County, Tennessee. https:\/\/www.eac.gov\/sites\/default\/files\/TestingCertification\/EAC_Report_of_Investigation_Dominion_DSuite_5.5_B.pdf (2022)"},{"key":"8_CR31","unstructured":"U.S. Senate Intelligence Committee: Russian targeting of election infrastructure during the 2016 election: summary of initial findings and recommendations. https:\/\/www.burr.senate.gov\/imo\/media\/doc\/RussRptInstlmt1-%20ElecSec%20Findings,Recs2.pdf (2018). Accessed 3 June 2020"},{"issue":"2","key":"8_CR32","first-page":"558","volume":"16","author":"D Wallach","year":"2020","unstructured":"Wallach, D.: On the security of ballot marking devices. Ohio State Technol. Law J. 16(2), 558\u2013586 (2020)","journal-title":"Ohio State Technol. Law J."},{"key":"8_CR33","unstructured":"Zetter, K.: Los Angeles County\u2019s risky voting experiment (2020). https:\/\/www.politico.com\/news\/2020\/03\/03\/los-angeles-county-voting-experiment-119157"}],"container-title":["Lecture Notes in Computer Science","Electronic Voting"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-15911-4_8","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,26]],"date-time":"2022-09-26T23:04:50Z","timestamp":1664233490000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-15911-4_8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"ISBN":["9783031159107","9783031159114"],"references-count":33,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-15911-4_8","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022]]},"assertion":[{"value":"3 September 2022","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"E-Vote-ID","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Joint Conference on Electronic Voting","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Bregenz","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Austria","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2022","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"4 October 2022","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"7 October 2022","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"7","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"evoteid2022","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/e-vote-id.org\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Double-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"EasyChair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"30","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"10","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"0","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"33% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"4","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"4","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"No","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}