{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T13:00:03Z","timestamp":1781614803234,"version":"3.54.5"},"reference-count":46,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,4,26]],"date-time":"2022-04-26T00:00:00Z","timestamp":1650931200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>This paper studies the problem of identifying piano music in various modalities using a single, unified approach called marketplace fingerprinting. The key defining characteristic of marketplace fingerprinting is choice: we consider a broad range of fingerprint designs based on a generalization of standard n-grams, and then select the fingerprint designs at runtime that are best for a specific query. We show that the large-scale retrieval problem can be framed as an economics problem in which a consumer and a store interact. In our analogy, the runtime search is like a consumer shopping in the store, the items for sale correspond to fingerprints, and purchasing an item corresponds to doing a fingerprint lookup in the database. Using basic principles of economics, we design an efficient marketplace in which the consumer has many options and adopts a rational buying strategy that explicitly considers the cost and expected utility of each item. We evaluate our marketplace fingerprinting approach on four different sheet music retrieval tasks involving sheet music images, MIDI files, and audio recordings. Using a database containing approximately 375,000 pages of sheet music, our method is able to achieve 0.91 mean reciprocal rank with sub-second average runtime on cell phone image queries. On all four retrieval tasks, the marketplace method substantially outperforms previous methods while simultaneously reducing average runtime. We present comprehensive experimental results, as well as detailed analyses to provide deeper intuition into system behavior.<\/jats:p>","DOI":"10.3390\/a15050146","type":"journal-article","created":{"date-parts":[[2022,4,26]],"date-time":"2022-04-26T11:45:55Z","timestamp":1650973555000},"page":"146","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Large-Scale Multimodal Piano Music Identification Using Marketplace Fingerprinting"],"prefix":"10.3390","volume":"15","author":[{"given":"Daniel","family":"Yang","sequence":"first","affiliation":[{"name":"Department of Engineering, Harvey Mudd College, 301 Platt Blvd, Claremont, CA 91711, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arya","family":"Goutam","sequence":"additional","affiliation":[{"name":"Department of Engineering, Harvey Mudd College, 301 Platt Blvd, Claremont, CA 91711, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kevin","family":"Ji","sequence":"additional","affiliation":[{"name":"Department of Engineering, Harvey Mudd College, 301 Platt Blvd, Claremont, CA 91711, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3832-496X","authenticated-orcid":false,"given":"TJ","family":"Tsai","sequence":"additional","affiliation":[{"name":"Department of Engineering, Harvey Mudd College, 301 Platt Blvd, Claremont, CA 91711, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,4,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1551","DOI":"10.1109\/TMM.2018.2883860","article-title":"Iterative Manifold Embedding Layer Learned by Incomplete Data for Large-Scale Image Retrieval","volume":"21","author":"Xu","year":"2018","journal-title":"IEEE Trans. Multimed."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"3128","DOI":"10.1109\/TMM.2020.2974326","article-title":"A Two-Stage Triplet Network Training Framework for Image Retrieval","volume":"22","author":"Min","year":"2020","journal-title":"IEEE Trans. Multimed."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2199","DOI":"10.1109\/TMM.2021.3065578","article-title":"Unsupervised adversarial instance-level image retrieval","volume":"23","author":"Bai","year":"2021","journal-title":"IEEE Trans. Multimed."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"3178","DOI":"10.1109\/TMM.2019.2920601","article-title":"Deep Progressive Hashing for Image Retrieval","volume":"21","author":"Bai","year":"2019","journal-title":"IEEE Trans. Multimed."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Philbin, J., Chum, O., Isard, M., and Sivic, J. (2007, January 17\u201322). Object Retrieval with Large Vocabularies and Fast Spatial Matching. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.","DOI":"10.1109\/CVPR.2007.383172"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Philbin, J., Chum, O., Isard, M., Sivic, J., and Zisserman, A. (2008, January 23\u201328). Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA.","DOI":"10.1109\/CVPR.2008.4587635"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"4541","DOI":"10.1109\/TMM.2020.3044473","article-title":"Semantic-Driven Interpretable Deep Multi-Modal Hashing for Large-Scale Multimedia Retrieval","volume":"23","author":"Lu","year":"2021","journal-title":"IEEE Trans. Multimed."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1109\/TMM.2020.2984081","article-title":"Mask Cross-Modal Hashing Networks","volume":"23","author":"Lin","year":"2020","journal-title":"IEEE Trans. Multimed."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"559","DOI":"10.1109\/TMM.2020.2985540","article-title":"Augmented Adversarial Training for Cross-Modal Retrieval","volume":"23","author":"Wu","year":"2020","journal-title":"IEEE Trans. Multimed."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"3101","DOI":"10.1109\/TMM.2020.2969792","article-title":"Multi-Level Correlation Adversarial Hashing for Cross-Modal Retrieval","volume":"22","author":"Ma","year":"2020","journal-title":"IEEE Trans. Multimed."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Huiskes, M.J., Thomee, B., and Lew, M.S. (2010, January 29\u201331). New Trends and Ideas in Visual Concept Detection: The MIR Flickr Retrieval Evaluation Initiative. Proceedings of the International Conference on Multimedia Information Retrieval, Philadelphia, PA, USA.","DOI":"10.1145\/1743384.1743475"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Chua, T.S., Tang, J., Hong, R., Li, H., Luo, Z., and Zheng, Y. (2009, January 8\u201310). NUS-WIDE: A Real-World Web Image Database from National University of Singapore. Proceedings of the ACM International Conference on Image and Video Retrieval, Santorini, Fira, Greece.","DOI":"10.1145\/1646396.1646452"},{"key":"ref_13","unstructured":"Thomas, V., Fremerey, C., M\u00fcller, M., and Clausen, M. (2022, March 20). Linking Sheet Music and Audio\u2014Challenges and New Approaches. In Dagstuhl Follow-Ups, Available online: https:\/\/drops.dagstuhl.de\/opus\/volltexte\/2012\/3463\/pdf\/2.pdf."},{"key":"ref_14","unstructured":"Kurth, F., M\u00fcller, M., Fremerey, C., Chang, Y.H., and Clausen, M. (2007, January 23\u201327). Automated Synchronization of Scanned Sheet Music with Audio Recordings. Proceedings of the International Society for Music Information Retrieval Conference, Vienna, Austria."},{"key":"ref_15","unstructured":"Fremerey, C., M\u00fcller, M., Kurth, F., and Clausen, M. (2008, January 14\u201318). Automatic Mapping of Scanned Sheet Music to Audio Recordings. Proceedings of the International Society for Music Information Retrieval Conference, Philadelphia, PA, USA."},{"key":"ref_16","unstructured":"Fremerey, C., Clausen, M., Ewert, S., and M\u00fcller, M. (2009, January 26\u201330). Sheet Music-Audio Identification. Proceedings of the International Society for Music Information Retrieval Conference, Kobe, Japan."},{"key":"ref_17","unstructured":"Dorfer, M., Arzt, A., and Widmer, G. (2017, January 23\u201327). Learning Audio-Sheet Music Correspondences for Score Identification and Offline Alignment. Proceedings of the International Society for Music Information Retrieval Conference, Suzhou, China."},{"key":"ref_18","first-page":"22","article-title":"Learning Audio-Sheet Music Correspondences for Cross-Modal Retrieval and Piece Identification","volume":"1","author":"Dorfer","year":"2018","journal-title":"Trans. Int. Soc. Music. Inf. Retr."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1007\/s13735-018-0151-5","article-title":"End-to-End Cross-Modality Retrieval with CCA Projections and Pairwise Ranking Loss","volume":"7","author":"Dorfer","year":"2018","journal-title":"Int. J. Multimed. Inf. Retr."},{"key":"ref_20","unstructured":"Dorfer, M., Arzt, A., and Widmer, G. (2016, January 7\u201311). Towards Score Following in Sheet Music Images. Proceedings of the International Society for Music Information Retrieval Conference, New York, NY, USA."},{"key":"ref_21","unstructured":"Dorfer, M., Henkel, F., and Widmer, G. (2018, January 23\u201327). Learning to Listen, Read, and Follow: Score following as a Reinforcement Learning Game. Proceedings of the International Society for Music Information Retrieval Conference, Paris, France."},{"key":"ref_22","first-page":"67","article-title":"Score Following as a Multi-Modal Reinforcement Learning Problem","volume":"2","author":"Henkel","year":"2019","journal-title":"Trans. Int. Soc. Music. Inf. Retr."},{"key":"ref_23","unstructured":"Henkel, F., Kelz, R., and Widmer, G. (2020, January 11\u201316). Learning to Read and Follow Music in Complete Score Sheet Images. Proceedings of the International Society for Music Information Retrieval Conference, Montreal, QC, Canada."},{"key":"ref_24","unstructured":"Henkel, F., Kelz, R., and Widmer, G. (2019, January 2). Audio-Conditioned U-Net for Position Estimation in Full Sheet Images. Proceedings of the International Workshop on Reading Music Systems, Delft, The Netherlands."},{"key":"ref_25","unstructured":"Yang, D., Tanprasert, T., Jenrungrot, T., Shan, M., and Tsai, T. (2019, January 4\u20138). MIDI Passage Retrieval Using Cell Phone Pictures of Sheet Music. Proceedings of the International Society for Music Information Retrieval Conference, Delft, The Netherlands."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3115","DOI":"10.1109\/TMM.2020.2973831","article-title":"Using Cell Phone Pictures of Sheet Music To Retrieve MIDI Passages","volume":"22","author":"Tsai","year":"2020","journal-title":"IEEE Trans. Multimed."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Tsai, T. (2020, January 4\u20138). Towards Linking the Lakh and IMSLP Datasets. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Toronto, ON, Canada.","DOI":"10.1109\/ICASSP40776.2020.9053815"},{"key":"ref_28","first-page":"42","article-title":"Piano Sheet Music Identification Using Dynamic N-gram Fingerprinting","volume":"4","author":"Yang","year":"2021","journal-title":"Trans. Int. Soc. Music. Inf. Retr."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Malik, R., Roy, P.P., Pal, U., and Kimura, F. (2013, January 23). Handwritten Musical Document Retrieval Using Music-Score Spotting. Proceedings of the IEEE International Conference on Document Analysis and Recognition, Washington, DC, USA.","DOI":"10.1109\/ICDAR.2013.170"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Calvo-Zaragoza, J., Toselli, A.H., and Vidal, E. (2018, January 5\u20138). Probabilistic Music-Symbol Spotting in Handwritten Scores. Proceedings of the IEEE International Conference on Frontiers in Handwriting Recognition, Niagara Falls, NY, USA.","DOI":"10.1109\/ICFHR-2018.2018.00103"},{"key":"ref_31","unstructured":"Thompson, J., Hankinson, A., and Fujinaga, I. (2011, January 24\u201328). Searching the Liber Usualis: Using CouchDB and ElasticSearch to Query Graphical Music Documents. Proceedings of the International Society for Music Information Retrieval Conference, Miami, FL, USA."},{"key":"ref_32","unstructured":"Achankunju, S.P. (2018, January 20). Music Search Engine from Noisy OMR Data. Proceedings of the International Workshop on Reading Music Systems, Paris, France."},{"key":"ref_33","unstructured":"Yang, D., and Tsai, T. (2020, January 11\u201316). Camera-Based Piano Sheet Music Identification. Proceedings of the International Society for Music Information Retrieval Conference, Montreal, QC, Canada."},{"key":"ref_34","unstructured":"Haji\u010d, J., Kol\u00e1rov\u00e1, M., Pacha, A., and Calvo-Zaragoza, J. (2018, January 28). How Current Optical Music Recognition Systems Are Becoming Useful For Digital Libraries. Proceedings of the 5th International Conference on Digital Libraries for Musicology, Paris, France."},{"key":"ref_35","unstructured":"Waloschek, S., Hadjakos, A., and Pacha, A. (2019, January 4\u20138). Identification and Cross-Document Alignment of Measures in Music Score Images. Proceedings of the International Society for Music Information Retrieval Conference, Delft, The Netherlands."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1984","DOI":"10.1109\/TMM.2017.2723846","article-title":"Audio Identification by Sampling Sub-Fingerprints and Counting Matches","volume":"19","author":"Yao","year":"2017","journal-title":"IEEE Trans. Multimed."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1569","DOI":"10.1109\/TMM.2017.2669864","article-title":"Known-Artist Live Song Identification Using Audio Hashprints","volume":"19","author":"Tsai","year":"2017","journal-title":"IEEE Trans. Multimed."},{"key":"ref_38","unstructured":"Ji, K., Yang, D., and Tsai, T. (2021, January 7\u201312). Piano Sheet Music Identification Using Marketplace Fingerprinting. Proceedings of the International Society for Music Information Retrieval Conference, Online."},{"key":"ref_39","unstructured":"Shan, M., and Tsai, T. (2020, January 11\u201316). Improved Handling of Repeats and Jumps in Audio-Sheet Image Synchronization. Proceedings of the International Society for Music Information Retrieval Conference, Montreal, QC, Canada."},{"key":"ref_40","first-page":"29","article-title":"Automatic Generation of Piano Score Following Videos","volume":"4","author":"Shan","year":"2021","journal-title":"Trans. Int. Soc. Music. Inf. Retr."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1270","DOI":"10.1109\/5.880083","article-title":"Two Decades of Statistical Language Modeling: Where Do We Go From Here?","volume":"88","author":"Rosenfeld","year":"2000","journal-title":"Proc. IEEE"},{"key":"ref_42","unstructured":"Wang, A. (2003, January 27\u201330). An Industrial Strength Audio Search Algorithm. Proceedings of the International Society for Music Information Retrieval Conference, Baltimore, MD, USA."},{"key":"ref_43","unstructured":"Hawthorne, C., Elsen, E., Song, J., Roberts, A., Simon, I., Raffel, C., Engel, J., Oore, S., and Eck, D. (2018, January 23\u201327). Onsets and Frames: Dual-Objective Piano Transcription. Proceedings of the International Society for Music Information Retrieval Conference, Paris, France."},{"key":"ref_44","unstructured":"Hawthorne, C., Stasyuk, A., Roberts, A., Simon, I., Huang, C.Z.A., Dieleman, S., Elsen, E., Engel, J., and Eck, D. (2019, January 6\u20139). Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_45","unstructured":"Gionis, A., Indyk, P., and Motwani, R. (1999, January 7\u201310). Similarity Search In High Dimensions Via Hashing. Proceedings of the Very Large Database (VLDB) Conference, Scotland, UK."},{"key":"ref_46","unstructured":"Jones, K.S. (2022, March 20). A Statistical Interpretation of Term Specificity and Its Application in Retrieval. J. Doc., Available online: http:\/\/courses.cs.tau.ac.il\/0368-4341\/shared\/Papers\/ksj_orig.pdf."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/5\/146\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:01:15Z","timestamp":1760137275000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/5\/146"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,26]]},"references-count":46,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,5]]}},"alternative-id":["a15050146"],"URL":"https:\/\/doi.org\/10.3390\/a15050146","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,26]]}}}