{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T21:58:34Z","timestamp":1790891914657,"version":"4.1.0"},"reference-count":28,"publisher":"SPIE-Intl Soc Optical Eng","issue":"04","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Electron. Imag."],"published-print":{"date-parts":[[2001,10,1]]},"abstract":"<jats:p>A necessary capability for content-based retrieval is to support the paradigm of query by example. In the past, there have been several attempts to use low-level features for video retrieval. However, most of the approaches support queries using image sequences only. We present an algorithm for matching multimodal (audio-visual) patterns for the purpose of content-based video retrieval. The novel ability of our approach to use the information content in multiple media coupled with a strong emphasis on temporal similarity differentiates it from the state of the art in content-based retrieval. At the core of the pattern matching scheme is a dynamic programming algorithm, which leads to a significant improvement in performance. Coupling the use of audio with video this algorithm can be applied to grouping of shots based on audio-visual similarity. We also support relevance feedback. The user can provide feedback to the system, by choosing clips, which are closer to the user\u2019s desired target. The system then automatically adjusts the relative weights or relevance of the media and fetches different sets of target clips accordingly. It is our observation that a few iterations of such feedback are generally sufficient, for retrieving the desired video clips.<\/jats:p>","DOI":"10.1117\/1.1407822","type":"journal-article","created":{"date-parts":[[2002,7,26]],"date-time":"2002-07-26T10:00:39Z","timestamp":1027677639000},"page":"861-870","source":"Crossref","is-referenced-by-count":8,"title":["Audio-visual query and retrieval: a system that uses dynamic programming and relevance feedback"],"prefix":"10.1117","volume":"10","author":[{"given":"Milind R.","family":"Naphade","sequence":"first","affiliation":[{"name":"IBM Thomas J. Watson Research Ctr."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roy R.","family":"Wang","sequence":"additional","affiliation":[{"name":"Univ. of Illinois\/Urbana-Champaign"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas S.","family":"Huang","sequence":"additional","affiliation":[{"name":"Univ. of Illinois\/Urbana-Champaign"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"189","reference":[{"key":"10.1117\/1.1407822_r1","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1109\/6046.909601","volume":"3","author":"Naphade","year":"2001","journal-title":"IEEE Trans. Multimedia","ISSN":"https:\/\/id.crossref.org\/issn\/1520-9210","issn-type":"print"},{"key":"10.1117\/1.1407822_r2","doi-asserted-by":"crossref","unstructured":"M. M. Yeung and B. Liu, \u201cEfficient matching and clustering of video shots,\u201d in\n                      Proceedings of IEEE International Conference on Image Processing\n                      , Vol. 1, pp. 338\u2013341, Washington, DC (Oct. 1995).","DOI":"10.1109\/ICIP.1995.529715"},{"key":"10.1117\/1.1407822_r3","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1109\/2.410146","volume":"28","author":"Flickner","year":"1995","journal-title":"IEEE Comput."},{"key":"10.1117\/1.1407822_r4","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1117\/12.234785","volume":"2670","author":"Bach","year":"1996","journal-title":"Proc. SPIE","ISSN":"https:\/\/id.crossref.org\/issn\/0277-786X","issn-type":"print"},{"key":"10.1117\/1.1407822_r5","doi-asserted-by":"crossref","first-page":"644","DOI":"10.1109\/76.718510","volume":"8","author":"Rui","year":"1998","journal-title":"IEEE Trans. Circuits Syst. Video Technol.","ISSN":"https:\/\/id.crossref.org\/issn\/1051-8215","issn-type":"print"},{"key":"10.1117\/1.1407822_r6","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1007\/s005300050121","volume":"7","author":"Ma","year":"1999","journal-title":"Multimedia Syst.","ISSN":"https:\/\/id.crossref.org\/issn\/0942-4962","issn-type":"print"},{"key":"10.1117\/1.1407822_r7","doi-asserted-by":"crossref","unstructured":"J. R. Smith and S. F. Chang, \u201cVisualseek: A fully automated content-based image query system,\u201d in\n                      Proceedings of ACM Multimedia\n                      , pp. 87\u201398, Boston, MA (Nov. 1996).","DOI":"10.1145\/244130.244151"},{"key":"10.1117\/1.1407822_r8","doi-asserted-by":"crossref","first-page":"564","DOI":"10.1117\/12.373590","volume":"3972","author":"Naphade","year":"2000","journal-title":"Proc. SPIE","ISSN":"https:\/\/id.crossref.org\/issn\/0277-786X","issn-type":"print"},{"key":"10.1117\/1.1407822_r9","doi-asserted-by":"crossref","unstructured":"S. Srinivasan, D. Ponceleon, A. Amir, and D. Petkovic, \u201cWhat is that video anyway? In search of better browsing,\u201d in\n                      Proceedings of IEEE International Conference on Multimedia and Expo\n                      , pp. 388\u2013392, New York (July 2000).","DOI":"10.1109\/MMCS.1999.779235"},{"key":"10.1117\/1.1407822_r10","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1007\/s005300050139","volume":"7","author":"Jain","year":"1999","journal-title":"Multimedia Systems, Special Issue on Video Libraries"},{"key":"10.1117\/1.1407822_r11","doi-asserted-by":"crossref","unstructured":"M. Naphade, T. Kristjansson, B. Frey, and T. S. Huang, \u201cProbabilistic multimedia objects (multijects): A novel approach to indexing and retrieval in multimedia systems,\u201d in\n                      Proceedings of IEEE International Conference on Image Processing\n                      , Vol. 3, pp. 536\u2013540, Chicago, IL (Oct. 1998).","DOI":"10.1109\/ICIP.1998.999041"},{"key":"10.1117\/1.1407822_r12","unstructured":"M. R. Naphade, I. Kozintsev, and T. S. Huang, \u201cOn probabilistic semantic video indexing,\u201d in\n                      Proceedings of Neural Information Processing Systems\n                      , Vol. 13, pp. 967\u2013973, Denver (Nov. 2000)."},{"key":"10.1117\/1.1407822_r13","doi-asserted-by":"crossref","unstructured":"M. Naphade, I. Kozintsev, T. Huang, and K. Ramchandran, \u201cA factor graph framework for semantic indexing and retrieval in video,\u201d in\n                      Proceedings of Workshop on Content Based Access to Image and Video Libraries Held in Conjunction with CVPR\n                      , pp. 35\u201339, Hilton Head, SC (June 2000).","DOI":"10.1109\/IVL.2000.853836"},{"key":"10.1117\/1.1407822_r14","doi-asserted-by":"crossref","unstructured":"M. R. Naphade and T. S. Huang, \u201cSemantic video indexing using a probabilistic framework,\u201d in\n                      Proceedings of IAPR International Conference on Pattern Recognition\n                      , Vol. 3, pp. 83\u201388, Barcelona, Spain (Sept. 2000).","DOI":"10.1109\/ICPR.2000.903490"},{"key":"10.1117\/1.1407822_r15","doi-asserted-by":"crossref","unstructured":"S. F. Chang, W. Chen, and H. Sundaram, \u201cSemantic visual templates\u2014linking features to semantics,\u201d in Ref. 11, pp. 531\u2013535.","DOI":"10.1109\/ICIP.1998.727321"},{"key":"10.1117\/1.1407822_r16","doi-asserted-by":"crossref","first-page":"506","DOI":"10.1117\/12.373583","volume":"3972","author":"Zhang","year":"2000","journal-title":"Proc. SPIE","ISSN":"https:\/\/id.crossref.org\/issn\/0277-786X","issn-type":"print"},{"key":"10.1117\/1.1407822_r17","unstructured":"N. Vasconcelos and A. Lippman, \u201cBayesian modeling of video editing and structure: Semantic features for video summarization and browsing,\u201d in Ref. 11, Vol. 2, pp. 550\u2013555."},{"key":"10.1117\/1.1407822_r18","doi-asserted-by":"crossref","first-page":"332","DOI":"10.1117\/12.373565","volume":"3972","author":"Kobla","year":"2000","journal-title":"Proc. SPIE","ISSN":"https:\/\/id.crossref.org\/issn\/0277-786X","issn-type":"print"},{"key":"10.1117\/1.1407822_r19","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1117\/12.263406","volume":"3022","author":"Saur","year":"1997","journal-title":"Proc. SPIE","ISSN":"https:\/\/id.crossref.org\/issn\/0277-786X","issn-type":"print"},{"key":"10.1117\/1.1407822_r20","doi-asserted-by":"crossref","unstructured":"R. Mohan, \u201cVideo sequence matching,\u201d in\n                      Proceedings of International Conference on Speech, Accoustics and Signal Processing\n                      , Vol. 6, pp. 3697\u20133700, Seattle, WA (1998).","DOI":"10.1109\/ICASSP.1998.679686"},{"key":"10.1117\/1.1407822_r21","doi-asserted-by":"crossref","unstructured":"D. Zhong and S. F. Chang, \u201cSpatio-temporal video search using the object-based video representation,\u201d in\n                      Proceedings of IEEE International Conference on Image Processing\n                      , vol. 1, pp. 21\u201324, Santa Barbara, CA (Oct. 1997).","DOI":"10.1109\/ICIP.1997.647374"},{"key":"10.1117\/1.1407822_r22","unstructured":"H. Zhang, A. Wang, and Y. Altunbasak, \u201cContent-based video retrieval and compression: A unified solution,\u201d in Ref. 21, pp. 13\u201316."},{"key":"10.1117\/1.1407822_r23","unstructured":"R. E. Bellman,\n                      Dynamic Programming\n                      , Princeton University Press, Princeton, NJ (1957)."},{"key":"10.1117\/1.1407822_r24","doi-asserted-by":"crossref","unstructured":"M. Naphade, R. Mehrotra, A. M. Ferman, J. Warnick, T. S. Huang, and A. M. Tekalp, \u201cA high performance shot boundary detection algorithm using multiple cues,\u201d in Ref. 11, Vol. 2, pp. 884\u2013887.","DOI":"10.1109\/ICIP.1998.723662"},{"key":"10.1117\/1.1407822_r25","unstructured":"L. Rabiner and B. H. Juang,\n                      Fundamentals of Speech Recognition\n                      , Prentice-Hall, Englewood Cliffs, NJ (1993)."},{"key":"10.1117\/1.1407822_r26","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1109\/TASSP.1978.1163055","volume":"26","author":"Sakoe","year":"1978","journal-title":"IEEE Trans. Acoust., Speech, Signal Process.","ISSN":"https:\/\/id.crossref.org\/issn\/0096-3518","issn-type":"print"},{"key":"10.1117\/1.1407822_r27","doi-asserted-by":"crossref","first-page":"1369","DOI":"10.1016\/S0031-3203(97)00131-3","volume":"31","author":"Jain","year":"1998","journal-title":"Pattern Recogn.","ISSN":"https:\/\/id.crossref.org\/issn\/0031-3203","issn-type":"print"},{"key":"10.1117\/1.1407822_r28","doi-asserted-by":"crossref","first-page":"644","DOI":"10.1109\/76.718510","volume":"8","author":"Rui","year":"1998","journal-title":"IEEE Trans. Circuits Syst. Video Technol. Special Issue on Segmentation, Description and Retrieval of Video Content","ISSN":"https:\/\/id.crossref.org\/issn\/1051-8215","issn-type":"print"}],"container-title":["Journal of Electronic Imaging"],"original-title":[],"link":[{"URL":"https:\/\/www.spiedigitallibrary.org\/journals\/journal-of-electronic-imaging\/volume-10\/issue-04\/861\/Audio-visual-query-and-retrieval--a-system-that-uses\/10.1117\/1.1407822.full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T21:28:56Z","timestamp":1790890136000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.spiedigitallibrary.org\/journals\/journal-of-electronic-imaging\/volume-10\/issue-04\/861\/Audio-visual-query-and-retrieval--a-system-that-uses\/10.1117\/1.1407822.full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2001,10,1]]},"references-count":28,"journal-issue":{"issue":"04"},"URL":"https:\/\/doi.org\/10.1117\/1.1407822","relation":{},"ISSN":["1017-9909","1560-229X"],"issn-type":[{"value":"1017-9909","type":"print"},{"value":"1560-229X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2001,10,1]]}}}