{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T01:31:48Z","timestamp":1784770308879,"version":"3.55.0"},"reference-count":72,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2018,1,8]],"date-time":"2018-01-08T00:00:00Z","timestamp":1515369600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"European Union's Horizon2020","award":["Grant No. 687698"],"award-info":[{"award-number":["Grant No. 687698"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2018,1,8]]},"abstract":"<jats:p>Wearables and mobile devices see the world through the lens of half a dozen low-power sensors, such as, barometers, accelerometers, microphones and proximity detectors. But differences between sensors ranging from sampling rates, discrete and continuous data or even the data type itself make principled approaches to integrating these streams challenging. How, for example, is barometric pressure best combined with an audio sample to infer if a user is in a car, plane or bike? Critically for applications, how successfully sensor devices are able to maximize the information contained across these multi-modal sensor streams often dictates the fidelity at which they can track user behaviors and context changes. This paper studies the benefits of adopting deep learning algorithms for interpreting user activity and context as captured by multi-sensor systems. Specifically, we focus on four variations of deep neural networks that are based either on fully-connected Deep Neural Networks (DNNs) or Convolutional Neural Networks (CNNs). Two of these architectures follow conventional deep models by performing feature representation learning from a concatenation of sensor types. This classic approach is contrasted with a promising deep model variant characterized by modality-specific partitions of the architecture to maximize intra-modality learning. Our exploration represents the first time these architectures have been evaluated for multimodal deep learning under wearable data -- and for convolutional layers within this architecture, it represents a novel architecture entirely. Experiments show these generic multimodal neural network models compete well with a rich variety of conventional hand-designed shallow methods (including feature extraction and classifier construction) and task-specific modeling pipelines, across a wide-range of sensor types and inference tasks (four different datasets). Although the training and inference overhead of these multimodal deep approaches is in some cases appreciable, we also demonstrate the feasibility of on-device mobile and wearable execution is not a barrier to adoption. This study is carefully constructed to focus on multimodal aspects of wearable data modeling for deep learning by providing a wide range of empirical observations, which we expect to have considerable value in the community. We summarize our observations into a series of practitioner rules-of-thumb and lessons learned that can guide the usage of multimodal deep learning for activity and context detection.<\/jats:p>","DOI":"10.1145\/3161174","type":"journal-article","created":{"date-parts":[[2018,1,9]],"date-time":"2018-01-09T13:26:11Z","timestamp":1515504371000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":211,"title":["Multimodal Deep Learning for Activity and Context Recognition"],"prefix":"10.1145","volume":"1","author":[{"given":"Valentin","family":"Radu","sequence":"first","affiliation":[{"name":"The University of Edinburgh"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Catherine","family":"Tong","sequence":"additional","affiliation":[{"name":"University of Oxford"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sourav","family":"Bhattacharya","sequence":"additional","affiliation":[{"name":"Nokia Bell Labs"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicholas D.","family":"Lane","sequence":"additional","affiliation":[{"name":"University of Oxford and Nokia Bell Labs"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cecilia","family":"Mascolo","sequence":"additional","affiliation":[{"name":"University of Cambridge"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahesh K.","family":"Marina","sequence":"additional","affiliation":[{"name":"The University of Edinburgh"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fahim","family":"Kawsar","sequence":"additional","affiliation":[{"name":"Nokia Bell Labs and TU Delft"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,1,8]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2968219.2971459"},{"key":"e_1_2_2_2_1","unstructured":"Yoshua Bengio Ian J. Goodfellow and Aaron Courville. 2015. Deep Learning. (2015). http:\/\/www.iro.umontreal.ca\/~bengioy\/dlbook Book in preparation for MIT Press.  Yoshua Bengio Ian J. Goodfellow and Aaron Courville. 2015. Deep Learning. (2015). http:\/\/www.iro.umontreal.ca\/~bengioy\/dlbook Book in preparation for MIT Press."},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/PERCOMW.2016.7457169"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2994551.2994564"},{"key":"e_1_2_2_5_1","volume-title":"Pattern Recognition and Machine Learning (Information Science and Statistics)","author":"Bishop Christopher M."},{"key":"e_1_2_2_6_1","volume-title":"Proceedings of the land warfare conference. 1--8.","author":"Bokareva Tatiana","year":"2006"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3123021.3123038"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/0031-3203(93)90060-A"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2134203.2134205"},{"key":"e_1_2_2_10_1","doi-asserted-by":"crossref","volume-title":"Mobile robot localization and map building: A multisensor fusion approach","author":"Castellanos Jose A","DOI":"10.1007\/978-1-4615-4405-0"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2014.6854370"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/BHI.2017.7897306"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/MPRV.2008.39"},{"key":"e_1_2_2_14_1","volume-title":"DEEP LEARNING: Methods and Applications. Technical Report MSR-TR-2014-21","author":"Deng Li","year":"2014"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12193-015-0195-2"},{"key":"e_1_2_2_16_1","volume-title":"Deep Bayesian Active Learning with Image Data. CoRR abs\/1703.02910","author":"Gal Yarin","year":"2017"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3131895"},{"key":"e_1_2_2_18_1","unstructured":"Github repository 2017. Multimodal Deep Learning Framework. https:\/\/github.com\/vradu10\/deepfusion.git. (2017).  Github repository 2017. Multimodal Deep Learning Framework. https:\/\/github.com\/vradu10\/deepfusion.git. (2017)."},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1161\/01.CIR.101.23.e215"},{"key":"e_1_2_2_20_1","volume-title":"Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 6645--6649","author":"Graves Alex","year":"2013"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2010.5540120"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2971648.2971708"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1656274.1656278"},{"key":"e_1_2_2_24_1","volume-title":"AAAI","author":"Hammerla Nils","year":"2015"},{"key":"e_1_2_2_25_1","volume-title":"Proceedings of IJCAI. ACM.","author":"Hammerla Nils","year":"2016"},{"key":"e_1_2_2_26_1","volume-title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149","author":"Han Song","year":"2015"},{"key":"e_1_2_2_27_1","volume-title":"Ng","author":"Hannun Awni Y.","year":"2014"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2517351.2517367"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3081333.3081360"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1101149.1101300"},{"key":"e_1_2_2_31_1","volume-title":"Activity recognition in beach volleyball using a Deep Convolutional Neural Network. Data Mining and Knowledge Discovery","author":"Kautz Thomas","year":"2017"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2851581.2892314"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2013.6638346"},{"key":"e_1_2_2_34_1","volume-title":"Zemel","author":"Kiros Ryan","year":"2014"},{"key":"e_1_2_2_35_1","volume-title":"Hinton","author":"Krizhevsky Alex","year":"2012"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3123024.3125616"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPSN.2016.7460664"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2820975.2820980"},{"key":"e_1_2_2_39_1","volume-title":"Lane and Petko Georgiev","author":"Nicholas","year":"2015"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2750858.2804262"},{"key":"e_1_2_2_41_1","volume-title":"Deep Learning. Nature","author":"LeCun Yann","year":"2015"},{"key":"e_1_2_2_42_1","volume-title":"Watch R 2017","author":"LG","year":"2017"},{"key":"e_1_2_2_43_1","volume-title":"Multimodal Emotion Recognition Using Multimodal Deep Learning. CoRR abs\/1602.08225","author":"Liu Wei","year":"2016"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1869983.1869992"},{"key":"e_1_2_2_45_1","unstructured":"Lumo Lift 2017. Lumo Lift. http:\/\/www.lumobodytech.com. (2017).  Lumo Lift 2017. Lumo Lift. http:\/\/www.lumobodytech.com. (2017)."},{"key":"e_1_2_2_46_1","unstructured":"J. Mao W. Xu Y. Yang J. Wang and A. L. Yuille. 2014. Explain Images with Multimodal Recurrent Neural Networks. ArXiv e-prints (Oct. 2014). arXiv:cs.CV\/1410.1090  J. Mao W. Xu Y. Yang J. Wang and A. L. Yuille. 2014. Explain Images with Multimodal Recurrent Neural Networks. ArXiv e-prints (Oct. 2014). arXiv:cs.CV\/1410.1090"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.5555\/3021319.3021339"},{"key":"e_1_2_2_48_1","unstructured":"Microsoft Band 2017. Microsoft Band. http:\/\/www.microsoft.com\/Microsoft-Band\/. (2017).  Microsoft Band 2017. Microsoft Band. http:\/\/www.microsoft.com\/Microsoft-Band\/. (2017)."},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2971763.2971764"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2015.7178347"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3123021.3123046"},{"key":"e_1_2_2_52_1","volume-title":"Proceedings of the 28th International Conference on Machine Learning, ICML 2011","author":"Ngiam Jiquan","year":"2011"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2014.10.012"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/CDC.2005.1583238"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2015.01.095"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2668332.2668347"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2968219.2971461"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPIN.2013.6817916"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46493-0_32"},{"key":"e_1_2_2_60_1","volume-title":"2013 AAAI Fall Symposium Series.","author":"Sachan Devendra Singh","year":"2013"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/1031495.1031497"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/1101149.1101236"},{"key":"e_1_2_2_63_1","volume-title":"Improved Multimodal Deep Learning with Variation of Information. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014","author":"Sohn Kihyuk","year":"2014"},{"key":"e_1_2_2_64_1","volume-title":"Advances in Neural Information Processing Systems 25","author":"Srivastava Nitish"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2809695.2809718"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.220"},{"key":"e_1_2_2_67_1","unstructured":"Torch 2017. Torch. http:\/\/torch.ch\/. (2017).  Torch 2017. Torch. http:\/\/torch.ch\/. (2017)."},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2014.6854363"},{"key":"e_1_2_2_69_1","volume-title":"Proceedings of the 32nd International Conference on Machine Learning (ICML-15)","author":"Wang Weiran","year":"2015"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2502081.2502112"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052577"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSNIP.2007.4496857"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3161174","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3161174","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:13:30Z","timestamp":1750212810000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3161174"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,1,8]]},"references-count":72,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2018,1,8]]}},"alternative-id":["10.1145\/3161174"],"URL":"https:\/\/doi.org\/10.1145\/3161174","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,1,8]]},"assertion":[{"value":"2017-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-10-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-01-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}