{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T07:20:07Z","timestamp":1740122407893,"version":"3.37.3"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2019,9,21]],"date-time":"2019-09-21T00:00:00Z","timestamp":1569024000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,9,21]],"date-time":"2019-09-21T00:00:00Z","timestamp":1569024000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100007601","name":"Horizon 2020","doi-asserted-by":"publisher","award":["731970","731970"],"award-info":[{"award-number":["731970","731970"]}],"id":[{"id":"10.13039\/501100007601","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005416","name":"Norges Forskningsr\u00e5d","doi-asserted-by":"publisher","award":["235530","235530"],"award-info":[{"award-number":["235530","235530"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Cluster Comput"],"published-print":{"date-parts":[[2020,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n<jats:p>Modern workloads often exceed the processing and I\/O capabilities provided by resource virtualization, requiring direct access to the physical hardware in order to reduce latency and computing overhead. For computers interconnected in a cluser, access to remote hardware resources often requires facilitation both in hardware and specialized drivers with virtualization support. This limits the availability of resources to specific devices and drivers that are supported by the virtualization technology being used, as well as what the interconnection technology supports. For PCI Express (PCIe) clusters, we have previously proposed Device Lending as a solution for enabling direct low latency access to remote devices. The method has extremely low computing overhead, and does not require any application- or device-specific distribution mechanisms. Any PCIe device, such as network cards disks, and GPUs, can easily be shared among the connected hosts. In this work, we have extended our solution with support for a virtual machine (VM) hypervisor. Physical remote devices can be \u201cpassed through\u201d to VM guests, enabling direct access to physical resources while still retaining the flexibility of virtualization. Additionally, we have also implemented multi-device support, enabling shortest-path peer-to-peer transfers between remote devices residing in different hosts.Our experimental results prove that multiple remote devices can be used, achieving bandwidth and latency close to native PCIe, and without requiring any additional support in device drivers. I\/O intensive workloads run seamlessly using both local and remote resources. With our added VM and multi-device support, Device Lending offers highly customizable configurations of remote devices that can be dynamically reassigned and shared to optimize resource utilization, thus enabling a flexible composable I\/O infrastructure for VMs as well as bare-metal machines. <\/jats:p>","DOI":"10.1007\/s10586-019-02988-0","type":"journal-article","created":{"date-parts":[[2019,9,21]],"date-time":"2019-09-21T10:02:40Z","timestamp":1569060160000},"page":"1211-1234","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Flexible device compositions and dynamic resource sharing in PCIe interconnected clusters using Device Lending"],"prefix":"10.1007","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3166-2480","authenticated-orcid":false,"given":"Jonas","family":"Markussen","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lars Bj\u00f8rlykke","family":"Kristiansen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rune Johan","family":"Borgli","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"H\u00e5kon Kvale","family":"Stensland","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Friedrich","family":"Seifert","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Riegler","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Carsten","family":"Griwodz","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"P\u00e5l","family":"Halvorsen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2019,9,21]]},"reference":[{"key":"2988_CR1","doi-asserted-by":"publisher","unstructured":"Kristiansen, L.B., Markussen, J., Stensland, H.K., Riegler, M., Kohmann, H., Seifert, F., Nordstr\u00f8m, R., Griwodz, C., Halvorsen, P.: Device Lending in PCI Express Networks. In: Proceedings of International Workshop on Network and Operating Systems Support for Digital Audio and Video, NOSSDAV, pp. 10:1\u201310:6 (2016). \nhttps:\/\/doi.org\/10.1145\/2910642.2910650","DOI":"10.1145\/2910642.2910650"},{"key":"2988_CR2","doi-asserted-by":"publisher","unstructured":"Markussen, J., Kristiansen, L.B., Stensland, H.K., Seifert, F., Griwodz, C., Halvorsen, P.: Flexible device sharing in pcie clusters using device lending. In: Proceedings of the International Conference on Parallel Processing Companion, ICPP Companion, pp. 48:1\u201348:10 (2018). \nhttps:\/\/doi.org\/10.1145\/3229710.3229759","DOI":"10.1145\/3229710.3229759"},{"key":"2988_CR3","unstructured":"Fountain, T., McCarthy, A., Peng, F.: PCI express: an overview of PCI express, cabled PCI express and PXI express. In: Proceedings of International Conference on Accelerator & Large Experimental Physics Control Systems, ICALEPCS (2005)"},{"key":"2988_CR4","unstructured":"Peripheral Component Interconnect Special Interest Group (PCI-SIG): PCI Express 3.1 Base Specification (2010). \nhttps:\/\/pcisig.com\/specifications"},{"key":"2988_CR5","doi-asserted-by":"publisher","unstructured":"Abramson, D., Jackson, J., Muthrasanallur, S., Neiger, G., Regnier, G., Sankaran, R., Schoinas, I., Uhlig, R., Vembu, B., Weigert, J.: Intel virtualization technology for directed I\/O. Intel Technol. J. 10(03) (2006) \nhttps:\/\/doi.org\/10.1535\/itj.1003.02","DOI":"10.1535\/itj.1003.02"},{"key":"2988_CR6","unstructured":"Linux IOMMU Support. \nhttps:\/\/www.kernel.org\/doc\/Documentation\/Intel-IOMMU.txt"},{"key":"2988_CR7","unstructured":"Nvidia Virtual GPU Technology (vGPU). \nhttp:\/\/www.nvidia.com\/object\/virtual-gpus.html"},{"key":"2988_CR8","unstructured":"Peripheral Component Interconnect Special Interest Group (PCI-SIG): Single-root I\/O Virtualization and Sharing Specification (2010). \nhttps:\/\/www.pcisig.com\/specifications\/iov\/single-root\/"},{"key":"2988_CR9","doi-asserted-by":"publisher","unstructured":"Ravindran, M.: Extending Cabled PCI Express to Connect Devices with Independent PCI Domains. In: Proceedings of the IEEE Systems Conference, SysCon, pp. 1\u20137 (2008). \nhttps:\/\/doi.org\/10.1109\/SYSTEMS.2008.4519048","DOI":"10.1109\/SYSTEMS.2008.4519048"},{"key":"2988_CR10","volume-title":"Using Non-transparent Bridging in PCI Express Systems","author":"J Regula","year":"2004","unstructured":"Regula, J.: Using Non-transparent Bridging in PCI Express Systems. PLX Technology Inc, Sunnyvale (2004)"},{"key":"2988_CR11","unstructured":"Sullivan, M.J.: Intel Xeon Processor C5500\/C3500 Series Non-Transparent Bridge. Specification, Intel Corporation (2010)"},{"key":"2988_CR12","unstructured":"Saito, K., Anai, K., Igarashi, K., Nishikawa, T., Himeno, R., Yoguchi, K.: ATM bus system (1998)"},{"key":"2988_CR13","doi-asserted-by":"publisher","unstructured":"Aln\u00e6s, K., Kristiansen, E.H., Gustavson, D.B., James, D.V.: Scalable coherent interface. In: Proceedings of International Conference on Computer Systems and Software Engineering, CompEuro, pp. 446\u2013453 (1990). \nhttps:\/\/doi.org\/10.1109\/CMPEUR.1990.113656","DOI":"10.1109\/CMPEUR.1990.113656"},{"key":"2988_CR14","unstructured":"The Case Against iWARP (2015). \nhttps:\/\/www.chelsio.com\/wp-content\/uploads\/resources\/iWARP-Myths.pdf"},{"key":"2988_CR15","unstructured":"RoCE vs. iWARP Competitive Analysis (2017). \nhttp:\/\/www.mellanox.com\/related-docs\/whitepapers\/WP_RoCE_vs_iWARP.pdf"},{"key":"2988_CR16","doi-asserted-by":"publisher","unstructured":"Trivedi, A., Metzler, B., Stuedi, P.: A case for RDMA in clouds. In: Proceedings of the Second Asia-Pacific Workshop on Systems, APSys, pp. 17:1\u201317:5 (2011). \nhttps:\/\/doi.org\/10.1145\/2103799.2103820","DOI":"10.1145\/2103799.2103820"},{"key":"2988_CR17","doi-asserted-by":"publisher","unstructured":"Huang, J., Ouyang, X., Jose, J., Wasi-Ur-Rahman, M., Wang, H., Luo, M., Subramoni, H., Murthy, C., Panda, D.K.: High-performance design of hbase with RDMA over InfiniBand. In: Proceedings of International Parallel and Distributed Processing Symposium, IPDPS, pp. 774\u2013785 (2012). \nhttps:\/\/doi.org\/10.1109\/IPDPS.2012.74","DOI":"10.1109\/IPDPS.2012.74"},{"key":"2988_CR18","doi-asserted-by":"publisher","unstructured":"Jiang, W., Liu, J., Jin, H.W., Panda, D.K., Gropp, W., Thakur, R.: High performance MPI-2 one-sided communication over InfiniBand. In: Proceedings of International Symposium on Cluster Computing and the Grid, CCGrid, pp. 531\u2013538 (2004). \nhttps:\/\/doi.org\/10.1109\/CCGrid.2004.1336648","DOI":"10.1109\/CCGrid.2004.1336648"},{"key":"2988_CR19","doi-asserted-by":"publisher","unstructured":"Duato, J., Pena, A., Silla, F., Mayo, R., Quintana-Ort\u00ed, E.: rCUDA: reducing the number of GPU-based accelerators in high performance clusters. In: Proceedings of International Conference on High Performance Computing and Simulation, HPCS pp. 224\u2013231 (2010). \nhttps:\/\/doi.org\/10.1109\/HPCS.2010.5547126","DOI":"10.1109\/HPCS.2010.5547126"},{"key":"2988_CR20","doi-asserted-by":"publisher","unstructured":"Venkatesh, A., Subramoni, H., Hamidouche, K., Panda, D.K.: A high performance broadcast design with hardware multicast and GPUDirect RDMA for streaming applications on Infiniband clusters. In: Proceedings of International Conference on High Performance Computing, HiPC (2014). \nhttps:\/\/doi.org\/10.1109\/HiPC.2014.7116875","DOI":"10.1109\/HiPC.2014.7116875"},{"key":"2988_CR21","unstructured":"Rosetti, D.: Benchmarking GPUDirect RDMA on Modern Server Platforms (2014). \nhttp:\/\/devblogs.nvidia.com\/parallelforall\/benchmarking-gpudirect-rdma-on-modern-server-platforms\/"},{"issue":"3","key":"2988_CR22","doi-asserted-by":"publisher","first-page":"567","DOI":"10.1145\/2872887.2750415","volume":"43","author":"A Daglis","year":"2015","unstructured":"Daglis, A., Novakovi\u0107, S., Bugnion, E., Falsafi, B., Grot, B.: Manycore network interfaces for in-memory rack-scale computing. ACM SIGARCH Comput. Archit. News 43(3), 567\u2013579 (2015). \nhttps:\/\/doi.org\/10.1145\/2872887.2750415","journal-title":"ACM SIGARCH Comput. Archit. News"},{"issue":"4","key":"2988_CR23","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1145\/2829988.2787492","volume":"45","author":"P Costa","year":"2015","unstructured":"Costa, P., Ballani, H., Razavi, K., Kash, I.: R2c2: a network stack for rack-scale computers. ACM SIGCOMM Comput. Commun. Rev. 45(4), 551\u2013564 (2015). \nhttps:\/\/doi.org\/10.1145\/2829988.2787492","journal-title":"ACM SIGCOMM Comput. Commun. Rev."},{"issue":"3","key":"2988_CR24","doi-asserted-by":"publisher","first-page":"292","DOI":"10.1145\/327070.327269","volume":"13","author":"C Whitby-Strevens","year":"1985","unstructured":"Whitby-Strevens, C.: The transputer. ACM SIGARCH Comput. Archit. News 13(3), 292\u2013300 (1985). \nhttps:\/\/doi.org\/10.1145\/327070.327269","journal-title":"ACM SIGARCH Comput. Archit. News"},{"issue":"5","key":"2988_CR25","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1109\/MM.1986.304707","volume":"6","author":"JP Hayes","year":"1986","unstructured":"Hayes, J.P., Mudge, T., Stout, Q.F., Colley, S., Palmer, J.: A microprocessor-based hypercube supercomputer. IEEE Micro 6(5), 6\u201317 (1986). \nhttps:\/\/doi.org\/10.1109\/MM.1986.304707","journal-title":"IEEE Micro"},{"key":"2988_CR26","unstructured":"Peripheral Component Interconnect Special Interest Group (PCI-SIG): Multi-root I\/O Virtualization and Sharing Specification (2008). \nhttps:\/\/www.pcisig.com\/specifications\/iov\/multi-root\/"},{"key":"2988_CR27","doi-asserted-by":"publisher","unstructured":"Suzuki, J., Hidaka, Y., Higuchi, J., Baba, T., Kami, N., Yoshikawa, T.: Multi-root Share of Single-Root I\/O Virtualization (SR-IOV) Compliant PCI Express Device. In: Proceedings of Symposium on High Performance Interconnects, HOTI, pp. 25\u201331 (2010). \nhttps:\/\/doi.org\/10.1109\/HOTI.2010.21","DOI":"10.1109\/HOTI.2010.21"},{"issue":"3","key":"2988_CR28","doi-asserted-by":"publisher","first-page":"108","DOI":"10.1145\/2508148.2485932","volume":"41","author":"CC Tu","year":"2013","unstructured":"Tu, C.C., Lee, Ct, Chiueh, Tc: Secure I\/O device sharing among virtual machines on multiple hosts. ACM SIGARCH Comput. Archit. News 41(3), 108\u2013119 (2013). \nhttps:\/\/doi.org\/10.1145\/2508148.2485932","journal-title":"ACM SIGARCH Comput. Archit. News"},{"key":"2988_CR29","doi-asserted-by":"publisher","unstructured":"Tu, C.C., Chiueh, T.c.: Seamless fail-over for PCIe switched networks. In: Proceedings of the International Systems and Storage Conference, SYSTOR, pp. 101\u2013111 (2018). \nhttps:\/\/doi.org\/10.1145\/3211890.3211895","DOI":"10.1145\/3211890.3211895"},{"key":"2988_CR30","unstructured":"Dilk, P.: Microsemi Switchtec PAX: Advanced fabric gen3 pcie switch (2017). \nhttps:\/\/www.youtube.com\/watch?v=OB7OuektR0E"},{"key":"2988_CR31","unstructured":"Wong, H.: PCI express multi-root switch reconfiguration during system operation (2011)"},{"key":"2988_CR32","unstructured":"VFIO\u2014\u201cVirtual Function I\/O\u201d. \nhttps:\/\/www.kernel.org\/doc\/Documentation\/vfio.txt"},{"key":"2988_CR33","unstructured":"Jia, N., Wankhede, K.: VFIO Mediated Devices. \nhttps:\/\/www.kernel.org\/doc\/Documentation\/vfio-mediated-device.txt"},{"key":"2988_CR34","unstructured":"Peripheral Component Interconnect Special Interest Group (PCI-SIG): Address Translation Services Revision 1.1 (2009). \nhttps:\/\/www.pcisig.com\/specifications\/iov\/ats\/"},{"key":"2988_CR35","unstructured":"PXH830 Gen3 PCI Express NTB Host Adapter. \nhttp:\/\/www.dolphinics.no\/products\/PXH830.html"},{"key":"2988_CR36","unstructured":"CUDA Toolkit Documentation v10.1.105 (2019). \nhttp:\/\/docs.nvidia.com\/cuda\/"},{"key":"2988_CR37","unstructured":"Keras (2015). \nhttps:\/\/keras.io"},{"key":"2988_CR38","unstructured":"TensorFlow: Large-scale machine learning on heterogeneous systems (2015). \nhttps:\/\/www.tensorflow.org\/"},{"key":"2988_CR39","unstructured":"Keras documentation: multi\\_gpu\\_model (2015). \nhttps:\/\/keras.io\/utils\/#multi_gpu_model"},{"key":"2988_CR40","unstructured":"Borgli, R., Halvorsen, P., Riegler, M., Stensland, H.K.: Automatic hyperparameter optimization in keras for the mediaeval 2018 medico multimedia task. In: Working Notes Proceedings of the MediaEval 2018 Workshop (2018)"},{"key":"2988_CR41","unstructured":"Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR arXiv (2014). \narXiv:abs\/1409.1556"},{"key":"2988_CR42","doi-asserted-by":"publisher","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A Large-Scale Hierarchical Image Database. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, CVPR (2009). \nhttps:\/\/doi.org\/10.1109\/CVPR.2009.5206848","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"2988_CR43","doi-asserted-by":"publisher","unstructured":"Pogorelov, K., Randel, K.R., Griwodz, C., Eskeland, S.L., de\u00a0Lange, T., Johansen, D., Spampinato, C., Dang-Nguyen, D.T., Lux, M., Schmidt, P.T., Riegler, M., Halvorsen, P.: KVASIR: A multi-class image dataset for computer aided gastrointestinal disease detection. In: Proceedings of the ACM Multimedia Systems Conference, MMSys, pp. 164\u2013169 (2017). \nhttps:\/\/doi.org\/10.1145\/3083187.3083212","DOI":"10.1145\/3083187.3083212"},{"key":"2988_CR44","doi-asserted-by":"publisher","unstructured":"Hicks, S.A., Riegler, M., Pogorelov, K., \u00c5nonsen, K.V., de\u00a0Lange, T., Johansen, D., Jeppsson, M., Randel, K.R., Eskeland, S., Halvorsen, P.: Dissecting deep neural networks for better medical image classification and classification understanding. In: Proceedings of International Symposium on Computer-Based Medical Systems, CBMS (2018). \nhttps:\/\/doi.org\/10.1109\/CBMS.2018.00070","DOI":"10.1109\/CBMS.2018.00070"},{"key":"2988_CR45","doi-asserted-by":"publisher","unstructured":"Pogorelov, K., Ostroukhova, O., Jeppsson, M., Espeland, H., Griwodz, C., de\u00a0Lange, T., Johansen, D., Riegler, M., Halvorsen, P.: Deep learning and hand-crafted feature based approaches for polyp detection in medical videos. In: Proceedings of International Symposium on Computer-Based Medical Systems, CBMS, pp. 381\u2013386 (2018). \nhttps:\/\/doi.org\/10.1109\/CBMS.2018.00073","DOI":"10.1109\/CBMS.2018.00073"},{"key":"2988_CR46","doi-asserted-by":"publisher","unstructured":"Neugebauer, R., Antichi, G., Zazo, J.F., Audzevich, Y., L\u00f3pez-Buedo, S., Moore, A.W.: Understanding PCIe performance for end host networking. In: Proceedings of the ACM Special Interest Group on Data Communication, SIGCOMM, pp. 327\u2013341 (2018). \nhttps:\/\/doi.org\/10.1145\/3230543.3230560","DOI":"10.1145\/3230543.3230560"}],"container-title":["Cluster Computing"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10586-019-02988-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s10586-019-02988-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10586-019-02988-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,9,19]],"date-time":"2020-09-19T23:23:32Z","timestamp":1600557812000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10586-019-02988-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,21]]},"references-count":46,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,6]]}},"alternative-id":["2988"],"URL":"https:\/\/doi.org\/10.1007\/s10586-019-02988-0","relation":{},"ISSN":["1386-7857","1573-7543"],"issn-type":[{"type":"print","value":"1386-7857"},{"type":"electronic","value":"1573-7543"}],"subject":[],"published":{"date-parts":[[2019,9,21]]},"assertion":[{"value":"29 March 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 July 2019","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 September 2019","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 September 2019","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}