{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T05:27:24Z","timestamp":1773811644225,"version":"3.50.1"},"reference-count":50,"publisher":"MDPI AG","issue":"20","license":[{"start":{"date-parts":[[2022,10,20]],"date-time":"2022-10-20T00:00:00Z","timestamp":1666224000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"ERDF\u2013European Regional Development Fund","award":["POCI-01-0145-FEDER-028611"],"award-info":[{"award-number":["POCI-01-0145-FEDER-028611"]}]},{"name":"National Funds","award":["POCI-01-0145-FEDER-028611"],"award-info":[{"award-number":["POCI-01-0145-FEDER-028611"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Mathematics"],"abstract":"<jats:p>In this work, we study a flexible compensation scheme for last-mile delivery where a company outsources part of the activity of delivering products to its customers to occasional drivers (ODs), under a scheme named crowdshipping. All deliveries are completed at the minimum total cost incurred with their vehicles and drivers plus the compensation paid to the ODs. The company decides on the best compensation scheme to offer to the ODs at the planning stage. We model our problem based on a stochastic and dynamic environment where delivery orders and ODs volunteering to make deliveries present themselves randomly within fixed time windows. The uncertainty is endogenous in the sense that the compensation paid to ODs influences their availability. We develop a deep reinforcement learning (DRL) algorithm that can deal with large instances while focusing on the quality of the solution: we combine the combinatorial structure of the action space with the neural network of the approximated value function, involving techniques from machine learning and integer optimization. The results show the effectiveness of the DRL approach by examining out-of-sample performance and that it is suitable to process large samples of uncertain data, which induces better solutions.<\/jats:p>","DOI":"10.3390\/math10203902","type":"journal-article","created":{"date-parts":[[2022,10,20]],"date-time":"2022-10-20T22:49:46Z","timestamp":1666306186000},"page":"3902","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Deep Reinforcement Learning for Crowdshipping Last-Mile Delivery with Endogenous Uncertainty"],"prefix":"10.3390","volume":"10","author":[{"given":"Marco","family":"Silva","sequence":"first","affiliation":[{"name":"Industrial Engineering and Management, Institute for Systems and Computer Engineering, Technology and Science, 4200-465 Porto, Portugal"}]},{"given":"Jo\u00e3o Pedro","family":"Pedroso","sequence":"additional","affiliation":[{"name":"Industrial Engineering and Management, Institute for Systems and Computer Engineering, Technology and Science, 4200-465 Porto, Portugal"},{"name":"Department of Computer Science, Faculty of Sciences, University of Porto, 4169-007 Porto, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,20]]},"reference":[{"key":"ref_1","unstructured":"ARC Advisory Group (2021, October 18). What Are Omni-Channel Fulfillment and Returns Management All about?. Available online: https:\/\/www.arcweb.com\/industry-best-practices\/what-omni-channel-fulfillment-returns-management-all-about."},{"key":"ref_2","unstructured":"Walmart (2021, October 18). Spark Driver Delivery. Available online: https:\/\/drive4spark.walmart.com\/."},{"key":"ref_3","unstructured":"Doordash (2021, October 18). Delivering with Doordash. Available online: https:\/\/www.doordash.com\/about\/."},{"key":"ref_4","unstructured":"JD-Dada (2021, October 18). Become a Dada Knight. Available online: https:\/\/www.imdada.cn\/."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"472","DOI":"10.1016\/j.ejor.2016.03.049","article-title":"The Vehicle Routing Problem with Occasional Drivers","volume":"254","author":"Archetti","year":"2016","journal-title":"Eur. J. Oper. Res."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1016\/j.trpro.2018.09.011","article-title":"Stochastic last-mile delivery with crowdshipping","volume":"30","author":"Gdowska","year":"2018","journal-title":"Transp. Res. Procedia"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Bekta\u015f, T., Coniglio, S., Martinez-Sykora, A., and Vo\u00df, S. (2017). The Vehicle Routing Problem with Dynamic Occasional Drivers. Proceedings of the Computational Logistics, Springer International Publishing.","DOI":"10.1007\/978-3-319-68496-3"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2153","DOI":"10.1111\/poms.13219","article-title":"Crowdshipping and Same-day Delivery: Employing In-store Customers to Deliver Online Orders","volume":"29","author":"Dayarian","year":"2020","journal-title":"Prod. Oper. Manag."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"222","DOI":"10.1287\/trsc.2017.0803","article-title":"Crowdsourced Delivery\u2014A Dynamic Pickup and Delivery Problem with Ad Hoc Drivers","volume":"53","author":"Arslan","year":"2019","journal-title":"Transp. Sci."},{"key":"ref_10","unstructured":"Barbosa, M. (2019). A Data-Driven Compensation Scheme for Last-Mile Delivery with Crowdsourcing. [Master\u2019s Thesis, Universidade do Porto]. Available online: https:\/\/repositorio-aberto.up.pt\/bitstream\/10216\/124212\/2\/367287.pdf."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Silva, M., Pedroso, J.P., and Viana, A. (2022, September 20). Deep Reinforcement Learning for Stochastic Last-Mile Delivery with Crowd Shipping. Available online: https:\/\/hal.archives-ouvertes.fr\/view\/index\/docid\/3821656.","DOI":"10.3390\/math10203902"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1016\/j.cor.2019.04.023","article-title":"The pickup and delivery problem with time windows and occasional drivers","volume":"109","author":"Dahle","year":"2019","journal-title":"Comput. Oper. Res."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"102139","DOI":"10.1016\/j.omega.2019.102139","article-title":"Crowdsourced delivery: A review of platforms and academic literature","volume":"98","author":"Alnaggar","year":"2019","journal-title":"Omega"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"929","DOI":"10.1287\/opre.36.6.929","article-title":"A Priori Solution of a Traveling Salesman Problem in Which a Random Subset of the Customers Are Visited","volume":"36","author":"Jaillet","year":"1988","journal-title":"Oper. Res."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"574","DOI":"10.1287\/opre.40.3.574","article-title":"A Vehicle Routing Problem with Stochastic Demand","volume":"40","author":"Bertsimas","year":"1992","journal-title":"Oper. Res."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1287\/opre.42.3.543","article-title":"A Priori Optimization of the Probabilistic Traveling Salesman Problem","volume":"42","author":"Laporte","year":"1994","journal-title":"Oper. Res."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1287\/trsc.29.2.143","article-title":"An Exact Algorithm for the Vehicle Routing Problem with Stochastic Demands and Customers","volume":"29","author":"Gendreau","year":"1995","journal-title":"Transp. Sci."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1287\/opre.50.3.415.7751","article-title":"An Integer L-Shaped Algorithm for the Capacitated Vehicle Routing Problem with Stochastic Demands","volume":"50","author":"Laporte","year":"2002","journal-title":"Oper. Res."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1016\/j.cor.2014.03.028","article-title":"A branch-cut-and-price algorithm for the vehicle routing problem with stochastic demands","volume":"50","author":"Gauvin","year":"2014","journal-title":"Comput. Oper. Res."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1414","DOI":"10.1016\/j.procs.2017.05.068","article-title":"An Exact Resolution for the Probabilistic Traveling Salesman Problem under the A Priori Strategy","volume":"108","author":"Amar","year":"2017","journal-title":"Procedia Comput. Sci."},{"key":"ref_21","unstructured":"Vaidya, J., and Li, J. (2018). A Parallel Branch and Bound Algorithm for the Probabilistic TSP. Algorithms and Architectures for Parallel Processing, Proceedings of the 18th International Conference, ICA3PP 2018, Guangzhou, China, 15\u201317 November 2018, Springer International Publishing."},{"key":"ref_22","unstructured":"Lagos, F., Klapp, M., and Toriello, A. (2022, September 20). Branch-and-Price for Probabilistic Vehicle Routing. Available online: http:\/\/www.optimization-online.org\/DB_HTML\/2017\/12\/6364.html."},{"key":"ref_23","unstructured":"Novoa, C., Berger, R., Linderoth, J., and Storer, R. (2022, September 20). A Set-Partitioning-Based Model for the Stochastic Vehicle Routing Problem. Available online: http:\/\/www.optimization-online.org\/DB_HTML\/2006\/12\/1542.html."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1058","DOI":"10.1287\/opre.1070.0441","article-title":"A Robust Optimization Perspective on Stochastic Programming","volume":"55","author":"Chen","year":"2007","journal-title":"Oper. Res."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"902","DOI":"10.1287\/opre.1090.0795","article-title":"Distributionally Robust Optimization and Its Tractable Approximations","volume":"58","author":"Goh","year":"2010","journal-title":"Oper. Res."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"595","DOI":"10.1287\/opre.1090.0741","article-title":"Distributionally Robust Optimization Under Moment Uncertainty with Application to Data-Driven Problems","volume":"58","author":"Delage","year":"2010","journal-title":"Oper. Res."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1007\/s10107-017-1151-6","article-title":"Exact algorithms for the chance-constrained vehicle routing problem","volume":"172","author":"Dinh","year":"2018","journal-title":"Math. Program."},{"key":"ref_28","unstructured":"Ghosal, S.K., and Wiesemann, W. (2022, September 20). The Distributionally Robust Chance Constrained Vehicle Routing Problem. Available online: http:\/\/www.optimization-online.org\/DB_FILE\/2018\/08\/6759.pdf."},{"key":"ref_29","unstructured":"Bello, I., Pham, H., Le, Q.V., Norouzi, M., and Bengio, S. (2016). Neural Combinatorial Optimization with Reinforcement Learning. arXiv."},{"key":"ref_30","unstructured":"Bengio, S., Wallach, H.M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (2018, January 3\u20138). Reinforcement Learning for Solving the Vehicle Routing Problem. Proceedings of the Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montr\u00e9al, QC, Canada, 3\u20138 December 2018, Montr\u00e9al, QC, Canada."},{"key":"ref_31","unstructured":"Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (2020, January 6\u201312). Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routing. Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, Virtual."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/s10107-020-01474-5","article-title":"Strong mixed-integer programming formulations for trained neural networks","volume":"183","author":"Anderson","year":"2020","journal-title":"Math. Program."},{"key":"ref_33","unstructured":"Chen, Y., Qian, Y., Yao, Y., Wu, Z., Li, R., Zhou, Y., Hu, H., and Xu, Y. (2019). Can Sophisticated Dispatching Strategy Acquired by Reinforcement Learning?\u2014A Case Study in Dynamic Courier Dispatching System. arXiv."},{"key":"ref_34","unstructured":"Chen, X., Ulmer, M.W., and Thomas, B.W. (2019). Deep Q-Learning for Same-Day Delivery with a Heterogeneous Fleet of Vehicles and Drones. arXiv."},{"key":"ref_35","unstructured":"Hildebrandt, F.D., Thomas, B.W., and Ulmer, M.W. (2021). Where the Action is: Let\u2019s make Reinforcement Learning for Stochastic Dynamic Vehicle Routing Problems work!. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1287\/moor.15.3.381","article-title":"On-Line Optimization of Simulated Markovian Processes","volume":"15","author":"Pflug","year":"1990","journal-title":"Math. Oper. Res."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1023\/A:1018943626786","article-title":"A Class of Stochastic Programs with Decision Dependent Random Elements","volume":"82","author":"Jonsbraten","year":"1998","journal-title":"Ann. Oper. Res."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1409","DOI":"10.1016\/j.compchemeng.2003.10.005","article-title":"A stochastic programming approach to planning of offshore gas field developments under uncertainty in reserves","volume":"28","author":"Goel","year":"2004","journal-title":"Comput. Chem. Eng."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1007\/s10107-006-0715-7","article-title":"A Class of stochastic programs with decision dependent uncertainty","volume":"108","author":"Goel","year":"2006","journal-title":"Math. Program."},{"key":"ref_40","unstructured":"Luo, F., and Mehrotra, S. (2018). Distributionally Robust Optimization with Decision Dependent Ambiguity Sets. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Devezas, T., Leit\u00e3o, J., and Sarygulov, A. (2021). Reinforcement Learning Approach for Dynamic Pricing. The Economics of Digital Transformation: Approaching Non-Stable and Uncertain Digitalized Production Systems, Springer International Publishing.","DOI":"10.1007\/978-3-030-59959-1"},{"key":"ref_42","unstructured":"Liu, J., Zhang, Y., Wang, X., Deng, Y., and Wu, X. (2019). Dynamic Pricing on E-commerce Platform with Deep Reinforcement Learning: A Field Experiment. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1186\/s40537-019-0197-0","article-title":"A survey on Image Data Augmentation for Deep Learning","volume":"6","author":"Shorten","year":"2019","journal-title":"J. Big Data"},{"key":"ref_44","first-page":"40","article-title":"If You Like It, GAN It\u2014Probabilistic Multivariate Times Series Forecast with GAN","volume":"5","author":"Koochali","year":"2021","journal-title":"Eng. Proc."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1146\/annurev-statistics-062713-085831","article-title":"Probabilistic Forecasting","volume":"1","author":"Gneiting","year":"2014","journal-title":"Annu. Rev. Stat. Its Appl."},{"key":"ref_46","first-page":"2672","article-title":"Generative Adversarial Networks","volume":"3","author":"Goodfellow","year":"2014","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_47","unstructured":"Mirza, M., and Osindero, S. (2014). Conditional Generative Adversarial Nets. arXiv."},{"key":"ref_48","unstructured":"Smith, K.E., and Smith, A.O. (2020). Conditional GAN for timeseries generation. arXiv."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Moreno-D\u00edaz, R., Pichler, F., and Quesada-Arencibia, A. (2015). Heuristic Approaches for the Probabilistic Traveling Salesman Problem. Computer Aided Systems Theory\u2014EUROCAST 2015, Proceedings of the 15th International Conference, Las Palmas de Gran Canaria, Spain, 8\u201313 February 2015, Springer International Publishing.","DOI":"10.1007\/978-3-319-27340-2"}],"container-title":["Mathematics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-7390\/10\/20\/3902\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:58:19Z","timestamp":1760144299000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-7390\/10\/20\/3902"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,20]]},"references-count":50,"journal-issue":{"issue":"20","published-online":{"date-parts":[[2022,10]]}},"alternative-id":["math10203902"],"URL":"https:\/\/doi.org\/10.3390\/math10203902","relation":{},"ISSN":["2227-7390"],"issn-type":[{"value":"2227-7390","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,20]]}}}