Reinforcement Learning-Based Dynamic Load Assignment for Automated 3PL Tendering Systems
DOI:
https://doi.org/10.15662/IJEETR.2024.0602015Keywords:
Reinforcement Learning, 3PL Tendering, Dynamic Load Assignment, Proximal Policy Optimization, EDI Automation, Oracle Transportation Management, Supply Chain Optimization, Carrier AssignmentAbstract
The use of third-party logistics (3PL) in freight transportation services attracts the issue of 3PL tender. In freight management, the submission of 3PL bids has been and can up to some degree continue to be a supplementary function of the technologies that have been deployed in the area industry. Consequently, companies have had to continue with the tedious routine of grading suppliers, marking a carrier with high marks yet, in real time, he is not available or he charges a high price. Therefore, this proposes an improvement of the existing 3PL system, namely, a reinforcement learning approach to dynamic cargo distribution, which has been experimented in the field. Such a method extends the possibilities of control of the 3PL system, allowing it to adapt to changing external requirements in the process of its functioning. The suggested development allows to improve 3PL services, making it more effective and efficient on the one hand, and operational on the other. The approach loads the agent with respect to an information space based on characteristics of the load, carrier rating, current market trends and accessibility of the transport network, to find the carrier assignment plan corresponding to the increase of aims achievement function. A 93.1% rate of first-tender acceptance (increased from 62.4%) has been verified in historical operational data received from Hub Group and this is further confirmed during its live deployment. It was also found that freight costs reduced by 14.2%, while manual intervention was reduced to just 8.3% of load assignments — a 91.7% reduction — with automation handling all standard freight loads while retaining human oversight for exceptions only. Therefore, based on the above results, the proposed dynamic load assignment system can successfully be deployed in the 3PL operations.
References
1. Bockweg, R., Caplice, C., & Sheffi, Y. (2021). Market-Aware Carrier Selection in Truckload Freight: A Benchmark Study. Transportation Science, 55(4), 831–847.
2. Chen, L., & Wang, Z. (2022). Proximal Policy Optimization for Dynamic Carrier Assignment in Simulated Freight Environments. International Journal of Logistics Research and Applications, 25(6), 792–811.
3. Jiang, Y., Zhou, M., & Liu, X. (2022). Actor-Critic Reinforcement Learning for Port Drayage Load Optimization. IEEE Transactions on Intelligent Transportation Systems, 23(9), 16201–16213.
4. Kumar, A., Patel, R., & Williams, J. (2020). Event-Driven EDI Processing Architecture for High-Throughput Freight Networks. Journal of Enterprise Information Management, 33(5), 1087–1105.
5. Li, S., Chen, H., & Zhao, Q. (2023). Graph Attention Networks for Lane-Level Carrier Performance Embedding. Computers & Operations Research, 152, 106147.
6. Liu, P., Zhang, Y., & Sun, W. (2021). Deep Q-Network Approaches to Carrier Selection in Third-Party Logistics. Expert Systems with Applications, 183, 115425.
7. Mnih, V., Kavukcuoglu, K., Silver, D., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533.
8. Nazari, M., Oroojlooy, A., Takáč, M., & Snyder, L. V. (2018). Reinforcement Learning for Solving the Vehicle Routing Problem. Advances in Neural Information Processing Systems (NeurIPS 2018), 31.
9. Oroojlooyjadid, A., Nazari, M., Snyder, L. V., & Takáč, M. (2022). A Deep Q-Network for the Beer Game: Deep Reinforcement Learning for Inventory Optimization. Manufacturing & Service Operations Management, 24(1), 285–304.
10. Park, J., & Kim, T. (2023). Multi-Agent Reinforcement Learning for Air Cargo Tendering with Stochastic Capacity. Transportation Research Part C: Emerging Technologies, 147, 103986.
11. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
12. Silver, D., Schrittwieser, J., Simonyan, K., et al. (2017). Mastering the game of Go without human knowledge. Nature, 550(7676), 354–359.
13. Oracle Corporation. (2023). Oracle Transportation Management Cloud Documentation. Oracle Help Center. https://docs.oracle.com/en/cloud/saas/transportation-management/
14. X12 Standards Organization. (2022). ANSI X12 Transaction Set Reference: 204 (Motor Carrier Load Tender), 210, 214, 850, 990. Data Interchange Standards Association.
15. Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2016). High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016.
16. Accorsi, R., Baruffaldi, G., & Manzini, R. (2019). A closed-loop supply chain model for multi-stage inventory management. International Journal of Production Research, 57(3), 742–760. https://doi.org/10.1080/00207543.2018.1471243
17. Al-Abbasi, A. O., Ghosh, A., & Aggarwal, V. (2020). DeepPool: Distributed model-free algorithm for ride-sharing using deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems, 21(12), 5151–5162. https://doi.org/10.1109/TITS.2019.2942280
18. Bahdanau, D., Brakel, P., Xu, K., et al. (2020). An actor-critic algorithm for sequence prediction. International Conference on Learning Representations (ICLR).
19. Belletti, F., Hazan, E., Madaan, D., et al. (2020). Expert level control of ramp metering based on multi-task deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems, 21(7), 2935–2945.
20. Chen, X., Wang, Y., & Li, M. (2021). Deep reinforcement learning for supply chain inventory optimization with stochastic demand. Computers & Industrial Engineering, 153, 107063.
21. Dutta, P., Choi, T. M., Somani, S., & Butala, R. (2020). Blockchain technology in supply chain operations: Applications, challenges and research opportunities. Transportation Research Part E, 142, 102067.
22. Gijsbrechts, J., Boute, R., & Van Mieghem, J. (2022). Can deep reinforcement learning improve inventory management? Performance on lost sales, dual sourcing, and multi-echelon problems. Manufacturing & Service Operations Management, 24(1), 134–152.
23. Goodfellow, I., Bengio, Y., & Courville, A. (2021). Deep learning (2nd ed.). MIT Press.
24. Hessel, M., Modayil, J., van Hasselt, H., et al. (2019). Rainbow: Combining improvements in deep reinforcement learning. Proceedings of AAAI Conference on Artificial Intelligence, 33(1), 3215–3222.
25. Ivanov, D., & Dolgui, A. (2020). Viability of intertwined supply networks: Extending the supply chain resilience angles towards survivability. International Journal of Production Research, 58(10), 2904–2915.
26. Kwon, H., Kim, J., & Kim, Y. (2021). Reinforcement learning-based dynamic routing for logistics networks. IEEE Access, 9, 102110–102124.
27. Mazyavkina, N., Sviridov, S., Ivanov, S., & Burnaev, E. (2021). Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research, 134, 105400.
28. Min, H. (2019). Blockchain technology for enhancing supply chain resilience. Business Horizons, 62(1), 35–45.
29. Oroojlooy, A., & Snyder, L. V. (2020). A review of deep reinforcement learning for inventory management. arXiv preprint arXiv:2005.10035.
30. Peng, B., Wang, Z., & Zhang, L. (2022). Multi-agent reinforcement learning for dynamic logistics resource allocation. Expert Systems with Applications, 197, 116674.
31. Sutton, R. S., & Barto, A. G. (2020). Reinforcement learning: An introduction (2nd ed.). MIT Press.
32. Tang, C. S., & Veelenturf, L. P. (2019). The strategic role of logistics in the industry 4.0 era. Transportation Research Part E, 129, 1–11.
