Routing Arena: A Benchmark Suite for Neural Routing Solvers
Daniela Thyssens Tim Dernedde Jonas K. Falkner Lars Schmidt-Thieme
Neural Combinatorial Optimization has been researched actively in the last
eight years. Even though many of the proposed Machine Learning based approaches
are compared on the same datasets, the evaluation protocol exhibits essential
flaws and the selection of baselines often neglects State-of-the-Art Operations
Research approaches. To improve on both of these shortcomings, we propose the
Routing Arena, a benchmark suite for Routing Problems that provides a seamless
integration of consistent evaluation and the provision of baselines and
benchmarks prevalent in the Machine Learning- and Operations Research field.
The proposed evaluation protocol considers the two most important evaluation
cases for different applications: First, the solution quality for an a priori
fixed time budget and secondly the anytime performance of the respective
methods. By setting the solution trajectory in perspective to a Best Known
Solution and a Base Solver's solutions trajectory, we furthermore propose the
Weighted Relative Average Performance (WRAP), a novel evaluation metric that
quantifies the often claimed runtime efficiency of Neural Routing Solvers. A
comprehensive first experimental evaluation demonstrates that the most recent
Operations Research solvers generate state-of-the-art results in terms of
solution quality and runtime efficiency when it comes to the vehicle routing
problem. Nevertheless, some findings highlight the advantages of neural
approaches and motivate a shift in how neural solvers should be conceptualized.