Almost every month, a new optimization algorithm is proposed, often accompanied by the claim that it is superior to all those that came before it. However, this claim is generally based on the algorithm's performance on a specific set of test cases, which are not necessarily representative of the types of problems the algorithm will face in real life. This book presents the theoretical analysis and practical methods (along with source codes) necessary to estimate the difficulty of problems in a test set, as well as to build bespoke test sets consisting of problems with varied difficulties. The book formally establishes a typology of optimization problems, from which a reliable test set can be deduced. At the same time, it highlights how classic test sets are skewed in favor of different classes of problems, and how, as a result, optimizers that have performed well on test problems may perform poorly in real life scenarios.
