Pruning at Initialization -- A Sketching Perspective

The study titled 'Pruning at Initialization -- A Sketching Perspective' delves into the lottery ticket hypothesis (LTH), which has garnered significant attention for its implications in pruning neural networks at initialization. By equating the search for a sparse mask to the sketching problem in efficient matrix multiplication, the authors provide a novel analytical framework for understanding LTH. This approach allows them to establish bounds on the approximation error of pruned linear models, enhancing the theoretical justification for previous empirical findings that suggest the search for sparse networks may be data independent. Furthermore, the research introduces a generic improvement to existing algorithms for pruning at initialization, demonstrating its advantages particularly in scenarios where data independence is a factor. This advancement could lead to more efficient neural network architectures, making the study a significant contribution to the field of artificial intell…