Pruning

Summary Research Objective There are recent found that, from a large neural trained network, we can prune and obtain a small sub network (even 90% of the parameters is being pruned), without compromising the performance. It natural to think that, if we could have a way, train a small network from the scratch, also obtains similar performance as the large network, saving the energy for training. According to current experience, a pruned sparse network is hard to train from start....

1. Abstract When using an over-parameterized model, the author found the model can be pruned without losing the most accuracy. The author identifies the essential weight using LWM to obtain the mask. For pruned model, train on source domain with regularization. Then, transfer the model to the target domain, freeze the un-pruned parameter on the source domain, and train only the pruned parameter on the target domain. 2. Contribution Very first using pruning in transfer learning....