7
The Lottery Ticket Hypothesis: finding sparse trainable NNs with 90% less params [2018] (arxiv.org)
submitted 10 months ago by to c/auai@programming.dev
4 comments fedilink hide all child comments
 

cross-posted from: https://lemmy.bestiver.se/post/844165

Comments

you are viewing a single comment's thread
view the rest of the comments
[โ€“] 4 points 10 months ago

Working pruning techniques are tested and seem at least good at maintaining coherent transformer MOE models. https://doi.org/10.48550/arXiv.2510.13999

There are several working examples of REAP pruned models HuggingFace and that method seems very good.

The op paper suggests a technique which starts with an arbitrary structured expers pruned during training. I'm not 100% understanding it, but I still don't think I've seen this exact technique which might be even more efficient

  • permalink
  • fedilink
  • source
  • parent