7
The Lottery Ticket Hypothesis: finding sparse trainable NNs with 90% less params [2018] (arxiv.org)
submitted 10 months ago by to c/auai@programming.dev
4 comments fedilink hide all child comments
 

cross-posted from: https://lemmy.bestiver.se/post/844165

Comments

you are viewing a single comment's thread
view the rest of the comments
[โ€“] [S] 3 points 10 months ago* (1 child)

A superficial search returned:

2020: https://github.com/rahulvigneswaran/Lottery-Ticket-Hypothesis-in-Pytorch

2024: https://arxiv.org/pdf/2403.04861

2025: https://github.com/gabrielolympie/moe-pruner

But yeah, in hindsight, I've been hearing about this stuff since 2019, it is not that new, given everything else. I added the paper date to the title.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [โ€“] 4 points 10 months ago

    Working pruning techniques are tested and seem at least good at maintaining coherent transformer MOE models. https://doi.org/10.48550/arXiv.2510.13999

    There are several working examples of REAP pruned models HuggingFace and that method seems very good.

    The op paper suggests a technique which starts with an arbitrary structured expers pruned during training. I'm not 100% understanding it, but I still don't think I've seen this exact technique which might be even more efficient

  • permalink
  • fedilink
  • source
  • parent