[–] 2 points 5 months ago (1 child)

Did you follow a guide for setting up Speculative Decoding? I haven't gotten it to work very well personally. Does the smaller model run on your CPU memory and the larger one fully on GPU?

  • source