you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 3 months ago

The offset is needed because this quantization dividing by 256 uses truncation whereas by 255 uses rounding.

So a value of 245 (for example) could be anything between 245.00000 and 245.99999. Therefore, when dequantizing, we set the value to 245.5, which is in the middle. Between 245.00000 and 245.99999.

  • source
  • parent