you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 3 months ago

My idempotent claim was too broad and wrong, up to aliasing that is reversible.

I think I’ve been talking about this differently than you, when you get 8bit into, 255 in/out is correct.

When generating data I do it in the natural [0,1] space, then map to whatever domain I’m quantizing for.

In that case, for N bins you divide by N+1 (and round however you want via correction) so 256. I was probably wrong earlier around this, 256 is correct for quantizing that way.

  • source
  • parent