Yeah. That's a huge issue rust has. However, it can't be solved with Rc.
You either do it in safe rust, by "cheating" the borrow checker and storing a size offset of the buffer instead of a reference. Or just use unsafe rust and store a raw pointer alongside the buffer.