IMO the biggest downside is that you need to use a Pixel: a $500-$900 phone with the build quality of a $200 phone. You can get them cheaper used or refurbished, but it's still not the best value proposition.
Battery life and camera quality (no manual focus, no focus locking, autofocus is slow and inaccurate) are pretty bad. It's also missing the physical slider for silent mode that many phones have. Also some usb-c aux adapters do not work. These problems are exactly the same in vanilla android as they are in Graphene OS, so the hardware is the weak link.
If you are happy with Android on a Pixel, you will be happy with GrapheneOS