The short summary is "Yeah, it's what pretty much anyone would call mostly harmless".

The long summary is "It is not completely without risk, especially for those rare people with a complicating factor that leaves them prone to B12 deficiency, but chronic symptoms only come from high risk individuals and chronic heavy exposure". And the reports of "nitrous psychosis" are pretty much just rumor, from what I know there have been no studies that found you could "go crazy" from nitrous oxide exposure. The recorded neurological effects are minor and reversible in all but the most extreme cases of complicating factors.

N2O MSDS for anyone curious. Aside from potential respiratory depression from inhalation, there are no side effects listed.

More recent literature has examined the association with N2O toxicity and B12 deficiency. Long term sustained exposure to high levels of N2O can in some rare circumstances cause B12 deficiency, leading to neuropathy/parathesia and associated neurological phenomena (side note that in all the literature I've seen it is treatable).

It's also quite uncommon to experience these side effects even with heavy use:

Findings from a survey conducted by Winstock and Ferris completed by over 16,000 recreational nitrous oxide users suggest that neurologic symptoms may be dose dependent with a 3.5% increase in reports of paresthesia with every 10% increase in nitrous oxide dose used

Incidentally the only deaths reported from N2O use have been from suffocation. However it's important to note that clinical and recreational use should be avoided by people with a condition that makes them prone to B12 deficiency:

It is also important to screen those with an underlying susceptibility to vitamin B12 deficiency for nitrous oxide use, as the clinical manifestations of vitamin B12 deficiency have been noted not only in long-term nitrous oxide users, but also after a single exposure in those with susceptibility to vitamin B12 deficiency.

Just make sure the room is ventilated because nitrogen is heavier than the air, it will slowly gather and you could suffocate without realizing it.