They are correct - nitrous oxide inhalation does not automatically cause brain damage.

When inhalation of high concentrations of nitrous oxide is discontinued, high partial pressure in blood transfers nitrous oxide to the alveoli rapidly. This dilutes the partial pressure of oxygen in the alveoli and may lead to hypoxemia. [...] This concern is more theoretical than clinical, however. Hypoxemia is significant for only a matter of minutes and has been documented only when high concentrations (>70%) have been delivered by full mask or by endotracheal tube. These conditions cannot be met with the use of conventional dental nitrous oxide machines with nasal masks, and any tendency for diffusion hypoxia is trivial.

^(^^source^^)^

(edit: clarity, typo)