Vectors extending beyond ascender/descender metrics

Working on a font of ampersands, a few of them have some big flourishes that extend vertically way past the ascender/descender metrics.

Should I be concerned about this? Should I adjust the metrics to cover the extreme case scenarios?

Depends on in which app they should work. You will likely run into trouble in Office software.