At a high level, it really shouldn’t matter much in this day and age. Personally I only go above 1000UPM if the design calls for it. Microsoft recommends a UPM value for TTF files that is a power of 2, but I think it’s safe to ignore this guidance
FYI, the Glyphs custom parameter description of the UPM:
…The value specified is the amount of units that will be used for the font size. A smaller value will cause the font to appear larger on screen, and vice versa. …
That is not wrong. It means if you just change that value, and don’t scale the rest of the font, The apparent font size will change. That is a good way to adjust the rendering size (e.g. to match it to other fonts).