Export layer font to one Color OTF

Working on a font with 4 masters, every master has a Master Color.
Exported with an extra instance and “Color Layers to SVG” checked.
The font looks good in Photoshop, except the spacing is very wide and irregular.

Is it impossible to have the same Glyphs file for making a layer font and a multicolor font?

It should just work. Can you send me the .glyphs and .otf file?