I believe I got into some trouble similar to what you’re describing when I didn’t have the numbers in my instances properly coordinated with those in my masters. I’m not sure I’m remembering exactly, but I think some of the numbers in my instances may have been out of range. So that’s one thing to check.
Your weight numbers look unusual. Why not use the standard numbers–300 for light, 700 for bold, etc. as in the Font Info → Instance panel? The recommendation for the Optical Size axis is to use point sizes. Is 12 really the smallest size you’re designing for?
Microsoft’s OpenType specification pages are a huge help when laying out your axes: