I am getting the same buggy behavior with bracket layers. It used to be that adding the bracket layers on composite glyphs would fix this; I posted a script for that here: Bracket layers in variable fonts? Bracket + brace?
I’m not sure what has changed but adding the bracket layers doesn’t work anymore
Perhaps something has changed on my end. I had been exploring this around the 2nd of July, if nothing has changed in that time, then my original solution must have been erroneous
Regardless, is there a way to get bracket layers working on a composite glyph when a glyph with bracket layers is used as a component?