It used to be that switching masters (cmd+number or clicking the icon) affected all glyphs in the Edit View, as opposed to clicking the layer in the layer panel. Now both does the same. Would be cool to have this back.
You need to switch several times to remove the manually set layers.