Groups in (custom) list filters

This is not possible as far as I know. You can add custom headings in the glyphOder custom parameter like described here: