Do you want something like this?
roundCornerFilter = NSClassFromString("GlyphsFilterRoundCorner")
roundCornerFilter.roundLayer_radius_checkSelection_visualCorrect_grid_( Layer, noodleRadius, False, True, False )
Feel free to poke around in the Noodler source code.
The runFilter methods are geared for different API approaches, e.g. as custom parameters etc., and may not lend itself for calling it from outside. What you want is probably NSClassFromString("RoundedFont").roundedLayer_error_(layer,None).
You can find stuff like that quickly with my Method Reporter script: PutNSClassFromString("RoundedFont") into the class field, and filter out the methods which contain layer.