Question: Is there a tool or script to order features.fea so longer ligatures are arranged by the longest substitution first, so they have the correct priority in ordering lookups?
Your first example is not incorrect, it is exactly what someone might want in some circumstances. So I don’t think an automated tool could guess where reordering is desired and where it should be left as is.
If you use the automatic feature generation for features such as liga, Glyphs will generate the code in the correct oder since Glyphs knows the intention of the code.
I did some more testing because this issue was confusing me more than I expected it to :) Automatic reordering is supported by Glyphs 2 and Glyphs 3, but it does not apply to the sample code from the first post.
A contiguous set of ligature rules is reordered (in makeOTF, Glyphs 2, and Glyphs 3), but only ligature rules, so once a single substitution rule is present the reordering does not work anymore.
Consider the following liga code:
sub a by X;
sub a b by Y;
sub a b c by Z;
This is shaped as follows (input text → displayed glyphs):
a → X
ab → Xb
abc → Xbc
The first feature rule (sub a by X) is not a ligature rule. If we remove it, we have the following liga code:
sub a b by Y;
sub a b c by Z;
This code is shaped as such:
a → a
ab → Y
abc → Z
So here the rules are reordered and abc is correctly displayed as Z and not as Yc. This applies to makeOTF/Glyphs 2 and Glyphs 3.
A lookup can only contain substitutions of one kind, so I guess that
sub a by X;
sub a b by Y;
sub a b c by Z;
is internally split into 2 lookups, one with a single substitution type:
sub a by X;
and one with a ligature substitution type:
sub a b by Y;
sub a b c by Z;
The rule in the first lookup is evaluated before the second lookup is being processed, so the a never appears in the input glyph sequence for the second lookup.
This may give the impression that the rules are not reordered when in fact they are, per lookup.