Creating a custom mask
A custom mask is basically just a node group. It's first input is the color of the channel it is effecting and the first output is the color that it gets changed to. All other inputs are then visible in the interface for the user to control the mask with.
In the example you can see the Color input and the Color output which represent the ones explained above. The Hue, Saturation, Value and the Fac are what is then visible in the interface:
You can create your own node groups this way and then load them into Layer Painter as explained below.
Input settings
The input settings are the same as for the custom filters which you can find here.
When you select texture for an input you have the option to choose a baked mask. This will then make that input require that mask to be baked for it to work.
Add to Layer Painter
You add the custom masks the same way as you add the filters. You can find the explanation here.