Creating a custom vector effect
A custom vector effect is basically just a node group. It's first input is the vector of the mapping node it is effecting and the first output is the vector that it gets changed to. All other inputs are then visible in the interface for the user to control the effect with.
In the example you can see the Color input and the Color output which represent the ones explained above. The Hue, Saturation, Value and the Fac are what is then visible in the interface:
You can create your own node groups this way and then load them into Layer Painter as explained below.
Input settings
The input settings are the same as in the custom filter section which you can find here.
Add to Layer Painter
You add a vector effect the same way you add a custom filter. Just make sure you are in the Vector FX section instead of the Filters in the addon preferences. You can find the explanation here.