...
Perform a dry run of the test case and once QTP is invoked, insert a debug point.
Navigate to the QTP menu Resources > Associated Function Libraries > {Open the custom lib to debug}
The custom file will be open, the custom action can now be debugged as required.