I don't see how it could be an infringement of anyone's rights.  Your client has implicitly signed up to this in their contract with the auditing organisation, and you have no contract with the auditing organisation, as your contract is solely with your client.

However I am baffled as to how an effective audit can be performed in the absence of the 'competent person'.

By the way, I am an employee, and I most certainly do challenge auditors about their interpretation, especially when they say X is a legal requirement, and I know it's not.  I see this as part of my job. Furthermore, it is part of the job of the auditor to respond to such challenges and either back them up or back down.  In my experience they back down :-)

Perhaps it is the 'competent person' in general they need to keep out of the audit instead of just those who are acting on a consultancy basis.

Do I guess correctly that your clients need these audits from that particular organisation and it's not a case where they could choose a different auditing organisation?

Edited by user 29 February 2020 17:14:54(UTC)  | Reason: "organiation" isn't a word

Maybe use 'quote' so that the original remains visible in replies would minimise this kind of truncation.

I find quotes tend to end up like TV programmes where you get a synopsis of what's happened before and after every ad break.

The missing OP read to me as a consultant engaged by a client to be their competent H&S person perceived they had a legal right to be present at any and all meetings. A certification provider (of what was not specified so whether H&S has any relevance was indeterminate) decided externals were not welcome to be present at their audits of said client.