CAS servers role for outlook clients only, control autodiscover XML

Is it possible to limit client connection to a few CAS servers only. Lets say CAS1 & CAS2 are hardware boxes & CAS3 is a virtual.

CAS1 -> Outlook clients

CAS2 -> Outlook clients

CAS3 -> for outgoing mail

Set-ClientAccessServer allows you to control AutoDiscoveryService URL, but this will not exclude the server as RPC/HTTP proxy. Clients keep connecting to CAS3.

So Set-ClientAccessServer will not allow you to exclude a CAS server, outlook clients will still connect to it, as server CAS3 is listed inside the autodiscover.xml.

If you look into the XML it says

Would creating a DNS record "cas.company.local" with CAS1 & CAS2 IP be enough to force the clients to the two CAS servers?

Set-outlookanywhere -server CAS1 -InternalHostname cas.company.local
Set-outlookanywhere -server CAS2 -InternalHostname cas.company.local
Set-outlookanywhere -server CAS2 -InternalHostname cas.company.local

Any other options to controle the behavior?