Yeah, there's no agent-side filtering, and the only manager-side filtering is not very granular (i.e. you can turn off ALL ObjectAudits from being sent to your console/database/correlation rules, but not only certain ones).
There is a related feature request (it talks about Windows Filtering Platform events, but could be applied to these as well): Windows Events Prefilter on the Agent