An audio system for customizing sound fields for increased user privacy. A microphone array of a headset detects sounds from one or more sound sources in a local area of the headset. The audio system estimates array transfer functions (ATFs) associated with the sounds, and determines determining sound field reproduction filters for a loudspeaker array of the headset using the ATFs. The audio system presents audio content, via the loudspeaker array, based in part on the sound field reproduction filters. The presented audio content has a sound field that has a reduced amplitude in a first damped region of the local area that includes a first sound source of the one or more sound sources.