An automatic headlamp control system for a vehicle which has an image sensor having a forward field of view that captures image data of a scene forward of the vehicle. The image processor processes the image data and detects objects of interest. A controller is responsive to the image processing to control a beam illumination state of the headlamps of the equipped vehicle. The controller is operable to detect at least a partial blocking condition at the image sensor and is operable to limit switching to a different beam illumination state in response to a signal indicative of an outside temperature at the equipped vehicle being below a threshold temperature level.