Systems and methods are disclosed for controlling a vehicle by generating a multi-dimensional model of a vehicle operating in a 3D environment; determining a hand control gesture as captured by a plurality of cameras or sensors in the vehicle, wherein a sequence of finger, palm or hand movements represents a vehicle control request; determining vehicle control options based on the model, a current state of the vehicle and the environment of the vehicle; and controlling the vehicle to operate based on the model and the 3D environment.