A driving handover control device includes a memory and a processor coupled to the memory. In a case in which driving is handed over from a first state in which a vehicle travels in accordance with instruction from a first remote operator of the vehicle to a second state in which the vehicle travels in accordance with instruction from a second remote operator who is different from the first remote operator, the processor implements a transition from the first state to a third state in which the vehicle travels autonomously without instruction from the first remote operator or the second remote operator, and, after the transition from the first state to the third state, implements a transition from the third state to the second state.