A driving handover control device includes a memory and a processor coupled to the memory. In a case in which driving is handed over from a first state in which a vehicle is traveling by remote driving by a first driver from outside the vehicle or by occupant driving by the first driver in the vehicle, to a second state in which the vehicle travels by the remote driving or the occupant driving by a second driver who is different from the first driver, the processor is configured to cause transition from the first state to a third state in which the vehicle is caused to travel by automatic driving, and then cause transition from the third state to the second state.