US Patent 12108184 Representing real-world objects with a virtual reality environment

An image processing system enables a user wearing a head-mounted display to experience a virtual environment combined with a representation of a real-world object. The image processing system receives a captured scene of a real-world environment that includes a target object. The image processing system identifies the target object in the captured scene and generates a representation of the target object. In some cases, the image processing system may include a graphical overlay with the representation of the target object. The image processing system can generate a combined scene that includes the target object and the virtual environment. The combined scene is presented to the user, thereby allowing the user to interact with the real-world target object (or a representation thereof) in combination with the virtual environment.