US Patent 10976804 Pointer-based interaction with a virtual surface using a peripheral device in artificial reality environments

In some examples, a method comprises: obtaining, by an artificial reality system including a head mounted display (HMD), image data via an image capture device, the HMD configured to output artificial reality content; detecting, by the artificial reality system, a physical peripheral device from the image data; detecting, by the artificial reality system, a pose of the peripheral device; generating, by the artificial reality system, based on the pose of the peripheral device, a virtual pointer along at least a portion of a line between the peripheral device and a virtual surface, the virtual pointer pointing at a location of the virtual surface; performing, by the artificial reality system, one or more actions based on the location of the virtual surface; and rendering, by the artificial reality system, the virtual pointer and the virtual surface for display at the HMD.