US Patent 11931114 Virtual interaction with instruments in augmented reality

Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to an Interaction Engine that generates within a unified three-dimensional (3D) coordinate space: (i) a 3D virtual medical model positioned according to a model pose and (ii) at least one virtual object associated with a physical instrument, the physical instrument having a current instrument pose based at least on current coordinates of one or more fiducial markers disposed on the physical instrument, in the unified 3D coordinate space. The Interaction Engine renders an Augmented Reality (AR) display that includes concurrent display of the 3D virtual medical model and the virtual object. The Interaction Engine modifies the AR display according to a virtual interaction related to the virtual object that incorporates the change of the physical instrument pose.