A modular building system consists of panels of various sizes and shapes that join together through a series of primary joints and secondary joints. The primary joint and secondary joint have key-to-keyway fasteners and wedge-to-slot fasteners. In this way any panel can attach to any other panel. Panels could be joined for form larger panels and cutouts or other features could be included in the panels.