A tool holder for a tool with a recess having detent disposed therein for holding the tool axially, and having a sleeve telescoped over a collet which has flexible fingers and with the sleeve and the fingers having camming surfaces for effecting clamping onto the tool. The axial locations of the recess and the detent and the camming surfaces are such that they are all in an engaged position when the sleeve is in one position along the axis. The sleeve and the collet have chamfers interengagable upon telescopic movement between the sleeve and the collet for flexing the fingers to thereby have the collet slide into the sleeve.