A method includes forming isolation regions extending into a semiconductor substrate, forming a plurality of semiconductor fins protruding higher than top surfaces of the isolation regions, forming a gate stack on the plurality of semiconductor fins, forming a gate spacer on a sidewall of the gate stack, and recessing the plurality of semiconductor fins to form a plurality of recesses on a side of the gate stack. The plurality of recesses extend to a level lower than top surfaces of the isolation regions. Epitaxy processes are performed to grow an epitaxy region, wherein the epitaxy region fills the plurality of recesses.