A first gate structure straddles one end of a staircase fin stack that contains a first semiconductor material fin, an insulator fin, and a second semiconductor material fin, a second gate structure straddles a portion of the staircase fin stack, a third gate structure straddles another end of the staircase fin stack, and a fourth gate structure straddles a portion of only the first semiconductor fin. A first contact structure is between the first and second gate structures, a second contact structure is between the second and third gate structures, and a third contact structure is between the third and fourth gate structures. The first contact structure has a contact metal that contacts the first and second semiconductor material fins. The second contact structure has a contact metal that contacts only the second semiconductor material fin, and the third contact structure has a contact metal that contacts only the first semiconductor fin.