Cseh was selected to the 2004 Hungarian Olympic Team, which were held in Athens, Greece. There he competed in the same events that he advanced to the championship final in of the 2003 World Championships — 400 IM, 200 IM and the 100 backstroke. While in Olympic training camp, Cseh actually suffered an ankle injury, but when it came to competition, the didn’t hold him back. Cseh won his first Olympic medal when he won a bronze medal in the 400 IM. He nearly missed the medal podium with a fourth-place finish in the 200 IM, and he also finished sixth overall in the 100 back.