Rose was born in Bedford–Stuyvesant, Brooklyn, and spent his childhood there and in Crown Heights in the same borough of New York City, and was raised by a single mother Sylvia Subin, his father having deserted the family by the time he was born. He and Woody Allen, then known as Allan Stewart Königsberg, first met at their high school, and became close friends, frequently skipping school, and playing jazz and baseball together. They matriculated at New York University, from which Rose earned a bachelor's degree in film, although Allen dropped out. The two remained friends for the remainder of Rose's life.
Rose's earliest material was for the ventriloquist Shari Lewis in her act with the sock-puppet Lamb Chop. After Allen had become a stand-up comedian, Rose co-wrote "The Moose" routine with him. Around this time, they collaborated with others on the English adaptation of a Japanese spy film, which was turned into What's Up, Tiger Lily? (1966), Allen's first film as director.
Later the two men collaborated on Allen's comedies Take the Money and Run (1969) and Bananas. After early work with Allen, Rose was a TV comedy writer, working for comedians such as Johnny Carson, while he was the host of The Tonight Show, Dean Martin (The Dean Martin Show, 1973) and Sid Caesar (1963). He also wrote for the Smothers Brothers and All in the Family. His other screenplays for films were for I Wonder Who's Killing Her Now? (1975) and Student Bodies (1981); he also directed the latter.
Rose was married to Judy Wolf from 1963 until her death in 2003; they were introduced by Allen and his first wife, Harlene Rosen. Allen was Rose's best man at the couple's wedding. Rose, with his wife and children, had relocated to the West Coast, settling in Southern California in 1970, and Rose died on 7 April 2013 at his home in Beverly Hills, California from colon cancer. He was survived by a daughter, son, and two grandchildren.
His son Quincy Rose is also a writer and filmmaker.