Holmes' first musical foray was with his wife Katherine in the folk pop parody duo, Allen & Grier. Following military service, he resumed his music career. Among the highlights: Holmes put lyrics to Bob Gaudio's music on The Four Seasons' 1969 Genuine Imitation Life Gazette album, after which the pair went on to compose Frank Sinatra's 1970 Watertown album. Coming during a relative low point in Sinatra's career, Watertown was his least successful album, but the song "I Would Be in Love (Anyway)" reached No. 4 on the Billboard Adult Contemporary chart. The song "What's Now is Now" reached No. 31 on that chart and was later included in Frank Sinatra's Greatest Hits Volume 2. Sinatra's recording of the Gaudio-Holmes composition "Lady Day" was left off the Watertown album, but was released as a single, and Don Costa later rearranged "Lady Day" for inclusion in Sinatra's Sinatra & Company album (1971). In 1985, Nina Simone recorded a cover version of "For a While", from the Watertown album, for her Nina's Back album. That same year, she recorded a live version of "For a While" for her Live And Kickin' album.
On his own, Holmes recorded during the 1960s two well-regarded albums for EMI's Tower Records label: A Letter to Katherine December, and "The Above Ground Sound" of Jake Holmes, which contained the aforementioned "Dazed and Confused" and "Genuine Imitation Life". The Four Seasons' Bob Gaudio heard Holmes sing "Genuine Imitation Life" at The Bitter End in New York's City's Greenwich Village, which led to their collaborations on The Four Seasons and Sinatra albums.
Between those projects, Holmes, who had landed a recording contract with Polydor, went to Nashville to record an album called Jake Holmes. That was followed by the most successful solo album of his career, So Close, So Very Far to Go. Released by Polydor in 1970, it reached No. 135 on the Billboard album chart, and the single "So Close" rose to No. 49 on Billboard's Hot 100. In 1970, Lena Horne sang Holmes' "It's Always Somewhere Else" on a TV special made with Harry Belafonte and released on the album Harry & Lena. Belafonte commenced recording Holmes' songs on The Warm Touch (1971), followed by Play Me (the song "So Close", 1973), and Loving You Is Where I Belong (1981), culminating in a whole album, 1988's Paradise in Gazankulu. These were followed by the live recordings Belafonte '89 and An Evening with Harry Belafonte and Friends (1997). In 1977, "So Close" became the title song of an album by Helen Schneider, a popular New York nightclub .