In 1966, he heard a song about unrequited love written for The Spinners and persuaded the writers that he should record it himself. His recording of "What Becomes of the Brokenhearted" became a major success. The song reached No. 7 on the Billboard Hot 100 and No. 6 on the R&B Chart. It also initially reached No. 8 in the UK Singles Chart, rising to No. 4 when it was reissued in the UK in 1974. "What Becomes of the Brokenhearted" remains Ruffin's best-known song. It was the lead single from his debut album Jimmy Ruffin Sings Top Ten (released as The Jimmy Ruffin Way in the UK), which was released on the Motown's Soul subsidiary label in 1967. Follow-up singles in America were successful, with "I've Passed This Way Before" and "Gonna Give Her All The Love I've Got" in late 1966 and early 1967.
Ruffin's second album, Ruff 'n' Ready, was released in 1969. It contained the song "Don't You Miss Me a Little Bit Baby", which made the lower parts of the Billboard Hot 100 (No. 68) and was a Top 30 hit on the R&B Chart, peaking at No. 27. As a solo artist, it would prove to be Ruffin's last significant chart appearance in America for many years, and his last significant charting record for Motown in the US. The song was also released as the B-side to "What Becomes of the Brokenhearted" when it was reissued in 1974.
As Ruffin found success in the United States difficult to sustain, he began to concentrate instead on the British market. In 1970, "Farewell Is a Lonely Sound", "I'll Say Forever My Love" and "It's Wonderful (To Be Loved by You)" each made the UK Top Ten, and he was voted the world's top singer in one British poll. He also teamed up with brother David to record the album I Am My Brother's Keeper, a modestly successful 1970 album for Motown that included the songs "When The Love Hand Comes Down", "Your Love Was Worth Waiting For" and a cover of Ben E. King's "Stand by Me". His third solo album for the label, The Groove Governor, was released in 1970, and did not fare as well as his previous two albums.
Following the success of his initial hits, Ruffin found it hard to maintain an identity, as most of his songs were later covered by other Motown artists, most prominently "Everybody Needs Love", a hit when covered by Gladys Knight & the Pips for their 1967 debut album of the same name, "Maria (You Were The Only One)", a hit for Michael Jackson and "If You Let Me," a minor hit for Eddie Kendricks. In addition, he had recorded the first version of The Temptations hit "Beauty Is Only Skin Deep".