James made his first credited film appearances in Night Beat and Black Memory in 1947, both crime dramas. He played the alcoholic hero's barman in Powell and Pressburger's The Small Back Room in 1949. The Lavender Hill Mob in 1951 was his first comedy film, ranked 17th out of the 100 best British films by the British Film Institute: with Alfie Bass, he made up the bullion robbery gang headed by Alec Guinness and Stanley Holloway. He also appeared in Lady Godiva Rides Again and The Galloping Major, both films were released in 1951, and as Harry Hawkins in The Titfield Thunderbolt (1953), and also had a lead role in The Wedding of Lilli Marlene. He featured in another Alec Guinness film, Father Brown (US: The Detective, 1954) and in Trapeze (1956) as Harry the snake charmer, a circus film which was one of the most successful films of its year, and he played Master Henry in "Outlaw Money" (also 1956), an episode of The Adventures of Robin Hood. He also appeared in a 1956 episode of the TV series The Buccaneers 'The Hand of the Hawk', starring Robert Shaw.
James had a supporting part as a TV advertisement producer in Charlie Chaplin's A King in New York, a non-comic supporting role as a journalist in the science-fiction film Quatermass 2, and he performed in Hell Drivers (all 1957), a film with Stanley Baker. This was one of many appearances James made for the Rank Organisation.
The next year, James starred with Miriam Karlin in East End, West End by Wolf Mankowitz, a half-hour comedy series for the ITV company Associated Rediffusion. Set within the Jewish community of London's East End, the series of six episodes was transmitted in February and March 1958, but plans for further episodes were abandoned after a disappointing response. For a while though, it had looked as if his commitment elsewhere might end his work with Tony Hancock, one of the most popular television comedians of the time.
He had begun working with Tony Hancock in 1954, in his BBC Radio series Hancock's Half Hour. Having seen him in The Lavender Hill Mob, it was the idea of Hancock's writers, Galton and Simpson, to cast James. He played a character called Sid James (but having the invented middle name Balmoral) who was a petty criminal and would usually manage to con Hancock in some way, although the character eventually ceased to be Hancock's adversary. With the exception of James, the other regular cast members of the radio series were dropped when the series made the transition to television. His part in the show now greatly increased, and many viewers came to think of Hancock and James as a double act.