Charles Hayman, an Eastbourne medical practitioner and member of the town's first council, together with other prominent local citizens, decided an independent school should be established and the support of William Cavendish, 7th Duke of Devonshire, was sought. He was supportive of the venture and provided 12 acres (5 hectares) of land for purchase at a modest price. This link with the Cavendish family is evidenced by the stag in the arms of the school.
From 1867 to 1869, it occupied Ellesmere Villa, now called Spencer Court; the location is now marked by a blue plaque. Architect Henry Currey was assigned by the duke to design a new school building, and College House, now School House, was built in 1870. The school chapel was constructed that same year.
During the 1880s, the school went through an impoverished period. Through the intervention of George Wallis, first mayor of Eastbourne and the work of new headmaster Charles Crowden, formerly of Cranbrook School, the school was saved from financial disaster.