Dover House was designed by James Paine as the London townhouse of Sir Matthew Fetherstonhaugh, Bart., MP, in the 1750s. It was remodelled by Henry Flitcroft, as “Montagu House”, for George Montagu, created 1st Duke of Montagu, who had moved from nearby Bloomsbury. It was refurbished once again, by Henry Holland for the Prince Frederick, Duke of York, from 1788 to 1792. The building belonged to the Melbourne family from 1793 to 1830.
It has also been home to a French ambassador and Lady Caroline Lamb, with whom the Romantic poet Lord Byron famously had an affair. Its most notable feature is an entrance hall in the form of a rotunda inserted into the former forecourt by Holland, which is a unique entrance to a London mansion. The last private owners were the family of the Whig politician George James Welbore Agar-Ellis, created (1831) Baron Dover, whose title it has retained.
The Agar-Ellis heirs owned Dover House from 1830 to 1885, when it became the Scottish Office, the UK government department responsible for Scottish affairs.
Although mostly used for the Scottish Office, this building was used by the Colonial Office for several years from 1941 onwards. In 1946 the Glasgow Herald speculated that “It will not be many years before the building will be put to demolition and new Government offices raised on the site.” It was still in use by the Colonial Office when Zionist terrorists planted a bomb there in April 1947.