Archaeological excavations have shown that the first castle on the site was a Norman motte and bailey, built sometime in the mid 11th century. Following the Norman Conquest, the Umfraville family took over control of the castle. Robert d?Umfraville was formally granted the barony of Prudhoe by Henry I but it is likely that the Umfravilles had already been granted Prudhoe in the closing years of the 11th century. The Umfravilles (probably Robert) initially replaced the wooden palisade with a massive rampart of clay and stones and subsequently constructed a stone curtain wall and gatehouse.
In 1173 William the Lion of Scotland invaded the North East to claim the earldom of Northumberland. The head of the Umfraville family, Odinel II, refused to support him and as a result the Scottish army tried to take Prudhoe Castle. The attempt failed as the Scots were not prepared to undertake a lengthy siege. The following year William attacked the castle again but found that Odinel had strengthened the garrison, and after a siege of just three days the Scottish army left. Following the siege, Odinel further improved the defences of the castle by adding a stone keep and a great hall.