Tradition relates that a castle was first constructed on this site by Rhodri the Great, but there are no remains from this period. Dinefwr later became the chief seat of Rhodri's grandson Hywel Dda, first ruler of Deheubarth and later king of most of Wales. Rhys ap Gruffydd, ruler of Deheubarth from 1155 to 1197, is thought to have rebuilt the castle. Giraldus Cambrensis tells a story about a plan by King Henry II of England to assault the castle during a campaign against Rhys. One of Henry's most trusted followers was sent on reconnaissance, guided by a local Welsh cleric, who was asked to lead him to the castle by the easiest route, but instead took the most difficult route he could find, ending the performance by stopping to eat grass with the explanation that this was the diet of the local people in times of hardship. The planned attack was duly abandoned.