As with many other places, the early manorial history of Broughton is somewhat sketchy. For the most part, one has to rely on the evidence to be found in such documents as have survived, and these do not always make it clear when land changed hands. What is certain is that the Saxon settlement of Brotone (derived from 'broc' and 'tun' meaning a settlement on a brook) was mentioned in the Domesday Book. It then consisted of two manors - one held by Walter Giffard, a loyal supporter of William the Conqueror, and the other by Countess Judith, a more shadowy figure.

Walter Giffard sub-let his land to Hugh de Bolebec whose main holding was at Whitchurch near Aylesbury, and he was the tenant at the time of the Domesday Survey. However, the de Bolebecs in their turn sub-let the estate to a family who took the name of the place, the de Broughtons. The Giffard manor then remained with this family until the end of the fourteenth century until it was acquired by the de Aylesbury family who also held nearby Milton Keynes. To complicate matters, Robert de Broughton retained a small holding in the village including that originally held by Countess Judith.

When Hugh de Aylesbury died in 1423 his land at Broughton passed to his aunt Eleanor who became the wife of Sir Humphrey Stafford of Grafton in Northamptonshire. Thus both Milton Keynes and Broughton came to be held by the Stafford family.

Meanwhile, the lands retained by the de Boughtons passed by marriage to the Howard family. Agnes Howard married William Paulet, and in 1572 they sold their holding to Thomas Duncombe. About a year later, Humphrey Stafford (the last Humphrey in the list below) also sold his interest in Broughton to Thomas Duncombe so that the manors were at last combined.

Listed below are some of the significant dates.olor