Lees-Milne came from a prosperous manufacturing family and grew up in Wickhamford, Worcestershire. He attended Lockers Park School in Hertfordshire, Eton and Oxford University. From 1931 to 1935, he was Private Secretary to George Lloyd, 1st Baron Lloyd of Dolobran.

In 1936 he was appointed secretary of the Country House Committee of the National Trust; he held that position until 1950, apart from a period of military service from 1939-1941. During that time he was a regular contributor to the Trust's member newsletter, penning various features. He was instrumental in the first large-scale transfer of country houses from private ownership to the Trust. After resigning his full-time position in 1950, he continued his connection with the National Trust as a part-time architectural consultant and member of committees.

Lees-Milne was visiting Diana Mosley when King Edward VIII abdicated. His visit there was to examine the seventeenth-century house she and her husband Sir Oswald Mosley were then renting; he recorded later how he and Diana (her husband was in London) had listened to the King's broadcast abdication speech with tears running down their faces. He had loved her brother Tom Mitford at Eton, and was devastated when Tom was killed in action in Burma in 1945.

He married Alvilde Chaplin, formerly Bridges, a prominent gardening and landscape expert, in 1951. Alvilde Lees-Milne died in 1994. Both Lees-Milne and Alvilde were bisexual, and for a period Alvilde had lesbian affairs with Vita Sackville-West and the wealthy Winnaretta Singer, among others.

After a period at Alderley Edge in Gloucestershire, and a brief sojourn in Bath he and Alvilde resided in their later years after 1974 at Essex House on the Badminton estate, also in Gloucestershire, while he worked most days in William Thomas Beckford's library at Lansdown Crescent. He was a friend of many of the most prominent British intellectual and social figures of his day, including Nancy Mitford, Harold Nicolson (about whom he wrote a two-volume biography), Deborah Mitford, and Cyril Connolly.

From 1947 Lees-Milne published a series of architectural works aimed primarily at the general reader. He was also a diarist, and his diaries were published in many volumes and were well received, in later years attracting a cult following. His other works included several biographies and an autobiographical novel.

Lees-Milne died in hospital at Tetbury on 28 December 1997. His ashes, together with those of Alvilde, were scattered in the grounds of Essex House.

An authorized biography by Michael Bloch, his friend and literary executor, was published by John Murray in September 2009 (ISBN 978-0-7195-6034-7).