His parents were Sir Lewis Leigh Fermor, a distinguished geologist, and Muriel Aeyleen (née Ambler).

Shortly after Patrick was born, his mother left to join his father in India, leaving him in England with another family. As a child, Leigh Fermor had problems with academic structure and limitations. As a result, he was sent to a school for difficult children. He was later expelled from The King's School, Canterbury when he was caught holding hands with a local greengrocer's daughter. He continued learning by reading texts on Greek, Latin, Shakespeare and History, with the intention of entering the Royal Military Academy Sandhurst.