Thomson was born in Oxford, where his father, William Thomson (who would later become Archbishop of York), was provost of The Queen's College. Thomson was educated at Worsley's School in Hendon and Eton College, and then attended New College, Oxford. Thomson ended his university studies after two terms, after suffering bouts of depression, and spent some time from 1881 to 1882 in the United States, working as a farmer in Iowa.

In 1883, with the promise of marriage to a Grace Webber should he be financially secure, Thomson secured a cadet position at the Colonial Office, where he assisted Sir William Des V?ux, then Governor of Fiji. Arriving in Fiji in early 1884, and set about learning the Fijian and Tongan languages while appointed as a stipendiary magistrate throughout the islands. When Sir William MacGregor was appointed administrator of British New Guinea, Thomson joined his staff until he was invalided back to England after contracting malaria.

Back in England, Thomson married Grace Webber in 1890, returning to Fiji with his wife in the middle of that year to serve as commissioner of native lands. When Sir John Thurston, the Governor of Fiji, dismissed the Premier of Tonga (Shirley Waldemar Baker) in his capacity as High Commissioner of the Western Pacific, Thomson was moved to Tonga, where he became assistant premier to Siaosi U. Tuku'aho, the pro-British chief appointed as Baker's replacement.

After three years at the Native Lands Office in Suva, Thomson resigned from colonial service, and returned to England in 1893, due in no small part to the deteriorating health of his wife. There he embarked on a career as a writer, drawing on his experiences in the South Sea Islands to produce South Sea Yarns (1894; written in Fiji), The Diversions of a Prime Minister (1894, about his government work in Tonga), and The Indiscretions of Lady Asenath (1898).

In the mid-1890s, Thomson read for the bar examinations at the Inner Temple, and was admitted to the bar in 1896. Instead of becoming a barrister, Thomson accepted the position of deputy governor at HM Prison Liverpool, after his name was suggested for the post due to a personal acquaintance with Sir Evelyn Ruggles-Brise, a fellow Old Etonian who had stayed with Thomson in Tonga. Over the next twelve years, he served as governor of Northampton, Cardiff, Dartmoor, and Wormwood Scrubs prisons.From 1908 to 1913, he served as secretary of the Prison Commission.

In 1899, the United Kingdom and Germany signed an agreement formalising each country's rights and claims over Tonga and Samoa respectively. Given his inside knowledge of Tongan politics, Thomson was tasked with expediting the establishment of a British protectorate over Tonga, which was established on 18 May 1900 despite Tongan resistance.

In June 1913, Thomson was appointed Assistant Commissioner "C" of London's Metropolitan Police, which made him the head of the Criminal Investigation Department (CID) at New Scotland Yard. When World War I broke out in 1914, the CID found itself acting as the enforcement arm for Britain's military intelligence apparatus: while the newly-formed Secret Service Bureau (later known as MI6, the Secret Intelligence Service), and the intelligence arms of the War Office and the Admiralty, collected intelligence on suspected spies in Britain, they had no arrest powers. As head of CID, Thomson was involved in the arrests in several high-profile espionage cases, including Lieutenant Carl Hans Lody and establishing himself a reputation as a "spycatcher". Thomson worked closely with the MI5, especially the MI5(g) headed by Vernon Kell and his works at the was key in dealing with the Indian nationalist movement in Europe. However, since the existence of the latter organisation was not acknowledged at the time, Thomson controversially claimed a large proportion of the credit in the successful British counter-espionage operations. In his memoirs, The Scene Changes, Thomson acknowledges only the works of Robert Nathan, who worked closely with him, and was involved in the interrogation of a number of Indian revolutionaries who worked with German Intelligence during the war. Thomson and Nathan's work at the time was key in identifying the plans by Ghadar Party and the Indian Independence Committee to assassinate Lord Kitchener in 1915 through an associate of Har Dayal, Gobind Behari Lal, as well as identifying the outlines of the Indian revolutionary conspiracy. Their efforts at the time also resulted in the capture of Harish Chandra (who was associated with the Berlin committee), and he was successfully turned into a double agent. Thomson's efforts were also key in uncovering the first concrete evidence of Turco-German agents operating in the middle east and attempting to destabillise Afghanistan and British India.

Thomson's work as Assistant Commissioner of Scotland Yard had its darker side. His natural conservatism was given full throttle against suffragettes, then against spies from Imperial Germany and its allies, then against Irish nationalists, and finally against British Marxists. Thomson was involved with the spreading of the "Black Diaries" used against Sir Roger Casement to prevent public support for a reduction of Casement's death sentence for high treason in 1916. He equated Bolsheviks only with Jews, and even wrote anti-Semitic articles about Jews for a newspaper, the Whitechapel Gazette, owned by the highly questionable social figure Maundy Gregory.

Thomson was appointed Companion of the Order of the Bath (CB) in 1916 and Knight Commander of the Order of the Bath (KCB) in 1919.

In 1919, while remaining Assistant Commissioner (Crime), he was appointed Director of Intelligence at the Home Office, in overall charge of every intelligence agency in the country, but in 1921 he fell out with Lloyd George and was asked to resign. The reasons for this remain mysterious.

In December 1925, Thomson was arrested in London's Hyde Park, and charged with "committing an act in violation of public decency" with a young woman, Miss Thelma de Lava. Thomson rejected the charges, insisting that he was engaged in conversation with the woman for the purposes of research for a book he was writing on London vice. Nonetheless, he was found guilty of public indecency, and fined £5.

In reality the story he gave the court (which his barrister, Sir Henry Curtis Bennett, probably did not support) sounded totally peculiar. Thomson apparently lied (or told a half-truth) regarding his name, calling himself "Home Thomson" when he was arrested with Miss de Lava. "Home" was one of his middle names. Unfortunately he was recognized by the police. He tried alternately to bluster or to offer a vague bribe to the constables. When he presented his version in the courtroom he said he was researching a book on the danger of left wing agitators in England and he was together with Miss de Lava awaiting for the speech to begin.Had this been true, Thomson should have revealed in court who the orator was he was awaiting for. He kept refusing. That plus the background of Miss de Lava as a prostitute ended any real credibility that Thomson thought would save him.

To be fair to Thomson, there is a slim chance he was partly set up. That is he was legitimately arrested, but given no chance to keep the matter quiet. The reason may be that Thomson was interested in an opening for a new Commissioner of Scotland Yard, a post for which he would have felt he was properly prepared. His enemies in the police may have made sure that the entire affair became public to discredit him.