Speer was born in Mannheim, Germany, into a wealthy middle class family. He was the second of three sons of Albert and Luise Speer. In 1918, the family moved permanently to their summer home, Schloss-Wolfsbrunnenweg, in Heidelberg. According to Henry T. King, deputy prosecutor at Nuremberg who later wrote a book about Speer, "Love and warmth were lacking in the household of Speer's youth." Speer wanted to become a mathematician, but his father said if Speer chose this occupation he would "lead a life without money, without a position, and without a future". Instead, Speer followed in the footsteps of his father and grandfather and studied architecture.

Speer began his architectural studies at the University of Karlsruhe instead of a more highly acclaimed institution because the hyperinflation crisis of 1923 limited his parents' income. In 1924 when the crisis had abated, he transferred to the "much more reputable" Technical University of Munich. In 1925 he transferred again, this time to the Technical University of Berlin where he studied under Heinrich Tessenow, whom Speer greatly admired. After passing his exams in 1927, Speer became Tessenow's assistant, a high honor for a man of 22. As such, Speer taught some of Tessenow's classes while continuing his own postgraduate studies. In Munich, and continuing in Berlin, Speer began a close friendship, ultimately spanning over 50 years, with Rudolf Wolters, who also studied under Tessenow.

In mid-1922, Speer began to date Margarete (Margret) Weber (1905—1987). The relationship was frowned upon by Speer's class-conscious mother, who felt that the Webers were socially inferior (Herr Weber was a successful craftsman who employed 50 workers). Despite this opposition, the two married in Berlin on August 28, 1928, though seven years were to elapse before Margarete Speer was invited to stay at her in-laws' home.

Joining the Nazis (1930—1934)

Speer stated he was apolitical when he was a young man, and that he attended a Berlin Nazi rally in December 1930 at the urging of some of his students. He was surprised to find Hitler dressed in a neat blue suit, rather than the brown uniform seen on Nazi Party posters, and was greatly impressed, not only with Hitler's proposals, but also with the man himself. Several weeks later he attended another rally, though this one was presided over by Joseph Goebbels. Speer was disturbed by the way Goebbels whipped the crowd into a frenzy. Despite this unease, Speer could not shake the impression Hitler had made on him. On March 1, 1931, he applied to join the Nazi Party and became member number 474,481.

Speer's first Nazi Party position was as head of the Party's motorist association for the Berlin suburb of Wannsee; he was in fact the only Nazi in the town with a car. Speer reported to the Party's leader for the West End of Berlin, Karl Hanke, who hired Speer — without fee — to redecorate a villa he had just rented. Hanke was enthusiastic about the resulting work.

In 1931, Speer surrendered his position as Tessenow's assistant because of pay cuts and moved to Mannheim, hoping to use his father's connections to get commissions. He had little success, and his father gave him a job as manager of the elder Speer's properties. In July 1932, the Speers visited Berlin to help out the Party prior to the Reichstag elections. While they were there, Hanke recommended the young architect to Goebbels to help renovate the Party's Berlin headquarters. Speer, who had been about to leave with his wife for a vacation in East Prussia, agreed to do the work. When the commission was completed, Speer returned to Mannheim and remained there as Hitler took office in January 1933.

After the Nazis took control, Hanke recalled Speer to Berlin. Goebbels, the new Propaganda Minister, commissioned Speer to renovate his Ministry's building on Wilhelmplatz. Speer also designed the 1933 May Day commemoration in Berlin. In Inside the Third Reich, he wrote that, on seeing the original design for the Berlin rally on Hanke's desk, he remarked that the site would resemble a Schützenfest — a rifle club meet. Hanke, now Goebbels' State Secretary, challenged him to create a better design. As Speer learned later, Hitler was enthusiastic about Speer's design (which used giant flags), though Goebbels took credit for it. Tessenow was dismissive: "Do you think you have created something? It's showy, that's all."

The organizers of the 1933 Nuremberg Nazi Party rally asked Speer to submit designs for the rally, bringing him into contact with Hitler for the first time. Neither the organizers nor Rudolf Hess were willing to decide whether to approve the plans, and Hess sent Speer to Hitler's Munich apartment to seek his approval. When Speer entered, the new Chancellor was busy cleaning a pistol, which he briefly laid aside to cast a short, interested glance at the plans, approving them without even looking at the young architect. This work won Speer his first national post, as Nazi Party "Commissioner for the Artistic and Technical Presentation of Party Rallies and Demonstrations".

Speer's next major assignment was as liaison to the Berlin building trades for Paul Troost's renovation of the Chancellery. As Chancellor, Hitler had a residence in the building and came by every day to be briefed by Speer and the building supervisor on the progress of the renovations. After one of these briefings, Hitler invited Speer to lunch, to the architect's great excitement. Hitler evinced considerable interest in Speer during the luncheon, and later told Speer that he had been looking for a young architect capable of carrying out his architectural dreams for the new Germany. Speer quickly became part of Hitler's inner circle; he was expected to call on Hitler in the morning for a walk or chat, to provide consultation on architectural matters, and to discuss Hitler's ideas. Most days he was invited to dinner.

The two men found much in common: Hitler spoke of Speer as a "kindred spirit" for whom he had always maintained "the warmest human feelings". The young, ambitious architect was dazzled by his rapid rise and close proximity to Hitler, which guaranteed him a flood of commissions from the government and from the highest ranks of the Party. Speer testified at Nuremberg, "I belonged to a circle which consisted of other artists and his personal staff. If Hitler had had any friends at all, I certainly would have been one of his close friends."

First Architect of the Third Reich (1934—1939)

When Troost died on January 21, 1934, Speer effectively replaced him as the Party's chief architect. Hitler appointed Speer as head of the Chief Office for Construction, which placed him nominally on Hess's staff.

One of Speer's first commissions after Troost's death was the Zeppelinfeld stadium...the Nuremberg parade grounds seen in Leni Riefenstahl's propaganda masterpiece Triumph of the Will. This huge work was capable of holding 340,000 people. The tribune was influenced by the Pergamon Altar in Anatolia, but was magnified to an enormous scale. Speer insisted that as many events as possible be held at night, both to give greater prominence to his lighting effects and to hide the individual Nazis, many of whom were overweight. Speer surrounded the site with 130 anti-aircraft searchlights. This created the effect of a "cathedral of light" or, as it was called by British Ambassador Sir Neville Henderson, a "cathedral of ice". Speer described this as his most beautiful work, and as the only one that stood the test of time.

Nuremberg was to be the site of many more official Nazi buildings, most of which were never built; for example, the German Stadium would have accommodated 400,000 spectators, while an even larger rally ground would have held half a million Nazis. While planning these structures, Speer invented the concept of "ruin value": that major buildings should be constructed in such a way that they would leave aesthetically pleasing ruins for thousands of years into the future. Such ruins would be a testament to the greatness of the Third Reich, just as ancient Greek or Roman ruins were symbols of the greatness of those civilizations. Hitler enthusiastically embraced this concept, and ordered that all the Reich's important buildings be constructed in accord with it.

Speer could not avoid seeing the brutal excesses of the Nazi regime. Shortly after the Night of the Long Knives, Hitler ordered Speer to take workmen and go to the building housing the offices of Vice-Chancellor Franz von Papen to begin its conversion into a security headquarters, even though it was still occupied by von Papen's officials. Speer and his group entered the building, to be confronted with a pool of blood, apparently from the body of Herbert von Bose, von Papen's secretary, who had been killed there. Speer related that the sight had no effect on him, other than to cause him to avoid that room.

When Hitler deprecated Werner March's design for the Olympic Stadium for the 1936 Summer Olympics as too modernistic, Speer modified the plans by adding a stone exterior. Speer designed the German Pavilion for the 1937 international exposition in Paris. The German and Soviet pavilion sites were opposite each other. On learning (through a clandestine look at the Soviet plans) that the Soviet design included two colossal figures seemingly about to overrun the German site, Speer modified his design to include a cubic mass which would check their advance, with a huge eagle on top looking down on the Soviet figures. Both pavilions were awarded gold medals for their designs. Speer would also receive, from Hitler Youth Leader and later fellow Spandau prisoner Baldur von Schirach, the Golden Hitler Youth Honor Badge with oak leaves.

In 1937, Hitler appointed Speer as General Building Inspector for the Reich Capital with the rank of undersecretary of state in the Reich government. The position carried with it extraordinary powers over the Berlin city government and made Speer answerable to Hitler alone. It also made Speer a member of the Reichstag, though the body by then had little effective power. Hitler ordered Speer to make plans to rebuild Berlin. The plans centered on a three-mile long grand boulevard running from north to south, which Speer called the Prachtstrasse, or Street of Magnificence; he also referred to it as the "North-South Axis". At the north end of the boulevard, Speer planned to build the Volkshalle, a huge assembly hall with a dome which would have been over high, with floor space for 180,000 people. At the southern end of the avenue would be a huge triumphal arch; it would be almost high, and able to fit the Arc de Triomphe inside its opening. The outbreak of World War II in 1939 led to the postponement, and eventual abandonment, of these plans. Part of the land for the boulevard was to be obtained by consolidating Berlin's railway system. Speer hired Wolters as part of his design team, with special responsibility for the Prachtstrasse. When Speer's father saw the model for the new Berlin, he said to his son, "You've all gone completely insane."

In January 1938, Hitler asked Speer to build a new Reich Chancellery on the same site as the existing structure, and said he needed it for urgent foreign policy reasons no later than his next New Year's reception for diplomats on January 10, 1939. This was a huge undertaking, especially since the existing Chancellery was in full operation. After consultation with his assistants, Speer agreed. Although the site could not be cleared until April, Speer was successful in building the large, impressive structure in nine months. The structure included the "Marble Gallery": at 146 metres long, almost twice as long as the Hall of Mirrors in the Palace of Versailles. Speer employed thousands of workers in two shifts. Hitler, who had remained away from the project, was overwhelmed when Speer turned it over, fully furnished, two days early. In appreciation for the architect's work on the Chancellery, Hitler awarded Speer the Nazi Golden Party Badge. Tessenow was less impressed, suggesting to Speer that he should have taken nine years over the project. The second Chancellery was damaged by the Battle of Berlin in 1945 and was eventually dismantled by the Soviets, its stone used for a war memorial.

During the Chancellery project, the pogrom of Kristallnacht took place. Speer would make no mention of it in the first draft of Inside the Third Reich, and it was only on the urgent advice of his publisher that he added a mention of seeing the ruins of the Central Synagogue in Berlin from his car.

Speer was under significant psychological pressure during this period of his life. He would later remember:

Soon after Hitler had given me the first large architectural commissions, I began to suffer from anxiety in long tunnels, in airplanes, or in small rooms. My heart would begin to race, I would become breathless, the diaphragm would seem to grow heavy, and I would get the impression that my blood pressure was rising tremendously... Anxiety amidst all my freedom and power!

Wartime architect (1939—1942)

Speer supported the German invasion of Poland and subsequent war, though he recognized that it would lead to the postponement, at the least, of his architectural dreams. In his later years, Speer, talking with his biographer-to-be Gitta Sereny, explained how he felt in 1939: "Of course I was perfectly aware that [Hitler] sought world domination ... [A]t that time I asked for nothing better. That was the whole point of my buildings. They would have looked grotesque if Hitler had sat still in Germany. All I wanted was for this great man to dominate the globe."

Speer placed his department at the disposal of the Wehrmacht. When Hitler remonstrated, and said it was not for Speer to decide how his workers should be used, Speer simply ignored him. Among Speer's innovations were quick-reaction squads to construct roads or clear away debris; before long, these units would be used to clear bomb sites. As the war progressed, initially to great German success, Speer continued preliminary work on the Berlin and Nuremberg plans, at Hitler's insistence, but failed to convince him of the need to suspend peacetime construction projects. Speer also oversaw the construction of buildings for the Wehrmacht and Luftwaffe, and developed a considerable organization to deal with this work.

In 1940, Joseph Stalin proposed that Speer pay a visit to Moscow. Stalin had been particularly impressed by Speer's work in Paris, and wished to meet the "Architect of the Reich". Hitler, alternating between amusement and anger, did not allow Speer to go, fearing that Stalin would put Speer in a "rat hole" until a new Moscow arose. When Germany invaded the Soviet Union in 1941, Speer came to doubt, despite Hitler's reassurances, that his projects for Berlin would ever be completed.

Appointment and increasing power

On February 8, 1942, Minister of Armaments Fritz Todt died in a plane crash shortly after taking off from Hitler's eastern headquarters at Rastenburg. Speer, who had arrived in Rastenburg the previous evening, had accepted Todt's offer to fly with him to Berlin, but had canceled some hours before takeoff (Speer stated in his memoirs that the cancellation was because of exhaustion from travel and a late-night meeting with Hitler). Later that day, Hitler appointed Speer as Todt's successor to all of his posts. In Inside the Third Reich, Speer recounts his meeting with Hitler and his reluctance to take ministerial office, only doing so because Hitler commanded it. Speer also states that Hermann Göring raced to Hitler's headquarters on hearing of Todt's death, hoping to claim Todt's powers. Hitler instead presented Göring with the fait accompli of Speer's appointment, causing Göring to leave without even attending Todt's funeral.

At the time of Speer's accession to the office, the German economy, unlike the British one, was not fully geared for war production. Consumer goods were still being produced at nearly as high a level as during peacetime. No fewer than five "Supreme Authorities" had jurisdiction over armament production — one of which, the Ministry of Economic Affairs, had declared in November 1941 that conditions did not permit an increase in armament production. Few women were employed in the factories, which were running only one shift. One evening soon after his appointment, Speer went to visit a Berlin armament factory; he found no one on the premises.

Speer overcame these difficulties by centralizing power over the war economy in himself. Factories were given autonomy, or as Speer put it, "self-responsibility", and each factory concentrated on a single product. Backed by Hitler's strong support (the dictator stated, "Speer, I'll sign anything that comes from you"), he divided the armament field according to weapon system, with experts rather than civil servants overseeing each department. No department head could be older than 55 — anyone older being susceptible to "routine and arrogance" — and no deputy older than 40. Over these departments was a central planning committee headed by Speer, which took increasing responsibility for war production, and as time went by, for the German economy itself. According to the minutes of a conference at Wehrmacht High Command in March 1942, "It is only Speer's word that counts nowadays. He can interfere in all departments. Already he overrides all departments ... On the whole, Speer's attitude is to the point." Goebbels would note in his diary in June 1943, "Speer is still tops with the Führer. He is truly a genius with organization." Speer was so successful in his position that by late 1943, he was widely regarded among the Nazi elite as a possible successor to Hitler.

While Speer had tremendous power, he was of course subordinate to Hitler. Nazi officials sometimes went around Speer by seeking direct orders from the dictator. When Speer ordered peacetime building work suspended, the Gauleiters (Nazi Party district leaders) obtained an exemption for their pet projects. When Speer sought the appointment of Hanke as a labor czar to optimize the use of German labor, Hitler, under the influence of Martin Bormann, instead appointed Fritz Sauckel. Rather than increasing female labor and taking other steps to better organize German labor, as Speer favored, Sauckel advocated importing labor from the occupied nations — and did so, obtaining workers for (among other things) Speer's armament factories, using the most brutal methods.

On December 10, 1943, Speer visited the underground Mittelwerk V-2 rocket factory that used concentration camp labor. Shocked by the conditions there (5.7 percent of the work force died that month), and to ensure the workers were in good enough shape to perform the labor, Speer ordered improved conditions for the workers and the construction of the above-ground Dora camp. In spite of these changes, half of the workers at Mittelwerk eventually died. Speer later commented, "[t]he conditions for these prisoners were in fact barbarous, and a sense of profound involvement and personal guilt seizes me whenever I think of them."

By 1943, the Allies had gained air superiority over Germany, and bombings of German cities and industry had become commonplace. However, the Allies in their strategic bombing campaign did not concentrate on industry, and Speer, with his improvisational skill, was able to overcome bombing losses. In spite of these losses, German production of tanks more than doubled in 1943, production of planes increased by 80 percent, and production time for Kriegsmarine's submarine was reduced from one year to two months. Production would continue to increase until the second half of 1944, by which time enough equipment to supply 270 army divisions was being produced...although the Wehrmacht had only 150 divisions in the field.

In January 1944, Speer fell ill with complications from an inflamed knee, and was away from the office for three months. During his absence, his political rivals mainly Göring, and Martin Bormann, attempted to have some of his powers permanently transferred to them. According to Speer, SS chief Heinrich Himmler tried to have him physically isolated by having Himmler's personal physician Karl Gebhardt treat him, though his "care" did not improve his health. His wife and friends managed to have his case transferred to his friend Dr. Karl Brandt, and Speer slowly recovered. In April, Speer's rivals for power succeeded in having him deprived of responsibility for construction, and Speer promptly sent Hitler a bitter letter, concluding with an offer of his resignation. Judging Speer indispensable to the war effort, Field Marshal Erhard Milch persuaded Hitler to try to get his minister to reconsider. Hitler sent Milch to Speer with a message not addressing the dispute but instead stating that he still regarded Speer as highly as ever. According to Milch, upon hearing the message, Speer burst out, "The Führer can kiss my ass!" After a lengthy argument, Milch persuaded Speer to withdraw his offer of resignation, on the condition his powers were restored. On April 23, 1944, Speer went to see Hitler who agreed that "everything [will] stay as it was, [Speer will] remain the head of all German construction". According to Speer, while he was successful in this debate, Hitler had also won, "because he wanted and needed me back in his corner, and he got me".

Fall of the Reich

Speer's name was included on the list of members of a post-Hitler government drawn up by the conspirators behind the July 1944 assassination plot to kill Hitler. The list had a question mark and the annotation "to be won over" by his name, which likely saved him from the extensive purges that followed the scheme's failure.

By February 1945, Speer, who had long concluded that the war was lost, was working to supply areas about to be occupied with food and materials to get them through the hard times ahead. On March 19, 1945, Hitler issued his Nero Decree, ordering a scorched earth policy in both Germany and the occupied territories. Hitler's order, by its terms, deprived Speer of any power to interfere with the decree, and Speer went to confront Hitler, telling him the war was lost. Hitler gave Speer 24 hours to reconsider his position, and when the two met the following day, Speer answered, "I stand unconditionally behind you." However, he demanded the exclusive power to implement the Nero Decree, and Hitler signed an order to that effect. Using this order, Speer worked to persuade generals and Gauleiters to evade the Nero Decree and avoid needless sacrifice of personnel and destruction of industry that would be needed after the war.

Speer managed to reach a relatively safe area near Hamburg as the Nazi regime finally collapsed, but decided on a final, risky visit to Berlin to see Hitler one more time. Speer stated at Nuremberg, "I felt that it was my duty not to run away like a coward, but to stand up to him again." Speer visited the Führerbunker on April 22. Hitler seemed calm and somewhat distracted, and the two had a long, disjointed conversation in which the dictator defended his actions and informed Speer of his intent to commit suicide and have his body burned. In the published edition of Inside the Third Reich, Speer relates that he confessed to Hitler that he had defied the Nero Decree, but then assured Hitler of his personal loyalty, bringing tears to the dictator's eyes. Speer biographer Gitta Sereny notes, "Psychologically, it is possible that this is the way he remembered the occasion, because it was how he would have liked to behave, and the way he would have liked Hitler to react. But the fact is that none of it happened; our witness to this is Speer himself." Sereny goes on to note that Speer's original draft of his memoirs lacks the confession and Hitler's tearful reaction, and contains an explicit denial that any confession or emotional exchange took place, as had been alleged in a French magazine article.

The following morning, Speer left the Führerbunker, with Hitler curtly bidding him farewell. Speer toured the damaged Chancellery one last time before leaving Berlin to return to Hamburg. On April 29, the day before his suicide, Hitler prepared his final political testament. That document excluded Speer from the Cabinet and specified that Speer was to be replaced by his subordinate, Karl-Otto Saur.

After Hitler's death, Speer offered his services to the so-called Flensburg Government, headed by Hitler's successor, Karl Dönitz, and took a significant role in that short-lived regime. On May 15, the Americans arrived and asked Speer if he would be willing to provide information on the effects of the air war. Speer agreed, and over the next several days, provided information on a broad range of subjects. On May 23, two weeks after the surrender of German troops, the Allies arrested the members of the Flensburg Government and brought Nazi Germany to a formal end.

Speer was taken to several internment centers for Nazi officials and interrogated. In September 1945, he was told that he would be tried for war crimes, and several days later, he was taken to Nuremberg and incarcerated there. Speer was indicted on all four possible counts: first, participating in a common plan or conspiracy for the accomplishment of crime against peace, second, planning, initiating and waging wars of aggression and other crimes against peace, third, war crimes, and lastly, crimes against humanity.

U.S. Supreme Court Justice Robert Jackson, the chief U.S. prosecutor at Nuremberg, alleged, "Speer joined in planning and executing the program to dragoon prisoners of war and foreign workers into German war industries, which waxed in output while the workers waned in starvation." Speer's attorney, Dr. Hans Flächsner, presented Speer as an artist thrust into political life, who had always remained a non-ideologue and who had been promised by Hitler that he could return to architecture after the war. During his testimony, Speer accepted responsibility for the Nazi regime's actions:

In political life, there is a responsibility for a man's own sector. For that he is of course fully responsible. But beyond that there is a collective responsibility when he has been one of the leaders. Who else is to be held responsible for the course of events, if not the closest associates around the Chief of State?

An observer at the trial, journalist and author William L. Shirer, wrote that, compared to his codefendants, Speer “made the most straightforward impression of all and ... during the long trial spoke honestly and with no attempt to shirk his responsibility and his guilt”. Speer also testified that he had planned to kill Hitler in early 1945 by dropping a canister of poison gas into the bunker's air intake. He said his efforts were frustrated by a high wall that had been built around the air intake. Speer stated his motive was despair at realizing that Hitler intended to take the German people down with him. Speer's supposed assassination plan subsequently met with some skepticism, with Speer's architectural rival Hermann Giesler sneering, "the second most powerful man in the state did not have a ladder."

Speer was found guilty of war crimes and crimes against humanity, though he was acquitted on the other two counts. On October 1, 1946, he was sentenced to 20 years' imprisonment. While three of the eight judges (two Soviet and one American) initially advocated the death penalty for Speer, the other judges did not, and a compromise sentence was reached "after two days' discussion and some rather bitter horse-trading".

The court's judgment stated that:

... in the closing stages of the war [Speer] was one of the few men who had the courage to tell Hitler that the war was lost and to take steps to prevent the senseless destruction of production facilities, both in occupied territories and in Germany. He carried out his opposition to Hitler's scorched earth program ... by deliberately sabotaging it at considerable personal risk.

Twelve of the defendants were sentenced to death (including Bormann, in absentia) and three acquitted; only seven of the defendants were sentenced to imprisonment. They remained in the cells at Nuremberg as the Allies debated where, and under what conditions, they should be incarcerated.

For additional detail on Speer's time at Spandau Prison, see Rudolf Wolters#Spandau years

On July 18, 1947, Speer and his six fellow prisoners, all former high officials of the Nazi regime, were flown from Nuremberg to Berlin under heavy guard. The prisoners were taken to Spandau Prison in the British Sector of what would become West Berlin, where they would be designated by number, with Speer given Number Five. Initially, the prisoners were kept in solitary confinement for all but half an hour a day, and were not permitted to address each other or their guards. As time passed, the strict regimen was relaxed, especially during the three months in four that the three Western powers were in control; the four occupying powers took overall control on a monthly rotation. Speer considered himself an outcast among his fellow prisoners for his acceptance of responsibility at Nuremberg.

Speer made a deliberate effort to make as productive a use of his time as possible. He wrote, "I am obsessed with the idea of using this time of confinement for writing a book of major importance ... That could mean transforming prison cell into scholar's den." The prisoners were forbidden to write memoirs, and mail was severely limited and censored. However, as a result of an offer from a sympathetic orderly, Speer was able to have his writings, which eventually amounted to 20,000 sheets, sent to Wolters. By 1954, Speer had completed his memoirs, which became the basis of Inside the Third Reich, and which Wolters arranged to have transcribed onto 1,100 typewritten pages. He was also able to send letters and financial instructions, and to obtain writing paper and letters from the outside. His many letters to his children, all secretly transmitted, eventually formed the basis for Spandau: The Secret Diaries.

With the draft memoir complete and clandestinely transmitted, Speer sought a new project. He found one while taking his daily exercise, walking in circles around the prison yard. Measuring the path's distance carefully, Speer set out to walk the distance from Berlin to Heidelberg. He then expanded his idea into a worldwide journey, visualizing the places he was "traveling" through while walking the path around the prison yard. Speer ordered guidebooks and other materials about the nations through which he imagined he was passing, so as to envision as accurate a picture as possible. Meticulously calculating every meter traveled, and mapping distances to the real-world geography, he began in northern Germany, passed through Asia by a southern route before entering Siberia, then crossed the Bering Strait and continued southwards, finally ending his sentence 35 kilometers south of Guadalajara, Mexico.

Speer devoted much of his time and energy to reading. Though the prisoners brought some books with them in their personal property, Spandau Prison had no library so books were sent from Spandau's municipal library.From 1952 the prisoners were also able to order books from the Berlin central library in Wilmersdorf. Speer was a voracious reader and he completed well over 500 books in the first three years at Spandau alone. He read classic novels, travelogues, books on ancient Egypt, and biographies of such figures as Lucas Cranach, Friedrich Preller, and Genghis Khan. Speer took to the prison garden for enjoyment and work, at first to do something constructive while afflicted with writer's block. He was allowed to build an ambitious garden, transforming what he initially described as a "wilderness" into what the American commander at Spandau described as "Speer's Garden of Eden".

Speer's supporters maintained a continual call for his release. Among those who pledged support for Speer's sentence to be commuted were Charles de Gaulle, U.S. diplomat George Ball, former U.S. High Commissioner John J. McCloy, and former Nuremberg prosecutor Hartley Shawcross. Willy Brandt was a strong advocate of Speer's, supporting his release, sending flowers to his daughter on the day of his release, and putting an end to the de-Nazification proceedings against Speer, which could have caused his property to be confiscated. A reduced sentence required the consent of all four of the occupying powers, and the Soviets adamantly opposed any such proposal. Speer served his full sentence, and was released on the stroke of midnight as October 1, 1966 began.

Speer's release from prison was a worldwide media event, as reporters and photographers crowded both the street outside Spandau and the lobby of the Berlin hotel where Speer spent his first hours of freedom in over 20 years. However, Speer said little, reserving most comments for a major interview published in Der Spiegel in November 1966, in which he again took personal responsibility for crimes of the Nazi regime. Abandoning plans to return to architecture (two proposed partners died shortly before his release), he revised his Spandau writings into two autobiographical books, and later researched and published a third work, about Himmler and the SS. His books, most notably Inside the Third Reich (in German, Erinnerungen, or Reminiscences) and Spandau: The Secret Diaries, provide a unique and personal look into the personalities of the Nazi era, and have become much valued by historians. Speer was aided in shaping the works by Joachim Fest and Wolf Jobst Siedler from the publishing house Ullstein. However, Speer found himself unable to re-establish his relationship with his children, even with his son Albert, who had also become an architect. According to Speer's daughter Hilde, "One by one my sister and brothers gave up. There was no communication."

Following the publication of his bestselling books, Speer donated a considerable amount of money to Jewish charities. According to Siedler, these donations were as high as 80% of his royalties. Speer kept the donations anonymous, both for fear of rejection, and for fear of being called a hypocrite.

As early as 1953, when Wolters strongly objected to Speer referring to Hitler in the memoirs draft as a criminal, Speer had predicted that were the writings to be published, he would lose a "good many friends". This came to pass, as following the publication of Inside the Third Reich, close friends, such as Wolters and sculptor Arno Breker, distanced themselves from him. Hans Baur, Hitler's personal pilot, suggested, "Speer must have taken leave of his senses." Wolters wondered that Speer did not now "walk through life in a hair shirt, distributing his fortune among the victims of National Socialism, forswear all the vanities and pleasures of life and live on locusts and wild honey".

Speer made himself widely available to historians and other enquirers. He did an extensive, in-depth interview for the June 1971 issue of Playboy magazine, in which he stated, "If I didn't see it, then it was because I didn't want to see it." In October 1973, Speer made his first trip to Britain, flying to London under an assumed name to be interviewed on the BBC Midweek programme by Ludovic Kennedy. Upon arrival, he was detained for almost 8 hours at Heathrow Airport when British immigration authorities discovered his true identity. The Home Secretary, Robert Carr, allowed Speer into the country for 48 hours. While in London eight years later to participate in the BBC Newsnight programme, Speer suffered a stroke and died on September 1, 1981. Speer had formed a relationship with a German woman living in England, and was with her at the time of his death.

Even to the end of his life, Speer continued to question his actions under Hitler. In his final book, Infiltration, he asks, "What would have happened if Hitler had asked me to make decisions that required the utmost hardness? ... How far would I have gone? ... If I had occupied a different position, to what extent would I have ordered atrocities if Hitler had told me to do so?" Speer leaves the questions unanswered.

Architectural legacy

Little remains of Speer's personal architectural works, other than the plans and photographs. No buildings designed by Speer in the Nazi era remain in Berlin; a double row of lampposts along the Strasse des 17. Juni designed by Speer still stands. The tribune of the Zeppelinfeld stadium in Nuremberg, though partly demolished, may also be seen. Speer's work may also be seen in London, where he redesigned the interior of the German Embassy to the United Kingdom, then located at 7—9 Carlton House Terrace. Since 1967, it has served as the offices of the Royal Society. His work there, stripped of its Nazi fixtures and partially covered by carpets, survives in part.

A perhaps more important legacy was the Arbeitsstab Wiederaufbau zerstörter Städte (Working group on Reconstruction of destroyed cities), authorized by Speer in 1943 to rebuild bombed German cities to make them more livable in the age of the automobile. Headed by Wolters, the working group took a possible military defeat into their calculations. The Arbeitsstab's recommendations served as the basis of the postwar redevelopment plans in many cities, and Arbeitsstab members became prominent in the rebuilding.

Actions regarding the Jews

As General Building Inspector, Speer was responsible for the Central Department for Resettlement. From 1939 onwards, the Department used the Nuremberg Laws to evict Jewish tenants of non-Jewish landlords in Berlin, to make way for non-Jewish tenants displaced by redevelopment or bombing. Eventually, 75,000 Jews were displaced by these measures. Speer was aware of these activities, and inquired as to their progress. At least one original memo from Speer so inquiring still exists, as does the Chronicle of the Department's activities, kept by Wolters.

Following his release from Spandau, Speer presented to the German Federal Archives an edited version of the Chronicle, stripped by Wolters of any mention of the Jews. When David Irving discovered discrepancies between the edited Chronicle and other documents, Wolters explained the situation to Speer, who responded by suggesting to Wolters that the relevant pages of the original Chronicle should "cease to exist". Wolters did not destroy the Chronicle, and, as his friendship with Speer deteriorated, allowed access to the original Chronicle to doctoral student Matthias Schmidt (who, after obtaining his doctorate, developed his thesis into a book, Albert Speer: The End of a Myth). Speer considered Wolters' actions to be a "betrayal" and a "stab in the back". The original Chronicle reached the Archives in 1983, after both Speer and Wolters had died.

Knowledge of the Holocaust

Speer maintained at Nuremberg and in his memoirs that he had no knowledge of the Holocaust. In Inside the Third Reich, he wrote that in mid-1944, he was told by Hanke (by then Gauleiter of Lower Silesia) that the minister should never accept an invitation to inspect a concentration camp in neighboring Upper Silesia, as "he had seen something there which he was not permitted to describe and moreover could not describe". Speer later concluded that Hanke must have been speaking of Auschwitz, and blamed himself for not inquiring further of Hanke or seeking information from Himmler or Hitler:

These seconds [when Hanke told Speer this, and Speer did not inquire] were uppermost in my mind when I stated to the international court at the Nuremberg Trial that, as an important member of the leadership of the Reich, I had to share the total responsibility for all that had happened. For from that moment on I was inescapably contaminated morally; from fear of discovering something which might have made me turn from my course, I had closed my eyes ... Because I failed at that time, I still feel, to this day, responsible for Auschwitz in a wholly personal sense.

Much of the controversy over Speer's knowledge of the Holocaust has centered on his presence at the Posen Conference on October 6, 1943, at which Himmler gave a speech detailing the ongoing Holocaust to Nazi leaders. Himmler said, "The grave decision had to be taken to cause this people to vanish from the earth ... In the lands we occupy, the Jewish question will be dealt with by the end of the year." Speer is mentioned several times in the speech, and Himmler seems to address him directly. In Inside the Third Reich, Speer mentions his own address to the officials (which took place earlier in the day) but does not mention Himmler's speech.

In 1971, American historian Erich Goldhagen published an article arguing that Speer was present for Himmler's speech. According to Fest in his biography of Speer, "Goldhagen's accusation certainly would have been more convincing" had he not placed supposed incriminating statements linking Speer with the Holocaust in quotation marks, attributed to Himmler, which were in fact invented by Goldhagen. In response, after considerable research in the German Federal Archives in Koblenz, Speer said he had left Posen around noon (long before Himmler's speech) in order to journey to Hitler's headquarters at Rastenburg. In Inside the Third Reich, published before the Goldhagen article, Speer recalled that on the evening after the conference, many Nazi officials were so drunk that they needed help boarding the special train which was to take them to a meeting with Hitler. One of his biographers, Dan van der Vat, suggests this necessarily implies he must have still been present at Posen then, and must have heard Himmler's speech. In response to Goldhagen's article, Speer had alleged that in writing Inside the Third Reich, he erred in reporting an incident that happened at another conference at Posen a year later, as happening in 1943.

In 2005, British newspaper The Daily Telegraph reported that documents had surfaced indicating that Speer had approved the allocation of materials for the expansion of Auschwitz after two of his assistants toured the facility on a day when almost a thousand Jews were murdered. The documents supposedly bore annotations in Speer's own handwriting. Speer biographer Gitta Sereny stated that, due to his workload, Speer would not have been personally aware of such activities.

The debate over Speer's knowledge of, or complicity in, the Holocaust made him a symbol for people who were involved with the Nazi regime yet did not have (or claimed not to have had) an active part in the regime's atrocities. As film director Heinrich Breloer remarked, "[Speer created] a market for people who said, 'Believe me, I didn't know anything about [the Holocaust]. Just look at the Führer's friend, he didn't know about it either.'"