Born in Carrollton, Mississippi, Spencer was valedictorian of her graduating class at local J.Z. George High School. She earned her Bachelor of Arts at Belhaven College in Jackson, Mississippi and her Master's Degree in Literature at Vanderbilt University in Nashville, Tennessee in 1943.

Spencer taught at the junior college level for two years before accepting a job with the Nashville Tennessean, but she soon returned to teaching, this time at the University of Mississippi in Oxford. In 1953 she was awarded a Guggenheim Fellowship and left Mississippi to live in Italy and pursue writing full time.

In 1957, the fiction jury of the Pulitzer Prize for Fiction recommended that she should be awarded that year's prize for her novel, The Voice at the Back Door, but the Pulitzer board, which has sole discretion for awarding the prize, made no award.

While in Italy, she met and married John Rusher of Cornwall, England. The couple moved to Canada in 1961, where they remained until moving to Chapel Hill, North Carolina in 1986. She taught creative writing at the University of North Carolina until her retirement. Rusher died in 1998, and Spencer continues to live in Chapel Hill.

Spencer's mother is the great aunt of John McCain.