Jarrell was a native of Nashville, Tennessee and graduated from Vanderbilt University. At Vanderbilt, he was acquainted with poets of the Fugitives group. Jarrell followed critic John Crowe Ransom from Vanderbilt to Kenyon College in Gambier, Ohio, where Jarrell wrote a masters thesis on the poetry of Alfred Edward Housman, and roomed with writers Robie Macauley and Peter Taylor, and poet Robert Lowell. Lowell and Jarrell remained good friends and peers until Jarrell's death. According to Lowell biographer Paul Mariani, "Jarrell was the first person of [Lowell's] own generation [whom he] genuinely held in awe" due to Jarrell's brilliance and confidence even at the age of 23.

Jarrell finished his Master's degree from Vanderbilt in 1938, then went on to teach at the University of Texas at Austin from 1939 to 1942, where he met his first wife, Mackie Langham. In 1942 he left the university to join the United States Army Air Forces, and according his obituary, he "[started] as a flying cadet, [then] he later became a celestial navigation tower operator, a job title he considered the most poetic in the Air Force." His early poetry would focus on the subject of his war-time experiences in the Air Force.

The Jarrell obit goes on to state that "after being discharged from the service he joined the faculty of Sarah Lawrence College in Bronxville, N.Y., for a year before going to the then Woman's College of the University of North Carolina where, as an associate professor of English, he taught modern poetry and imaginative writing."

Some of the awards that he received during his lifetime included a Guggenheim Fellowship for 1947-48, a grant from the National Institute of Arts and Letters in 1951, and the National Book Award in 1961.

On October 14, 1965, while walking along a road in Chapel Hill near dusk, Jarrell was struck by a car and killed. The coroner ruled the death accidental, but Jarrell had recently been treated for mental illness and a previous suicide attempt, so some of the people closest to Jarrell suspected that he might have committed suicide. In a letter to Elizabeth Bishop about a week after Jarrell's death, Robert Lowell wrote, "There's a small chance [that Jarrell's death] was an accident. . . [but] I think it was suicide, and so does everyone else, who knew him well." Still, Jarrell's second wife Mary (whom he'd married in 1952) held that his death was an accident.

On February 28, 1966, a memorial service was held in Jarrell's honor at Yale University, and some of the best-known poets in the country attended and spoke at the event, including Robert Lowell, Richard Wilbur, John Berryman, Stanley Kunitz, and Robert Penn Warren. Reporting on the memorial service, the New York Times quoted Lowell who said that Jarrell was, "'the most heartbreaking poet of our time'. . . [and] had written 'the best poetry in English about the Second World War.'"

In 2004, the Metropolitan Nashville Historical Commission approved placement of a historical marker in his honor, to be placed at Hume-Fogg High School, which he attended.

His first collection of poetry, Blood from a Stranger, which was heavily influenced by W.H. Auden, was published in 1942 — the same year he enlisted in the United States Army Air Corps. His second and third books, Little Friend, Little Friend (1945) and Losses (1948), drew heavily on his Army experiences. It was in these books that Jarrell broke free of Auden's influence and developed his own style and poetic philosophy which he would later document in his critical essays. "The Death of the Ball Turret Gunner" is the most famous of Jarrell's war-poems and one that is frequently anthologized. It presents the soldier as innocent and child-like, placing blame for war on "the State."

However, during this period, he earned a reputation primarily as a critic, rather than as a poet. Encouraged by Edmund Wilson, who published Jarrell's criticism in The New Republic, Jarrell quickly became a fiercely humorous critic of fellow poets. In the post-war period, his criticism began to change, showing a more positive emphasis. His appreciations of Robert Lowell, Elizabeth Bishop, and William Carlos Williams helped to establish or resuscitate their reputations as significant American poets, and his poet/friends often returned the favor, as when Lowell wrote a review of Jarrell's book of poems, The Seven League Crutches in 1951. Lowell wrote that Jarrell was "the most talented poet under forty, and one whose wit, pathos, and grace remind us more of Pope or Matthew Arnold than of any of his contemporaries." In the same review, Lowell calls Jarrell's first book of poems, Blood for A Stranger "a tour-de-force in the manner of Auden." And in another book review for Jarrell's Selected Poems, a few years later, fellow-poet Karl Shapiro compared Jarrell to "the great modern Rainer Maria Rilke" and stated that the book "should certainly influence our poetry for the better. It should become a point of reference, not only for younger poets, but for all readers of twentieth-century poetry."

Jarrell is also noted for his essays on Robert Frost ... whose poetry was a large influence on Jarrell's own ... Walt Whitman, Marianne Moore, Wallace Stevens, and others, which were mostly collected in Poetry and the Age (1953). Many scholars consider him the most astute poetry critic of his generation, and in 1979, the author and poet Peter Levi went so far as to advise younger writers, "Take more notice of Randall Jarrell than you do of any academic critic."

His reputation as a poet was not firmly established until 1960, when his National Book Award-winning collection The Woman at the Washington Zoo was published. His final volume, The Lost World, published in 1965, cemented that reputation; many critics consider it his best work. The book's subject, one of Jarrell's favorites, is childhood. Jarrell also published a satiric novel, Pictures from an Institution, in 1954 (nominated for the 1955 National Book Award) ... drawing upon his teaching experiences at Sarah Lawrence College, which served as the model for the fictional Benton College ... and several children's stories, among which The Bat-Poet (1964) and The Animal Family (1965) are considered prominent. He translated poems by Rainer Maria Rilke and others, a play by Anton Chekhov, and several Grimm fairy tales. He also served as Consultant in Poetry to the Library of Congress ... a position today known as Poet Laureate Consultant in Poetry ... from 1956-1958.