Langston retired in 1894, after which he wrote "From the Virginia Plantation to the National Capital", his autobiography. He died in Washington, D.C., on Nov. 15, 1897. The town of Langston, Okla., home of Langston University, is named after him.

The site for Langston Golf Course, named for John Mercer Langston, was selected in 1929 as a reaction to the demand for an African American golf course in the District.Langston was officially opened in 1939, as a nine hole course, with the strong support of Harold Ickes, Secretary of the Interior.

Langston's home near Oberlin College in Ohio was placed on the National Register in 1975. In 1983, Oberlin's middle school on North Main Street was named for him.