Washington married three times. A private and complex man, he had the trauma of losing two wives. He married one of his Malden school pupils, Fanny Norton Smith in 1882. Their daughter Portia was born in 1883. Fanny died in 1884.
He then married Olivia Davidson in 1885. A Hampton graduate, Olivia was the assistant principal of Tuskegee. She had great influence on Washington and the development of his Northern philanthropic support. They had two sons, Booker T. Washington Jr. and Ernest Davidson Washington. Olivia died in 1889.
Washington then married Margaret James Murray in 1892. A teacher, Margaret became the Lady Principal of Tuskegee after Olivia's death. Margaret and Booker did not have children. In addition to her professional role on campus, Margaret ran a home for the entire Washington family at The Oaks. She died in 1925.