Audley Moore was born in New Iberia, Louisiana, on July 27, 1898. Her parents were both dead by the time she was 14, and she became the primary support for her two sisters. She worked as a nurse during World War I, and after moving to Anniston, Alabama, organized the first USO for African American soldiers who had been denied entrance by the official USO organization. She also assisted them in receiving medical care and food.