According to the 1920 United States Federal Census, he was married with his own family. He married Daisy Morris Smith. Xavier was twenty-two and his wife Daisy was eighteen. They had one child living with them at the time name Helen Smith. Ten years later in 1930, he and Daisy other children were: Xavier, Jr., Glen, Dorothy and Clarence Smith. By 1940 United States Federal Census they had given birth to two more children: Earl Smith and Yvonne Smith. They were living in Ried's Quarter. Xavier owned his own home and worked as a laborer.
Xavier Smith, Sr. was a very active and dedicated member at Grant Chapel A.M.E. Church. He joined the congregation of Grant Chapel in1916. He served 70 years of continued service and was considered to be the church's historian and father. He passed away on November 10, 1993 in Amite, Louisiana. He is buried at Butler Cemetery in Amite, Louisiana.