Joseph Fossett was the grandson of Elizabeth (Betty) Hemings and the son of Mary Hemings Bell. According to overseer Edmond Bacon, Fossett, a blacksmith, was "a very fine workman; could do anything it was necessary to do with steel or iron."
Joseph and his wife, Edith, Monticello's chief cook from 1809, had ten children. Fossett was freed in Jefferson's will, but the rest of his family was sold at the Monticello estate sale in 1827. He continued to work as a blacksmith and, with the help of his mother and other free family members, was able to purchase the freedom of his wife and some of their children. They moved to Cincinnati, Ohio, about 1840.