James Jory, Senior, was born in the village of St. Cleer, in Cornwall, England, on August 7, 1787. His father was gamekeeper and gardener on and English estate, and James himself became a carpenter and mechanic. He married Mary Stephens Jory on September 28, 1814, and all of their eight children — two daughters and six sons — save their youngest child, Hugh, were born in England.