Throughout his life, James Madison's views on slavery and his ownership of slaves were complex. James Madison grew up on a plantation that made use of slave labor. He viewed slavery as a necessary part of the Southern economy, though he was troubled by the instability of a society that depended on a large slave population.