Tooyama Yasumitsu was a vassal of the Houjou Clan, son of Tooyama Naokage and younger brother of Tooyama Tsunakage. He was master of Shinjou Castle in Sagami. His wife was Uesugi Kagetora's aunt (mother's older sister).

In 1569, Yasumitsu and his son Tooyama Yasuhide were sent by Houjou Ujiyasu to participate in peace negotiations with Uesugi Kenshin in the Echigo-Sagami Alliance, and when Ujiyasu's son Saburou was adopted by Kenshin in 1570, he went with Saburou to Echigo. When Kagetora was defeated by Uesugi Kagekatsu in the Otate no Ran, Yasumitsu committed suicide alongside him.