Friedrich Hölderlin was cared for by the Tübingen master carpenter Ernst Friedrich Zimmer (and later by his daughter Lotte) from 1807 until his death in 1843 due to his unstable mental condition. He lived in the chamber in a fortified tower that had formerly belonged to the medieval city fortifications and which Zimmer had acquired at the beginning of 1807. In 1875, the building, which had been remodelled several times, burned down to the ground floor. A successor building was erected on the old foundation walls. In 1921, the city of Tübingen acquired the house. In 1954 they handed over the ground floor and the first floor to the Hölderlin Society, which had been founded in the meantime and set up a first exhibition. After the tower room was also made accessible in 1978, the building was thoroughly renovated in 1983/84. The new permanent exhibition opened in 1985. Further renovation work from 2017 onwards made it possible to include another floor as well as the garden.

Even during Hölderlin's lifetime, many fellow writers travelled to Tübingen to visit the poet, who was considered insane, in the tower.