There was a farm at Katholm from the fifteenth century. In 1545 Christen Fasti raised it to the status of a manor, and his son Thomas Fasti built the house that still stands - the east wing completed in 1588, the north wing in 1591. Thomas died in 1600. His widow Christence Bryske stayed on at Katholm for another eleven years, running the estate until her own death in 1611, and the couple were buried together in a chapel at the local parish church. The inheritance then went sideways and badly. Christence's brother took the estate, died within a few years, and left it to his two sons, Gert and Truid Bryske, who could not make the finances work. In 1616 they sold.

The buyer was Albret Skeel, and 1616 was his year: he was appointed Admiral of the Realm and admitted to the Rigsråd, the Privy Council, in the same twelve months. He finished what the Fastis had started. A low west wing went up in 1622, closing the courtyard on its third side, and a substantial working farm was laid out to the south - a barn in 1618, a stable in 1619, both still there. The architecture is deliberately plain for its date. Red brick, no ornament to speak of, two capped corner towers and a courtyard staircase surviving from the original build. The east wing carries a Dutch gable with curved contours; the younger, lower west wing has a crow-stepped gable instead, so the two ends of the house speak slightly different dialects. The courtyard opens south, where a masonry bridge crosses the moat toward the farm.

For most of the next two centuries Katholm passed through hands rather than families. The Skeels held it for generations until 1681, when Albret Skeel's granddaughter married Jens Maltesen Sehested; she died in 1684 and he bought out her mother and her brother-in-law to keep the place. In 1690 it went to Palle Krag, a soldier who had distinguished himself in the Scanian War and would do so again in the Great Northern War, and who had just married a wealthy widow, Helle Nielsdatter Trolle. They had no children. Krag died in the early 1720s, a year after his wife, and Katholm was sold to Major-General Poul Rosenørn. Three generations of Rosenørns followed before Mathias Peter Otto Rosenørn sold to Jens Jørgensen, a man who had spent years as a tenant at Rosenholm. Then came the collapse. The early nineteenth century broke Danish landowners in numbers, and in 1823 the State was obliged to take Katholm over along with a string of other estates.

In December 1839 the State put Katholm up at public auction, and First Lieutenant Adolph Wilhelm Dinesen bought it. He became a councillor and a member of parliament, and he had eight children. One of them, born in 1845, grew up on this island in its lake: Wilhelm Dinesen, who fought in the Second Schleswig War, then for France against Prussia, then spent sixteen months from August 1872 living among the Chippewa in Wisconsin, and came home to write Boganis' Jagtbreve - hunting letters that became a minor classic of Danish literature. He did not inherit Katholm; when his father died in 1876 the estate went to his elder brother Wentzel Laurentzius Dinesen. Wilhelm bought Rungstedlund north of Copenhagen instead, and there on 17 April 1885 his daughter Karen was born. She would publish as Isak Dinesen and be known in Denmark as Karen Blixen. Her father took his own life in March 1895, when she was nine.

In 1916 Holger Collet bought Katholm from a Dinesen daughter, and the family has held it ever since. Collet built a new stable, then a tenant residence and another stable with a garage - the routine expansion of a working estate. In 1935 fire tore through the farm buildings and did substantial damage. Carl Frederik Collet took over in 1942 and reorganised the operation on more efficient lines; Peter Collet succeeded him in 1971. Since 2007 Katholm has belonged to Marie Therese Collet, the fourth generation of her family here, running an estate of 1,170 hectares. The house is private, but the land outside the moat and the open roads across the property are accessible - including Hulvejen, which remains the best and the most alarming way to arrive.