The earliest Hald went up somewhere between 1250 and 1300 and was apparently never completed. What survives is the earthwork: a rampart roughly eighty metres across, the kind of investment that only makes sense if you intend to hold ground. It is usually attributed to the Eberstein family, and the first documentary mention of Hald comes in 1328, when it belonged to the Rigsmarsk — the marshal of the realm — Ludvig Albertsen Eberstein. It was known then as Brattingsborg and stood east of the present main building. Then it stopped being used, and the next builder chose a different spot.

Niels Bugge bought Hald in 1346 and built himself a new main house, taking up residence there. Dendrochronology confirms him: oak piles driven beneath the surviving ruins were felled in the winter of 1347-48. Bugge became a leader of the Jutland nobility's rising against King Valdemar IV, and it killed him — he was murdered on the road home from peace negotiations at Slagelse that had come to nothing. His son-in-law Skarpenberg inherited Hald and soon had to sell it to Queen Margaret I, who in 1383 handed the estate over to the bishop's seat at Viborg. Within forty years of its building, the second Hald had passed from a rebel to the Crown to the Church.

The third Hald was raised in 1528 on a small peninsula reaching out into the lake, built for Jørgen Friis, Bishop of Viborg — and, as it turned out, the last Catholic one. He was building modern fortifications at exactly the moment when the thing he was fortifying against was not an army but an argument. Hans Tausen had begun preaching Lutheran reform in Viborg three years earlier. When the Reformation came, Friis retreated to Hald, and in 1536 he was imprisoned in his own castle along with Denmark's other Catholic bishops. He was released in 1540 on his oath to accept the new order and lived out his life as a landowner. The ruins on the peninsula, ringed by tall earthworks, are the most legible remains at Hald; the masonry tower there is partly a reconstruction.

The fourth Hald was completed in 1703 for General Gregers Daa, a large whitewashed half-timbered establishment of four wings with barns, stables and a paved courtyard, built partly from the stone of the demolished castle and standing a little south of the current house in what is now parkland. Nothing of it is left at all. The building that survives began as an outbuilding to that complex — a gatehouse, single-storeyed with a three-storey centre block, put up in the late eighteenth century, with two pavilions in the park dating from around 1795. It was converted into a dwelling in the early 1800s. Ove Høegh-Guldberg, who had effectively run the Danish government from 1772 to 1784, is the name attached to this fifth and final Hald.

The estate belongs to the Danish state now, 973 hectares of it, part farmed organically by the Danish Nature Agency and part forest under the Fussingø State Forest District. The manor is a conference and meeting venue, and since 2002 it has housed the Danish Centre for Writers and Translators, which gives writers, translators and illustrators free stays to work in peace and runs public literary events. A barn from the mid-eighteenth century, restored in 2008, holds an exhibition on the area's geology, nature and history, and Viborg Museum runs guided tours across the ruins. The English ghost-story writer M. R. James, who set his tale "Number 13" (1904) in Viborg, called Hald "accounted one of the prettiest things in Denmark" — a judgement that has held up rather better than any of the buildings did.