Aalholm enters the written record in 1329, and not in a flattering way: King Christopher II pawned it to Count John the Mild, one of a string of humiliations in a reign spent mortgaging Denmark piece by piece. The castle was almost certainly older, probably raised around 1200 as part of a network of royal strongholds - Koldinghus, Riberhus, Nyborg, Gurre, Ravnsborg in northern Lolland - built to project a king's authority into the provinces. Its site was chosen with care. Aalholm stood on a small island in Nysted Nor, an inlet sheltered from the open Baltic but deep enough for ships whose crews knew the channel. Fertile land lay to the north and west, and between Nakskov at Lolland's western end and Stubbekøbing on Falster there was nowhere else remotely as good to put a castle. That the surrounding district mattered is confirmed by the Franciscans, who founded a monastery at Nysted in 1286 - the only one on these islands, and the order rarely settled anywhere that could not feed it.

No written source describes the first castle, and whatever archaeology survives lies buried under the present building. What can be dated is the north wing, thought to be 14th- and 15th-century work. After that the documentation becomes abundant, and much of it is about decay. In the 1550s the castle was patched using bricks robbed from the Franciscan monastery in Nysted. After the Swedish Wars of 1657-1660 it stood with no windows in its walls and no roofs on its towers. The 18th century brought demolition of part of the south wing and a full renovation of the east. Then in 1889 the architects Hans Jørgen Holm and Gotfred Tvede took another bite out of the south wing and added two lateral wings to the south side of the north wing. Seen on a plan, the once-rectangular castle now resembles a pair of spectacles. Nothing significant has changed outside since.

In 1725 Frederik IV sold Aalholm for 95,000 rigsdaler to Emerentia von Levetzow, widow of Johan Otto Raben, who went on to become one of the largest landowners on Lolland. Nine years later, by testament, she created the countship of Christiansholm for her grandson Christian Raben. He died childless at twenty-five. His younger brother Otto Ludvig Raben inherited, bought more land, and set up a factory on the estate making starch and face powder. Otto Ludvig's son Frederik Christian installed an astronomical observatory in the castle. Then the male line simply ran out: Frederik Christian's sons Gregers Christian and Julius both died without issue, and the estate crossed sideways to a descendant of Emerentia's own brother, Josias Levetzau, who took the compound name Raben-Levetzau. Under that name the family held Aalholm until 1995 - a run of well over a century that ended, as these things often do, in a sale.

In 1964 the Raben-Levetzau family opened something odder than a manor house tour: a veteran car museum in the outbuildings of Stubberup Farm, about a kilometre west of the castle. The castle itself followed in the 1970s, when parts of it opened to visitors as a manor house museum. The car collection grew into one of the largest private holdings in Europe. When Aalholm was sold to Stig Husted-Andersen in 1995, the castle's furnishings went under the hammer and the buyer took over the cars. The museum closed in 2008 because the buildings housing it had deteriorated too far, and the vehicles sat in the dark. After Husted-Andersen's death his estate put them up. RM Auctions took 181 of them on 12 August 2012; the estate administrator's inventory ran to 224 vehicles in all. Since 2019 the castle has belonged to Bo Ladegaard Bendtsen.

The landscape around Aalholm has been rearranged almost as often as the building. An S-shaped causeway ran to the castle from Søstræde and Slotsgade in Nysted, probably from the beginning. A small garden appeared on the west side in 1796; a park was laid out in Hestehave Woods in 1814 and again in the 1830s; a walled garden with greenhouses and a water tower went up west of the castle, perhaps in the 1890s. At the 1889 change of ownership the southern dam was closed to the public and a new, smaller dam built to the north, so that part of the lake could be drained and grazed. Then in 2008 - the same year the cars were locked away - the pumps were switched off. The water has been coming back ever since, and the pasture north of the dam is a shallow lake again.