Legend tells that the castle was built around 540 AD by the Goth, Genseric, on the ruins of the previous manor house: Genseric was a mysterious character and the right-hand man to Amalasuntha, queen of the Ostrogoths . After the tragic death of the sovereign, killed on an island on Lake Bolsena by assassins sent by her husband Theodahad, Genseric fled to the north with Eudosia, a lady-in-waiting, whom he married and with whom he settled down in the district of Mel. He rebuilt a "castle" here and was given the title of "comes" (count) by Vitiges, the new king of the Ostrogoths, as well as full jurisdiction over the area: "in the morning the water of the Ardo, at the midday the top of the mountains, in the evening a water course known as Rivo Bianco, 5 north of the river Anaxo, now called Piave".

According to Bonifacio, the family coat of arms was represented by a gold sphere on a turquoise background, surmounted by twins in an embrace, as a reminder of the happiness that was bestowed on Genseric's new household with the birth of his twin sons, Iusprando and Goffredo. The castle of Zumelle takes it name from this legend: 'castrum zumellarum' – castle of the twins.