Panemunė Castle was at first called thus after the manor on the land of which it stood. Later, the castle was taken over by the Gelgaudai Family and renamed to Gelgaudai Castle. Following the change of owners after the Uprising of 1831, the castle gained the name of Zamkus Castle. When the Vytėnai Village was established nearby in the beginning of the 20th century, the castle received the name of Vytėnai. Presently, it carries its original name.
