I spent a few hours one night trying to figure out where the association of wizards with towers came from. It was mostly theories about why a wizard would want or end up in a tower. Some people concluded it was just another thing that was in D&D and became widespread that way.
There's the scholastic aspect, long study. There's the "study the skies" element... and just getting CLOSER to the skies. Towers are great for all sorts of isolation. Also being "above" everyone else: symbolic superiority.
Before D&D there was Orthanc (and Barad-dur, though we don't see much detail); and various "Minas
this" and "Minas
that" ("minas" being "tower") -- e.g. Minas Morgul where the Witch King dwelt. I'm pretty sure Jack Vance wrote some wizard-in-a-tower tropes, and that's another of D&D's sourcematter. REH/Conan put some wizards in towers.
The "ivory tower" imagery (going back to "scholasticism") is I think over a century old.
But going back even further, I
think there's a classic "wizard in tower" in Spenser's
Faerie Queene.
It's also worth noting that "tower" in many of the older sources wouldn't refer to a singular standalone structure, e.g. "the Tower of London" is an entire castle, really --
Her Majesty's Royal Palace and Fortress of the Tower of London. Certainly many of Tolkien's "Minas" sites were entire castles, or even cities!
In the end, yeah: in fantasy literature & RPG'ish use, I think Tolkien and D&D are our proximate culprits. I don't think Spenser and the evolutionary use of language enter much into it...