Full-disclosure, I was a big HP fan to begin with.
But the key thing is that I was really only a casual Universal fan. I'd been to the parks several times, usually for HHN, but I probably wouldn't now have an Annual Pass if it weren't for the Wizarding World. And I think the consensus here has been that Potter has more appeal to non-fans than a lot of other properties.
Remember that, so far, HP has only replaced 1/3 (Merlinwood) of LC in IOA and one ride plus carnival games in USF.
I understand why some might see further expansion as overkill, but the overwhelming success and continued popularity of Phase 1 and undoubtedly of Phase 2 is really unprecedented and may call for unprecedented expansion.