I'm the furthest from a Potterhead, but that doesn't affect my enjoyment of the rides. Sure, it'd be nice to know the backstories, or who some of the characters are, but that doesn't hinder the enjoyment. I'd say it's even easier with Hagrid's, because they're not directly referencing any place we've seen in the films, rather a fabrication specifically for the park itself. It being self-contained kind of helps it for the GP.