In Goblet of Fire, Hermione explains that Muggles cannot see Hogwarts:
‘[Hogwarts is] bewitched,’ said Hermione. ‘If a Muggle looks at it, all they see is a mouldering old ruin with a sign over the entrance saying DANGER, DO NOT ENTER, UNSAFE.’
Goblet of Fire — page 148 — Bloomsbury — chapter 11, Aboard the Hogwarts Express
Edit: This may help explain the presence of "ruins" around Hogsmeade. Also, we've had ruins in Hogsmeade for 8 years now in the form of the Dragon Challenge queue. No?