It's funny you should ask that. I happen to be an Air Traffic Controller for the FAA.
There are generally pretty hard/fast rules on the FAA's FARs (Federal Aviation Regulations), but things can get a bit fuzzy at times and helicopters almost ALWAYS have an exception to the hard/fast rules. Here's the word straight from the FAR's:
Except when necessary for takeoff or landing, no person may operate an aircraft below the following altitudes:
(a) Anywhere. An altitude allowing, if a power unit fails, an emergency landing without undue hazard to persons or property on the surface.
(b) Over congested areas. Over any congested area of a city, town, or settlement, or over any open air assembly of persons, an altitude of 1,000 feet above the highest obstacle within a horizontal radius of 2,000 feet of the aircraft.
(c) Over other than congested areas. An altitude of 500 feet above the surface, except over open water or sparsely populated areas. In those cases, the aircraft may not be operated closer than 500 feet to any person, vessel, vehicle, or structure.
(d) Helicopters. Helicopters may be operated at less than the minimums prescribed in paragraph (b) or (c) of this section if the operation is conducted without hazard to persons or property on the surface. In addition, each person operating a helicopter shall comply with any routes or altitudes specifically prescribed for helicopters by the Administrator.
Note the helicopter section...essentially, you can fly as low as Orange County will allow you (not sure what that is or IF they have rules for that), otherwise, as low as you can go until someone starts complaining and calls the FAA with your tail number. And trust me, they will if you blow the shingles off of their house, blow their patio furniture over, or scare their little annoying yappy lap dog.
Keep in mind that there are places you simply cannot fly and these areas have names like "Restricted Areas" (Disney World is one, for example) and "Prohibited Areas" (Camp David and the Groom Lake "Area-51" section of Nevada are 2 examples). I could go deeper into the details on these, but I don't think it would be pertinent for this discussion.