The nature of the material and its contemporary use as a veneer. Water gets behind masonry and if you look closely you'll see weep holes that allow moisture to escape. That same hole can let in humid air. Masonry also does not breathe well and will retain the heat it absorbs during the day. Compounding that with other materials that would act in a similar manner, such as concrete, only makes the problem worse.