Difficulty | Damage Dealt | Damage Taken
-----------------------------------------
Novice | x2 | x0.5
Apprentice | x1.5 | x0.75
Adept | x1 | x1
Expert | x0.75 | x1.5
Master | x0.5 | x2
Based on this, there are a few things you can extrapolate as the difficulty level increases:
Skills based on damage increase more slowly, and in turn, the slower you level up (thanks yx.)
Followers can be knocked out, and in turn, killed, more easily
Your effective health decreases
Enemies' effective health increases