I just switched from the Moonshadow robe set to the Drake Leather armour on my ES and none of the effects from the Light Armor Mastery passive apply.
In fact, I get more Cast speed and MP regen with the Moonshadow robe, which doesn't have bonuses to those stats.
Robe Mastery passive:
PDef
Max HP
Light Armor Mastery:
PDef
Casting speed
Attack speed
MP Recovery.
None of the light armor mastery stats apply