I always took it as only your models recovered during your recovery phase.
From Rulebook:
"During the recovery phase, warriors in your warband who have been stunned become knocked down instead and warriors who have been knocked down may stand up (see the Injuries section)."
Since you and your opponent each have a recovery phase, I'd take that to mean each person recovers their own models in their own recovery turn, and not in the opposite players phase.
Yeah, that means it's really bad if your model gets stunned during your own combat phase.