Going from my own experience as a former reenactor, It'd depend what sort of state the powder was in when the fire reached it, and how it was stored.
For example, if the impact had smashed the barrels and scattered it, it would burn quickly, but not explode - an explosion is essentially pressure building up in a confined space (Ie storage barrels or musket breech)
If the powder was scattered onto a fire, it'd effectivley sparkle - looks quite pretty if your using the amount in an American Civil War cartridge.
So anyway, before I ramble on too much, I'd say definately burn the place, take the roof off (After all, fireworks factory are VERY thick walls, light roofs as the saying goes
) And maybe blow the windows & door out.
Good luck with it, and I look forward to seeing the results