Cher, sorry to hear the bad news. Life is tough “raising” these little birds since we cannot intervene much to help them. I have had the same experience, to a lesser extent (2 boxes vs. 4) but the effect was the same. I did remove both boxes after the second nest robbery, cleaned them and put them away. I did but the house that is now up back at the onset of winter. My blues use it over the winter as a roosting spot.
Stick with it and watch out for that wren. They can be mean.