I always remove the old nest and clean the nestbox with soap and water to prepare for the next brood. Many of the bluebird sites recommend this as a way to remove ants, mites etc. that may have hatched during the growth of the young birds. I think that is good to have a fresh room ready for the nest family.
David
Stafford, VA