I would wait before cleaning it out. Bluebirds sometimes build a nest then wait a week or even more before laying any eggs. They could be waiting for the weather or food supply to improve, or they’re taking a break. Also check to see if any wasps are building a nest in the upper corner of the roof inside. That will cause them to abandon the nest. If no wasp nest then I would wait longer to see if they return. In my case I have found early in the mornings are when they are most active around the nest site. Also try putting out some mealworms to entice them to return. If they don’t return at all (which is a BIG if) try and save the nest by placing it in a ziplock sandwich bag. This way you can use it for future nest changes if you ever encounter a wet nest with baby bluebirds in it. One question for you…did you notice any eggs BEFORE the nest appeared abandoned? Snakes will swallow an entire clutch of eggs and leave no trace of the nest being disturbed. This will also cause the birds to abandon the nest. Hope this info helps some..