I think that’s wise. I leave them in a long time and most of the time don’t ever remove them. I find them later at the bottom of the nest. But at some point you could reach in and very lightly pick it up with your fingers or a spoon if you are worried about breakage.
