I think it’s a miracle every time a bird lives to fledge. Kind of like healthy human births are a miracle. When you think about it, the odds are against a 100% successful nest because so much can go wrong. People who always witness total success year after year lead some kind of charmed life!
What I would do is wedge a piece of grass in the hole and see if it is removed within 24 hours. You could also carefully put a dot on each egg with a sharpie. No rotation and the grass stays in place means they are truly abandoned. I always hate to remove eggs and a nest but I have done it more than once.