Do you have a baffle on the pole, if not get one up asap. Having said that if it was a raccoon and there were eggs you wouldn’t have found them on the ground. Opossums do not get into nestboxes. Most likely a sparrow or a wren. I realize you said you had a spooker up but that isn’t fullproof.
I myself would remove the nest and possibly turn the box a bit so they may try again. Typically they won’t renest where they have had a failure unless there are no other housing options available.
Good luck
Donna in WI