15′ from the shed is a fine distance for the box but I wouldn’t recommend 5-10 feet from something.  That’s asking for a tragedy.  Baffle the pole and apply grease on the pole inside the baffle.  
Squirrels will eat mealworms if they can get them, but I have never seen them eat the dried ones.  They love the live ones though.  
For the birds to find your worms easily, put a few in a light colored smooth sided dish and place the dish on the ground under where they perch to hunt.  They will find that and then you can move the dish to where you want it.  Move it a few feet at a time and they will follow because the things are addictive!