Wasps and bees seldom usurp boxes from nesting birds. They are mostly found in empty boxes. If these insects are found in a box, it is best to let them be and not take any active measures to exterminate them. Instead, wait to clean them out in the fall when the weather is cooler and their activity h