Let's reason about this from four angles- the three most common reasons for being a vegetarians and a rhetorical dodge to avoid confronting vegetarianism.
Insects make up a tiny proportion of the weight of a fruit. As far as I know, they don't harbor harmful bacteria, like say, chicken. Eating insects in fruit is likely to have no particular impact on health because the volumes are so small.
Vegetarians get the bulk of their calories from grains, which are less impactful than animal husbandry in most ecosystems. Fruit is about 100 calories a pound, and is eaten in about the same quantity by omnivores as vegetarians. My suspicion is that the worst problems with the environment are not the orchards and the number of orchards would change that much.
If the insect is already dead by the time you get the fruit it isn't a problem. (By this line of reasoning, road kill is vegan!) Almost all figs have a bug in them of some sort- wasps, etc. No intentional act of a human led to that insects death, so it isn't our fault it crawled into a piece of fruit, ate, and died.
Deciding what to do about other small life that is making trouble for us, but isn't dead yet is more complicated and calls for case by case reasoning. This essay is a fascinating example. The strategies that bugs use to get through life put them in to direct conflict with our desires to live- they eat our houses, drink our blood and end up in our all manners of potential food sources. So even if we value the life of an insect as much as a cow or ourselves, we will still end up in the place of deciding which one will live.
In my opinion this is a sort of "trolley problem" where our intuitive sense of ethics fails us and it isn't clear what answer is correct. In my opinion, this doesn't support ethical nihilism when it comes to other questions like, should we eat cows and chickens.
Rhetoric to avoid facing Vegetarianism
This is an argument about the supposed futility of vegetarianism, i.e. if you can't avoid killing the small, then you should bother refraining from killing the large. The usual vegan response is that the project of veganism is about harm reduction and not a perfectionist project of never killing anything. Also, if we have to choose the killing of the small (insects, mice) is less problematic that killing the large who generally more intelligent and aware of their suffering, if only because killing a short lived insect is depriving it of fewer of it's remaining days of life than a normally long lived cow.