The fact that such products may be labelled vegan and receive, for example, the UK Vegan Society (an ethics-based organisation) trademark indicates, in my opinion, that the consensus insofar as there can be one on the matter, is that the ethical implications are not significant. That is, as an ethical vegan, you should apparently not be troubled about it in and of itself.
I still look for a reliable suitability label (a trademark of some vegan organisation) as this provides some degree of reassurance that ingredients were not tested on animals and so on.
However, I don't want to write the issue off so simply; the Vegan Society, for instance, are happy to trademark products that individually meet their standards even though the company makes non-vegan products. Some people prefer not to give their money to such companies. The question arises as to why those traces are there. When I see for example
produced in a facility that handles dairy, eggs...
I wonder why the item had to be produced in such a facility. Does the company use dairy and eggs in their other products, and if so, should I consider buying from someone else instead?
Of course, it's possible that a small company that makes only vegan products might have no choice about using such a facility, and as you suggest allergy warnings are included on so many products because it's extremely difficult to guarantee that no traces of some substance may be present.
You can always try to get in touch with the maker to see if they are able to comment on ethical concerns about their ingredients or process yourself.