You have asked about animal cruelty, animal product or something else. I would argue that because of each one of these things:
Animal cruelty: To satisfy the demand of our booming population, milk industry is far far away from where it was just a few centuries ago. Milk, primarily intended for calves, is being harvested by machines in an infinite loop to be drunk by humans. Male cows are redundant and therefore eaten/killed. Normal bonds between calves and cows are destroyed for the sake of production. Not speaking of less than ideal living conditions in many cases. As any other animal we use, cows are being exploited terribly and, literally, treated as pieces of dead meat, not living beings.
Animal product: As such, milk from cows (or any other animal, for that matter), is not made for us. It has its own primary recipients and there are some clues about them being rather underfed. When left to fed as they please, calves drank two to three times the amount that is recommended (and widely administered to them) by industry. We are actually stealing it and other than the taste and our convenience, there is no real reason for us to do so. We do not need it to survive and I believe that our consumption of it is not seen as perverse only because of how deeply ingrained it is in our culture.
Something else: Here I would mention ecological impact of dairy industry for which rain forests and other biotops are being destroyed, mainly for producing food for cows. For various reasons, the water cost per litre of milk can be as high as 1,200 litres of water. With the population of about 1.3 billion, large quantity of which is bred for the dairy industry, the combined biomass of cattle is roughly 520 million tonnes (this is rather conservative estimate based on average cow size of 400kg). There is no other big animal on this planet with bigger biomass. And it is anything, but natural.
On the other hand, there is human breastfeeding. Milk which is produced precisely for humans. It has different nutritional composition compared to other milks, eg. is much lower in casein. It targets the needs of the human baby by design and the act of feeding the baby serves as one of the early life bonding moments for the mother and the child. There is also the immunological aspect of breastfeeding, because numerous studies show passive, long-term benefits with regards to immunity. This is also one of the reasons for which brestfeeding is preferred over infant milk substitutes.
To put things into perspective, there is no added animal cruelty or ecological impact of human breastfeeding. It is the most natural way of feeding human infants.
This is not about things being or not being called milk. Word milk is but a label which we use to abstract rather large group of things with similiar properties. While this labeling may add to the confusion, milk from other animals and human milk are inherently different. There is no single reason for vegans to even consider not to breastfeed because of their diet.
For the purposes of this answer, I assumed that by milk is meant the most produced type of milk which is cow's. The reasons given in the answer would stay largely the same for any type of milk though.