It's actually possible for children to be vegan from any age, assuming they are breastfed or appropriate formula can be sourced (vegan infant formula is rather difficult to get hold of though).
You can introduce vegan "milk" products (use one fortified with calcium) from age 1 (soya, rice, oat, almond milk etc) Soya or pea milk are better than nut milks as they have more protein and calories. If your baby continues to breastfeed frequently after age 1, you don't especially need to introduce any milk. As breast (or formula) feeding decreases, getting enough calcium from the diet alone can be a challenge depending on your little one's appetite and food preferences, so milk (including fortified plant milk) is a convenient source of calcium for them. I like to thin out banana smoothies with fortified soya milk. You can also buy food grade calcium carbonate and add 1/4 tsp (provides about 300mg calcium) to a serving of (vegan) yogurt, smoothie etc.
For other sources of calcium see Vegans: Besides supplements, what can substitute for Calcium? and make sure you introduce potentially allergenic foods such as nuts, seeds and soya for the first time in small amounts. It is now recommended to introduce allergenic foods as early as possible (from 6 months onwards) as this reduces the risk of developing allergies.
As for ghee - while babies and toddlers need a high fat diet, that fat doesn't have to be ghee. You can just as well add coconut oil or olive oil to baby's food. Nut butter (spread on bread or mixed into smoothies, yogurts, oatmeal or other food) or nut powder is a great source of fat and also provides protein and some vitamins and minerals. Depending on availability you might be able to offer fat-rich fruits such as avocado (a baby-led-weaning favourite), and acai. Just try to avoid foods with hydrogenated vegetable oils that contain trans fats, as these are harmful to health.
Vegan babies (like all other vegans) should take a B12 supplement once they stop drinking breast milk or formula. Breastfeeding vegans should take a DHA supplement and babies may benefit from a DHA supplement once weaned.
Another nutrient worth paying extra attention to in vegetarian and vegan (and even omnivorous) diets is iron. There are many plant sources of iron, but this iron is less bioavailable than that in meat (dairy products are not sources of iron). To improve absorption of iron, serve iron-rich foods such as beans, tofu, green vegetables, peanut butter, chia seeds etc to your baby alongside a source of vitamin C such as strawberries, capsicum, kiwifruit, pineapple, papaya, citrus fruits etc.
Also related: What vegan food should I feed to a baby?