Land used for grazing that was previously not used for commercial food production may be former rainforest, or hilltop regions (this may certainly not be an exhaustive list of the possibilities). At first sight these appear very different, but I believe they have common issues. Here I will only deal with hilltop areas in temperate zones as I have knowledge about it. I hope other answers can discuss different conditions.
Hilltop areas in temperate zones
I will use my native land as an example here. As pointed out in this and other articles by George Monbiot, land used for grazing on hilltops would naturally be forested in temperate areas:
In Britain, strangely, the lowlands are largely bare and the uplands
are even barer. This is not a natural condition. In Norway, at the
same latitudes as northern Scotland, in similar climatic conditions,
trees cover the high mountainsides[3]. The uplands of Britain would
once have been largely forested. But, aside from plantations of exotic
conifers, there are few trees in this country above 200m.
Our bare hills are an artefact of three principal activities: sheep
farming, deer stalking and grouse shooting. Sheep and deer selectively
browse out tree seedlings, ensuring that existing forests cannot
regenerate and trees cannot repopulate bare land.
Sheep are a fully automated system for environmental destruction. Once
they are released into the hills, no further human agency is required
to prevent ecological recovery.
While the rich soil of the lowlands is excellent for cultivation of food crops, the highlands have poor soil which can support trees, but is not suitable for profitable crop cultivation. It is important to note that sheep farming on this land is also very unprofitable and in my country is very heavily subsidised:
Without farm subsidies, there would be scarcely any hill farming in
Europe: the activity is uneconomic almost everywhere and will remain
so under all conceivable market conditions, as a result of poor soils
and harsh climate. Anyone who wishes to farm there must rely on
subsidies.
The loss of trees causes soil erosion from the hills and significantly increases the risk of flash flooding and drought in the lowlands. Trees and woodland retain water and soil and regulate the water cycle, thus protecting fertile lowlands and their crops from flood (which may catastrophically remove soil into water courses and the sea) and drought.
Thus, the best way to use former grazing lands to increase food production may well be to simply allow natural forest growth whether or not the soil is too poor to support sustainable cultivation.
The benefits of allowing natural forest growth extend beyond food production; it also protects settlements from flooding, regulates the water table (for drinking and other uses), and provides habitats to support biodiversity (which creates an exciting lovely world to live in for humans, if you need a justification on that basis - I think biodiversity is an end in itself). It may well be possible to introduce crops that can tolerate poor soil, exposed conditions, relative drought and so on that could be grown profitably on hilltops, or to use artificial techniques to fertilise the soil, but taking a holistic, long term view, allowing natural forest would be more sustainable and beneficial.