Ministers have been urged to get more houses built in the rural South West so people can "live in the villages in which they were born".
Labour peer Lord Whitty told the House of Lords that low wages in the region are pricing local people out of the market in the countryside.
House prices have rocketed across Cornwall, Devon and Somerset in the last decade, fuelled by a boom in second home-owners in pretty rural and coastal communities.
His comments came after Communities Minister Lady Hanham said that between April 2010 and September 2012 just 7,519 affordable homes were built in rural communities through the Homes and Communities Agency's Affordable Homes Programme.
Labour claimed the figures "do not compare well" given the need for 11,000 additional houses a year in rural areas to meet demand.
Lord Whitty said the ratio between house prices and incomes "is at its worst" in the South West, and was urged to "address holistically the problem of affordable housing in areas such as the South West by having a coherent regional housing policy which allows people to live in the villages in which they were born".
Coalition ministers have made clear they want more houses built on green spaces.
David Cameron in December backed plans to build on hundreds of square miles of open countryside.
The Prime Minister backed Nick Boles, Planning Minister, after he came under fire for suggesting that the 9% of England built on so far should be increased to 12% to meet demand.
This would see 1,500 square miles of open countryside – over twice the area covered by greater London – covered with new housing.
In the debate, Baroness Hanham said it is "important that people in rural areas are able to stay in rural areas". She admitted parts of the green belt that are "not absolutely brilliant land" should be used for housing developments. There is no official "green belt" in Devon or Cornwall.
"We have made it clear, in the National Planning Policy Framework, that the green belt is virtually sacrosanct, but we recognise that occasionally green belt land needs to be used for affordable housing, although that will need to be replaced," Lady Hanham told the House of Lords.
"Some green belt land… is not absolutely brilliant land, so where you can use that rather than going into real open space, it should be used. However, we need affordable housing, and we recognise that."
She added that the Government "expect rural delivery in the next two years to account for nearly 10% of anticipated completions of the programme outside London".
He comments came as Labour accused Eric Pickles, Communities and Local Government Secretary, of giving a "misleading impression" over the state of the country's house building programme.