The listing system began by assessing the architectural significance of Blitz-damaged buildings and has since grown into the leading mechanism for protecting good buildings of all ages. But the system now looks to be under critical threat from a lethal combination of government indifference and financial pressure due to the huge profitability of redevelopment.

The fate of the Commonwealth Institute in London (1960-62) shows the pattern only too clearly. It was one of only 0.2 per cent of English listed buildings built since 1945, and held the second-highest listing designation, Grade II*. But when it was converted into a new home for the Design Museum, its original architecture was devastated. The interior, although a beautiful exhibition space, was almost entirely destroyed. A wing was demolished, and flats built over its original gardens. If anyone ought to be looking after good modernist design, surely it is the Design Museum.

Yet even the retained paraboloid roof has lost its impact now the mutilated carcass of the old building has been overstuffed with boxy new galleries. I would not entrust a goldfish to people who believe this is how to look after a building. With a few orange scales floating in the tank, they might insist the essence of Goldy was still being respected.

Strangely, the Commonwealth Institute case was explicitly celebrated by a former CEO of English Heritage, Simon Thurley. A scholarly and serious historian of earlier phases of English architecture, Thurley’s blind spot where it comes to modernism has posed problems for the listing system he once oversaw. He explained his unorthodox view in an interview with the Observer: for twentieth-century architecture “the whole listed-building system, the legislation and everything based around keeping the fabric, is not relevant.” His justification? “These buildings are about ideas and other things.”

The idea that physical fabric became unimportant to buildings after 1900 is baffling. Skilled craft remained a cornerstone of good architecture throughout the twentieth century. If anything, the focus on materials was more intense than in some earlier periods as technological change produced almost obsessive consideration of what buildings were made of and how they were put together. Thurley argues that fewer postwar buildings would have been listed without compromise. He may be right, but the effects of downplaying the importance of original fabric are also problematic. The Commonwealth Institute is just one of a group of II*-listed buildings that has recently been through, or is currently poised for, terrible changes.

Three affected projects make a decent history of England’s world-leading architectural ideas on housing. The Alton estate, Roehampton (1952-61), was the London County Council’s foremost showcase of how cars, buses and inexpensive concrete and steel construction could enable ordinary people to escape crowded slums for magnificent suburban parkland. In recognition of its exceptional importance, most of the estate is a conservation area, with numerous buildings listed at Grades II and II*. By baffling omission the centre of the estate, including its original entrance sequence and the best of all its blocks, Allbrook House, were left unprotected.

The unprotected area now looks as if it might be replaced by very dense, high blocks that will destroy the unity of such an important estate and loom large in views from Richmond Park.

The next generation of radical housing was spearheaded by Sheffield, which commissioned a vast single building containing almost a thousand flats from two gifted young designers just out of architecture school. Park Hill (1957-61) rejected the arcadian ideals of the Alton Estate in favour of a conceptual continuation of the terraced street. The sloping site meant that almost everyone (and a little milk float) could enter at ground level at the top of the hill, even if they were ten storeys up by the time they reached the other end of the long walkway.

The result was a set of heroic, inhabited concrete canyons and cliffs overlooking the city centre. The original façade saw the concrete frame infilled with various shades of brick to the designs of an artist, in the confident belief that abstract modern art was not too good for ordinary people. The building was visited and discussed by architects, sociologists and politicians from all over the world.

From 2009 the façades of a large section of the II*-listed building were ripped out, and the interiors destroyed. Replacement infill panels were provided in a range of garish colours. Would anyone think it acceptable to do the same to the II*-listed façades of the (internationally unimportant) Tudor houses on High Holborn?

A third generation of masterly council housing to receive II* listing included the two great concrete towers by Ernö Goldfinger — Balfron Tower (1965-7) in Poplar, east London, and Trellick Tower (1968-72) in Westbourne Park, west London. In these towers Goldfinger brought to blue-collar residents the attention to detail, architectural beauty, quality, sound-proofing, views and comfort that had long been the preserve of the rich. However, the residents for whom Balfron was intended have been “decanted” — an odious euphemism for “forced out” — and the building has now been gutted and its windows ruined. The new flats are for sale to rich new occupants and, no doubt, buy-to-leave property speculators.

The architecture of these wonderful projects was fundamentally driven by materials. In the case of the Commonwealth Institute, the materials were derived from all over the Commonwealth — the fabric becoming the very essence of the building.

The differences between the handling of concrete in the three estates was of central significance to their design. At Alton the exteriors are clad in good-quality precast panels that produce an elegant, orderly neatness. The slender concrete balconies form a delicate grid, a three-dimensional tartan. The architects of Park Hill and Balfron Tower, by contrast, brought their cast-in-place concrete structures chunkily to the forefront.

The Sheffield architects preserved the wooden marks of the formwork in which the concrete was moulded, whilst at Balfron Goldfinger had the outer surface carefully hammered off to reveal the stones that are at the heart of all concrete. In each case the façades have a muscularity that derives immediately and expressively from their physical fabric. The fabric is the idea.

The materials in which these skilled and dedicated architects manifested their competing artistic ideas were at least as formative to their architectural thinking as Victorian architects’ debates on the handling of brick and terracotta, iron and glass.

If you are reading this with a gleeful right-wing chuckle, gratified to see that the architecture of a more left-wing period is suffering so badly, you should be ashamed of yourself. One does not need to approve of Flavian politics, or indeed of mass murder as politically-expedient spectacle, in order to admire the engineering and architecture of the Colosseum and to wish to see it preserved. If you dislike the idea of council housing (I would passionately disagree with you), it should not mean that you wish to see all trace of it expunged. You are following some unappetising historical precedents if you support an architectural damnatio memoriae.

A lack of general architectural education makes it particularly crucial that listing is directed by expertise rather than popular vote

Yet the reasons for the devastation of these highly protected buildings may be even more worrying than a deliberate right-wing cultural revolution. Ignorance, indifference and a lack of aesthetic sophistication, more than political ill-will, risk bringing about the collapse of the listing system.

A lack of general architectural education makes it particularly crucial that listing is directed by expertise rather than popular vote. Listing is not there only to preserve what is universally popular at a given moment. Just as importantly it exists to save the public from short-term blindness. Without expert judgments overruling popular opinion, St Pancras Station, the Hoover Building, Battersea Power Station and many other fine buildings would have been destroyed before the general public came to see how good they were.

There are very few cases where later opinion has come to decide that a listing was a mistake (and such a mistake is easily reversed). There are many instances where listing was controversial, but the building has subsequently come to be valued. Yet the preservation of buildings seems to be becoming another victim of the current political hostility towards expertise.

The ultimate listing authority is the Westminster government. So how is it responding to the pressure on post-war listed architecture? Well, at the moment parliament was dissolved for the election, the executive was showing its absolute contempt for its own advisers and its own recent listing decisions by proposing to demolish all but the entrance of Richmond House, a fine 1980s government building which was listed at Grade II* less than four years ago.

They want the site for a stand-in House of Commons chamber during the refurbishment of Pugin and Barry’s Palace of Westminster. Other sites have been suggested, but have been dismissed because of alleged security problems. Given that one proposed alternative is within another newish government building, Portcullis House, it is hard to see how a demolished Richmond House would be more secure.

Meanwhile, the effect is a grotesque undermining of the value of listed buildings. Could the government not even have had the competent, forward-looking dishonesty to refuse to list Richmond House? The result for the building would be equally disastrous, but the precedent for the listing system itself would have been considerably less catastrophic than to list and then swiftly demolish, not because of some failing in the building but simply because they want the site.

If the authority which underpins listing can demolish a Grade II*-listed building simply because it suits their estate development plans, why should any developer, institution or homeowner be bound by that same system?

Richmond House is more than just a test case. It is a wonderful building. Unless you are a civil servant you probably do not know any of it other than its gothic entrance on Whitehall, but it is just as good behind that. Built by a leading British ex-modernist, William Whitfield, in the modernism-hating 1980s, it is self-effacing and sensitive in its treatment of its surroundings. Even the entrance is stepped back so that its exuberant Perpendicular does not claim excessive focus on the street front.

One wing of the new offices is grafted onto the back of a carefully-conserved Georgian terrace, and the commercial street-front buildings of Whitehall are also preserved, screening the new addition. Round the back, the architecture responds with humour and charm to the striped brick and stone of Norman Shaw’s New Scotland Yard building — now known as Norman Shaw North — cladding staircase and lift cores with the same palette. Between these towers is a modernist megastructure of concrete offices, stepping back as they rise, with lead roof cladding that again picks up on neighbouring buildings. The clever planning manages to squeeze a big brief into a tightly hemmed-in site, yet brings daylight to every desk. The architecture is tough, lively, and very well executed.

Internally, too, the finish is remarkably good – fine stone, brass and hardwood. It is clearly expensive, and designed to age gracefully for several centuries, rather than being smashed up and landfilled after 36 years like a superannuated portacabin. Staircases, lobbies and corridors have a monumental dignity of architectural treatment that would make them a marvellous film set. Or indeed an excellent office building for a government department.

There is a great deal of money to be made from weakening the listing system to infect older periods of architecture

The fact that the building nods to history, was built under a Conservative government, and is unobtrusive on the street, suggests that there is probably little if any animus against Richmond House. Instead, one can only assume, the short-lived governments who have presided over the decision simply do not care at all about relatively recent architectural heritage.

If you are still unmoved by these losses, consider one further threat posed by the sad fate of some key post-war listed buildings. The successes enjoyed by developers in hacking around II*-listed structures can only embolden them. Immense profits are available from the demolition of lower-rise, lower-density older buildings and their replacement with the denser, higher buildings possible under today’s construction technologies and planning regimes.

If I were a developer, or one of their consultants, I would be preparing my arguments for demonising Georgian, Victorian, Edwardian and inter-war architecture in the way that modernism has been demonised. They should be getting together their specious arguments for claiming that these older buildings are unsustainable, unhealthy, unloved and unviable. They need, too, to find an equivalent to asbestos to terrify planners and the public into accepting the demolition of loved and eminently retainable structures.

The elements of jerry-building behind so much older architecture can surely be exaggerated to argue for the destruction of Georgian squares and Victorian terraces – could “concrete cancer” not have its analogue in a newly-invented (and equally spurious) “brick arthritis”? Why not use the flammability of wooden floors to justify the replacement of almost everything through to 1860, and much thereafter?

There is a great deal of money to be made from pushing the current weakening of the listing system onwards to infect older periods of architecture. The busyness, spinelessness and ignorance of many on local authority planning committees, abetted by the developer-friendly passivity of central government, is surely a first-rate opportunity for a new demolition gold rush.

Whatever your preferred flavour of architecture, Georgian, Victorian or twentieth century, join and support the relevant amenity societies. They are likely to be in for more and tougher fights over the next twenty years than over the last twenty. The attack on our built heritage is short-sighted in any terms; with an uncertain economic future, we would be ill-advised to throw away our appeal as a film set and a historical theme park for the world’s emerging rich. Far more important, however, is the wealth of embodied history, and the aesthetic and intellectual pleasure, that our diverse and varied cityscapes offer us every day. We must fight for it or lose it.