Recently it was revealed that the Queen’s personal diaries and personal correspondence were going to be assessed, with a view to eventually making them public. The diaries, which she kept daily, go back to the beginning of her reign. They will be an important source for historians, giving her views, hitherto largely unknown, on world events and leaders.

As someone who had the highest security clearance, was briefed by everyone from prime ministers and intelligence chiefs, and knew a worldwide network of statesmen, her views and reactions will be fascinating and possibly sensational. That is before one even considers what she might have written about her own family. If published without any redactions, they would be bestsellers, putting her grandson’s memoir in the shade.

Of course, they will never be publicly available in full and will need to be vetted. Which eminent and trusted historian has been chosen for this task? Perhaps Sir David Cannadine, a former Regius Professor, or an historian peer in the House of Lords. Maybe an official biographer has already been selected. Indeed, someone has been chosen, someone who knew the Queen well, was loyal and discreet — her footman, Paul Wybrew, who has been in royal service for over forty years.

Wybrew, for all his qualities, is not an historian who would understand the nuances and significance of such an important historical source. The worry is that without wider scrutiny, historically important material may not just be closed but destroyed. There are precedents for this. Queen Victoria’s daughter Beatrice destroyed many of her diaries, and Princess Margaret burnt many of the papers of the Queen Mother.

It all goes back to the question of whether material in the Royal Archives qualifies as private or state papers. Clearly, the Royal Family is entitled to privacy, but where does private behaviour cross over into justified public interest? It is a problem that has bedevilled royal biography.

Communications with the Sovereign are exempted under the Freedom of Information Act, and even the most trivial references in historical documents to the Royal Family in the National Archives and other collections are redacted, which is to say censored. The result is that most royal biographies are based on newspaper cuttings and briefings from “sources”.

Prince Harry can share the most intimate and recent details of life in the Royal Family, and Royal households can brief against each other, yet historians cannot see archives that are a century old. It is an absurd situation, and it needs to change if our history is to be written accurately.

This is why documents, the building blocks from which historians construct the past, are so important and why, with a new reign, it is hoped there will be a little more transparency in the Royal Archives. Currently, the institution has no public inventory — rather like a restaurant with no menu. One asks for what they have, and, if one is lucky, files will be produced.

All too often it is a mystery why files were closed in the first place

As a royal biographer, it is a question with which I have often grappled. For my biography of Dickie and Edwina Mountbatten, I sought access to their personal diaries and letters, which had been widely quoted in several previous books. I was told by the University of Southampton, which had bought the papers for almost £3 million of public monies, that they were closed.

Eventually — after several years, numerous FOI requests, the intervention of the Information Commissioner and the unprecedented threat of contempt proceedings against the University — in 2019 a Decision Notice was issued ordering the release of the material. Southampton and the Cabinet Office appealed the decision, but then, just before the November 2021 hearing, dumped 99.9 per cent of the material (more than 30,000 pages) on the Internet. The material that they had kept closed for a decade and fought so hard to prevent being made publicly available proved to be entirely innocuous.

All too often it is a mystery why files were closed in the first place. A 1932 protection file on the future Edward VIII remains closed, I was told, because releasing it might jeopardise the current safety of the Royal Family. A file on 1978 parachute training by Prince Andrew, requested under FOI over a year ago for my current book, remains closed on the grounds of national security, law enforcement, and health and safety.

Recently the campaigning organisation Index on Censorship published a report on royal records pointing out that almost 500 files at the National Archives were closed. They included such controversial subjects as “Family name of Royal Family members 1952-1960” and “Visits overseas by members of Royal Family 1954. Record opening date 1st January 2055”. The website Declassified UK recently reported that over 200 files on overseas trips made by King Charles, going back to the 1970s, remain closed. They include a 1983 visit to Australia that will only be released when the King is 121 years old.

This culture of unnecessary secrecy needs to stop. With a new reign, there is an opportunity to review and open up the files in the Royal Archives, National Archives and other repositories. Their closure creates a vacuum for speculation and fantasists; their release would go some way to restoring trust in institutions, not least the monarchy. I hope Mr Wybrew takes note.