We were informed last week, with all of the self-effacing modesty of Moses bringing down the Ten Commandments from Mount Sinai, that Paul McCartney will be releasing a new book this November. Those of us who were hoping for a sequel to his piquant late-era masterpiece Hey, Grandude! may be disappointed to find that McCartney’s latest publication is entitled, portentously, The Lyrics: 1956 to the Present.

In this “extraordinary book”, McCartney, “with unparalleled candour”, will dissect 154 songs from his career, from his boyhood days up to his most recent album, McCartney III. Presented alphabetically rather than chronologically, we are promised that “the voice and personality of Paul McCartney sings off every page”, and fate is tempted by the assurance: “There has never been a book about a great musician like it.”

The new book should be an invaluable acquisition for any of McCartney’s millions of admirers

Even those of us who still believe that McCartney was the most talented and dynamic of the Beatles may detect something of the whiff of mutton dressed as lamb about this project. Despite the weighty presence of the Pulitzer Prize-winning Irish poet Paul Muldoon as McCartney’s interlocutor and editor – Muldoon compared their collaboration to “the afternoon sessions [McCartney] had with John Lennon when they wrote for the Beatles”, and said, “We were determined never to leave the room without something interesting” – those of us who are not wholly swayed by PR hype might remember firstly that McCartney released a similarly high-profile collection of his lyrics and poetry, Blackbird Singing, in 2001, and that his choice of editor in that instance was another great poet, Adrian Mitchell.

Likewise, those of us who long for McCartney’s more conventional autobiographical disclosures may well have been satisfied with his 1997 book Many Years From Now, his “official” biography drawn from hundreds of hours of conversations with the writer Barry Miles. The collaboration was sufficiently close for McCartney not only to have vetted the manuscript before publication, but also for him to have retained 75 per cent of the royalties of the sale proceeds. Amusingly, its mild revisionism led to a stinging attack from Yoko Ono, who claimed that McCartney had chosen to present himself as the envious, second-rate Antonio Salieri to John Lennon’s brilliant Mozart, but the suspicion has remained that McCartney’s career-long interest in presenting himself in a favourable light led to a partial and inevitably biased account.

Nonetheless, McCartney’s life has been far from uneventful over the past two decades. Even if we leave behind the tawdry and rather sad saga of his marriage (and subsequent highly public divorce) to Heather Mills, he has continued to record albums and tour with the vigour of a much younger man. He has remained interested in everything from classical recording (as in 2006’s Ecce Cor Meum, an elegy for his late wife Linda) to experimental electronic music recorded under the pseudonym “The Fireman”, a nod to his much-cherished status as the “true” innovator behind the Beatles. Therefore, should Muldoon have elicited some candid and interesting insights from him, the new book should be an invaluable acquisition for any of McCartney’s millions of admirers.

Looking for autobiographical truth in a pop song is ultimately a doomed endeavour

In choosing to eschew the conventional autobiographical format in favour of an annotated selection of lyrics, McCartney follows in the footsteps of the Pet Shop Boys’ Neil Tennant and Pulp’s Jarvis Cocker, both of whom released their own acclaimed collections via the legendary publishing house Faber and Faber. If either Cocker or Tennant, two of the most literate and wittiest men in popular music, felt any sense of inadequacy at placing themselves in the footsteps of Eliot, Larkin or Auden, their insightful commentaries on their work doubled as autobiography lightly worn and a fascinating insight into the songwriting process. If personal revelations were relatively unforthcoming (although it’s fascinating that the Pet Shop Boys’ songs “It Couldn’t Happen Here”, “Being Boring” and “Your Funny Uncle” form a lyrical trilogy about a friend of Tennant’s who died of AIDS), the writers would be quick to respond that looking for autobiographical truth in a pop song is ultimately a doomed endeavour, and one destined to drive lyrical trainspotters to desperation.

Instead, if a reader wants the unvarnished truth behind the rock and roll life, they have to purchase one of the revelatory memoirs that appear on an annual basis (usually in time for Christmas), presumably on the grounds that tales of drug overdoses and unfeasibly vigorous sex with endless numbers of willing fans sit nicely with a surfeit of turkey and mince pies. Some of these books have been exceptionally good. I read Elton John’s Me back to back with Anne Glenconner’s Lady in Waiting, and the two memoirs had a remarkable amount in common with one another, in that I spent most of the time reading them with my mouth agape, constantly skipping back over a page to check that, yes, I had read that particularly piquant anecdote correctly.

In the case of Sir Elton’s autobiography, he was aided by his excellent choice of The Guardian’s rock critic Alexis Petridis as his ghost-writer, meaning that the jokes and callbacks grow in mounting hilarity over the course of the book. There is a great deal to be said for a man, famously not short of vanity, including an anecdote about how, surprised and wigless, he bears an unfortunate resemblance to Uncle Fester from The Addams Family, just as regular references to “our old friend, the Dwight Family Temper”, only get funnier each time that they appear. Refreshingly short of specious nonsense about “the creative process”, Me instead offers such indelible details as how Elton preferred voyeurism to sexual participation in his Seventies heyday, thus avoiding contracting the AIDS that killed Freddie Mercury, but this too had its downside: while watching various comely youths have sex on his snooker table, the houseproud musician snapped at them, “Make sure you don’t come on the baize!”

In this, Me belongs to a fine tradition of rock ‘n’ roll autobiographies that include Keith Richards’ Life, Slash’s eponymous memoir and Ronnie Spector’s account of her marriage to the murderous pop producer Phil Spector. All of these are rich in vivid detail about the consequences of untrammelled and narcotic-fuelled ego, but manage to make their protagonists seem likeable and sympathetic, even while they are committing yet another crime against nature.

Some of the memoirs of musicians that are regarded as literary masterpieces are not tell-all accounts

It is notable that Jimmy Page, whose most debauched adventures in the Seventies are still talked of in terms of hushed horror, has yet to publish a conventional memoir, while Mick Jagger returned the substantial advance for his on the grounds that he was no longer able to remember the events that he was being paid to describe. The work of memoir that he did write in the Eighties is owned by the publisher John Blake, who simultaneously calls it “a little masterpiece”, while admitting that it is “light on sex and drugs”. Perhaps, instead, it features lengthy accounts of his business dealings with the Stones’ aristocratic financial manager Prince Rupert Loewenstein, which would have their own charms, if less of the longed-for debauchery.

Certainly, some of the memoirs of musicians that are regarded as literary masterpieces are not tell-all accounts. One of the very finest, Bob Dylan’s Chronicles: Volume One – we await the second volume still – is a richly poetic account of three moments in his career, 1961, 1970 and 1989, marred only by its description of Bono as possessing “the soul of an ancient poet”. And Bruce Springsteen’s 2016 autobiography Born To Run eschewed self-indulgent stories about excess in favour of a deeply affecting insight into what it is like to achieve worldwide fame at the price of, in Thomas Wolfe’s description, never being able to go “home” again.

All of these books have earned their place on our bookshelves because they remain fascinating narratives with larger-than-life protagonists at their centre, and we enjoy their more unsavoury antics with the vicarious shudder of a spectator watching a horror film. It is hard to imagine that many of today’s rock stars will produce so engaging a tale when their turns come. The likes of Ed Sheeran, Coldplay’s Chris Martin and Taylor Swift are all personable, intelligent and talented, but unless they have successfully hidden a far darker side from the media, it is impossible to imagine them snorting a line of live ants in lieu of drugs, as Ozzy Osborne once did, or holding a record launch party along the lines of Freddie Mercury’s 1978 “Saturday Night in Sodom”, in which dwarves were sent around the room with trays of cocaine attached to their heads.

The most rock ‘n’ roll thing that Swift has recently done is to get involved in an intellectual property dispute with her former record company that has led to her re-recording her old songs, album by album, to regain control of the publishing rights. Which is undeniably important, but hardly on a par with Keith Richards snorting the ashes of his deceased father after having mixed them with cocaine. Yet in our increasingly public world, outrageous exploits taking place behind closed doors are all too easy to capture on smartphones and could go viral on social media before their practitioners had even finished getting dressed.

There is an inexhaustible public appetite for learning about the private lives of our idols

It remains to be seen whether McCartney’s new book touches on any of his own less salubrious exploits, such as his well-documented 1980 nine-day stay in a Japanese jail for drug possession, or Heather Mills’ allegations that he abused both her and Linda McCartney during his marriages to them. In the latter case, Mills’ own credibility is now regarded as virtually non-existent (Jonathan Ross famously quipped, “She’s such a fucking liar that I wouldn’t be surprised if we found out that she’s actually got two legs”) and so it is unlikely that he would feel the need to add anything to his 2006 statement that he would defend himself against her allegations “vigorously and appropriately” in court.

Yet perhaps Muldoon will elicit some new revelations from McCartney and therefore make the book essential, rather than merely diverting. Otherwise, there is always the risk that it might follow the path described by Morrissey in his comically lugubrious Autobiography, in which, informed by his idol David Bowie at breakfast that, “You know, I’ve had so much sex and drugs that I can’t believe I’m still alive,” Morrissey ‘loudly’ responds, “You know, I’ve had SO LITTLE sex and drugs that I can’t believe I’m still alive.” And yet, for all that, he still got a bestselling book out of his life, showing that, after all, there is an inexhaustible public appetite for learning about the private lives of our idols. Until, that is, they prove to have feet of clay.