Gilbert and George, the identically-dressed bank-clerk duo who have been a staple of British art for more than 50 years, live in a beautiful 1720s townhouse in Fournier Street in Spitalfields in London, sandwiched between Brick Lane and Liverpool Street Station. They moved there in the mid-1960s and, like the house itself, they have been an unchanging fixture in an area that has adapted dramatically to the passing years.

Nevertheless, my first sensation on knocking on the door was disappointment. George — tall, balding, bespectacled and English — opened the door, and behind him, in the posture of the admirable Crichton, stood Gilbert — shorter, trim and South Tyrolean. Both, though, were wearing their timeless uniform of tweed suits and brogues (the tweed used to be Scottish but they switched to Irish in disapproval at the 2014 Scottish independence referendum). I had hoped to find them in mufti: the red corduroys beloved of middle-class men of a certain stamp and a nice cashmere sweater perhaps. But they offered no hint as to whether they are ever off duty. When they step outside their door they are living sculptures but what about when there are no eyes to watch what they do?

I had a quick glimpse of wood panelling and rows of 1930s cut-glass vases on an Arts and Crafts sideboard as they ushered me through the house to their large, crisp and brilliantly white studio and archive space behind. None of their pictures are on display, just reproductions for their latest show and a series of scale models of exhibition spaces in San Francisco and Oslo with miniature versions of their artworks in their allotted places.

The switch between ancient and modern makes sense. “We want to be part of the past, present and future,” George says in his immaculate 1930s received pronunciation. “We are living in a French street which had a synagogue which used to be a church and is now a mosque. We’re living on top of a Roman cemetery. When we came here it used to be Yiddish-speaking. There was an opium den here. There were the Russian vapour baths. We can walk to the grave of William Blake, Daniel Defoe, John Bunyan.” Now it is a predominantly Muslim area — cut with a sprinkling of hipsters — famous for the curry houses of Brick Lane and the street signs in English and Bangladeshi.

Their tweed suits used to be Scottish but they switched to Irish in disapproval at the 2014 Scottish referendum

When G&G, as they refer to themselves, first moved in there were only two curry houses on Brick Lane and tramps used to rush in, grab handfuls of masala and rush out again. “We became very big friends with all the waiters,” says George. “The only other people who went to the curry houses when we did were the police, for a takeaway, which they got free. I agonised over who I should report them to.”

At that point in the 1960s, “The men were happy to be here, away from their villages. They could be free, they could buy bell-bottom trousers here. They used to have fancy hairstyles, they wanted to join in.” The idyll was not to last. “Everything changed once the mullahs came. In the early days, when the men went to the mosque they carried their hat in their pocket and as soon as they stepped inside they put it on and as they came out they whipped it off. Now they wear it 24 hours a day.”

The two men pick up each other’s sentences, not finishing what the other says but adding a coda — each expressed thought is a splicing. So it is Gilbert, with his still heavy Tyrolean accent, who picks up the theme: “In 1990 we did some pictures called The New Democratic Works and the Indians and Pakistanis would model, we used to pay them £30, there was a queue outside. They loved it so much. They wouldn’t do it now.” Things have got much worse: not long ago the doors of seven houses were kicked in one night. Nothing was taken. They were, says George, all non-Muslim houses, and “there were leaflets outside with drawings of the Twin Towers with the airplanes going in.” Some things in the area have, however, changed more subtly.

Their part of the East End used to be frequented by tramps, many of whom were war wounded and others gay men from before the legalisation of homosexuality in 1967 (the year G&G moved in). “The police would visit them and give them a warning: ‘If you are not here tomorrow then we won’t be able to charge you.’ So they got out and gravitated to the capital,” says George. Now there are more drug addicts than tramps but, according to Gilbert, “They are supporters of ours. We became the artists of the disenfranchised.” These unlikely fans won’t have seen their pictures but they nevertheless know who G&G are and what they do.

According to George, “Last week we were walking down Whitechapel and a tramp came along, completely skinny, staggering, don’t know how he could stand up. And as he came by he shouted out ‘Hey, I like your shit pieces best’” (referring to the Naked Shit Pictures of 1995, which show exactly what you’d expect). “I came home and cried. It was very moving.” What was it the tramp liked? I ask: “Our art is a global language to do with humanism — pain and suffering.” Not quite an answer.

G&G have another dishevelled fan too. They are men of routine and, since they don’t have a kitchen in the house, go out to eat every night, usually at the same restaurant. For years they went to a Korean restaurant in Highbury and Islington, a place named after an island that, according to George, was famous for weddings and gigolos (“When’s the next flight?”).

One man would often come in to pick up a takeaway and put his bicycle on the floor. It was only when that cyclist became mayor of London that they realised who he was — or so they say. Later, reckons Gilbert, Boris Johnson would cycle past them and shout “Gilbert! George!” “That’s more than enough for us. We know whose side he’s on,” says Gilbert. “People are trying to bring him down with all this liar, liar stuff. It’s quite appalling.”

Nevertheless, it is Michael Gove who they most admire. “He’s brilliant, a genius,” offers Gilbert. It is certainly not Jeremy Corbyn: “Can you imagine if it was he who decided what art is? It would be all Nicaraguan posters.”

G&G are unrepentant Conservatives — staunch supporters of the Queen, the union and Margaret Thatcher, but not of religion, Anglican or otherwise. “We say ban religion and decriminalise sex,” proffers George. “Though we are not against decent Christian behaviour, we think that’s very important.” Their politics puts them at odds with the rest of the art world, they reckon. “In the art world, us saying we were Thatcherites was the worst thing you could say. Like saying you were a Nazi paedophile or something. They still don’t forgive us for that.” The art herd, they say, is “all left-wingers who want to be millionaires. In 1969 Richard Hamilton [the Pop artist] took us out to a Chinese restaurant and came to collect us in a Porsche with a Labour sticker on the back window. Even multi-millionaire American artists like to tell you how marvellous the education system is in Cuba. But art only survives by capitalism.”

Indeed G&G are extraordinarily keen not to be seen as part of the art world. They don’t go to galleries or mingle with other artists. “We invented blinkers for ourselves. We have to do our vision and nothing else. We don’t want to be socially involved,” says Gilbert. When I ask if they actually like other people, Gilbert responds: “Having friends is compromising. We stopped seeing friends in 1979. We realised it was all just about gossip and money.” When pressed he admits, “We do have a small group of nice friends.”

This self-imposed isolation is partly to remain “more private, more pure”: pure is a word that crops up often in their talk. They fear that greater interaction would taint them: “If we saw more art, more artists, we’d become normal. We don’t want to be normal like them. We don’t want to be normal because everybody is and we don’t want to be weird because the artists all like to be weird.”

G&G are, however, weird in the art world in being strongly pro-Brexit. When I raise the topic George states, with a hint of weariness, that his standard response is: “Good afternoon, Trump, Trump, Trump, Brexit, Brexit, Brexit. Now can we address the exhibition please.” But they are nevertheless quick to warm to the theme. When Gilbert asks what is wrong with being a sovereign country, George expands: “We Brexited from Rome 500 years ago and did very nicely out of it thank you. We wouldn’t have had the industrial revolution without it. And to our American friends we say, ‘You Brexited from us, and you’ve done fine.’ To all of our young friends, who come from all over the world, Australia and New Zealand and wherever, we say, ‘Well, you’ve Brexited to Britain.’ Anyway, you can’t leave Europe, it’s a continent, look at an atlas. We can leave Brussels, though.”

What then do they make of the current situation and the rift in politics and society Brexit has caused? “Brexit will happen or it will be a disaster and we will be a damaged colony of Europe,” reckons George. The damage has already manifested itself in other ways. “Perfectly normal people, gentlemanly and ladylike people, turn into spitting dogs when the subject comes up, or like cats when they go mad. Nicely dressed women — it’s wrong.” What’s more, they believe there’s incivility on the European side too. “If a man resigns from his club in the West End the club secretary says: ‘Oh we’re so sorry Mr George, is there something we can do?’ But Europe has not behaved like that.

“We think Europe is building a wall for itself, and you are either in or out. They want selective free movement of people but only of the right sort of people.” What’s more, “They are anti the Anglo-Saxon world, whether by design or chance.” They see Brexit as the chance to leave “some United States of Europe” that is not really an agglomeration of 28 member states but the plaything of just two, Merkel and Macron.

Indeed, perhaps unsurprisingly for a duo who define themselves as “two people, one artist”, they see binaries everywhere. In 2007 they had a major retrospective at Tate Modern but have grown to resent the place, not least because it doesn’t have a G&G work on display in its permanent collection. But, as modern British artists, they also dislike the way the London Tates split art into modern and British: it leaves them stateless. “You need a passport,” says George. “We advised [Nick] Serota not to do it that way.” They once asked a taxi driver about the Tates and he said that if one was Tate Modern the other must be Tate Old-fashioned. “You see they, the Tate, fucked it. They confused time and nationality.” It is the only time George swears during our conversation, despite the fact that G&G pictures are blushingly full of profanities and bodily matter.

They deny all artistic influence other than Mother, Jesus, Van Gogh and Terry- Thomas

Even when they first met as “baby art students” at St Martin’s School of Art in the late 1960s, their tutors told them to look to Europe and then slightly later they were made to swivel towards America. The message they took was that “You are not supposed to be what you are. It decided us immediately to stick our feet in the mud and be here.” So they deny all artistic influences other than, as George says, “Mother, Jesus, Van Gogh and Terry-Thomas.”

G&G are strangely defensive about their art given their long and immensely lucrative careers and, despite their disdain for the art world, feel disappointingly unloved in return. “Museums don’t show us,” says George, so they are building their own gallery nearby. So they are concerned with legacy. “We want to live forever. We have not been accepted. We make art because we want to win and be loved.”

When I suggest that some of their pictures are funny, that doesn’t go down too well. They refuse to be labelled satirists. “We don’t do fun. But we don’t do angst either.” They stress that their art, all those huge stained-glass-like photomontages, has a serious purpose. “Our art is speaking to the viewer,” they say, even if they don’t explain just what it is it is saying. “A lot of modern art is not speaking to the viewer, it’s just mucking about.” But then they rather undermine their case by reporting gleefully, “We had this show in Oslo recently and all the girls wanted to be photographed in front of our piece Bumholes.”

These switches from the grand to the childish or the quirky seem typical of them. In one sentence they can say “Our champion is Darwin who liberated us from Rome”, only to follow it by relating how, when they went to Darwin’s house in Kent they bumped into a retired army-type chap in the tearoom who said, much to their delight: “Saw you on the Jonathan Ross Show last night. Marvellous! Marvellous!” They clearly love recognition and believe they have earned it. Their joint life as a piece of performance art, those thousands of pictures, every one of which they meticulously record, and every bit of press coverage since 1969 cut out and kept, are proof for G&G that they have amounted to something. And not just anything: “We feel part, a small part, of the great Western success story.”