Many people arrive at Dover, only to leave immediately. England’s ancient gateway to the world — one of the original five mediaeval Cinque Ports — seems to embody the Brexit psychodrama: post-industrial neglect, gridlocked lorries, paperwork, small boats.

In Dover Beach, Matthew Arnold’s 1867 poem on a crisis of faith beneath the chalk-white cliffs, the sound of the sea here is the “grating roar of pebbles … the eternal note of sadness in”. Why would anyone spend a day here?

Dover is one of those post-industrial towns, like Croydon, Swindon and Dudley, that is sometimes treated more like a punchline than a place. Perhaps it is easier, more comforting to joke about neglect than to question it. This year, though, the Kentish town is trying to shift that perspective with a new kind of summer festival.

You Can Reach The World From Dover is mid-way through its first, five-month programme, with less than £100,000 in funding from the Arts Council and a few other bodies. The festival’s outward-facing title is not the only unexpected thing about it. There is a radical, notably avant-garde bent to its programme, which has been curated by an artist-led collective: more happenings, conceptual art and experimental music than end-of-the-pier turns.

As an industrial port with a surrounding landscape riddled with abandoned military structures, “Dover does not have hipster, faded-glory appeal,” says Charles Holland. He is the architect, writer and co-organiser of the festival, which is run by Dover Arts Development. Its purpose is “a cultural approach to building community resilience”.

Neither is it an elegantly windswept wilderness, like Dungenees, nor a borderline twee day-trip favourite, like Rye. “It doesn’t fit the narrative of coffee shops and sourdough,” says Holland, drily.

For locals, the Citadel is loaded with self-destructive association

Dover’s gnarly, industrial appeal may not be obvious, but that means it can afford to experiment with the offbeat, the unexpected.

Like Swindon, Croydon and the rest, Dover has magnificent buildings and tunnels as big as cathedrals carved within its cliffs. They often sit alongside a motley array of unpromising industrial and military structures, sometimes languishing or abandoned. Some of them have been commandeered by the festival to surprising — and deliberately jarring — effect.

One is Shocked into Presence, an exhibition of paintings by Joanna Jones, is staged in the vast, sleeping-giant fortress known as the Citadel, a mostly derelict former prison crouched on the edifice of the Western Heights. Jones’ show is on display not in the pretty Napoleonic-era ramparts overlooking the sea, but deep within the Citadel complex, in the former prison canteen — a gloomy, late-20th century slab.

Holland points out that, for locals, the Citadel is loaded with self-destructive association. It was built during the Napoleonic era to defend the town from French attack. Had Dover been captured, its weapons would have turned on the town, reducing the invader’s prize to rubble. As a temporary art gallery, the building that was at once defender and potential attacker becomes something else entirely, an elevated place.

The festival’s guided walks take in more forgotten buildings. Dover Crossings, a walk around Dover’s neglected architecture led by the artist Simon Bill, makes participants clamber through thick brambles to reach the abandoned Farthingloe anti-aircraft battery. This string of concrete munitions stores and gun mounts is strewn across the nearby north downs.

They were, says Bill, built to fire at the Luftwaffe — less with the intention of shooting the bombers down, more to deplete their fuel reserves by forcing them to fly higher, out of range. Countless holes for mounting weaponry are bored in the ground here, ringed with rusting iron in which the guns would slot. Bill tells me there are so many concrete structures in this overgrown landscape that no one knows what to do with them. They do not feature on maps.

Concrete storage structures stand about in circles, like mute henges. One has store records still stencilled on the walls. Bill says he often leaves piles of pebbles along routes so he can find his way back, like a fairy tale.

Dover’s festival anchor project is I’m Looking Out for You, a candy-coloured temporary tower on the port seafront that lights up at night, designed by Holland, whose practice is based in the town. It is reminiscent of the two Roman-era Pharos that guarded the harbour, one of which survives at Dover Castle.

Unlike its near neighbours Margate and Folkestone, the town is less accustomed to art installations. Now, on a hot, mid-August afternoon, a live band is playing beneath the tower. A few beer stalls have sprung up, and a substantial crowd has gathered. People look to have settled in for the day. Perhaps Dover’s new annual summer celebration of its gritty, rich, conflicted history might help shift the national psychodrama elsewhere.