It may be just over an hour from New York’s Grand Central Station, but New Canaan, Connecticut is a different world.

On a flawless October afternoon, it’s a full-immersion American fantasia: leafy hills coloured russet and gold, light as luscious as that of a pre-revolutionary Fragonard, the air scented with cinnamon and hubris. New Canaan boasts not only one of the most affluent communities in the US – don’t stand in the way of the terrifying women stocking up on tiny Pierrot-styled cashmere cardigans in Crewcuts – but also the east coast’s largest concentrations of mid-century architecture. (And only third to Los Angeles and Palm Springs.)

You could take a trip to this pretty slice of haute-suburbia just to soak up the Rick Moody mise-en-scène, and peek at the odd-master work from the road. But every summer and fall, hordes of architecture-tourists now come to see celebrated architect Philip Johnson’s Glass House at close hand. This tiny but extravagantly glazed jewel in the Connecticut modernist crown has, since the architect and partner David Whitney’s deaths, been under the protection of the National Trust for Historic Preservation and open to the public since 2007. The tours have proved so popular the organisation has had to continually extend their programs to meet demand.

While Johnson’s ultimate legacy in architectural circles is as an educator and populariser, his 1949 Glass House is iconic enough a work to make it into high school art and design textbooks the world over. It’s a house pared back to its essence, with minimal framing and a flat roof; it both underscores and subverts what we normally recognise as shelter, substituting glass for brink or wood or concrete. Mies ‘less is more’ van der Rohe’s extreme clarity, his much vaunted ‘skin and bones’, is here rendered literally – the house is transparent. Dividing walls are all but dispensed with, although traditional zones to eat, rest, cook, sleep and wash are still demarcated, almost stridently so, and the interior materials – wood, leather, tiles and brick – are seductively warm, surprisingly cottage-like. A central cylindrical unit houses a fireplace and the bathroom; its archetypal chimney shape firmly earths the surrounding lightness.

While the house’s aggressive minimalism has become shorthand for American modernism itself, it’s the site – an equally iconic east coast landscape of some 47 acres – that is its defining feature, and what makes visiting such an extraordinary experience. Johnson used a prized Poussin as a screen in the living area, and the painting is key to the whole shebang. This idealised Baroque landscape served as both inspiration for the ever growing, ever manipulated estate that surrounds the house, as well as being a formal, but characteristically pert, declaration of the architect’s intent.

The Glass House is, for all its many ascribed meanings, essentially a pavilion from which to simply be, to exist for a time in a perfected visual world. Johnson was enamoured with the constructed wildness of Paris’ Parc des Buttes-Chaumont and neo-classical English gardens of the 18th-century, and the view from the house combines these old world influences with rambling fields and meadows that recall those of his Ohio childhood.

As well as a swimming pool, a neo-Baroque lake pavilion, a ghostly house-shaped ‘folly’ and a large sculpture by Donald Judd, the grounds are dotted with several other structures. These include the original brick guesthouse, solidly facing off with the Glass House itself; a campy, mad (and maddening) Frank Stella collaboration, ‘Da Monsta’; and a cell-like po-mo library. Johnson and Whitney’s collection of paintings by Andy Warhol, Robert Rauschenberg, David Salle, Cindy Sherman, Frank Stella and Julian Schnabel inhabit an underground art lair that was built in 1962; although Johnson claimed it was inspired by a Greek tomb, it feels like the quintessential cold war bunker. The works are kept on revolving ‘poster racks’, making for an oddly hermetic, if luxurious, way to experience a collection, a few paintings at a time. It could almost be one of the basement home entertainment zones of the surrounding McMansions, only for more rarefied tastes.

The sculpture gallery, with a booty that includes a Dan Flavin and a Bruce Nauman, has an almost opposite aesthetic; with its glass ceiling and tubular steel rafters, it’s light to the point of feeling like a garden shed, or, with its continual play of light across walls and floor, rather more like an actual garden glass house.

What of the architect himself? The tour is flush with sparkling anecdotes about infamous society soirees that took place at the house, Johnson’s propensity for snobbery and arch repartee, his prodigious work ethic and his often fractious relationship with his mentors Mies van der Rohe and Frank Lloyd Wright. One thing about Johnson that isn’t mentioned in the tour, nor in any take-home literature, is the rather galling fact that, once upon a time, he wasn’t just your garden-variety design fascist. He was, in fact, a gung-ho Nazi sympathiser, involved in the far right’s cause deeply enough to be welcomed as a guest of the Third Reich (to witness the invasion of Poland no less).

Unlike like the German Mies, Studio BBPR in Milan or Finland’s Alvar Aalto, his politics were not conveniently contrived for commissions’ sake, or survival. Johnson only really tamped down his domestic propagandising when, in 1940, the FBI had a quiet word about possible consequences if the US was to enter the war, which, of course, it soon did. The expat Australian art critic Robert Hughes recalls interviewing Albert Speer, Hitler’s chief architect, in 1978, and being asked by Speer to take a cheerfully inscribed copy of his recently published monograph back to Johnson in New York. On unwrapping the gift (naturally while lunching at the Four Seasons restaurant, Johnson’s Seagram Building collaboration with Mies), he paled and furiously stuffed the book under the banquette. ‘You haven’t shown this to anyone?’ Johnson murmured, according to an amused and ever provocative Hughes.

Looking at Johnson’s spectacular American idyll with the knowledge of his early (and later regretted) political obsessions isn’t vital, or even necessary, to appreciating or understanding it. But the motivations that drove modernism on both the left and the right throughout the 20th-century, not to mention architects’ often-Faustian pact with power, are, for me at least, worth pondering.