SANTA BARBARA, California — This singular Pacific coast town, Spanish at heart, haunts me — and why would it not? It is sometimes sultry and sometimes misty, but because it is also my hometown, Santa Barbara is  always more or less a ghost. 

Flinging frisbees across the broad lawn in front of the Mission Santa Barbara, to my mind the most magical of all of California’s 21 original Spanish missions; sidling up to thick, sugar-dusted French toast at a restaurant once unapologetically called Sambo’s; gazing up at the tall palm trees standing sentry over East Beach; or looking out across the channel to Santa Cruz Island in the hope of spotting a whale’s tail: Santa Barbara makes room for memories.

The frenzy of Los Angeles is just two hours down the coast, but a million miles away. There is no doubt that had I never left the ravishing confines of “America’s Riviera,” I would be a far more relaxed person than I am today. 

It is a ghost of a path once presented but only partially taken, and the rattling of chains comes in the form of dawn breaking over the Santa Ynez Mountains. The oak- and chaparral-covered slopes rise to heights of more than 4,000 feet and form the backdrop to the town, with the Pacific shore for sparkling center stage.

The deep channel offshore teems with dolphins and whales. For those who have the time and don’t mind waters that can get a little choppy, it’s worth looking into a whale watching excursion. From Butterfly Beach, more in the direction of Montecito, it is often possible to spot dolphins darting above the surface of the water. Behind that beach, the views from the Channel Road Drive coastal promenade and bicycle path are stunning, with vistas stretching to the Channel Islands in one direction and the lush slopes of Montecito in the other. 

Steep prices for things like nice houses and good quality hotel rooms tend to keep the tourist hordes at bay in this corner of Southern California, which is more or less the way the locals like it. Without a large port such as San Francisco’s or the naval nexus such as at San Diego, and little in the way of industry, Santa Barbara was always going to be something of a rarefied urban experience. 

West of the center, the multimillion-dollar mansions of Montecito speak to a stupendous amount of wealth, but as at Hope Ranch just to the east, it is all about discretion with approximately zero of the Hollywood flash and hustle.

The history runs deep. For a quick dive into it, stop at the Presidio Real de Santa Barbara, which Spanish missionaries built in 1782 to protect Spain’s colonial claims in these parts. The Spanish empire’s last military-style outpost in the New World retains the original adobe structure and it was built to last.

Even more emblematic of the city is the Mission Santa Barbara, founded by a Spanish Franciscan order in 1786 and set in a lush park setting. The 10th mission to be founded at Alta California, it was considered the “queen” of the missions that the Spanish built along El Camino Real, the royal road that once ran 965 miles north to south along the California coast. An earthquake in 1812 destroyed the mission (and most of Santa Barbara), but a new version with the signature double bell towers was completed by 1820.

The compact city of Santa Barbara itself was founded by the Spanish and the California Mission style (also called Spanish Colonial Revival) prevails. Another earthquake, in 1925, destroyed a vast number of original buildings but also heightened civic efforts to maintain a uniform historical aesthetic thereafter. 

Soak up the atmosphere of the white stucco walls and shaded arcades by taking a leisurely stroll up and down State Street, Santa Barbara’s main commercial thoroughfare, with particular attention to the 400, 500, and 600 blocks. Buildings with Moorish-flavored flourishes surprise on cross-streets like Ortega and elsewhere. 

In the 700 block there is McConnell’s ice cream, which is one of America’s best purveyors of fine ice creams There are never any fillers used, and aficionados of mint ice cream will want to know that the peppermint stick flavor is exceptional: Once you’ve had a scoop of this — let it melt a little —  it is difficult to settle for lesser mints. 

There are great restaurants aplenty, such as Jane for fresh homemade American bistro fare, also on State Street. My short sojourn scarcely left enough time for a taco run, but the Cal-Ital restaurant Convivo in my hotel, the breezy Santa Barbara Inn, left me in good stead. On the first night, after a four-hour drive up the coast from the Mojave Desert, I ordered a wood-fired spicy soppressata pizza and brought it up to the room to enjoy on the balcony with a view of the Pacific Ocean. 

Breakfast in Santa Barbara should never be rushed, and mine at Convivo was not. A frittata of fresh mozzarella, chives, and crispy jamón Serrano would not leave anyone hungry, while international tastes came through in an asparagus tartine with buttery brioche, two fried eggs, and labneh. A freshly baked spiced pecan cinnamon roll was big enough to feed Prince Harry’s ego.

On my last night at the inn, after a day of gourmet grazing and daydreaming in the Montecito section of Santa Barbara, I opened the sliding glass door to listen to the waves gnashing the dark beach on the far side of East Cabrillo Boulevard. Tiny lights from the offshore oil rigs twinkled on the horizon. With a little  imagination it could have been the ghostly outline of a Spanish galleon, on the prowl or just making its long, long way home. 

Unpacked is a periodic travel column from Mr. Grant in which he highlights or otherwise deconstructs a destination or experience of potential interest to today’s curious traveler.