Poor Mitt Romney. He already had one big strike against him — his Mormonism — even before he contemplated his second run for the presidency. But thanks to the Tea Party, he’s also under serious fire for his 2006 Massachusetts health care plan, which his conservative critics liken to Obamacare — dubbing it “Romneycare” — since both plans mandate individual health coverage and expand the reach of Medicaid.

Romney has done his best not to repudiate one of his signature achievements as governor by suggesting that it’s just the kind of “state-based” solution uniquely tailored to local conditions that conservatives should support as an alternative to President Obama’s “federally mandated” plan. But no one’s actually buying that argument. Except, of course, Obama, and his close friend and political ally, current Massachusetts Governor Deval Patrick, who says Romney’s plan is working “brilliantly,” expanding affordable health coverage to nearly 98% of the state’s residents.

The White House, of course, knows a promising “divide-and-rule” strategy when it sees one. Under withering fire from conservatives — and with a majority of independents still opposed to Obamacare — it relishes the opportunity to cite the record of the presumptive GOP nominee as evidence that its own policies, notwithstanding Tea Party charges of “creeping socialism,” are broadly in tune with voter preferences, at least in a state that’s still — Scott Brown, notwithstanding — one of the bluest of the blue.

But as polls show, Romney’s clearly paying the price with conservatives. As the second-place finisher to John McCain in 2008, he’s the GOP’s heir apparent, but, by all appearances, he seems to be fading fast. Former Arkansas Gov. Mike Huckabee, who gave McCain and Romney an unexpected run for their money in 2008, besting both in a handful of critical primaries, has been leading Romney in the polls for weeks. And that was before the party’s new gadfly, Donald Trump, entered the fray, pushing Romney to an increasingly distant third nationally, and even lower in some statewide polls.

Romney, from the beginning, has counted on first winning a commanding victory in New Hampshire, then, like the proverbial oil spot, slowly spreading his influence to other states. But last week, in addition to Trump’s surge, he suffered more bad news: his key 2008 New Hampshire political operative, Wally Stickney, has publicly defected to GOP long-shot Jon Huntsman, who’s not only the former governor of Utah and, like Romney, a Mormon, but also Obama’s ambassador to China. The son of a billionaire, Huntsman is perhaps the most moderate GOP candidate currently contemplating a presidential bid — fiscally conservative and broadly pro-life, but still close to Obama and moderate Democrats on issues like cap and trade, immigration, and gay marriage. In fact, he even supported the stimulus package, which is likely to cost him dearly with conservatives.

Still, Stickney’s defection to Huntsman suggests that a shift may be underway among a critical segment of New Hampshire primary voters, who are notoriously independent and unpredictable. McCain won New Hampshire in 2008, but thanks largely to Stickney, Romney did well in and around Salem, which allowed him to finish a close second, earning him GOP contender status. Huntsman, it turns out, is a close ally of McCain and even has friendly ties to former Alaska Governor Sarah Palin. Huntsman was Utah’s governor when Palin was Alaska’s — both had stellar political records while in office — and it was Huntsman who gave the GOP convention speech formally nominating Palin as McCain’s running mate in 2008.

Palin hasn’t yet commented on Huntsman’s presidential bid, which won’t become official until he formally resigns from his ambassadorial post on April 30th. But despite her reputation for promoting Tea Party candidates, she’s frequently backed relative moderates — including McCain — out of personal loyalty or when she wanted to ingratiate herself with the GOP establishment.

Huntsman has also won support from GOP political architect Karl Rove, in part because both men have strong ties to the Bush family: Rove because he personally steered George W. Bush to the Oval Office, and Huntsman because he served as U.S. ambassador to Singapore under George H.W. Bush. Rove’s no friend of Romney, and is hostile to Trump’s candidacy, so in a GOP field dominated by the Tea Party conservatives, Huntsman could well turn out to be the establishment’s “dark horse.”

In addition to pressure from Huntsman (and Trump) in New Hampshire, Romney also faces a serious challenge on his Southern flank from Mississippi Governor Haley Barbour, who just hammered Romney, 22%-12%, in a recent South Carolina straw poll. Romney fared even worse in a second state poll, where Ron Raul came out on top. South Carolina is the most important early primary state next to New Hampshire and Iowa, and in Iowa, Romney is far behind other prospective GOP candidates, including Huckabee and Minnesota Governor Tim Pawlenty, in establishing a campaign base.

In fact, Romney’s not even safe in his own backyard. Pawlenty just visited Massachusetts to attend a huge Tea Party rally where there was widespread and vocal criticism of Romney on health care. An influential radio talk-show host has even set up a website, www.anyonebutmitt.com, and is urging Massachusetts conservatives not to back Romney, who managed to edge past McCain in the Bay State in 2008. Romney still has a strong lead in early straw polls in states like Michigan (based largely on his father’s reputation as the state’s popular former governor), and in Nevada, a critical Western primary, but all of those polls were conducted in January when Romney was still the GOP frontrunner everywhere.

Should Mitt call it quits? No one’s said so, and undoubtedly no one will, this early in the contest. But with Huckabee hinting at a potential “bloodbath” on the right, and Huntsman, Barbour, and Trump encroaching on his political turf, the writing may be on the wall. And there’s also the Mormon factor: the entry of Huntsman into the race is likely to highlight Romney’s own Mormonism, which he’s done his best to downplay, largely successfully, to date. Americans, generally, may be prepared to embrace a Romney or a Huntsman, but are conservatives? There’s always been a deep undercurrent of evangelical hostility towards Mormons that could easily bubble to the surface if Huntsman starts gaining traction — which, ironically, could provide just one more reason for conservatives to withdraw their support from Romney.

Romney still has an impressive war chest, and the GOP field is still so fragmented that he’s undoubtedly planning to weather the current challenge. Then, too, he may well remember that McCain was all but counted out in 2008, but managed to hold on long enough to launch a successful comeback. But McCain, of course, first won in New Hampshire. If current trends hold, Romney’s own front-loaded primary strategy may soon unravel completely.

Stewart J. Lawrence is a Washington, D.C.-based public policy analyst who writes frequently on immigration and Latino affairs. He is also founder and managing director of Puentes & Associates, Inc., a bilingual survey research and communications firm.