On Monday, fact-checking organization PolitiFact tweeted a link to one of the entries in its “Trump-O-Meter” campaign-promise tracker. In doing so, the organization highlighted a longstanding problem with its own work.

PolitiFact tweeted: “As a candidate, Trump said, ‘We’re going to be saying Merry Christmas at every store. You can leave (happy holidays) at the corner.’ Retailers continue to use the phrase ‘Happy Holidays,’ but as president, Trump has stayed away from using the term,” and it marked the statement a “promise kept."

While PolitiFact quietly deleted the self-contradictory tweet, much of the text comes straight from the “Trump-O-Meter” ruling, which is itself poorly reasoned and generally misleading. It’s also a missed opportunity for a site with a mission of giving “citizens the information they need to govern themselves in a democracy.”

“Trump pledges to break from Obama, say ‘Merry Christmas,’” reads the Inauguration Day entry in the “Trump-O-Meter.” The problem with this statement, aside from it being a totally ludicrous promise that has nothing to do with policy or governance, is that President Barack Obama frequently said “Merry Christmas.” The argument that Obama was waging some sort of “war on Christmas” during his time in office was a totally baseless right-wing talking point. Still, the PolitiFact entry on Trump’s promise takes the false premise at face value. Additionally, even though the quote being fact-checked was, “If I become president, we’re going to be saying Merry Christmas at every store,” PolitiFact’s rating implies that he wasn’t actually promising that “every store” would say “Merry Christmas” to customers. 

The “Merry Christmas” fact-check is pretty insignificant on the surface, but it’s important for what it shows about the system in place. It’s a “brown M&Ms” moment.

In the 1980s, rock band Van Halen famously included an odd item in its touring contract rider: M&Ms, but “absolutely no brown ones.” The point of the hyper-specific request had nothing to do with the candy itself, but was a way for the band to determine whether concert promoters had actually read the contract. Singer David Lee Roth explained in a 2012 video that the line item helped him know whether proper safety precautions were being taken with regard to staging and lighting. If the venue couldn’t get something as simple as the catering correct, how could it be trusted to ensure that the larger and much more significant items weren’t handled in a similarly sloppy way?

The “Trump-O-Meter” mistake highlights deeper flaws and biases at play in PolitiFact’s operation. “We rate campaign promises based on verifiable outcomes, not on intentions or effort,” reads the “Our Process” page on PolitiFact’s website. But Trump was given a “promise kept” for an outcome he didn’t (and couldn’t reasonably) achieve. Sure, the effort and intentions may have been there, but that doesn’t matter, according to the organization. The promise that “we’re going to be saying Merry Christmas at every store” has a verifiable outcome, and it wasn’t realized.

Perhaps this seems a bit persnickety, but PolitiFact didn’t offer Obama the same level of leniency.

One entry on the site’s “Obameter” is a check on Obama’s 2008 promise to “place the full weight of (his) administration behind ... a fully inclusive Employment Non-Discrimination Act to outlaw workplace discrimination on the basis of sexual orientation and gender identity.”

The entry notes that Obama supported the 2013 version of the bill, which cleared the Senate 64-32, and issued a statement reading in part, “Workers should not fear being fired from their jobs, harassed at their workplaces, or otherwise denied the chance to earn a living for themselves and their families, simply because of sexual orientation or gender identity.” The bill stalled out in the Republican-controlled House of Representatives.

When a new version of the bill came up during the next Congress, the Obama administration supported that, as well, though it failed to make it through either congressional chamber. Undeterred, Obama signed an executive order prohibiting federal contractors from engaging in anti-LGBTQ discrimination, issued a May 2016 memo to public schools against anti-transgender discrimination, and put the administration’s weight behind a lawsuit against North Carolina’s anti-trans “bathroom bill.”

So did he make good on the promise to “place the full weight of (his) administration” in support of bills and policies that would protect LGBTQ people from discrimination? Based on the information included in the entry itself, the answer is a clear yes, as his promise was to advocate for passage rather than will it into existence. Bafflingly, PolitiFact rated this a “promise broken,” as the bill never became law, even though that wasn’t part of the statement being checked.

A similar example can be found in PolitiFact’s “promise broken” ruling on a claim that Obama would work to “get an assault weapons ban reintroduced” in Congress. Such bans were introduced in the House and Senate, but as a bill never made its way to his desk for signature, he was awarded a “promise broken” despite making good on the specific statement being fact-checked.

Many of the “Obameter” and “Trump-O-Meter” rulings are entirely fair. No, Obama wasn’t able to double the number of American exports between 2012 and 2016. No, Obama wasn’t able to make good on his promise to “institute a common standard for securing [personal] data across industries.” No, it’s safe to say that Obama was not able to “bring Democrats and Republicans together to pass an agenda that works for the American people.”

But Trump gets an absurd amount of leeway compared to his predecessor. “As soon as I take office I will ask Congress to fully eliminate the defense sequester and will submit a new budget to rebuild our military,” reads one of Trump’s promises. PolitiFact rated this a “promise kept,” though the sequester “remains on the books.” Instead of eliminating it, Trump and Congress have simply raised the spending caps to levels where it has little effect. It’s understandable how PolitiFact might reach such a conclusion, but it’s inconsistent with how it rated Obama.

PolitiFact often doesn’t paint a complete picture of what it is fact-checking.

“I’m not going to cut Social Security like every other Republican and I’m not going to cut Medicare or Medicaid,” Trump said during the 2016 campaign. Since taking office, he’s tried repeatedly to gut the social safety net. Even so, the pledge to “make no cuts to Social Security” received a “promise kept.”

Here’s PolitiFact’s convoluted explanation for how it arrived at this decision.

Trump’s promise to save an Indiana Carrier plant received a “promise kept” rating based mostly on a technicality. In a May 2018 interview with Popular Mechanics, United Steelworkers Local chief Robert James explained that he and other Carrier workers had been misled by the company and Trump in their much-hyped promise to save union jobs.

In fact, 400 of those 1,100 jobs “saved” were never at risk of being cut. Hundreds of factory employees lost their jobs in the years since, and according to a New York Times article in August 2018, remaining workers “share a looming sense that a factory shutdown is inevitable — that Carrier has merely postponed the closing until a more politically opportune moment.”

Is the factory still open? Yes. Does that matter in any sort of practical sense to the hundreds of people who lost their jobs after being under the impression that Trump had saved them? Probably not.

Yet another example of a promise that was technically “kept” but practically screams out for context is Trump’s pledge not to take a salary as president, which received a “promise kept” rating. He may not be collecting his $400,000 annual salary as president, but that doesn’t mean Trump’s not raking in cash from the federal government. The “Trump-O-Meter” entry runs through a list of some of the recipients of Trump’s donated paychecks, but it makes no mention of the fact that Trump’s properties have been charging the government millions of dollars since taking office. The ruling aside, it’s strange that PolitiFact wouldn’t even include this important context in its write-up. This is another example of something that seems unlikely to have happened had a previous president done it.

Trump has a way of exposing the flaws in media institutions, fact-checkers included.

How do you handle a president who lies on a scale unlike that of anyone to hold that office before him? It’s a question journalists have had to ask themselves repeatedly for the past three and a half years, and two of the most popular conclusions seem to be: grade him on a curve, or risk the wrath of his Twitter account and bad-faith conservative media operatives, who are all-too-eager to scream “Liberal bias!” While the latter displays integrity and bravery, many news organizations have defaulted to the former.

On PolitiFact, this approach can be seen in how the site grades some of Trump’s more outlandish claims and conspiracy theories. The site defines “false” claims as simply “not accurate,” but “pants on fire” as “not accurate and makes a ridiculous claim,” yet the application of the terms often makes little sense. For instance, recent “pants on fire” claims include his statement that Joe Biden wants to “abolish our police departments” and “abolish our prisons,” a claim that he was the one who gave former Defense Secretary James Mattis the nickname “Mad Dog,” and his bizarre assertion that everyone who takes a flight or rides a train is receiving a COVID-19 test. Yet somehow, Trump’s wildly untrue claim that Biden wants to ban windows in all new construction buildings only received a “false,” as did his complete fabrication that Biden wrote a “letter of apology” for something he never actually said. Lies about charter schools, the cost of insulin, Stacey Abrams’ beliefs about voting, and his electoral performance with women were considered only “mostly false.”

If you go through Trump’s PolitiFact page, it’s apparent that numerous things that should have been labeled “pants on fire” were instead listed as simply “false,” some of the things that should have been listed as “false” were instead listed as just “mostly false,” and multiple things that should have been listed as “mostly false” were instead categorized as “half true.”

Meanwhile, certain politicians seem to get the opposite treatment. For instance, PolitiFact rated Sen. Bernie Sanders’ claim that Mike Bloomberg said in 2015, “I, for example, am not in favor, have never been in favor of raising the minimum wage,” as just “mostly true.” Sanders made a factual claim about something one of his then-opponents in the Democratic primary said in the past, but PolitiFact dinged him on it because Bloomberg’s policy positions in 2020 were different. That’s not how fact-checking works. Sanders’ statement was true. Similarly, Sanders’ claim during a February 19 debate that “Mike Bloomberg owns more wealth than the bottom 125 million Americans” was listed as just “mostly true” because “the roughly 123 million Americans at the bottom of the wealth spectrum are collectively $44 billion in the red” and thus “millions of Americans can also say they're richer than more than 100 million Americans.” Setting aside how odd it is to simply say that people in debt don’t count in this scenario, it’s another example of PolitiFact seeking out something that wasn’t argued by a politician in order to bump the ruling back a level.

While PolitiFact’s coverage of Trump is not particularly flattering, bias makes it seem better than it actually is. The opposite bias bumps progressive politicians in the other way.

Facts don’t care about balance. Facts don’t care about whether you’ve already labeled five claims this month as “pants on fire” so maybe it’d be better to put the sixth in “false.” Facts don’t care if your “lie of the year” consistently comes from one party or ideology over another. Facts are facts, and it’s time to stop letting fear of being called biased dictate how they’re presented.