New Hugging Face Vulnerability Exposes AI Models to Supply Chain Attacks

Cybersecurity researchers have found that it's possible to compromise the Hugging Face Safetensors conversion service to ultimately hijack the models submitted by users and result in supply chain attacks.

"It's possible to send malicious pull requests with attacker-controlled data from the Hugging Face service to any repository on the platform, as well as hijack any models that are submitted through the conversion service," HiddenLayer said in a report published last week.

This, in turn, can be accomplished using a hijacked model that's meant to be converted by the service, thereby allowing malicious actors to request changes to any repository on the platform by masquerading as the conversion bot.

Hugging Face is a popular collaboration platform that helps users host pre-trained machine learning models and datasets, as well as build, deploy, and train them.

Safetensors is a format devised by the company to store tensors keeping security in mind, as opposed to pickles, which has been likely weaponized by threat actors to execute arbitrary code and deploy Cobalt Strike, Mythic, and Metasploit stagers.

It also comes with a conversion service that enables users to convert any PyTorch model (i.e., pickle) to its Safetensor equivalent via a pull request.

HiddenLayer's analysis of this module found that it's hypothetically possible for an attacker to hijack the hosted conversion service using a malicious PyTorch binary and compromise the system hosting it.

What's more, the token associated with SFConvertbot – an official bot designed to generate the pull request – could be exfiltrated to send a malicious pull request to any repository on the site, leading to a scenario where a threat actor could tamper with the model and implant neural backdoors.

"An attacker could run any arbitrary code any time someone attempted to convert their model," researchers Eoin Wickens and Kasimir Schulz noted. "Without any indication to the user themselves, their models could be hijacked upon conversion."

Should a user attempt to convert their own private repository, the attack could pave the way for the theft of their Hugging Face token, access otherwise internal models and datasets, and even poison them.

Complicating matters further, an adversary could take advantage of the fact that any user can submit a conversion request for a public repository to hijack or alter a widely used model, potentially resulting in a considerable supply chain risk.

"Despite the best intentions to secure machine learning models in the Hugging Face ecosystem, the conversion service has proven to be vulnerable and has had the potential to cause a widespread supply chain attack via the Hugging Face official service," the researchers said.

"An attacker could gain a foothold into the container running the service and compromise any model converted by the service."

The development comes a little over a month after Trail of Bits disclosed LeftoverLocals (CVE-2023-4969, CVSS score: 6.5), a vulnerability that allows recovery of data from Apple, Qualcomm, AMD, and Imagination general-purpose graphics processing units (GPGPUs).

The memory leak flaw, which stems from a failure to adequately isolate process memory, enables a local attacker to read memory from other processes, including another user's interactive session with a large language model (LLM).

"This data leaking can have severe security consequences, especially given the rise of ML systems, where local memory is used to store model inputs, outputs, and weights," security researchers Tyler Sorensen and Heidy Khlaaf said.