The Threat of Supply Chain Attacks For ML: Insecure Files and Antivirus Software Size Issues in Machine Learning

I predict that an AI supply chain attack will occur within the next year.

As a long-time cybersecurity professional and founder of a new business focused on generative AI, I have a unique perspective on this topic. I've observed several concerning developments related to model file creation, sharing, distribution, and use in recent months. While some of these concerns have been widely discussed, others have not received enough attention. One reason for writing this article is the frequency with which I see comments in forums about models being flagged by antivirus software.

The risk of supply chain attacks continues to grow, with hackers exploiting vulnerabilities in trusted software and hardware components to compromise system integrity and security. In this article, I aim to shed light on two specific aspects of supply chain attacks in the context of machine learning and the booming field of generative AI: insecure files and the size issues associated with running large files through antivirus software.

Insecure Files:

Checkpoint files, commonly abbreviated as CKPT files, are critical and commonly used components in machine learning workflows. They serve as snapshots of trained models, enabling researchers and practitioners to save and restore the state of a model during the training process. Although there are newer and more secure standards for model files, such as safetensor files, CKPT files are still widely used and supported. Unfortunately, these and other trained model files are potential targets for supply chain attacks.

Supply chain attacks involving model files typically involve adversaries tampering with the files before distribution, often through sharing on common model repository sites. We have witnessed similar attacks on popular software repositories in the past. Once the files are compromised, they can be distributed through trusted channels, leading to widespread compromise when users unknowingly download and utilize the tampered files.

These attacks pose serious risks to machine learning practitioners. Tampered files can introduce backdoors, malware, or altered model weights that compromise the integrity of trained models. Adversaries can exploit these vulnerabilities to conduct unauthorized activities, including stealing sensitive data, manipulating predictions, or using compromised models as a stepping stone for further attacks.

Size Issues with Antivirus Software:

Machine learning models have become increasingly complex and large. For example, the average size for Stable Diffusion files at @eightbuffalomediagroup ranges between 2GB to 8GB, with some models reaching 16GB. With the rise in file sizes, traditional antivirus software faces challenges in effectively scanning and analyzing these large files. Online file scanners often have hard limits, with 25MB being a common threshold at the time of this writing. Different desktop antivirus software may have varying limits, but most have file size limitations for scanning.

Many antivirus solutions rely on file scanning mechanisms to detect and mitigate threats. However, the sheer size and quantity of large files, especially in the Stable Diffusion Communities, can cause performance degradation and potential false negatives. Additionally, if the output of the machine learning process is the attack target, identifying the issue becomes even more challenging.

Mitigation:

To mitigate the risk of supply chain attacks involving files, it is crucial to ensure secure distribution channels, implement robust authentication mechanisms, and regularly verify the integrity of downloaded files using cryptographic checksums or digital signatures. While newer file types support encryption and improved security, blindly trusting the source is still not advisable. Machine learning practitioners should stay updated with the latest security practices and follow recommendations from framework developers and trusted sources.

It is necessary to adapt checks and scanning techniques to handle large machine learning files promptly. This may involve implementing intelligent scanning algorithms, leveraging distributed computing resources, or utilizing specialized machine learning models to analyze file security without compromising performance. In their Ethics and Society Newsletter #4, @Huggingface mentioned red-teaming models, which I applaud, but I am concerned that security might lag behind in this fast-moving field.

We, as machine learning practitioners, can adopt complementary security measures to enhance the protection of our systems. These measures may include isolating machine learning environments, implementing network segmentation, and establishing strict access controls. These practices help limit the attack surface, prevent unauthorized modifications to the files, and reduce the risk of downloading models from untrusted sources.

Conclusion:

Supply chain attacks pose a significant threat to the integrity and security of machine learning workflows. The compromise of insecure files and the size issues associated with scanning large machine learning files through antivirus software highlight the evolving challenges faced by both practitioners and cybersecurity experts. To mitigate these risks, a multi-faceted approach is required, encompassing secure distribution channels, robust authentication mechanisms, regular integrity verification, adaptive antivirus scanning techniques, and complementary security measures. By being proactive and vigilant, machine learning practitioners can enhance the resilience of their systems and protect against the stealthy threat of supply chain attacks. It is crucial to prioritize security in the rapidly evolving field of AI and take the necessary steps to safeguard our models, data, and systems.