Analyzing the Current State of AI Use in Malware

Executive Summary

Unit 42 researchers searched through open-source intelligence (OSINT) and our internal telemetry for potential signs of malware made to any degree with large language models (LLMs). This includes either using LLMs to create the malware entirely or to assist with their functionality. This article examines two samples, both of which originated from our OSINT hunts.

The rise of AI has sparked considerable interest in its potential applications within cybersecurity, both from the defender and attacker perspectives. We currently consider three primary use cases for AI as applied by the creators of malware:

1. Leveraging AI to write malware
2. Leveraging AI for remote decision making (e.g. augment or replace a command-and-control operator)
3. Leveraging AI for local decision making (e.g. locally executed agentic attack flows)

Unit 42 has analyzed malware that fits the first two categories: AI-written malware and malware controlled by an AI command-and-control (C2) for remote decision making. We are not aware of any examples in the wild of the third category: locally executed agentic attack flows.

We believe that threat actors are leveraging AI to help write malware, and that AI enables lower-skilled threat actors to create functional malware. However, we still see attackers having significant challenges in deploying local models to a target environment for malicious use or embedding them directly into a malware sample for local decision making and execution.

This article focuses on our analysis of samples that leverage AI for remote decision making. We’ll discuss the following two cases that represent the current state of AI in malware:

• AI Theater: An Infostealer’s Illusory LLM Features
  • A trio of highly similar .NET information stealer samples that incorporate the OpenAI GPT-3.5-Turbo model via HTTP API. We will explore the implementation and assess the practical impact of its AI integration.
• AI-Gated Execution: Malware Dropper's LLM-Based Environment Assessment
  • A malware dropper written in Golang that leverages an LLM to evaluate a system and provide a decision on whether to proceed with an infection. The sample was initially highlighted on X as a dropper for Sliver malware.

Palo Alto Networks customers are better protected from the threats discussed in this article through the following products and services:

• Advanced Threat Prevention
• Advanced WildFire
• Cortex XDR and XSIAM

The Unit 42 AI Security Assessment can help empower safe AI use and development.

If you think you might have been compromised or have an urgent matter, contact the Unit 42 Incident Response team.

AI Theater: A .NET Infostealer’s Illusory LLM Features

The first sample we’ll discuss is an information stealer that integrates its functionality with OpenAI's GPT-3.5-Turbo via HTTP API. Encountering .NET malware packed with ConfuserEx 2 and observing calls to OpenAI was certainly exciting for a researcher, as it likely indicates a malware sample using an AI integration for remote C2.

This integration with OpenAI indicates the malware may enable a lower skilled threat actor to interact with an infected environment without having to learn lateral movement, data collection and persistence techniques themselves. However, as we discuss later in this post, the integration with OpenAI is poorly implemented and not fully functional for some of the API calls that are available to the malware. This may indicate early testing or a low skilled actor.

Artifacts such as the console log generated by the malware suggest that it may have the following capabilities:

• Dynamically generating a scare message without supporting functionality
• Analyzing target environments
• Creating host endpoint detection and response (EDR)/antivirus (AV) evasion content

Examining the sample will reveal if these capabilities align with the sample's actual functionality.

The malware is written in C# (.NET Framework 4.0) and has been obfuscated with ConfuserEx 2. The obfuscation allows the malware author to potentially hinder both analysis and detection efforts. This sample is a functional information stealer and begins by collecting and saving data to disk, like system information, browser cookies and file listings. This data is then exfiltrated to a C2 server.

We found two similar samples of this malware, both with the same functionality. Both samples feature the same type of LLM use.

LLM Use Represented Through Four Function Calls

References and requests to the OpenAI LLM API are contained in four function calls. None of these calls positively impact the malware’s operation. In fact, these calls add noise, which defenders are likely to notice. This specific implementation of these requests and references is a nonsensical use of an LLM in malware.

These four function calls are:

• GenerateEvasionTechnique()
• AnalyzeTargetEnvironment()
• SendToC2ServerWithLLM()
• GenerateObfuscatedCommunication()

Method One of Four: GenerateEvasionTechnique()

This method sends the following prompt to the OpenAI GPT-3.5-Turbo model using the standard API:

As instructed, the LLM returns a technique name (e.g., Random Delay, Process Spoofing). The malware author set a default technique name of Random Delay in case this API call fails.

The technique name returned from the LLM is simply written to victim_logs.txt on the victim's desktop directory. An example of content from one of the victim_logs.txt files is:

It is important to note that technique names returned from the LLM are not actually implemented. They appear to be for logging purposes only. Realistically, the LLM could return any three words for an evasion technique name, so implementing this technique correctly would require one of two options to succeed:

1. The malware would require handler code to execute based on the string returned from the LLM.
2. The LLM would have to send data back that could be converted to executable code at runtime.

These are both feasible options, but the malware samples we've discovered using this API call do not implement either option.

Method Two of Four: AnalyzeTargetEnvironment()

This method sends the following prompt to the OpenAI GPT-3.5-Turbo model using the standard API:

The LLM response from this prompt is different from the GenerateEvasionTechnique() method, because the malware actually implements the result and sleeps for anywhere between 1-5 seconds (1,000-5,000 milliseconds). If the LLM fails to respond, the malware samples use a default value of 2 seconds for the sleep duration.

From a malware reverse engineering perspective, this is a nonsensical use of an LLM because the response has no practical impact. The author (human or otherwise) of this malware sample does not appear to have any tangible experience in the design of tooling evasion to draw from, nor the knowledge to reasonably speculate on evasions.

Method Three of Four: GenerateObfuscatedCommunication()

This method sends the following prompt to the OpenAI GPT-3.5-Turbo model using the standard API:

Similar to GenerateEvasionTechnique(), the LLM returns an obfuscation technique name, which is ultimately written to a log file. The malware creates a simple structure as shown below. The timestamp is randomly generated before it is encoded as a Base64 string.

It may be tempting to consider that perhaps the timestamp was Base64 encoded, as the LLM suggested in the above example. However, we could not find any implementation of Base64 that the malware leverages. The technique name is simply copied to the console output and a JSON log file. An example of the console output from this technique is:

Once again, it is important to recognize that the output of this method is yet another unimplemented feature. There is no code to dynamically enforce a data obfuscation algorithm that is used in the C2 protocol. This is certainly feasible to implement, but the developer has not done so in the samples of this malware we reviewed.

Method Four of Four: SendToC2ServerWithLLM()

This is the method that is responsible for sending data back to the C2 server. The malware sends the following prompt to the OpenAI GPT-3.5-Turbo model using the standard API:

The LLM returns with a legitimate-sounding message (e.g., "Routine system diagnostics completed successfully. Data transmitted for analysis."). The malware prepares an HTTP request and modifies the HTTP request header based on the response. The following are lines added to the HTTP request headers by this method:

The malware sends the stolen data in JSON format to hxxp[:]//localhost:3002/crypto-data. Like most of the parameters used in the previous three methods by these malware samples, the C2 URL is simply a default value. This could indicate that the sample was not intended for actual use or was merely built for testing locally. The LLM-generated message is sent with every attempt at C2 communication.

This function is different from the others in that an action is taken. Data could be successfully exfiltrated if a legitimate C2 server address or domain is provided. On the other hand, the additional HTTP request headers add no functionality, and they only appear to highlight that an LLM is being leveraged.

What These Methods Tell Us

The primary purpose of this malware is to:

• Extract sensitive data from victim systems (browser cookies, system information, file listings)
• Use AI/LLM capabilities to dynamically adapt its behavior in an attempt to evade detection
• Exfiltrate stolen data to a C2 server with LLM-enhanced communication
• Attempt to evade detection through extensive logging that impersonates legitimate activity

These samples may have been generated with AI assistance, or they may have been simply guided by an inexperienced individual or team. Artifacts produced by these samples suggest interesting possibilities for the future of AI integration into C2 management, but its use of LLMs only provides an illusion of practicality. Ultimately, we can consider this AI theater.

AI-Gated Execution: A Malware Dropper's LLM-Based Safety Assessment

The second malware sample acts as a dropper for Sliver, an open-source adversary emulation and red team framework. Before deploying the payload, the malware sample gathers system information, including its own process name and that of its parent. It then decrypts Donut shellcode, but instead of immediately executing the shellcode, this dropper uses the collected data to assess the environment's "safety" via an LLM.

The following information about the victim host is collected in the system survey:

• Hostname
• Process list
• Network information
• USB drives
• System uptime

This information is inserted into a prompt and sent to OpenAI’s GPT-4 model using an HTTP API. The prompt offloads the decision-making to the LLM for determining if it considers the environment safe to drop the Sliver payload.

Traditionally, this step is handled efficiently within the malware by carefully crafted heuristics, often combined with allow lists and deny lists, which is common practice in ransomware. However, using an LLM to make the verdict is a new approach that may prevent defenders from determining which process or system setting the malware authors are hiding from.

Details

The prompt clearly states its intention, provides inputs and gives general guidance on how to interpret the system survey data. An example of this prompt is shown below.

Upon execution, the sample parses the response as JSON data and checks the execute key generated by the LLM. This response reveals that the LLM effectively reviews the submitted data and passes a verdict on the safety of the environment.

If the execute key is true, the dropper proceeds to launch its Sliver payload. The dropper also writes a log file to disk (opsec.log) in the same folder it is located in during execution. An example of the opsec.log content is shown below. Note that the log output states the Sliver payload is “AI-powered,” though this is not the case.

What This Use of AI Tells Us

This malware dropper is notable for its use of an LLM to make execution decisions. While the LLM is hosted remotely, delegating the determination of a safe environment to AI is an interesting concept. Traditionally, this is achieved through hard-coded allow lists and deny lists. However, leveraging an LLM allows for potentially more intelligent connections between system data points, leading to a more accurate verdict. A logical next step could be to evolve to locally execute a small language model or a simple ML model trained to classify the safety of a host environment based on its features.

Conclusion

The current landscape of AI in malware is characterized by experimentation and uneven integration. The .NET infostealer samples demonstrate a superficial and ultimately ineffective use of LLMs as AI theater. The malware dropper showcases an interesting approach by leveraging AI for environment assessment as AI-gated execution.

While we cannot yet conclusively determine if developers used AI to create these malware samples, the potential for AI to aid in malware creation highlights a concerning issue of lowering the barrier to entry for less-skilled threat actors.

Looking ahead, we anticipate a future where AI plays a greater role in both malware creation and execution. As local model deployment becomes more feasible, we may see malware samples with embedded AI capabilities (especially code generation) that can more dynamically adapt to their environment, evade detection and optimize malicious activities in real-time.

The rise of AI-assisted malware could manifest in the form of increased feature cadence and reliability. It will be crucial to monitor these advancements and develop defenses that can effectively counter an evolving AI-driven threat landscape.

Palo Alto Networks customers are better protected from the threats discussed in this article through the following products:

• Advanced Threat Prevention is designed to defend networks against both commodity threats and targeted threats, including the Sliver dropper.
• The Advanced WildFire machine-learning models and analysis techniques have been reviewed and updated in light of the indicators shared in this research.
• Cortex XDR and XSIAM help to prevent the threats described in this article, by employing the Malware Prevention Engine. This approach combines several layers of protection, including Advanced WildFire, Behavioral Threat Protection and the Local Analysis module, to prevent both known and unknown malware from causing harm to endpoints.

The Unit 42 AI Security Assessment can help empower safe AI use and development.

If you think you may have been compromised or have an urgent matter, get in touch with the Unit 42 Incident Response team or call:

• North America: Toll Free: +1 (866) 486-4842 (866.4.UNIT42)
• UK: +44.20.3743.3660
• Europe and Middle East: +31.20.299.3130
• Asia: +65.6983.8730
• Japan: +81.50.1790.0200
• Australia: +61.2.4062.7950
• India: 000 800 050 45107
• South Korea: +82.080.467.8774

Palo Alto Networks has shared these findings with our fellow Cyber Threat Alliance (CTA) members. CTA members use this intelligence to rapidly deploy protections to their customers and to systematically disrupt malicious cyber actors. Learn more about the Cyber Threat Alliance.

Indicators of Compromise

• SHA256 hash for .NET-based infostealer, sample 1 of 3: 1b6326857fa635d396851a9031949cfdf6c806130767c399727d78a1c2a0126c
• SHA256 hash for .NET-based infostealer, sample 2 of 3: 02ce798981fb2aa68776e53672a24103579ca77a1d3e7f8aaeccf6166d1a9cc6
• SHA256 hash for .NET-based infostealer, sample 3 of 3: 7c7b7b99f248662a1f9aea1563e60f90d19b0ee95934e476c423d0bf373f6493
• SHA256 hash for malware dropper: 052d5220529b6bd4b01e5e375b5dc3ffd50c4b137e242bbfb26655fd7f475ac6