loader image

Reply To: Set up a Local AI like ChatGPT on your own machine!

What makes us different from other similar websites? › Forums › Tech › Set up a Local AI like ChatGPT on your own machine! › Reply To: Set up a Local AI like ChatGPT on your own machine!

June 17, 2025 at 7:45 pm #8078
thumbtak
Moderator

How to Get Shell GPT (sgpt) Working with Local Ollama Models on Kali Linux

Hey everyone!

I wanted to share a comprehensive guide on how I got sgpt (Shell GPT) to work with local Ollama models on my Kali Linux setup. This is super useful for privacy, control, and avoiding those recurring cloud API costs. It was a bit of a journey, but here are all the steps that finally got it working for me!

Goal

To use sgpt from the command line to interact with Large Language Models (LLMs) hosted locally via Ollama.

I. Initial Setup: Kali Linux, Ollama, and shell-gpt Installation

  1. Ensure Kali Linux is installed and updated:
    • This is your base operating system.
    • (Assumed you have this ready!)
  2. Install Ollama: This command downloads and installs the Ollama server, which is essential for running LLMs locally.
    curl -fsSL https://ollama.com/install.sh | sh
  3. Download your desired LLM(s) using Ollama: This pulls the model files onto your system. I used mistral as an example, but you can choose others from Ollama’s library.
    ollama pull mistral # Or 'ollama pull llama3', 'ollama pull codellama', etc.

    • Verify Ollama is running:
      systemctl status ollama
      (Expected output will show Active: active (running))
  4. Install shell-gpt with LiteLLM support: LiteLLM is crucial because it enables sgpt to communicate with various LLM providers, including Ollama’s local API.
    sudo apt update
    sudo apt install pipx
    pipx ensurepath
    pipx install "shell-gpt[litellm]"

    • Initial sgpt run (and dummy API key entry): When you first run sgpt after installation, it might prompt for an OpenAI API key. I entered a dummy string (e.g., dummykey123, potato) and pressed Enter to bypass this. This allows sgpt to create its initial configuration file.

II. Critical Configuration Adjustments for Ollama Integration

This was the trickiest part! It involves precisely editing sgpt‘s configuration file (~/.config/shell_gpt/.sgptrc) to direct it to Ollama.

  1. Open the shell-gpt configuration file using sudo nano: Using sudo nano is vital to ensure you have the necessary permissions to save changes, especially if the file was previously owned by root.
    sudo nano ~/.config/shell_gpt/.sgptrc
    (Enter your password when prompted.)
  2. Make these precise edits in the nano editor:
    • DEFAULT_MODEL: Set this to your local Ollama model.
      DEFAULT_MODEL=ollama/mistral # Make sure 'mistral' matches your downloaded model
    • OPENAI_USE_FUNCTIONS: Disable OpenAI-specific function calls as Ollama doesn’t use them in the same way.
      OPENAI_USE_FUNCTIONS=false
    • USE_LITELLM: Enable LiteLLM as the backend. This is essential for Ollama compatibility.
      USE_LITELLM=true
    • LITELLM_API_BASE: Explicitly tell LiteLLM where your local Ollama server is running (default is http://localhost:11434).
      LITELLM_API_BASE=http://localhost:11434
    • OPENAI_API_KEY: Keep your dummy key here. sgpt still expects a value in this field, but LiteLLM will ignore it when routing to Ollama.
      OPENAI_API_KEY=dummykey123 # Or whatever dummy key you used
    • CRITICAL: REMOVE / DELETE THIS LINE: This was the most persistent and problematic line that kept causing the OpenAI AuthenticationError. Ensure it is COMPLETELY GONE from the file. Do not just comment it out for now; delete it entirely if it’s present.
      # API_BASE_URL=default
      (If you found this line, delete it. If it was never there, great!)
  3. Save the file in nano:
    • Press Ctrl+O (Write Out)
    • Press Enter to confirm the filename
    • Press Ctrl+X (Exit)
  4. Verify the changes (Crucial for troubleshooting): Immediately after saving, I used cat to confirm the file content from the terminal. This helped ensure the edits actually stuck.
    cat ~/.config/shell_gpt/.sgptrc

III. Resolving the PermissionError

The final hurdle was a PermissionError when sgpt tried to run. This happened because sudo nano had set root as the owner of the config file, preventing my regular user from writing to it.

  1. Change the ownership of the config file and its directory back to your user (USER in my case):
    sudo chown USER:USER /home/USER/.config/shell_gpt/.sgptrc
    sudo chown USER:USER /home/USER/.config/shell_gpt/
    (Replace USER with your actual Kali username if it’s different).
  2. Verify ownership (optional):
    ls -l /home/USER/.config/shell_gpt/.sgptrc
    ls -ld /home/USER/.config/shell_gpt/
    (Expected output for owner and group: yourusername yourusername)

IV. Final Test

  1. Run sgpt with a query:
    sgpt "Tell me a joke about cybersecurity."
    This should now work, with sgpt communicating successfully with your local Ollama model!

Hopefully, this detailed guide helps others facing similar issues! It’s rewarding to get a powerful local AI assistant working right in your Kali terminal.

Current time: 2025-06-17 08:46:18 PM EDT

  • This reply was modified 1 month ago by thumbtak.
TAKs Shack