GGUF Models

GGUF models are a popular open format for running large language models efficiently on a wide range of hardware, including Mac, Windows, and Linux. Enjoy fast, local inference with broad model compatibility!

⚙️ Prerequisites

• If you haven’t already, install the nexa-SDK.
• Below are the GGUF-compatible model types you can experiment with right away.

LLM - Language Models

📝 Language models in GGUF format. Try out this quick example:

Try it out:

bash

nexa infer NexaAI/Qwen3-0.6B

⌨️ This will spawn an interactive REPL conversation session with the model.

---

LMM - Multimodal Models

🖼️ Multimodal models that accept vision and/or audio inputs. Try out this quick example:

bash

nexa infer NexaAI/Qwen2.5-Omni-3B-GGUF

⌨️ Drag images and audio files to the conversation input to chat with images / audio.

Supported Model List

We curated a list of top, high quality models in GGUF format.

LLMs for GGUF

View Huggingface Collection 🤗

Multimodal for GGUF

View Huggingface Collection 🤗

To try other GGUF models, visit Hugging Face, copy the path of any compatible GGUF model (e.g., unsloth/Qwen2.5-VL-3B-Instruct-GGUF), and replace the model path in the command above.

For more advanced models, you may visit the Nexa Model Hub. Also, access token is required to download and use these models. To get access token:

• Create an account at sdk.nexa.ai
• Generate a token: Go to Deployment → Create Token
• Activate your SDK: Run the following command on the terminal to set your license:

bash

nexa config set license '<your_token_here>'

🙋 Request New Models

Missing a model? Vote for it on the Nexa Wishlist — we build the most-voted models fast!

You can also submit an issue on the nexa-sdk GitHub or request in our Discord/Slack community.

Was this page helpful?

Yes

No