Serving the model
Ollama: We've released a Q4_K_M GGUF quantization of Instinct for efficient local inference. Try it with Continue's Ollama integration, or just run ollama run nate/instinct.
You can also serve the model using either of the below options, then connect it with Continue.
SGLang: python3 -m sglang.launch_server --model-path continuedev/instinct --load-format safetensors
vLLM: vllm serve continuedev/instinct --served-model-name instinct --load-format safetensors
Learn more
For more information on the work behind Instinct, please refer to our blog.