​
What are Endpoints?

Endpoints are the actual deployments of your models on a dedicated GPU resource. They provide a stable and efficient interface to serve your models in real-world applications, ensuring high availability and optimized performance. With endpoints, you can manage model versions, scale resources, and seamlessly integrate your model into production environments.

​
Key Capabilities of Endpoints:

  • Efficient Model Serving: Deploy models on powerful GPU instances optimized for your use case.
  • Flexibility with Multi-LoRA Models: Serve multiple fine-tuned adapters alongside base models.
  • Autoscaling: Automatically adjust resources to handle varying workloads, ensuring optimal performance and cost efficiency.
  • Monitoring and Management: Check endpoint health, adjust configurations, and view logs directly from the platform.
  • Interactive Testing: Use the integrated playground to test your models before integrating them into applications.
  • API Integration: Access your models via robust OpenAI-compatible APIs, enabling easy integration into any system.

​
Creating Endpoints

You can create your endpoint by specifying the name, the model, and the instance configuration, consisting of your desired GPU specification.
Endpoint Create

​
Selecting Instance

Instance selection depends on your model size and workload.
endpoint_instance_warnings
If the selected GPU type or count is insufficient for the model size, the instance may not be selectable. If the configuration is expected to prevent you from fully leveraging the capabilities of the Friendli Inference Engine, you’ll see a TIGHT MEMORY warning. In such cases, we recommend enabling Online Quantization or increasing the GPU count.

​
Available Features

Endpoint features

​
Online Quantization

Skip the hassle of preparing a quantized model. By enabling Online Quantization, your model will be automatically quantized at runtime using Friendli’s proprietary method—preserving quality while improving speed and cost-efficiency.
This allows you to select lower-VRAM GPU instances without performance loss.
Some models (e.g., those already quantized) may not be compatible with Online Quantization.
In certain cases, specific GPU instance types may not be available when this option is enabled.

​
Speculative decoding

​
N-gram speculative decoding

You may toggle the switch to activate N-gram speculative decoding. When enabled, past tokens are leveraged to pre-generate future tokens. For predictable tasks, this can deliver substantial performance gains. You can also set the Maximum N-gram Size, which defines how many tokens are predicted in advance. We recommend keeping the default value of 3.
Higher values can further reduce latency when successful. However, predicting too many tokens at once may lower prediction efficiency and, in extreme cases, even increase latency.

​
Advanced speculative decoding (Coming soon)

We’ll soon be releasing an advanced speculative decoding feature that requires training but delivers better performance in most cases compared to the N-gram method. If you’re interested, please Contact us for more details.

​
Intelligent Autoscaling

Autoscaling Config
Our autoscaling system automatically adjusts computational resources based on your traffic patterns, helping you optimize both performance and costs.

​
How Autoscaling Works

  • Minimum Replicas:
    • When set to 0, the endpoint enters sleeping status during periods of inactivity, helping to minimize costs
    • When set to a value greater than 0, the endpoint maintains at least that number of active replicas at all times
  • Maximum Replicas: Defines the upper limit of replicas that can be created to handle increased traffic load
  • Cooldown Period: Measured in seconds; if no requests are received during this period, the endpoint transitions to sleeping status.

​
Autoscaling types

We highly recommend using the Default autoscaling type, as it performs stable for most workloads. Please note that performance degradation or unexpected charges may occur with other configurations without a proper understanding of your workload characteristics.
We provide 2 types of autoscaling, but only the Default option is available for non-Enterprise plans.
  • Default (Recommended): This is the best choice for the majority of users. It operates reliably across most workloads with no configuration required, leveraging our internal expertise to provide a balanced approach to performance and cost.
  • Request count (Enterprise plan only): This is an advanced option for users who have a deep understanding of their workload characteristics and require granular control over scaling behavior.
    • As users define the number of requests a single worker will handle, cost prediction becomes more straightforward and intuitive.
    • This method can serve as a foundation for implementing your own custom autoscaling logic by dynamically changing the threshold via an API, targeting custom metrics.

​
Benefits of Autoscaling

  • Cost Optimization: Pay only for the resources you need by automatically scaling to zero during idle periods
  • Performance Management: Handle traffic spikes efficiently by automatically adding replicas
  • Resource Efficiency: Maintain optimal resource utilization across varying workload patterns

​
Serving Multi-LoRA Models

You can serve Multi-LoRA models using Friendli Dedicated Endpoints. For an overview of Multi-LoRA models, refer to our document on serving Multi-LoRA models with Friendli Container. In Friendli Dedicated Endpoints, Multi-LoRA model is supported only in Enterprise plan. For pricing and availability, Contact sales.

​
Checking Endpoint Status

After creating the Endpoint, you can view its health status and Endpoint URL on the Endpoint’s details page.
Endpoint Detail
The cost of using dedicated endpoints accumulates from the INITIALIZING status. Specifically, charges begin after the Initializing GPU phase, where the endpoint waits to acquire the GPU. The endpoint then downloads and loads the model onto the GPU, which usually takes less than a minute.

​
Model max context length and KV cache size

Model max context length is determined by how the model was trained, while KV cache size depends on memory and is affected by your instance type and Online Quantization setting. In some workloads, having a KV cache size smaller than the model max context length may still work fine.
However, to fully leverage the performance of the Friendli Inference Engine, we recommend first enabling Online Quantization (as it doesn’t require changing your instance), and then, if needed, selecting a GPU with more VRAM or increasing the number of GPUs.

​
Using Playgrounds

To test the deployed model via the web, we provide a playground interface where you can interact with the model using a user-friendly chat interface. Simply enter your query, adjust your settings, and generate your responses!
Endpoint Playground
Send inference queries to your model through our API at the given endpoint address, accessible on the endpoint information tab.