The Video feature is available when the model supports video understanding capabilities.
Friendli supports video input functionality that enables models to understand and analyze video content, extending multimodal capabilities beyond text and images. This feature allows you to process video files and receive intelligent responses about their content.

​
Use Cases

  • Video content description and summarization
  • Character and object identification
  • Scene analysis and understanding
  • Comparative analysis across multiple videos

​
Code examples

Prerequisites:
  • A Friendli endpoint with video understanding capabilities
  • Valid Friendli Personal Access Token
Video Requirements:
  • Videos must be hosted at publicly accessible URLs
  • HTTPS URLs are recommended for security
  • Consider video file size and processing time implications
  • Some models may have specific resolution or duration requirements
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.friendli.ai/dedicated/v1",
    api_key=os.environ.get("FRIENDLI_TOKEN"),
)

video_url = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4"

completion = client.chat.completions.create(
    # Replace YOUR_ENDPOINT_ID with the ID of your endpoint, e.g. "zbimjgovmlcb"
    model="YOUR_ENDPOINT_ID",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What's in this video?",
                },
                {
                    "type": "video_url",
                    "video_url": {"url": video_url},
                },
            ],
        },
    ],
    temperature=0,
    max_tokens=100,
)

print(completion.choices[0].message.content)

​
What just happened?

When leveraging the video input feature: Single Video Analysis:
  • The model will analyze the provided video content
  • Responses typically include descriptions of visual elements, actions, characters, or scenes
  • Processing time depends on video length and model capabilities
Multi-Video Analysis:
  • The model can compare and contrast content across multiple videos
  • Useful for identifying similarities, differences, or relationships between video content
  • The order of videos in the request may influence the analysis structure

​
Best Practices

  • Processing Speed: Shorter videos generally process faster than longer ones
  • Compression Trade-offs: Consider video compression and quality trade-offs
  • URL Accessibility: Ensure video URLs are publicly accessible and don’t require authentication
  • Video Quality: Higher quality videos generally provide better analysis results
  • Prompt Clarity: Provide clear, specific questions about what you want to analyze in the video
  • Timeout Configuration: Contact support to adjust video fetch timeout for large files or slow connections
  • Model Selection: Choose models with proven video understanding capabilities for your specific use case
By default, video fetching through HTTP URLs is limited to 30 seconds. If you require support for longer timeout values, please contact us.