SauravP97/tooling-gemma-270M-inst API & Inference Endpoint

Introduction

Tooling Gemma is a fine-tuned version of Gemma 3 270M base model tuned for function calling. Our model will take APIs / Functions documentation (structure) as an input from the user along with user's query and will perform Function Call with right arguments wherever needed.

Note: The Functions / APIs structure is generally provided as a part of the System Instructions.

An Example User-Model communication will look like this:

markdown
SYSTEM: You are a helpful assistant with access to the following functions. Use them if required -
{
  "name": "calculate_factorial",
  "description": "Calculate the factorial of a given number",
  "parameters": {
    "type": "object",
    "properties": {
      "number": {
        "type": "integer",
        "description": "The number to calculate factorial for"
      }
    },
    "required": [
      "number"
    ],
  }
}


USER: Hi, I need to calculate the factorial of 5. Can you help me with that?

ASSISTANT: <functioncall> {"name": "calculate_factorial", "arguments": '{"number": 5}'} <|endoftext|>

FUNCTION RESPONSE: {"result": 120}

ASSISTANT: The factorial of 5 is 120. <|endoftext|>

USER: That was quick! Can you also calculate the factorial of 7?

ASSISTANT: <functioncall> {"name": "calculate_factorial", "arguments": '{"number": 7}'} <|endoftext|>

Using the model

python
class ToolingGemma:
    def __init__(self, system_instructions):
        self.chat_history = ''
        self.model = AutoModelForCausalLM.from_pretrained('SauravP97/tooling-gemma-270M-inst', device_map="cpu")
        self.tokenizer = AutoTokenizer.from_pretrained('google/gemma-3-270m')
        self.system_instructions = system_instructions
        self.stop_word = "<|endoftext|>"

    def generate(self, user_query):
        user_query = 'USER: ' + user_query
        if self.chat_history:
            prompt = self.chat_history + '\n' + user_query + '\n' + 'ASSISTANT:'
        else:
            prompt = self.system_instructions + '\n\n' + user_query + '\n' + 'ASSISTANT:'

        input_ids = self.tokenizer(prompt, return_tensors="pt")

        agent_response = self.model.generate(
            **input_ids, 
            generation_config=GenerationConfig.from_dict({"max_new_tokens": 1000}),
            stop_strings=[self.stop_word],
            tokenizer=self.tokenizer,
        )

        decoded_agent_response = self.tokenizer.decode(agent_response[0])
        self.chat_history = decoded_agent_response

        return decoded_agent_response

system_instructions = '''
SYSTEM: You are a helpful assistant with access to the following functions. Use them if required -
{
    "name": "calculate_discount",
    "description": "Calculate the discounted price of a product",
    "parameters": {
        "type": "object",
        "properties": {
            "original_price": {
                "type": "number",
                "description": "The original price of the product"
            },
            "discount_percentage": {
                "type": "number",
                "description": "The discount percentage"
            }
        },
        "required": [
            "original_price",
            "discount_percentage"
        ]
    }
}
'''

tooling_gemma_model = ToolingGemma(system_instructions=system_instructions)

agent_response = tooling_gemma_model.generate('Can you please book a flight for me from New York to London?')

Model Output:

markdown
ASSISTANT: I'm sorry, but I'm unable to assist with booking flights. My current capabilities are limited to calculating discounted prices based on original price and discount percentage. If you need help with that, feel free to ask! <|endoftext|>

Continue talking to the model:

python
agent_response = tooling_gemma_model.generate('Calculate the discounted price for 100 dollars at a discount of 30%')

Model Output:

markdown
ASSISTANT: <functioncall> {"name": "calculate_discount", "arguments": '{"original_price": 100, "discount_percentage": 30}'} <|endoftext|>

Introduction

Note: The Functions / APIs structure is generally provided as a part of the System Instructions.

An Example User-Model communication will look like this:

markdown

SYSTEM: You are a helpful assistant with access to the following functions. Use them if required -
{
  "name": "calculate_factorial",
  "description": "Calculate the factorial of a given number",
  "parameters": {
    "type": "object",
    "properties": {
      "number": {
        "type": "integer",
        "description": "The number to calculate factorial for"
      }
    },
    "required": [
      "number"
    ],
  }
}


USER: Hi, I need to calculate the factorial of 5. Can you help me with that?

ASSISTANT: <functioncall> {"name": "calculate_factorial", "arguments": '{"number": 5}'} <|endoftext|>

FUNCTION RESPONSE: {"result": 120}

ASSISTANT: The factorial of 5 is 120. <|endoftext|>

USER: That was quick! Can you also calculate the factorial of 7?

ASSISTANT: <functioncall> {"name": "calculate_factorial", "arguments": '{"number": 7}'} <|endoftext|>

Using the model

python

class ToolingGemma:
    def __init__(self, system_instructions):
        self.chat_history = ''
        self.model = AutoModelForCausalLM.from_pretrained('SauravP97/tooling-gemma-270M-inst', device_map="cpu")
        self.tokenizer = AutoTokenizer.from_pretrained('google/gemma-3-270m')
        self.system_instructions = system_instructions
        self.stop_word = "<|endoftext|>"

    def generate(self, user_query):
        user_query = 'USER: ' + user_query
        if self.chat_history:
            prompt = self.chat_history + '\n' + user_query + '\n' + 'ASSISTANT:'
        else:
            prompt = self.system_instructions + '\n\n' + user_query + '\n' + 'ASSISTANT:'

        input_ids = self.tokenizer(prompt, return_tensors="pt")

        agent_response = self.model.generate(
            **input_ids, 
            generation_config=GenerationConfig.from_dict({"max_new_tokens": 1000}),
            stop_strings=[self.stop_word],
            tokenizer=self.tokenizer,
        )

        decoded_agent_response = self.tokenizer.decode(agent_response[0])
        self.chat_history = decoded_agent_response

        return decoded_agent_response

system_instructions = '''
SYSTEM: You are a helpful assistant with access to the following functions. Use them if required -
{
    "name": "calculate_discount",
    "description": "Calculate the discounted price of a product",
    "parameters": {
        "type": "object",
        "properties": {
            "original_price": {
                "type": "number",
                "description": "The original price of the product"
            },
            "discount_percentage": {
                "type": "number",
                "description": "The discount percentage"
            }
        },
        "required": [
            "original_price",
            "discount_percentage"
        ]
    }
}
'''

tooling_gemma_model = ToolingGemma(system_instructions=system_instructions)

agent_response = tooling_gemma_model.generate('Can you please book a flight for me from New York to London?')

Model Output:

markdown

ASSISTANT: I'm sorry, but I'm unable to assist with booking flights. My current capabilities are limited to calculating discounted prices based on original price and discount percentage. If you need help with that, feel free to ask! <|endoftext|>

Continue talking to the model:

python

agent_response = tooling_gemma_model.generate('Calculate the discounted price for 100 dollars at a discount of 30%')

Model Output:

markdown

ASSISTANT: <functioncall> {"name": "calculate_discount", "arguments": '{"original_price": 100, "discount_percentage": 30}'} <|endoftext|>

tooling-gemma-270M-inst

Get help setting up a custom Dedicated Endpoints.

README

Introduction

Explore FriendliAI today

README

Introduction