"
+ ],
+ "text/html": [
+ "\n",
+ " \n",
+ " \n",
+ "
\n",
+ " [30/30 03:57, Epoch 0/1]\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | Step | \n",
+ " Training Loss | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | 1 | \n",
+ " 7.838200 | \n",
+ "
\n",
+ " \n",
+ " | 2 | \n",
+ " 7.321400 | \n",
+ "
\n",
+ " \n",
+ " | 3 | \n",
+ " 5.612000 | \n",
+ "
\n",
+ " \n",
+ " | 4 | \n",
+ " 6.834700 | \n",
+ "
\n",
+ " \n",
+ " | 5 | \n",
+ " 6.055700 | \n",
+ "
\n",
+ " \n",
+ " | 6 | \n",
+ " 5.708900 | \n",
+ "
\n",
+ " \n",
+ " | 7 | \n",
+ " 5.883100 | \n",
+ "
\n",
+ " \n",
+ " | 8 | \n",
+ " 6.726200 | \n",
+ "
\n",
+ " \n",
+ " | 9 | \n",
+ " 6.340900 | \n",
+ "
\n",
+ " \n",
+ " | 10 | \n",
+ " 6.254000 | \n",
+ "
\n",
+ " \n",
+ " | 11 | \n",
+ " 8.979900 | \n",
+ "
\n",
+ " \n",
+ " | 12 | \n",
+ " 6.578000 | \n",
+ "
\n",
+ " \n",
+ " | 13 | \n",
+ " 8.251400 | \n",
+ "
\n",
+ " \n",
+ " | 14 | \n",
+ " 9.181700 | \n",
+ "
\n",
+ " \n",
+ " | 15 | \n",
+ " 6.717700 | \n",
+ "
\n",
+ " \n",
+ " | 16 | \n",
+ " 8.310600 | \n",
+ "
\n",
+ " \n",
+ " | 17 | \n",
+ " 7.068600 | \n",
+ "
\n",
+ " \n",
+ " | 18 | \n",
+ " 7.544400 | \n",
+ "
\n",
+ " \n",
+ " | 19 | \n",
+ " 4.389900 | \n",
+ "
\n",
+ " \n",
+ " | 20 | \n",
+ " 3.215800 | \n",
+ "
\n",
+ " \n",
+ " | 21 | \n",
+ " 3.547600 | \n",
+ "
\n",
+ " \n",
+ " | 22 | \n",
+ " 4.041100 | \n",
+ "
\n",
+ " \n",
+ " | 23 | \n",
+ " 3.491700 | \n",
+ "
\n",
+ " \n",
+ " | 24 | \n",
+ " 3.100900 | \n",
+ "
\n",
+ " \n",
+ " | 25 | \n",
+ " 3.017400 | \n",
+ "
\n",
+ " \n",
+ " | 26 | \n",
+ " 2.235200 | \n",
+ "
\n",
+ " \n",
+ " | 27 | \n",
+ " 2.572800 | \n",
+ "
\n",
+ " \n",
+ " | 28 | \n",
+ " 2.524600 | \n",
+ "
\n",
+ " \n",
+ " | 29 | \n",
+ " 2.193600 | \n",
+ "
\n",
+ " \n",
+ " | 30 | \n",
+ " 2.651600 | \n",
+ "
\n",
+ " \n",
+ "
"
+ ]
+ },
+ "metadata": {}
+ }
+ ],
+ "source": [
+ "trainer_stats = trainer.train()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "pCqnaKmlO1U9",
+ "outputId": "bf1534bd-8217-47ac-aefe-f655c1d0941b"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "338.048 seconds used for training.\n",
+ "5.63 minutes used for training.\n",
+ "Peak reserved memory = 12.592 GB.\n",
+ "Peak reserved memory for training = 0.0 GB.\n",
+ "Peak reserved memory % of max memory = 85.422 %.\n",
+ "Peak reserved memory for training % of max memory = 0.0 %.\n"
+ ]
+ }
+ ],
+ "source": [
+ "# @title Show final memory and time stats\n",
+ "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
+ "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",
+ "used_percentage = round(used_memory / max_memory * 100, 3)\n",
+ "lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n",
+ "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n",
+ "print(\n",
+ " f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\"\n",
+ ")\n",
+ "print(f\"Peak reserved memory = {used_memory} GB.\")\n",
+ "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n",
+ "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n",
+ "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ekOmTR1hSNcr"
+ },
+ "source": [
+ "\n",
+ "### Inference\n",
+ "Let's run the model via Unsloth native inference! According to the `Gemma-3` team, the recommended settings for inference are `temperature = 1.0, top_p = 0.95, top_k = 64`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "kR3gIAX-SM2q",
+ "outputId": "098ca4bf-5590-4e46-a098-c1f04bbd7b1c"
+ },
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "['user\\nContinue the sequence: 1, 1, 2, 3, 5, 8,\\nmodel\\nThe sequence is the Fibonacci sequence. Each number is the sum of the two preceding numbers.\\n\\n1, 1, 2, 3, 5, 8, **13, 21, 34...**\\n\\nSo the next number is 13.\\n']"
+ ]
+ },
+ "metadata": {},
+ "execution_count": 25
+ }
+ ],
+ "source": [
+ "from unsloth.chat_templates import get_chat_template\n",
+ "tokenizer = get_chat_template(\n",
+ " tokenizer,\n",
+ " chat_template = \"gemma-3\",\n",
+ ")\n",
+ "messages = [{\n",
+ " \"role\": \"user\",\n",
+ " \"content\": [{\n",
+ " \"type\" : \"text\",\n",
+ " \"text\" : \"Continue the sequence: 1, 1, 2, 3, 5, 8,\",\n",
+ " }]\n",
+ "}]\n",
+ "inputs = tokenizer.apply_chat_template(\n",
+ " messages,\n",
+ " add_generation_prompt = True, # Must add for generation\n",
+ " return_tensors = \"pt\",\n",
+ " tokenize = True,\n",
+ " return_dict = True,\n",
+ ").to(\"cuda\")\n",
+ "outputs = model.generate(\n",
+ " **inputs,\n",
+ " max_new_tokens = 64, # Increase for longer outputs!\n",
+ " # Recommended Gemma-3 settings!\n",
+ " temperature = 1.0, top_p = 0.95, top_k = 64,\n",
+ ")\n",
+ "tokenizer.batch_decode(outputs)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "CrSvZObor0lY"
+ },
+ "source": [
+ " You can also use a `TextStreamer` for continuous inference - so you can see the generation token by token, instead of waiting the whole time!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "e2pEuRb1r2Vg",
+ "outputId": "1a8234b0-ac1e-4d75-bf57-65da5a4f7f67"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "The sky is blue because of a phenomenon called **Rayleigh scattering**. Here's a breakdown of what happens:\n",
+ "\n",
+ "* **Sunlight is made of all colors:** White sunlight is actually composed of all the colors of the rainbow (red, orange, yellow, green, blue, indigo, violet).\n",
+ "* **\n"
+ ]
+ }
+ ],
+ "source": [
+ "messages = [{\n",
+ " \"role\": \"user\",\n",
+ " \"content\": [{\"type\" : \"text\", \"text\" : \"Why is the sky blue?\",}]\n",
+ "}]\n",
+ "inputs = tokenizer.apply_chat_template(\n",
+ " messages,\n",
+ " add_generation_prompt = True, # Must add for generation\n",
+ " return_tensors = \"pt\",\n",
+ " tokenize = True,\n",
+ " return_dict = True,\n",
+ ").to(\"cuda\")\n",
+ "\n",
+ "from transformers import TextStreamer\n",
+ "_ = model.generate(\n",
+ " **inputs,\n",
+ " max_new_tokens = 64, # Increase for longer outputs!\n",
+ " # Recommended Gemma-3 settings!\n",
+ " temperature = 1.0, top_p = 0.95, top_k = 64,\n",
+ " streamer = TextStreamer(tokenizer, skip_prompt = True),\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "uMuVrWbjAzhc"
+ },
+ "source": [
+ "\n",
+ "### Saving, loading finetuned models\n",
+ "To save the final model as LoRA adapters, either use Huggingface's `push_to_hub` for an online save or `save_pretrained` for a local save.\n",
+ "\n",
+ "**[NOTE]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "upcOlWe7A1vc",
+ "outputId": "2c2aa03d-6116-4bb5-fdfa-0527718f8ffd"
+ },
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "['gemma-3n/processor_config.json']"
+ ]
+ },
+ "metadata": {},
+ "execution_count": 27
+ }
+ ],
+ "source": [
+ "model.save_pretrained(\"gemma-3n\") # Local saving\n",
+ "tokenizer.save_pretrained(\"gemma-3n\")\n",
+ "# model.push_to_hub(\"HF_ACCOUNT/gemma-3\", token = \"...\") # Online saving\n",
+ "# tokenizer.push_to_hub(\"HF_ACCOUNT/gemma-3\", token = \"...\") # Online saving"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "AEEcJ4qfC7Lp"
+ },
+ "source": [
+ "Now if you want to load the LoRA adapters we just saved for inference, set `False` to `True`:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "MKX_XKs_BNZR",
+ "outputId": "3c3453e6-24ce-4080-a9de-e4936872eb20"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Gemma-3N is a family of open-weights large language models (LLMs) created by the Gemma team at Google DeepMind. Here's a breakdown of what that means:\n",
+ "\n",
+ "* **Open-weights:** This is a key aspect. \"Open-weights\" means the model's parameters (the learned values within the model) are publicly available. This allows researchers, developers, and enthusiasts to download, use, and fine-tune the model for their own purposes, fostering innovation and transparency in the field of AI.\n",
+ "* **Large Language Model (LLM):** Gemma-3N is a\n"
+ ]
+ }
+ ],
+ "source": [
+ "if False:\n",
+ " from unsloth import FastModel\n",
+ " model, tokenizer = FastModel.from_pretrained(\n",
+ " model_name = \"lora_model\", # YOUR MODEL YOU USED FOR TRAINING\n",
+ " max_seq_length = 2048,\n",
+ " load_in_4bit = True,\n",
+ " )\n",
+ "\n",
+ "messages = [{\n",
+ " \"role\": \"user\",\n",
+ " \"content\": [{\"type\" : \"text\", \"text\" : \"What is Gemma-3N?\",}]\n",
+ "}]\n",
+ "inputs = tokenizer.apply_chat_template(\n",
+ " messages,\n",
+ " add_generation_prompt = True, # Must add for generation\n",
+ " return_tensors = \"pt\",\n",
+ " tokenize = True,\n",
+ " return_dict = True,\n",
+ ").to(\"cuda\")\n",
+ "\n",
+ "from transformers import TextStreamer\n",
+ "_ = model.generate(\n",
+ " **inputs,\n",
+ " max_new_tokens = 128, # Increase for longer outputs!\n",
+ " # Recommended Gemma-3 settings!\n",
+ " temperature = 1.0, top_p = 0.95, top_k = 64,\n",
+ " streamer = TextStreamer(tokenizer, skip_prompt = True),\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "f422JgM9sdVT"
+ },
+ "source": [
+ "### Saving to float16 for VLLM\n",
+ "\n",
+ "We also support saving to `float16` directly for deployment! We save it in the folder `gemma-3N-finetune`. Set `if False` to `if True` to let it run!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "iHjt_SMYsd3P"
+ },
+ "outputs": [],
+ "source": [
+ "if False: # Change to True to save finetune!\n",
+ " model.save_pretrained_merged(\"gemma-3N-finetune\", tokenizer)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "z6O48DbNIAr0"
+ },
+ "source": [
+ "If you want to upload / push to your Hugging Face account, set `if False` to `if True` and add your Hugging Face token and upload location!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ZV-CiKPrIFG0"
+ },
+ "outputs": [],
+ "source": [
+ "if False: # Change to True to upload finetune\n",
+ " model.push_to_hub_merged(\n",
+ " \"HF_ACCOUNT/gemma-3N-finetune\", tokenizer,\n",
+ " token = \"hf_...\"\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "TCv4vXHd61i7"
+ },
+ "source": [
+ "### GGUF / llama.cpp Conversion\n",
+ "To save to `GGUF` / `llama.cpp`, we support it natively now for all models! For now, you can convert easily to `Q8_0, F16 or BF16` precision. `Q4_K_M` for 4bit will come later!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "FqfebeAdT073"
+ },
+ "outputs": [],
+ "source": [
+ "if False: # Change to True to save to GGUF\n",
+ " model.save_pretrained_gguf(\n",
+ " \"gemma-3N-finetune\",\n",
+ " quantization_type = \"Q8_0\", # For now only Q8_0, BF16, F16 supported\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Q974YEVPI7JS"
+ },
+ "source": [
+ "Likewise, if you want to instead push to GGUF to your Hugging Face account, set `if False` to `if True` and add your Hugging Face token and upload location!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ZgcJIhJ0I_es"
+ },
+ "outputs": [],
+ "source": [
+ "if False: # Change to True to upload GGUF\n",
+ " model.push_to_hub_gguf(\n",
+ " \"gemma-3N-finetune\",\n",
+ " quantization_type = \"Q8_0\", # Only Q8_0, BF16, F16 supported\n",
+ " repo_id = \"HF_ACCOUNT/gemma-3N-finetune-gguf\",\n",
+ " token = \"hf_...\",\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "pnz9QOYTMvbH"
+ },
+ "source": [
+ "Now, use the `gemma-3N-finetune.gguf` file or `gemma-3N-finetune-Q4_K_M.gguf` file in llama.cpp or a UI based system like Jan or Open WebUI. You can install Jan [here](https://github.com/janhq/jan) and Open WebUI [here](https://github.com/open-webui/open-webui)\n",
+ "\n",
+ "And we're done! If you have any questions on Unsloth, we have a [Discord](https://discord.gg/unsloth) channel! If you find any bugs or want to keep updated with the latest LLM stuff, or need help, join projects etc, feel free to join our Discord!\n",
+ "\n",
+ "Some other links:\n",
+ "1. Train your own reasoning model - Llama GRPO notebook [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb)\n",
+ "2. Saving finetunes to Ollama. [Free notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb)\n",
+ "3. Llama 3.2 Vision finetuning - Radiography use case. [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb)\n",
+ "6. See notebooks for DPO, ORPO, Continued pretraining, conversational finetuning and more on our [documentation](https://docs.unsloth.ai/get-started/unsloth-notebooks)!\n",
+ "\n",
+ "\n",
+ "
![](\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\")
\n",
+ "
![](\"https://github.com/unslothai/unsloth/raw/main/images/Discord.png\")
\n",
+ "
![](\"https://github.com/unslothai/unsloth/blob/main/images/documentation%20green%20button.png?raw=true\")
\n",
+ "\n",
+ " Join Discord if you need help + ⭐️
Star us on Github ⭐️\n",
+ "
![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}