Migrate Gemma variants [1B, 4B, 27B] to TT-Transformers Library

[MohammedTaherMcW]

Ticket

Link to JIRA Issue

Problem description

Migration of Gemma-3-27b-it from experimental setup to TT-Transformers Library

What's changed

Moved the experimental Gemma-3-27b-it to TT-Transformers
Added sliding_window support in Attention.
Added pre_feedforward_norm and post_feedforward_norm support in decoder

Checklist

• All post commit CI passes
• Blackhole Post commit CI with demo tests passes (if applicable)
• Model regression CI passes (if applicable)
• Device performance regression CI passes (if applicable)
• (For models and ops writers) Single-card demo tests CI passes (if applicable) See recommended dev flow.
• Galaxy quick CI passes (if applicable)
• TG demo tests, for Llama CI passes, if applicable, because of current Llama work
• (For runtime and ops writers) T3000 unit tests CI passes (if applicable, since this is run on push to main)
• (For models and ops writers) T3000 demo tests CI passes (if applicable, since this is required for release)
• New/Existing tests provide coverage for changes

[Copilot]

Pull Request Overview

This PR migrates Gemma-3 multimodal models (1B, 4B, 27B) from the experimental codebase to the unified TT-Transformers library, consolidating functionality and enabling better code reuse.

• Migrates all Gemma-3 multimodal components to tt_transformers structure
• Adds sliding window attention support and additional normalization layers
• Consolidates experimental code into the main transformer library

Reviewed Changes

Copilot reviewed 49 out of 52 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
models/tt_transformers/tt/multimodal/gemma3/*.py	Migrated vision components from experimental to tt_transformers
models/tt_transformers/tt/model_config.py	Added Gemma-3 specific configurations and vision parameters
models/tt_transformers/tt/model.py	Added attention mask support for sliding window attention
models/tt_transformers/tt/decoder.py	Added pre/post feedforward normalization support
models/tt_transformers/tt/attention.py	Enhanced with sliding window attention and mask parameters
models/tt_transformers/tt/generator.py	Extended to support multimodal inputs and Gemma-3 models
models/experimental/gemma3/tt/*.py	Removed experimental files (migrated to tt_transformers)

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

[nitpick] The hardcoded empty string for Gemma-3 models could be replaced with a more explicit constant or method to improve maintainability and make the special case clearer.

[Copilot]

Double .model access looks suspicious and could lead to AttributeError if the structure is not as expected. This should be verified against the actual model structure.

Suggested change

	layer = reference_model.model.model.embed_tokens
	layer = reference_model.model.embed_tokens

[Copilot]

[nitpick] The multiplication followed by division pattern (line 254 and 275) suggests a scaling operation that could be abstracted into a helper function to reduce code duplication and improve clarity.

[Copilot]

[nitpick] The comment should explain why pre_feedforward_layernorm might not be present and the implications of setting it to None for model behavior.

Suggested change

	# If pre_feedforward_layernorm is not in state_dict, we do not use it
	# pre_feedforward_layernorm may be absent from state_dict for certain model variants or checkpoint formats.
	# When it is missing, self.pre_ff_norm is set to None and the model skips this normalization step,
	# which may affect output consistency or compatibility with architectures that expect this layernorm.

[Copilot]

The comment mentions "Reshape output back to original shape" but no reshaping operation follows. This comment should be removed or the intended operation should be implemented.

Suggested change

	if seq_len >= MAX_MM_SEQ_LEN:
	output = ttnn.reshape(output, [1, seq_len, -1])