refactor(go-segmenter): replace custom GoSegmenter with Tree-Sitter implementation

[vbelouso]

Summary

This PR replaces the legacy regex-based Go segmenter with a native Tree-Sitter parser (GoSegmenterExtended), enabling syntax-aware extraction of Go functions, methods, anonymous functions, and types with deterministic and reproducible results.

Rationale

• The previous segmenter was purely heuristic and text-based, which led to:
• Incomplete parsing of modern Go syntax (e.g. generics, inline funcs, nested declarations).
• Non-deterministic document order and inconsistent call-chain results.
• Fragile regex logic that required frequent maintenance as Go evolved.

Implementation highlights

• Introduced GoSegmenterExtended, powered by the official Tree-Sitter Go grammar.
• Integrated it into ExtendedLanguageParser as the default segmenter for .go files.
• Added deterministic sorting for reproducible dependency-chain resolution.
• Reworked tests to cover generics, pointer receivers, multiline methods, and anonymous functions.

Architectural impact

The segmentation layer now uses a structured syntax tree (Tree-Sitter) instead of regex parsing.

Downstream modules such as ChainOfCallsRetriever and function analyzers still operate on text chunks, but now those chunks are syntactically well-formed and consistent across runs.

This refactor lays the foundation for future AST-based semantic analysis (e.g. variable/type inference, symbol resolution).

Benchmark

Tested on https://github.com/openshift/origin with 35001 Go files

Metric	Legacy (Regex)	New (Tree-Sitter)	Δ
Total runtime	145.11 s	93.59 s	−35 %
Extracted chunks	411 732	593 793	+44 % coverage
Function calls (profiling)	15.2 M	9.8 M	−36 % overhead

[zvigrinberg]

Hi @vbelouso, Can you please rebase and resolve conflicts before i'm starting reviewing it?
Thanks.

[vbelouso]

Hi @vbelouso, Can you please rebase and resolve conflicts before i'm starting reviewing it? Thanks.

Done

[zvigrinberg]

@vbelouso This is an example of something that is not working correctly ( the example test is failing) , get_function_name should return the variable name containing the anonymous function.

@pytest.mark.asyncio
async def test_transitive_search_golang_generic():
    parser = GoLanguageFunctionsParser()
    doc1 = Document(page_content=("greet := func() { // Assigning anonymous function to a variable 'greet'\n"
                                  "		fmt.Println(\"Greetings from a variable-assigned anonymous function!\")\n"
                                  "	}"))
    name = parser.get_function_name(doc1)
    print(f"name_of_function={name}")
    assert name == "greet"

Your revised GoSegmenter with TreeSitter parse the anonymous function assigned to a variable correctly,
But instead of taking the name of the variable in this case, it return :=, which is incorrect , please check.

[zvigrinberg]

[vbelouso]

@zvigrinberg
Updated.
I also increased the number of test cases.