Skip to content
This repository was archived by the owner on Jan 23, 2026. It is now read-only.

Feature Evals β­πŸ“ŠπŸ§ͺπŸ’―πŸ’° #446

Closed
potofpie wants to merge 10 commits intomainfrom
feature-eval
Closed

Feature Evals β­πŸ“ŠπŸ§ͺπŸ’―πŸ’° #446
potofpie wants to merge 10 commits intomainfrom
feature-eval

Conversation

@potofpie
Copy link
Copy Markdown
Member

@potofpie potofpie commented Sep 15, 2025

No description provided.

* added patch

* added evals function

* added eval cmd
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Sep 15, 2025

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches
πŸ§ͺ Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feature-eval

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Member

@jhaynie jhaynie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can talk next week and i can help here but just some cursory review

return sb.String()
}

// escapeString escapes a string for use in TypeScript string literals
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have Interpolate in go-common string we should probably use.

cmd.Stdout = ctx.Writer
cmd.Stderr = ctx.Writer
if err := cmd.Run(); err != nil {
if ctx.DevMode {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

}

func copyPromptsFromSrc(logger logger.Logger, projectDir, outdir string) error {
srcRoot := filepath.Join(projectDir, "src")
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't use source directly like this. must go through the ProjectContext to get the correct source folder otherwise this will break for python

// generatePromptTypeDeclarations generates only TypeScript declarations for prompt methods
func generatePromptTypeDeclarations(logger logger.Logger, projectDir string) error {
// Find all prompts.yaml files in the source directory
srcRoot := filepath.Join(projectDir, "src")
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use src like this. use project context

methodSignatures.WriteString(fmt.Sprintf(" * @param variables - Template variables to substitute\n"))
methodSignatures.WriteString(fmt.Sprintf(" * @returns Object with system and prompt strings, both with attached metadata properties\n"))
methodSignatures.WriteString(fmt.Sprintf(" */\n"))
methodSignatures.WriteString(fmt.Sprintf(" %s(variables?: Record<string, unknown>): Promise<{\n", methodName))
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can probably just use a otel span here with the right prompt id

@potofpie potofpie closed this Oct 3, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants