Skip to content

feat: initial paq writeup #1385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

feat: initial paq writeup #1385

wants to merge 16 commits into from
+85 −30

Conversation

@GowriH-1
Copy link

@GowriH-1 GowriH-1 commented Nov 12, 2025 •
edited
Loading

Overview

Pairwise Annotation Queues

Type of change

Type:
Updating existing annotation queues with PAQ information

Related issues/PRs

  • GitHub issue:
  • Feature PR:
  • Linear issue:
  • Slack thread:

Checklist

  • I have read the contributing guidelines
  • I have tested my changes locally using docs dev
  • All code examples have been tested and work correctly
  • I have used root relative paths for internal links
  • I have updated navigation in src/docs.json if needed
  • I have gotten approval from the relevant reviewers
  • (Internal team members only / optional) I have created a preview deployment using the Create Preview Branch workflow

Additional notes

@GowriH-1 GowriH-1 requested a review from katmayb as a code owner November 12, 2025 17:21
@github-actions github-actions bot added the langsmith For docs changes to LangSmith label Nov 12, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 12, 2025

Mintlify preview ID generated: preview-gowrip-1762968133-6c34622

katmayb
katmayb reviewed Nov 12, 2025
View reviewed changes
Copy link
Contributor

@katmayb katmayb left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @GowriH-1, added some suggested changes. Let me know when you have added the screenshots in to the current plaeholders.

Globally on the page, can you reinstate/use the markdown ordered numbering to "1." throughout the list because it'll render automatically.

GowriH-1 and others added 7 commits November 12, 2025 11:02
@GowriH-1 @katmayb
Update src/langsmith/annotation-queues.mdx
dab885e 
Co-authored-by: Kathryn May <44557882+katmayb@users.noreply.github.com>
@GowriH-1 @katmayb
Update src/langsmith/annotation-queues.mdx
2f9257e 
Co-authored-by: Kathryn May <44557882+katmayb@users.noreply.github.com>
@GowriH-1 @katmayb
Update src/langsmith/annotation-queues.mdx
b78b754 
Co-authored-by: Kathryn May <44557882+katmayb@users.noreply.github.com>
@GowriH-1 @katmayb
Update src/langsmith/annotation-queues.mdx
3abd801 
Co-authored-by: Kathryn May <44557882+katmayb@users.noreply.github.com>
@GowriH-1 @katmayb
Update src/langsmith/annotation-queues.mdx
85df7d5 
Co-authored-by: Kathryn May <44557882+katmayb@users.noreply.github.com>
@GowriH-1 @katmayb
Update src/langsmith/annotation-queues.mdx
40f515b 
Co-authored-by: Kathryn May <44557882+katmayb@users.noreply.github.com>
@GowriH-1 @katmayb
Update src/langsmith/annotation-queues.mdx
eb5fa4b 
Co-authored-by: Kathryn May <44557882+katmayb@users.noreply.github.com>
@github-actions
Copy link
Contributor

github-actions bot commented Nov 12, 2025

Mintlify preview ID generated: preview-gowrip-1762974340-72173c5

@github-actions
Copy link
Contributor

github-actions bot commented Nov 12, 2025

Mintlify preview ID generated: preview-gowrip-1762991403-07c0af6

@github-actions
Copy link
Contributor

github-actions bot commented Nov 13, 2025

Mintlify preview ID generated: preview-gowrip-1762992248-dcf6848

katmayb
katmayb approved these changes Nov 13, 2025
View reviewed changes
Copy link
Contributor

@katmayb katmayb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of small nits, and double check the images as they're not rendering. (I think it might be the file name.)

Otherwise, awesome, LGTM!

@GowriH-1 @katmayb
Update src/langsmith/annotation-queues.mdx
8154424 
Co-authored-by: Kathryn May <44557882+katmayb@users.noreply.github.com>
@github-actions
Copy link
Contributor

github-actions bot commented Nov 13, 2025

Mintlify preview ID generated: preview-gowrip-1763055037-4db26d9

@github-actions
Copy link
Contributor

github-actions bot commented Nov 13, 2025

Mintlify preview ID generated: preview-gowrip-1763057757-680a4ea

@github-actions
Copy link
Contributor

github-actions bot commented Nov 13, 2025

Mintlify preview ID generated: preview-gowrip-1763062096-d455489

katmayb
katmayb approved these changes Nov 13, 2025
View reviewed changes
Copy link
Contributor

@katmayb katmayb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@GowriH-1 Take a look at the preview here: https://langchain-5e9cc07a-preview-gowrip-1763062096-d455489.mintlify.app/langsmith/annotation-queues#create-a-pairwise-annotation-queue

The screenshots are stacked and hard to differentiate. I've made two suggestions to delete two of the images anyway because I don't think they're 100% necessary.

Otherwise this LGTM!

src/langsmith/annotation-queues.mdx Outdated
Comment on lines 79 to 81
![Popover showing the “Add to Pairwise Annotation Queue” card highlighted after two experiments are selected.](/langsmith/images/pairwise-annotation-queue-popup.png)

![Pairwise creation pane.](/langsmith/images/pairwise-annotation-queue-create-form.png)
Copy link
Contributor

@katmayb katmayb Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These images are stacked oddly in the preview. I would recommend deleting /langsmith/images/pairwise-annotation-queue-create-form.png I don't really think it's necessary, you've got great instructions there already (it's not adding much).

Keeping the queue-popup image makes sense, this is a bit of a different view compared to a regular annotation queue.

src/langsmith/annotation-queues.mdx Outdated
Comment on lines 153 to 155
![Pairwise review screen showing runs side-by-side with the floating feedback bar containing A/B/Equal buttons and keyboard shortcuts.](/langsmith/images/pairwise-annotation-queue-review-screen.png)

![Pairwise review screen showing runs side-by-side with the feedback pane containing A/B/Equal buttons and keyboard shortcuts.](/langsmith/images/pairwise-annotation-queue-review-feedback-pane.png)
Copy link
Contributor

@katmayb katmayb Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are also stacked and hard to decipher in the rendered view.

I think you could drop one of these — the feedback-pane screenshot seems more useful.

The ALT text is also identical?

@github-actions
Copy link
Contributor

github-actions bot commented Nov 13, 2025

Mintlify preview ID generated: preview-gowrip-1763063196-3f3a4c3

@github-actions
Copy link
Contributor

github-actions bot commented Nov 13, 2025

Mintlify preview ID generated: preview-gowrip-1763075169-d7084f7

@github-actions
Copy link
Contributor

github-actions bot commented Nov 17, 2025

Mintlify preview ID generated: preview-gowrip-1763404313-26b8cbb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ayoung19 ayoung19 ayoung19 approved these changes

@katmayb katmayb katmayb approved these changes

Assignees

No one assigned

Labels

langsmith For docs changes to LangSmith

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@GowriH-1 @ayoung19 @katmayb