Feature/Implement Semantic Search for BigQuery resources

[srinivasan-acn]

Upgraded SearchController to support bigquery_semantic_search with pagination, project-based scoping, and location filtering.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the BigQuery search functionality within the Jupyter plugin by introducing a new semantic search API. The changes enable users to perform more precise and scalable searches for BigQuery assets, incorporating features like project-specific scoping, location filtering, and efficient data retrieval through pagination. This upgrade improves the user experience by providing more control and better performance for discovering relevant BigQuery resources.

Highlights

• Semantic Search API Integration: The SearchController now utilizes a new bigquery_semantic_search method, enabling more advanced and context-aware searches for BigQuery assets through the Dataplex API.
• Enhanced Filtering Capabilities: Implemented support for project-based scoping, allowing users to restrict searches to specific projects, and location-based filtering to narrow down results by geographical region.
• Pagination Support: Added pagination parameters (page_size and page_token) to the search API, enabling efficient retrieval of large result sets by fetching data in manageable chunks.
• Flexible Project Specification: The search now allows projects to be explicitly provided as an argument, or defaults to listing all available BigQuery projects if not specified.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• dataproc_jupyter_plugin/controllers/bigquery.py
  • Modified the post method to accept new query parameters: scope, projects, locations, page_size, and page_token.
  • Renamed the type argument to search_type for improved clarity.
  • Replaced the call to bigquery_search with the new bigquery_semantic_search method, passing all relevant new parameters.
  • Implemented logic to parse projects argument (if provided) or fetch all BigQuery projects.
• dataproc_jupyter_plugin/services/bigquery.py
  • Introduced a new asynchronous method bigquery_semantic_search to handle advanced search requests.
  • Constructed dynamic Dataplex API queries based on search_string, system, type, projects, and locations parameters.
  • Implemented logic for pagination using pageSize and pageToken in the Dataplex API request.
  • Added support for project-based scope in the Dataplex API payload.
  • Included robust error handling for Dataplex API calls and network issues.
  • Parsed and returned search results, nextPageToken, and totalSize from the Dataplex API response.

Activity

• No human activity has been recorded on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request implements a new semantic search API endpoint by adding bigquery_semantic_search to the bigquery service and updating the SearchController. The changes introduce support for pagination, project-based scoping, and location filtering.

My review focuses on the new bigquery_semantic_search method in dataproc_jupyter_plugin/services/bigquery.py. I've identified several areas for improvement, including fixing a potential bug in the search query syntax, removing redundant and unreachable code, and renaming a parameter to avoid shadowing a Python built-in. These changes will improve the code's correctness, readability, and maintainability.