bulkquery requests don't deduplicate packages

[G-Rath]

osv-scanner/cmd/osv-scanner/scan/source/testdata/cassettes/TestCommand.yaml

Lines 62 to 122 in 30ab0fe

	body: |
	{
	"queries": [
	{
	"package": {
	"ecosystem": "RubyGems",
	"name": "ast"
	},
	"version": "2.4.2"
	},
	{
	"package": {
	"ecosystem": "Packagist",
	"name": "sentry/sdk"
	},
	"version": "2.0.4"
	},
	{
	"package": {
	"ecosystem": "RubyGems",
	"name": "ast"
	},
	"version": "2.4.2"
	},
	{
	"package": {
	"ecosystem": "npm",
	"name": "balanced-match"
	},
	"version": "1.0.2"
	},
	{
	"package": {
	"ecosystem": "RubyGems",
	"name": "ast"
	},
	"version": "2.4.2"
	},
	{
	"package": {
	"ecosystem": "Packagist",
	"name": "sentry/sdk"
	},
	"version": "2.0.4"
	},
	{
	"package": {
	"ecosystem": "npm",
	"name": "balanced-match"
	},
	"version": "1.0.2"
	},
	{
	"package": {
	"ecosystem": "npm",
	"name": "balanced-match"
	},
	"version": "1.0.2"
	}
	]
	}

This cassette recording shows we're requesting vuln info for the same packages multiple times in the same request, which is silly.

This is probably related to (iirc) we use the index to map results from the bulk query so if we were to just filter out duplicates before doing the querying we'd end up with a bunch of "off by 1" index type bugs and whatnot

[G-Rath]

Just making an issue now so I don't forget, but I've not actually looked into the code - I suspect this might actually be something for osv.dev as it could be actionable within the osv client