Add per catalog metastore metrics to QueryStats #26900
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reviewer's GuideThis PR extends the metadata API to collect and expose per-catalog metastore call metrics in QueryStats and QueryInfo. It adds SPI methods for listing active catalogs and fetching connector metrics, wraps Hive metastore calls with timing and failure counting, integrates metrics capture into QueryStateMachine on query completion or failure, and updates connectors and tests to support and validate the new metrics field. Class diagram for MeasuredHiveMetastore and metastore metrics integrationclassDiagram
class MeasuredHiveMetastore {
-HiveMetastore delegate
-MetastoreApiCallStats allApiCallsStats
-Map<String, MetastoreApiCallStats> apiCallStats
-Ticker ticker
+Metrics getMetrics()
+<all HiveMetastore methods> (wrapped)
}
class MetastoreApiCallStats {
-TDigest timeNanosDistribution
-long totalTimeNanos
-long totalFailures
+addTime(long)
+addFailure()
+put(ImmutableMap.Builder<String, Metric<?>>, String)
}
class MeasuredMetastoreFactory {
-HiveMetastoreFactory metastoreFactory
+createMetastore(Optional<ConnectorIdentity>)
+isImpersonationEnabled()
}
MeasuredHiveMetastore --> HiveMetastore : delegates
MeasuredHiveMetastore --> MetastoreApiCallStats : uses
MeasuredMetastoreFactory --> MeasuredHiveMetastore : creates
MeasuredMetastoreFactory --> HiveMetastoreFactory : delegates
class HiveMetastore {
<<interface>>
+getMetrics() : Metrics
+<other methods>
}
MeasuredHiveMetastore ..|> HiveMetastore
class Metrics {
+Map<String, Metric<?>> metrics
}
MeasuredHiveMetastore --> Metrics : returns
MetastoreApiCallStats --> Metric : builds
class Metric {
<<interface>>
}
Class diagram for QueryStats and catalogMetadataMetrics fieldclassDiagram
class QueryStats {
+Map<String, Metrics> catalogMetadataMetrics
+getCatalogMetadataMetrics()
+<other fields and methods>
}
QueryStats --> Metrics : contains
class Metrics {
+Map<String, Metric<?>> metrics
}
Metrics --> Metric : contains
class Metric {
<<interface>>
}
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey there - I've reviewed your changes - here's some feedback:
- Consolidate the duplicated assertCountMetricExists/assertDistributionMetricExists helpers in BaseHiveConnectorTest and BaseIcebergConnectorTest into a shared test utility to reduce code duplication.
- Centralize the collectCatalogMetadataMetrics invocation in QueryStateMachine (rather than calling it separately in both transitionToFinishing and transitionToFailed) to DRY up the code and ensure consistency.
- Consider refactoring the very large MeasuredHiveMetastore class by extracting the stats‐collection logic into smaller, focused components or utility classes to improve readability and maintainability.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- Consolidate the duplicated assertCountMetricExists/assertDistributionMetricExists helpers in BaseHiveConnectorTest and BaseIcebergConnectorTest into a shared test utility to reduce code duplication.
- Centralize the collectCatalogMetadataMetrics invocation in QueryStateMachine (rather than calling it separately in both transitionToFinishing and transitionToFailed) to DRY up the code and ensure consistency.
- Consider refactoring the very large MeasuredHiveMetastore class by extracting the stats‐collection logic into smaller, focused components or utility classes to improve readability and maintainability.
## Individual Comments
### Comment 1
<location> `plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java:9225-9226` </location>
<code_context>
assertQuerySucceeds("CALL system.flush_metadata_cache()");
}
+ @Test
+ public void testCatalogMetadataMetrics()
+ {
+ MaterializedResultWithPlan result = getQueryRunner().executeWithPlan(
</code_context>
<issue_to_address>
**suggestion (testing):** Missing test for metrics with multiple catalogs.
Please add a test that runs a query across multiple catalogs to ensure metrics are tracked separately for each.
Suggested implementation:
```java
@Test
public void testCatalogMetadataMetrics()
{
MaterializedResultWithPlan result = getQueryRunner().executeWithPlan(
getSession(),
"SELECT count(*) FROM region r, nation n WHERE r.regionkey = n.regionkey");
Map<String, Metrics> metrics = getCatalogMetadataMetrics(result.queryId());
assertCountMetricExists(metrics, "iceberg", "metastore.all.time.total");
assertDistributionMetricExists(metrics, "iceberg", "metastore.all.time.distribution");
assertCountMetricExists(metrics, "iceberg", "metastore.getTable.time.total");
assertDistributionMetricExists(metrics, "iceberg", "metastore.getTable.time.distribution");
}
@Test
public void testCatalogMetadataMetricsWithMultipleCatalogs()
{
// Assume "iceberg" and "tpch" catalogs are available for testing
MaterializedResultWithPlan result = getQueryRunner().executeWithPlan(
getSession(),
"SELECT count(*) FROM iceberg.region r JOIN tpch.nation n ON r.regionkey = n.regionkey");
Map<String, Metrics> metrics = getCatalogMetadataMetrics(result.queryId());
// Assert metrics for iceberg catalog
assertCountMetricExists(metrics, "iceberg", "metastore.all.time.total");
assertDistributionMetricExists(metrics, "iceberg", "metastore.all.time.distribution");
assertCountMetricExists(metrics, "iceberg", "metastore.getTable.time.total");
assertDistributionMetricExists(metrics, "iceberg", "metastore.getTable.time.distribution");
// Assert metrics for tpch catalog (replace with actual metric names if different)
assertCountMetricExists(metrics, "tpch", "metastore.all.time.total");
assertDistributionMetricExists(metrics, "tpch", "metastore.all.time.distribution");
assertCountMetricExists(metrics, "tpch", "metastore.getTable.time.total");
assertDistributionMetricExists(metrics, "tpch", "metastore.getTable.time.distribution");
}
```
- If the "tpch" catalog does not support the same metrics, adjust the metric names or assertions accordingly.
- Ensure that the catalogs "iceberg" and "tpch" are available and configured in your test environment.
- If you use different catalogs, update the catalog names in the test.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java
Show resolved
Hide resolved
|
There are related CI failures. Moving to draft until I fix it |
lukasz-stec
force-pushed
the
ls/2510/01-catalog-metastore-metrics
branch
from
57c65e9 to
2b8cd5c
Compare
lukasz-stec
force-pushed
the
ls/2510/01-catalog-metastore-metrics
branch
from
2b8cd5c to
0c78e13
Compare
lukasz-stec
force-pushed
the
ls/2510/01-catalog-metastore-metrics
branch
2 times, most recently
from
a120650 to
75ab2b8
Compare
There was a problem hiding this comment.
Hey there - I've reviewed your changes - here's some feedback:
- The MeasuredHiveMetastore class manually wraps every HiveMetastore method, which leads to a lot of boilerplate; consider using a dynamic proxy or an abstract base wrapper to automatically instrument all methods and reduce duplication.
- The Hive and Iceberg connector tests duplicate the same metric‐assertion logic; extracting the
assertCountMetricExistsandassertDistributionMetricExistshelpers into a shared base test would DRY up the code and centralize metric validation. - Since connectors now rely on the new getMetrics/listActiveCatalogs SPI methods, add a quick check or lint to ensure every connector overrides these (or explicitly opts out) so no catalog is left without metrics by accident.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The MeasuredHiveMetastore class manually wraps every HiveMetastore method, which leads to a lot of boilerplate; consider using a dynamic proxy or an abstract base wrapper to automatically instrument all methods and reduce duplication.
- The Hive and Iceberg connector tests duplicate the same metric‐assertion logic; extracting the `assertCountMetricExists` and `assertDistributionMetricExists` helpers into a shared base test would DRY up the code and centralize metric validation.
- Since connectors now rely on the new getMetrics/listActiveCatalogs SPI methods, add a quick check or lint to ensure every connector overrides these (or explicitly opts out) so no catalog is left without metrics by accident.
## Individual Comments
### Comment 1
<location> `core/trino-main/src/main/java/io/trino/execution/QueryStateMachine.java:392-390` </location>
<code_context>
return queryStateMachine;
}
+ private void collectCatalogMetadataMetrics()
+ {
+ // collect the metrics only once. This avoid issue with transaction being removed
+ // after the check but before the metrics collection
+ if (catalogMetadataMetricsCollected.compareAndSet(false, true)) {
+ if (session.getTransactionId().filter(transactionManager::transactionExists).isEmpty()) {
+ // The metrics collection depends on active transaction as the metrics
+ // are stored in the transactional ConnectorMetadata, but the collection can be
+ // run after the query has failed e.g., via cancel.
+ return;
+ }
+
+ ImmutableMap.Builder<String, Metrics> catalogMetadataMetrics = ImmutableMap.builder();
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Consider handling exceptions during metrics collection to avoid masking query state transitions.
Unexpected exceptions during metrics collection may interrupt query state transitions. Wrapping this logic in a try-catch and logging errors will improve robustness, especially during transitions like finishing or failure.
Suggested implementation:
```java
private void collectCatalogMetadataMetrics()
{
// collect the metrics only once. This avoid issue with transaction being removed
// after the check but before the metrics collection
if (catalogMetadataMetricsCollected.compareAndSet(false, true)) {
if (session.getTransactionId().filter(transactionManager::transactionExists).isEmpty()) {
// The metrics collection depends on active transaction as the metrics
// are stored in the transactional ConnectorMetadata, but the collection can be
// run after the query has failed e.g., via cancel.
return;
}
try {
ImmutableMap.Builder<String, Metrics> catalogMetadataMetrics = ImmutableMap.builder();
for (CatalogInfo activeCatalog : metadata.listActiveCatalogs(session)) {
Metrics metrics = metadata.getMetrics(session, activeCatalog.catalogName());
if (!metrics.getMetrics().isEmpty()) {
catalogMetadataMetrics.put(activeCatalog.catalogName(), metrics);
}
}
this.catalogMetadataMetrics.set(catalogMetadataMetrics.buildOrThrow());
}
catch (Exception e) {
log.error(e, "Error collecting catalog metadata metrics for query %s", queryId);
}
}
}
```
If the logger (`log`) is not already defined in this class, you should add:
```java
private static final Logger log = Logger.get(QueryStateMachine.class);
```
at the top of the class, with the appropriate import:
```java
import io.airlift.log.Logger;
```
</issue_to_address>
### Comment 2
<location> `lib/trino-metastore/src/main/java/io/trino/metastore/MeasuredHiveMetastore.java:46-54` </location>
<code_context>
+ private final HiveMetastore delegate;
+ private final MetastoreApiCallStats allApiCallsStats = new MetastoreApiCallStats();
+ private final Map<String, MetastoreApiCallStats> apiCallStats = new ConcurrentHashMap<>();
+ private final Ticker ticker = Ticker.systemTicker();
+
+ public MeasuredHiveMetastore(HiveMetastore delegate)
</code_context>
<issue_to_address>
**suggestion:** Consider allowing ticker injection for testability.
Injecting the ticker through the constructor would make it easier to test time-dependent logic.
```suggestion
private final HiveMetastore delegate;
private final MetastoreApiCallStats allApiCallsStats = new MetastoreApiCallStats();
private final Map<String, MetastoreApiCallStats> apiCallStats = new ConcurrentHashMap<>();
private final Ticker ticker;
public MeasuredHiveMetastore(HiveMetastore delegate)
{
this(delegate, Ticker.systemTicker());
}
public MeasuredHiveMetastore(HiveMetastore delegate, Ticker ticker)
{
this.delegate = requireNonNull(delegate, "delegate is null");
this.ticker = requireNonNull(ticker, "ticker is null");
}
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
lib/trino-metastore/src/main/java/io/trino/metastore/MeasuredHiveMetastore.java
Show resolved
Hide resolved
|
@findepi @raunaqmorarka This is ready for review. There is one CI failure, but it is unrelated. |
|
test (plugin/trino-lakehouse) this job hanged it contains a couple errors like this and then are they related? |
lukasz-stec
force-pushed
the
ls/2510/01-catalog-metastore-metrics
branch
from
75ab2b8 to
15f1657
Compare
Yes, the impl for the |
findepi
reviewed
Oct 13, 2025
core/trino-main/src/main/java/io/trino/execution/QueryStateMachine.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/QueryStateMachine.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/test/java/io/trino/execution/TestQueryStateMachine.java
Show resolved
Hide resolved
...rino-tests/src/test/java/io/trino/connector/system/metadata/TestSystemMetadataConnector.java
Outdated
Show resolved
Hide resolved
lib/trino-metastore/src/main/java/io/trino/metastore/MeasuredHiveMetastore.java
Outdated
Show resolved
Hide resolved
lib/trino-metastore/src/main/java/io/trino/metastore/MeasuredHiveMetastore.java
Show resolved
Hide resolved
lib/trino-metastore/src/main/java/io/trino/metastore/MeasuredHiveMetastore.java
Show resolved
Hide resolved
lib/trino-metastore/src/main/java/io/trino/metastore/MeasuredHiveMetastore.java
Show resolved
Hide resolved
plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseHiveConnectorTest.java
Outdated
Show resolved
Hide resolved
lukasz-stec
force-pushed
the
ls/2510/01-catalog-metastore-metrics
branch
2 times, most recently
from
c3d6dcb to
8be4339
Compare
lukasz-stec
commented
Oct 13, 2025
core/trino-main/src/main/java/io/trino/execution/QueryStateMachine.java
Outdated
Show resolved
Hide resolved
lukasz-stec
force-pushed
the
ls/2510/01-catalog-metastore-metrics
branch
from
8be4339 to
879926b
Compare
lukasz-stec
commented
Oct 13, 2025
lib/trino-metastore/src/main/java/io/trino/metastore/MeasuredHiveMetastore.java
Show resolved
Hide resolved
lib/trino-metastore/src/main/java/io/trino/metastore/MeasuredHiveMetastore.java
Show resolved
Hide resolved
lib/trino-metastore/src/main/java/io/trino/metastore/MeasuredHiveMetastore.java
Show resolved
Hide resolved
lib/trino-metastore/src/main/java/io/trino/metastore/MeasuredHiveMetastore.java
Show resolved
Hide resolved
plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseHiveConnectorTest.java
Outdated
Show resolved
Hide resolved
lukasz-stec
force-pushed
the
ls/2510/01-catalog-metastore-metrics
branch
3 times, most recently
from
d014746 to
a4aa981
Compare
lukasz-stec
force-pushed
the
ls/2510/01-catalog-metastore-metrics
branch
from
a4aa981 to
a5d9fe1
Compare
|
@findepi @raunaqmorarka There were issues with collecting the metadata metrics concurrently with updating the final query info. The query info could be triggered asynchronously, and thus, missing the collected metrics. To fix this, I added a metadata metrics collection before every query stats collection. This also makes the metadata metrics available before the query is done, which is an additional benefit. |
findepi
approved these changes
Oct 15, 2025
core/trino-main/src/main/java/io/trino/execution/QueryStateMachine.java
Outdated
Show resolved
Hide resolved
lukasz-stec
force-pushed
the
ls/2510/01-catalog-metastore-metrics
branch
2 times, most recently
from
c0fddfe to
10d41ae
Compare
| transitionToFailed(e); | ||
| return true; | ||
| } | ||
| collectCatalogMetadataMetrics(); |
There was a problem hiding this comment.
We make metadata calls during FINISHING state as well, so we should collect metrics at the end of that
There was a problem hiding this comment.
We remove the transaction right after this call, so there is no way to collect the metrics anymore.
| Optional<Object> getInfo(Session session, TableHandle handle); | ||
|
|
||
| /** | ||
| * Return connector-specific, metadata operations metrics for the given session. |
There was a problem hiding this comment.
Well, the metrics are connector-specific, but we return them for the provided catalog.
| @Override | ||
| public Metrics getMetrics(ConnectorSession session) | ||
| { | ||
| Span span = startSpan("getMetrics"); |
There was a problem hiding this comment.
Do we really want to have Span for get metrics collection call ?
If we're calling this every time QueryStats is constructed, the traces are going to show this Span a lot
There was a problem hiding this comment.
There should be only a few of those calls made + since this is connector specific, we don't really know what is there. I think it is ok to keep it consistent with the rest of the class.
There was a problem hiding this comment.
btw I tested this and those spans are not part of the query trace but only as a part of a standalone Metadata.getMetrics trace. Seems like a bug.
There was a problem hiding this comment.
I would still suggest skipping tracing for this API. It should always only collect in-memory objects and not make remote calls. There's no real value in tracing this. You can add a code comment explaining why it is not consistent with the rest of the code. We know that the biggest problem when working with traces today is the sheer amount of things in it.
lib/trino-metastore/src/main/java/io/trino/metastore/MeasuredHiveMetastore.java
Outdated
Show resolved
Hide resolved
| { | ||
| // cross TX metastore cache is enabled wrapper with caching metastore | ||
| return sharedHiveMetastoreCache.createCachingHiveMetastoreFactory(metastoreFactory); | ||
| return sharedHiveMetastoreCache.createCachingHiveMetastoreFactory(new MeasuredMetastoreFactory(metastoreFactory)); |
There was a problem hiding this comment.
Will cached metastore calls show up in the metrics ?
Also, please check that the metrics tracking is working for Glue metastore as well, the way that is setup is a but different.
There was a problem hiding this comment.
No, cached calls are not measured.
I will check the glue
There was a problem hiding this comment.
It works for
glue_20251016_184141_00003_txnrc.json
glue
"activeCatalogs": [
{
"name": "hive",
"version": "95607a88f6655db4de4dbfdc0c255acf246148f0550ddf891c903b8aaea7f7b6",
"connectorName": "hive",
"properties": {
"hive.metastore": "glue",
"hive.security": "allow-all",
"hive.metastore.glue.default-warehouse-dir": "local:///glue",
"hive.max-partitions-per-scan": "1000",
"fs.hadoop.enabled": "true",
"hive.max-partitions-for-eager-load": "1000"
}
}
],
"catalogMetadataMetrics": {
"hive": {
"metastore.getPartitionNamesByFilter.time.distribution": {
"@class": "io.trino.plugin.base.metrics.DistributionSnapshot",
"total": 1,
"min": 1600526792,
"max": 1600526792,
"p01": 1600526792,
"p05": 1600526792,
"p10": 1600526792,
"p25": 1600526792,
"p50": 1600526792,
"p75": 1600526792,
"p90": 1600526792,
"p95": 1600526792,
"p99": 1600526792
},
"metastore.getTable.time.total": {
"@class": "io.trino.plugin.base.metrics.LongCount",
"total": 210967292
},
"metastore.all.time.distribution": {
"@class": "io.trino.plugin.base.metrics.DistributionSnapshot",
"total": 15,
"min": 169969959,
"max": 1600526792,
"p01": 169969959,
"p05": 169969959,
"p10": 176701709,
"p25": 194937458,
"p50": 206969875,
"p75": 214762042,
"p90": 362961542,
"p95": 1600526792,
"p99": 1600526792
},
"metastore.all.time.total": {
"@class": "io.trino.plugin.base.metrics.LongCount",
"total": 4566443044
},
"metastore.getTable.time.distribution": {
"@class": "io.trino.plugin.base.metrics.DistributionSnapshot",
"total": 1,
"min": 210967292,
"max": 210967292,
"p01": 210967292,
"p05": 210967292,
"p10": 210967292,
"p25": 210967292,
"p50": 210967292,
"p75": 210967292,
"p90": 210967292,
"p95": 210967292,
"p99": 210967292
},
"metastore.getPartitionNamesByFilter.time.total": {
"@class": "io.trino.plugin.base.metrics.LongCount",
"total": 1600526792
},
"metastore.getPartitionsByNames.time.total": {
"@class": "io.trino.plugin.base.metrics.LongCount",
"total": 2754948960
},
"metastore.getPartitionsByNames.time.distribution": {
"@class": "io.trino.plugin.base.metrics.DistributionSnapshot",
"total": 13,
"min": 169969959,
"max": 362961542,
"p01": 169969959,
"p05": 169969959,
"p10": 176701709,
"p25": 194937458,
"p50": 203406792,
"p75": 210784958,
"p90": 219027333,
"p95": 362961542,
"p99": 362961542
}
}
},
plugin/trino-lakehouse/src/main/java/io/trino/plugin/lakehouse/LakehouseMetadata.java
Outdated
Show resolved
Hide resolved
| @Override | ||
| public Metrics getMetrics() | ||
| { | ||
| return metastore.getMetrics(); |
There was a problem hiding this comment.
I think we would at least want tracking in REST and Glue catalogs as well
There was a problem hiding this comment.
The goal of this PR is to add Hive metastore metrics, as this is often the source of perf problems.
Support for REST and glue catalogs can be added as a follow-up.
| @Override | ||
| public io.trino.spi.metrics.Metrics getMetrics(ConnectorSession session) | ||
| { | ||
| return catalog.getMetrics(); |
There was a problem hiding this comment.
This approach is only catching the metadata calls made to the metastore, which is relatively few operations in iceberg. The bulk of metadata fetching in iceberg is happening through the use of iceberg APIs in this class. You'll have to register InMemoryMetricsReporter using org.apache.iceberg.Scan#metricsReporter and collect those metrics like in IcebergSplitSource.
That will still not catch the metadata calls in commit operations, but I don't see a way in iceberg APIs to register metrics reporter for commit operations. Maybe something to improve in upstream iceberg.
fyi @ebyhr @findepi
There was a problem hiding this comment.
Yes, for now, I'm only adding support to metastore metrics. This could be later extended.
lukasz-stec
force-pushed
the
ls/2510/01-catalog-metastore-metrics
branch
2 times, most recently
from
823da64 to
107045c
Compare
lukasz-stec
commented
Oct 16, 2025
| @Override | ||
| public Metrics getMetrics() | ||
| { | ||
| return metastore.getMetrics(); |
There was a problem hiding this comment.
The goal of this PR is to add Hive metastore metrics, as this is often the source of perf problems.
Support for REST and glue catalogs can be added as a follow-up.
...tests/src/test/java/io/trino/connector/informationschema/TestInformationSchemaConnector.java
Show resolved
Hide resolved
| @Override | ||
| public Metrics getMetrics(ConnectorSession session) | ||
| { | ||
| Span span = startSpan("getMetrics"); |
lukasz-stec
force-pushed
the
ls/2510/01-catalog-metastore-metrics
branch
from
107045c to
dc0bbec
Compare
lukasz-stec
force-pushed
the
ls/2510/01-catalog-metastore-metrics
branch
from
dc0bbec to
b1b0f26
Compare
|
rebased on master to resolve conflicts |
lukasz-stec
force-pushed
the
ls/2510/01-catalog-metastore-metrics
branch
from
b1b0f26 to
b1bac33
Compare
Before Benchmark (valuesPerHistogram) Mode Cnt Score Error Units BenchmarkHistogram.benchmarkTDigestSnapshot 10 avgt 10 152.097 ± 22.783 ns/op BenchmarkHistogram.benchmarkTDigestSnapshot 100 avgt 10 350.355 ± 13.313 ns/op BenchmarkHistogram.benchmarkTDigestSnapshot 1000 avgt 10 413.406 ± 6.479 ns/op After Benchmark (valuesPerHistogram) Mode Cnt Score Error Units BenchmarkHistogram.benchmarkTDigestSnapshot 10 avgt 10 52.963 ± 23.871 ns/op BenchmarkHistogram.benchmarkTDigestSnapshot 100 avgt 10 106.029 ± 0.304 ns/op BenchmarkHistogram.benchmarkTDigestSnapshot 1000 avgt 10 109.722 ± 0.327 ns/op
Metadata and QueryStateMachine must use the same `TransactionManager` instance.
The goal is to expose in QueryStats, per catalog, connector-specific metrics like metastore api call stats.
lukasz-stec
force-pushed
the
ls/2510/01-catalog-metastore-metrics
branch
from
b1bac33 to
bdb5bb0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Slow metastore can be a root cause of slow analysis or planning. This adds explicit metrics to the
QueryStatswith remote metastore call stats made for a given query.This is what this looks like in the query.json:
Additional context and related issues
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( X) Release notes are required, with the following suggested text:
Summary by Sourcery
Add per-catalog metastore metrics to QueryStats by extending the Metadata API and collecting metrics from each active catalog on query completion or failure, instrument connectors with MeasuredHiveMetastore, and verify behavior with new connector tests.
New Features:
Enhancements:
Build:
Tests:
Summary by Sourcery
Add per-catalog metastore metrics tracking to QueryStats by instrumenting connectors and extending Metadata APIs to gather and report connector-specific metadata call metrics on query finish or failure.
New Features:
Enhancements:
Build:
Tests: