ASP.NET CORE API running on ServiceFabric does not have service annotations when remoting #117
Description
We are running a couple of Service Fabric clusters on Azure VMSS, have integrated Application Insights instrumentation for performance data and have run into a technical issue with our integration. The APIs for our applications are implemented as ASP.NET CORE APIs served by Kestrel. They then make backend calls to the other microservices on the Service Fabric application using SF service remoting.
We integrated both the Application Insights .Net SDK and the Application Insights ServiceFabric SDK and they initialise correctly. However, we are ending up with a problem. While calls to the API are annotated correctly as coming from the ServiceFabric application on the correct nodes and everything, the Service Fabric remoting calls that occur on the API are detected as entirely separate calls and are not annotated correctly, so they appear to be coming from the default source named after the AppInsights instance.
Moreover, remoting calls seem to be correctly annotated with the target service but they do not resolve to the actual service on the Application Map, so they are not tracked past the actual remoting call itself. For example, if there is a call to to an API that makes a remoting call to service A, which calls service B, which calls service A, it looks like:
ApplicationInsightsInstance (missing the service and application annotation) -> A Remoting
A -> B Remoting
B -> A Remoting
A
Initialisation is done as follows:
In the service constructors:
m_TelemetryConfiguration.TelemetryInitializers.Add(FabricTelemetryInitializerExtension.CreateFabricTelemetryInitializer(context));
m_TelemetryConfiguration.TelemetryInitializers.Add(new OperationCorrelationTelemetryInitializer());
m_TelemetryConfiguration.TelemetryInitializers.Add(new HttpDependenciesParsingTelemetryInitializer());
m_TelemetryConfiguration.TelemetryInitializers.Add(new CodePackageVersionTelemetryInitializer());
var dependencyTrackingModule = new DependencyTrackingTelemetryModule();
var requestTrackingModule = new ServiceRemotingRequestTrackingTelemetryModule();
var remotingDependencyTrackingModule = new ServiceRemotingDependencyTrackingTelemetryModule();
dependencyTrackingModule.Initialize(m_TelemetryConfiguration);
requestTrackingModule.Initialize(m_TelemetryConfiguration);
remotingDependencyTrackingModule.Initialize(m_TelemetryConfiguration);
In the ASP.NET Core services:
In the constructor:
m_TelemetryConfiguration = TelemetryConfiguration.CreateDefault();
m_TelemetryConfiguration.InstrumentationKey = m_AppInsightsKey;
Further initialisation happens in ConfigureServices:
services.AddSingleton<ITelemetryInitializer>(provider => FabricTelemetryInitializerExtension.CreateFabricTelemetryInitializer(serviceContext));
services.AddSingleton<ITelemetryInitializer>(provider => new CodePackageVersionTelemetryInitializer());
services.AddSingleton<ITelemetryModule>(serviceProvider =>
{
var requestTrackingModule = new ServiceRemotingRequestTrackingTelemetryModule();
requestTrackingModule.Initialize(m_TelemetryConfiguration);
return requestTrackingModule;
});
services.AddSingleton<ITelemetryModule>(serviceProvider =>
{
var dependencyTrackingModule = new ServiceRemotingDependencyTrackingTelemetryModule();
dependencyTrackingModule.Initialize(m_TelemetryConfiguration);
return dependencyTrackingModule;
});
services.AddApplicationInsightsTelemetry(options =>
{
options.InstrumentationKey = m_TelemetryConfiguration.InstrumentationKey;
options.EnableActiveTelemetryConfigurationSetup = true;
});
These evolved multiple times before they ended up as they are, as I have been chasing this issue for a while and I have left out some details that I feel are not relevant (such as configuration actually being done by methods that are also called by the configuration package changed event etc.). I was directed to submit an issue here by the AppInsights support team.
I can point at the relevant resource and provide any more information as needed.