Typegen loads nuget packages with the incorrect framework version when resolving assemblies

[alundporchsoftware]

My team found some inconsistent behavior with how nuget dependencies are loaded when resolving assemblies in typegen. Unfortunately it took us three days and much confusion to figure out the problem, but I'm glad to have identified the issue in spite of that. We're still pretty confused about what changed on our side to start this, because we've building without , but nonetheless I'd like request a bugfix for this issue, since the issue is pretty directly caused by typegen, and I'd like to spare other people the trouble of tracking this down.

The error we encountered is the following:

Unhandled exception: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
File name: 'System.Runtime, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at System.ModuleHandle.ResolveType(QCallModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
   at System.ModuleHandle.ResolveTypeHandle(Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
   at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
   at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(MetadataToken caCtorToken, MetadataImport& scope, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1& derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctorWithParameters, Boolean& isVarArg)
   at System.Reflection.CustomAttribute.AddCustomAttributes(ListBuilder`1& attributes, RuntimeModule decoratedModule, Int32 decoratedMetadataToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1 derivedAttributes)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeType type, RuntimeType caType, Boolean inherit)
   at System.Attribute.GetCustomAttributes(MemberInfo element, Type attributeType, Boolean inherit)
   at System.Attribute.GetCustomAttribute(MemberInfo element, Type attributeType, Boolean inherit)
   at System.Reflection.CustomAttributeExtensions.GetCustomAttribute[T](MemberInfo element)
   at TypeGen.Core.Metadata.AttributeMetadataReader.GetAttribute[TAttribute](Type type) in C:\projects\typegen\src\TypeGen\TypeGen.Core\Metadata\AttributeMetadataReader.cs:line 13
   at TypeGen.Core.Extensions.TypeExtensions.HasExportAttribute(Type type, IMetadataReader reader) in C:\projects\typegen\src\TypeGen\TypeGen.Core\Extensions\TypeExtensions.cs:line 27
   at TypeGen.Core.Extensions.TypeExtensions.<>c__DisplayClass1_0.<GetExportMarkedTypes>b__0(Type t) in C:\projects\typegen\src\TypeGen\TypeGen.Core\Extensions\TypeExtensions.cs:line 43
   at System.Linq.Enumerable.WhereArrayIterator`1.MoveNext()
   at TypeGen.Core.Generator.Services.GenerationSpecProvider.GetGenerationSpec(IEnumerable`1 assemblies) in C:\projects\typegen\src\TypeGen\TypeGen.Core\Generator\Services\GenerationSpecProvider.cs:line 28
   at TypeGen.Core.Generator.Generator.Generate(IEnumerable`1 assemblies) in C:\projects\typegen\src\TypeGen\TypeGen.Core\Generator\Generator.cs:line 220

We've found other issues in the repo where this was reported for Typegen and we made certain that we had the right framework and Typegen versions present. For reference, we are targeting net8.0 for our framework and are using Typegen 5.0.1. The thing that possibly makes our situation unique is that we are executing our builds in dotnet-docker images when we're generating TS files with typegen.

Here is what we've determined is going on (and have more or less confirmed by experimentation):

1. Some nuget packages are downloaded with multiple frameworks, e.g., net8.0, net9.0, and netstandard2.0 will all be downloaded for System.Text.Json 9.0.4. This is normal behavior for nuget.
2. We build our code and explicitly target net8.0. Assemblies are produced that only link to net8.0 or netstandard2.0 packages as needed.
3. We run typegen. Typegen locates the nuget package paths for our assemblies and their dependencies.
4. Typegen attempts to locate each referenced assembly by file name in the directories included in _nugetPackagesFolders.
5. Typegen finds multiple instances of some assemblies under multiple paths, because assemblies with the same filename exist under net8.0, net9.0, and netstandard2.0.
6. Typegen selects the each found version of the assembly, loads that assembly (src/TypeGen/TypeGen.Cli/TypeResolution/AssemblyResolver.cs:150), and if the loaded assembly version matches the expected version, then it selects that assembly for future type resolution
7. Depending on the order of files returned from Directory.GetFiles (src/TypeGen/TypeGen.Core/Storage/FileSystem.cs:48)`, the net9.0 assembly might be loaded, or the net8.0. If you look the dotnet documentation, you will see that the order of files returned by GetFiles is not guranteed to be consistency.
8. Later, when resolving whether or not a System.Runtime type has any Typegen attributes, it chokes because the net9.0 runtime isn't loaded or available locally to get the TypeInfo (because the dotnet-docker container doesn't have the 9.0 runtime or sdk, only net8.0).

Basically, the problem is that the selected framework version of the nuget packages loaded by typegen is stochastic. The key is that Assembly.GetName().Version returns the same result for all of the net8.0, net9.0, and netstandard2.0 versions of the nuget package, so even though there are checks being made to ensure the correct nuget package version is being loaded, by itself there isn't any way to discriminate whether the assembly with the correct framework version was loaded in AssemblyResolver.ResolveFromPaths. This may explain some instances of encountering Could not resolve assembly: System.Runtime, Version=9.0.0.0 that have been reported in other issues.

In our experiments trying to resolve the issue, what we did that eventually resolved the problem for us, was running our build, deleting all net9.0 directories in the /lib directories in our nuget packages, and then running typegen, which immediately fixed the issue for us. I would provide a solution that reproduces this, except that the conditions required to reproduce this are unpredictable anyway, because of Directory.GetFiles being the way it is.

I think having to delete elements from my nuget package is the jankiest of solutions, because I can't predict which version if the library I might need while linking in the event I legitimately need a netstandard2.0 version of the library, and I can't predict what other framework versions may be present in a downloaded nuget package. It also means I have to delete the files after every build, because the packages will be constantly restored. I also am not eager to install all the required runtimes in our docker image, because the point of building in a targeted docker image is to keep things lean and avoid weird side effects from having multiple runtimes installed. It would be much better to load the assembly with the right framework.

In searching around it looks like there are two possible ways to check for matching frameworks based on assembly info

1. Detect if the TargetFrameworkAttribute is present in the assembly attributes. This attribute seems to be applied automatically in the modern dotnet ecosystem so it might be reliable.
2. Check the Referenced assemblies to see which version of System.Runtime is referenced by the assembly.

On one hand both of these options seem like they would make good heuristic for ensuring framework version compatiiblity, because I can always match framework versions. On the other hand, I would predict immediately running into problems depending on which version of System.Runtime is required, and if in the dependency chain, the framework version legitimately switches from net9.0 to netstandard2.0 or similar.

The only other alternatives would be to implement a mechanism that forces typegen to respect framework versions by some sort of inclusion/exclusion rules in the TypeGen config, or detecting what frameworks are available locally and only loading assemblies that match available frameworks. This seems more complicated then inferring framework version from loaded assemblies, but might be more consistent in the wider nuget ecosystem.

Regardless of whether you can provide a solution given the above, I appreciate the work that's been put into this library, since it saves my team a ton of trouble. It's a good library.