As part of an eventual move to nuget packages, it would be better to split the code into distinct modules that are not all required. For example, OpenGL support should be a separate package so that we don't include OpenGL.NET etc for projects that don't make use of it. Similarly, DirectX and SDL support should both be modules, and potentially Win32.
This may require a bit of refactoring, particularly to handle SDL being an optional component in scenes, but it should also make the project much cleaner.