How to manage third-party asemblies in a (nuget-)

or: How to automate nuspec/nupkgs creation?

TL; DR;

Our project teams need to manage many third party-libraries for which no nuget-packages exist, along with (of course) many nuget-based dependencies. We´d like to manage these dependencies in the most nuget-consistent way, but handcrafting nuspecs kindof sucks. How can we automate the creation of nuget-packages for our third-party dependencies?

Background

The project teams in our company are using certain third-party libraries in many of our projects, for which no public available nuget-packages exist, like DevExpress DevExtreme controls libraries, or similar.

We often are having difficulties setting up Continous Integration (CI) / - Deployment (CD) pipelines for these projects, as our build-agents (a "co-located-on-premise" Team Foundation Server (TFS) atm) often lack the required assemblies / components in order to successfully build these projects. This often leeds to unsuccessfull builds when setting up new build-agents (e.g. when moving to a newer TFS version) or when setting up new projects. This further results in tideous installation of component-suits, most of the time even with a load of different versions of the respective libraries and "reference-fiddling" in order to support building (legacy) applications.

This problem was traditionally solved most of the time by manually managing dependencies on code-repository level in the past, e.g. by checking in dependencies in a "dependencies"-folder in the code-repository root - but this approach is

bloating the repo with every version-update by hundreds of megabytes at least, from day one
tideous in figuring out the dependency-hierarchy and identifying the needed assemblies
error-prone when it comes to re-reference assemblies from e.g. the GAC to said dependency-directory
a style breaker in sense of "modern" dependency management "the nuget way"

I understand that licensing-concerns keep library-vendors from publishing their components via public nuget feeds in the first place, but this stylistic inconsistency in managing references totally sucks. (The rise of nuget and package managers in general say enough about the "traditional approach" of dependency management)

And even if such vendors eventually decide to support a "native" packages for referenceing their stuff in the future, there still is the question about legacy-apps which e.g. use older versions of the libs.

Of course there is the option to setting up our dev-machines as build agents. But this is no solution for us, as our projects are rather long-living in sense of support and our dev-machines are kind of fragile in comparison (these easyly get wiped at least once a year, if not more often here and there; Our projects often need to build even after longer inactivity periods after e.g. five (5), or even ten (10) years).

I think many of you guys and gals out their are facing similar issues, expecially in larger orgs.

One way to address this is to maually craft nuget-packages for this libraries. This limits the tideous work of making sense of the assembliey-hierarchie to a "one timer" (at least per library-version) and "magically" takes care of the assembly-"re-referencing", but is also error prone (at least at authoring-time of the respective nuspec) and a really (really) painful task, especially when it comes to newer library-versions, which often come with more dependencies)

How do you folks address these kind of issues? Isn´t there something out there to automate this?

Yes, there definitely is a way.

Introducing TeGun

from the TeGun github-repository:

TeGun (NuGet reversed) is a simple, yet handy cross-platform(.netcore) commandline-utility for creating nuget packages (nupkg) from a bunch of ordinary, unpackaged assemblies. The main goal is to provide a fast tooling experience when dealing with third-party assemblies that do not come packaged via e.g. Nuget.org.

Just point (dotnet-)tegun, to a directory containing the third-party assemblies. TeGun will scan the assemblies, analyze their dependencies and create **.nuspecs* for them, respecting internal dependencies to prevent packages with "duplicate/redundant" assemblies in it.

You can influence naming- and versioning-schemes of packages and even pre-define bundle-packages via wildcard-patterns, to bundle e.g. Theme-Assemblies for UI-Components.

This helps at least in the initial creation of * .nuspecs The produced **.nuspecs* can either be further tweeked to suit your needs or be used out-of-the-box to produce consumable packages via NuGet.exe-tooling. From that on, you are free to decide over which channel you consume them (like a local Network-Share or Nuget-Feed), and creating packages e.g. based on updated libraries is a breeze.

After TeGun´s initial release I created a total of 1200 packages weighin almost 1.5 GB for a large component-suite , which spans over 5 major versions, in under 30 Minutes (and from this timeframe, TeGun itself was doing most of the work). And this was just for one vendor of one library.

I even managed to create packages that upgdrade from on major-version to another major-version via nuget-package-explorer, and another variant that "locks" the major-version, but respects vendor-updates from package explorer (just because I could, and because some of our projects are kind of precautios regarding major version upgrades of e.G. UI component suits)

This way, the packages provides a kind of "fast-ring" and "slow-ring" in the sense of library-updates.

Sample

This sample is also available on GitHub

Let´s assume we have an app that does some MS Office-automation. Hence it requires certain assemblies from the Office-SDKs. We identify a bunch of available assemblies and copy them to c:\temp\officeSDK. The contents of the directory might look like his:

Microsoft.Office.Interop.OneNote.dll
Microsoft.Office.Interop.Outlook.dll
Microsoft.Office.Interop.OutlookViewCtl.dll
Microsoft.Office.Interop.PowerPoint.dll
Microsoft.Office.Interop.Publisher.dll
...

In order to invoke TeGun, we utilize the dotnet cli extension point as described here. So let´s create a TeGun-enabled project

dotnet new tegun
dotnet restore

This will create a project with a DotNetCliToolReference to the actual TeGun-Executable, which will be downloaded upon restore

TeGun uses a config file for most of it´s settings atm (see #1). Let´s initialize a new config with:

dotnet tegun init office

This wil create an initial office.config.json pointing to assemblies in the current directory.

Open the config to tweak the package-creation to your needs:

{
  "assemblies": {
    "sources": [ 
      "."  //scan the local directory for assemblies
    ],
    "excludePatterns": [ 
      "System.*" //exclude matching assemblies
    ],
    "bundles": [
      {
        "searchPattern": "Microsoft.Office.*", //Bundle all matching assemblies
        "packageId": "OfficeBundle.Sample",    //into a package
        "packageVersion": "1.2.3.4"            //with this version
      }
    ],
    "substitutions": [
      {
        "replace": "Interop",
        "with": "xxx"
      },
    ]
  },
  //general nuspec-creation settings
  "package": { 
    "authors": "earloc",
    "owners": "earloc",
    "tags": [ //additional tags
      "tags",
      "go",
      "here"
    ],
    "targetframeworks": [
      "net45"
    ],
    "versionFormat": "{Major}.{Minor}.{Build}.{Revision}", //Use full version number from scanned assembly as package version
    "title": {
      "prefix": "earloc.", //prefix package names
      "suffix": ""
    },
    "nuspecTemplate": null
  }
}

If we now invoke the command:

dotnet tegun nuspec office

TeGun will create nuspec-files according to the above settings in the subfolder office/nuspecs:

OfficeBundle.Sample.1.2.3.nuspec
earloc.Microsoft.Office.xxx.OneNote.14.0.0.nuspec
earloc.Microsoft.Office.xxx.Outlook.14.0.0.nuspec
earloc.Microsoft.Office.xxx.OutlookViewCtl.14.0.0.nuspec
earloc.Microsoft.Office.xxx.PowerPoint.14.0.0.nuspec
earloc.Microsoft.Office.xxx.Publisher.14.0.0.nuspec
...

These nuspecs can now be further tweaked to your needs and be used to produce packages, e.g. with a simple batch file in the office subfolder:

set nuget=nuget.exe
for /R nuspecs %%f in (*.nuspec) do (
    %nuget% pack %%f  -OutputDirectory packages
)

will produce the following packages at office/packages:

OfficeBundle.Sample.1.2.3.nupkg
earloc.Microsoft.Office.xxx.OneNote.14.0.0.nupkg
earloc.Microsoft.Office.xxx.Outlook.14.0.0.nupkg
earloc.Microsoft.Office.xxx.OutlookViewCtl.14.0.0.nupkg
earloc.Microsoft.Office.xxx.PowerPoint.14.0.0.nupkg
earloc.Microsoft.Office.xxx.Publisher.14.0.0.nupkg
...

Limitations

TeGun does not support automatic inclusion of satelite assemblies atm, but it´s a feature I am working on when I find the time (contributions accepted ;) ).