Hosting Nuget Feeds

In this post we are going to see how we can create nuget feeds for intra-organization needs. Since these packages are not intended to be shared outside the organization, we don't need to host them on nuget.org. Nuget feeds can be directly created from file system folder or they can be hosted on a web server.

Package Feeds from Network Share
We can simply setup a Nuget repository on a server. We can simply create a folder which could be shared across the network. This is called Poor Man Nuget Repository. Let's use MyTestPackage.1.0.0.nupkg created earlier and copy that into the folder used as the repository. The configuration for the repository can be done in Tools -> Options. The developers need to provide this configuration in their Visual Studio 2012.

After configuring the repository, they can use it to source their nuget packages. You should be able to see the repository in the Nuget Package Manager. Please also notice the metadata for the package.

Hosting Nuget Packages on a Web Server
We can also host nuget packages on a web server. We can easily create a web application and use it to host nuget packages. This technique is specified here [http://docs.nuget.org/docs/creating-packages/hosting-your-own-nuget-feeds]. Let's create an empty ASP.net web application in Visual Studio 2012 as follows:

The most amazing part is that Microsoft has provided a nuget package to help organizations hosting their nuget repositories for intra-organization consumption. The package is available on nuget.org. We can add the package to our web application as follows:

Look at the package dependencies for this package.

In order to run the application I needed to update <system.web> section as follows for hosting in the selected framework:


Visual Studio 2012 Ultimate can host it directly in IIS Express. Let's run the application and see the following page:

The application will source the packages from "Packages" folder in the solution. This is configurable. We can update the following app setting in web.config.


This is the same folder we were using to host the packages directly from file system. Let's run the application now. This should update the view as follows:

We can add this as a package source for client applications.

Packages can be added from the source now as follows:

We can also support clients to publish nuget packages and contribute to the feed. The web server host is secured through the following settings:


For our testing, let's just disable it. Now let's create another package and use Package Explorer to publish it to the nuget feed as follows:

Now you can see that the new package is available to the feed now:

Download:

Using Nuget Packages in an Enterprise Software

Nuget has become the technology standard to include the dependencies of a project. These dependencies can simply be assemblies referenced in order to use a particular framework. We can also create our own nuget packages and make it available in the nuget repository. The packages can be installed using Nuget Package Manager which would download the packages from the repository. When we include a Nuget package, the assemblies are referenced from the "packages" folder in the solution directory. This can simply be understood as the same "Lib" folder we used to have in our projects to include common libraries which are then referenced by all the projects in a solution, except nuget packages can be a lot more than a bunch of assemblies.

In an organizational setup, the source code is maintained under a source control where it is shared between all the developers. If the organization has embraced continuous integration, there must be a build server which would keep the project working based on certain events [code check-ins, periodic intervals etc]. We might not want to check-in the "packages" folder to the source control and other developers and build server to automatically get it from the source. But we would not want other developers to get it directly from the repository as there are great chances that they might select a wrong version of the package. Then you could end-up in a situation like "It's working on my machine" which would later be identified as "Oops, wrong package version". We just want to avoid this situation. So there must be some way so that other developers and build server automatically install the packages as specified in "packages.config" and they don't have to reference the projects themselves. This is exactly the topic of our discussion today.

We will be using Visual Studio 2012 Ultimate for our example.

Automatic Package Restore Configuration
Visual Studio 2012 supports automatic download of referenced Nuget apps if they are missing from packages folder. Let's create a WPF project in Visual Studio 2012 and name it AutoNugetApp.

First we need to allow Visual Studio to automatically download Nuget packages. This setting is configured in Tools -> Options. Please select PackageManager -> General Tab. You will notice the following setting:

Just enable that. The same config page also supports clearing and browsing Nuget Cache [to be discussed later in this post]. You can also clear recent packages list. This is available when you are adding packages to your projects so that you don't have to search them each time. Now Visual Studio is all setup to download missing packages from Package Source. We also need to support restoration of packages on solution level. Just right click the solution in Solution Explorer and select Enable Package Restore.

This should add a solution folder .nuget. This also contains a build target file. The related nuget configuration is provided in nuget.config file. For our solution, you can only find disableSourceControlIntegration setting. This enables / disables the addition of packages folder to source control. You can find other related configuration here [http://docs.nuget.org/docs/reference/nuget-config-settings]. We can even change the packages downloaded into some folder other than packages folder.

Let's add MVVMLight package to our WPF application. This would add packages.config file with the details about the added packages.

Now let us build the project and notice the packages folder in the solution folder.

You can also notice the nuget.target build file to the project configuration. Just unload the project and see the project configuration by selecting Edit Project in solution explorer.

Let's delete the packages folder and build the project again. This should still build. You can notice that it has created the packages folder again for the packages added. Let's call it a success.

Manual Package Restore Configuration:
Let's see how we can do it manually without any package related support from visual studio. Let's create a sample WPF project. Since we have heard that [MVVM is cool], let's base our design on it. We add MVVM Light Package to the solution as follows:

This should add packages.config to our project as follows:

Additionally, this should add a packages folder to the project directory. This folder contains the assemblies and other related files added by package provider.

Now let us build the project. It should be building successfully. Now we need to see how we can enable automatic restoration of packages. Let us delete packages folder from the solution directory. This should cause the build to fail as the referenced assemblies are not available. We need a mechanism for the automatic download of the assemblies from the package source. When we build the project, you should see the following:

Let us get create a folder named Nuget in the solution directory, and add nuget.exe from AutoNugetApp. You can also get it from codeplex.

In order to avoid build failures we need to download the packages referenced before build. We can define a pre-build event to install the package. The install package command downloads the packages in the packages folder in solution directory.

You might have noticed double quotes. They are just to accommodate any specified in the path. Just open the Property window for the project and update the pre-build event as above. This should look like this:

That's it. Now we can build the project. The pre-build event would cause the installation of the packages as specified in Packages.config file for the project. Since all the assemblies are available in the "packages" folder, as expected by the project, it should build fine now.

[Note]: Now when the other developers and build server would get the code from the source control, the [packages]folder would be missing. As they start building the project, Nuget.exe would install the necessary packages which would then be used by the project build. You do need to check-in the Nuget directory created under the solution folder. [http://docs.nuget.org/docs/reference/command-line-reference]

Nuget Cache
Nuget also supports an offline cache for faster access to the missing nuget packages required for a project. This cache is maintained in AppData folder. For our solution we are using MVVM Light package. The package can be found as follows:

In case of non-availability of network, this can be really useful. We can even use this as a Nuget Package source for the other projects. Or we can just update their packages.config and this can be automatically used as they would be found missing. If we need to verify if it is really using the packages from the cache, just delete "Packages" folder and get your machine off the network. With the current configuration, the packages are used from cache. We can hookup Process Monitor to verify this.

Creating Nuget Package
The easiest way to create nuget packages is to use a tool. Nuget packages can be created using Nuget Package Explorer. The tool is available on CodePlex [http://npe.codeplex.com/]

Let's create a sample package using libraries from our CoffeeHouseApp. We are adding libraries for portable models and view models. This also contains a config file. For the sample package, it is just an empty file but you might support some configuration which could help better use the libraries in your package.

Remember we have just seen the nuget download cache. The explorer supports viewing the cache. It also supports clearing the contents so that the packages are downloaded directly from the source. Selecting View Nuget Cache would open the same cache from the AppData folder.

Package Visualizer
Package Visualizer is an amazing tool to view the packages referenced by all the projects in a solution. You can just select [Tools -> Library Package Manager -> Package Visualizer] to open the use this. This should open a dgml file with the details of your projects and their referenced packages. If two or more projects reference the same package, it would be displayed as a single node with references shown from all such projects. Let's add another console application project. We are referencing nunit package to both projects.

It also lets you visualize the dependencies of a package on other packages. Here you can see the dependencies of nuget.server package after installing it to your project.

Download

[MPGO] Managed Profile Guided Optimization - .net framework 4.5

In the previous post we discussed about what are native assembly images and how it can help us in improved application startup and facilitate code sharing. In this post, we are going to discuss another CLR feature provided in .net framework 4.5. The feature is called Profile Guided Optimization. The only difference is that this profiling would not be based on historical usage of the application by the user but would be based on some test scenarios. The profile optimization is about improving JIT compilation, on the other hand, profile guided optimization is improving the performance by the layout of native assemblies.

It must be remembered that this is the same optimization that is used by .net framework assemblies. Profilers have been helping the software developers with code optimization. They are used to measure various statistics for the running software including memory footprints, memory leaks, CPU and I/O usage. The feature just creates the profile data for assemblies for generation of better native binary images by NGEN [Native Image Generator].

How it works?
The workflow involving the use of MPGO is very simple. It is fed with the list of IL assemblies with the scenarios to run. It generates the updated IL images in the specified folder. The simplified workflow can be presented as follows:

The first part of this procedure happens on vendor side creating providing the software. The vendor uses [MPGO] and associates profile data with the IL assemblies. Now the assemblies are provided to a client in a deployment package. Here they can be NGENed manually during installation. They can also be queued for NGENing for later when the machine is idle to be taken care of by Native Image Service [ Or Native Image Task for Windows 8 machines]. They can also be automatically NGENed by the framework when no such specification is provided by the installation package.

Running MPGO
MPGO can be used with Developer Command Prompt for Visual Studio 2012 Ultimate. Please make sure that you are running the command prompt with administrator privileges.

It might ask for administrator privileges if your Developer Command prompt is not running with it. You will get an access denied for not granting them.

MPGO & Application Types
It must be remembered that neither the native images nor MPGO are the silver bullets to improve application performance. They are recommended for very large applications where start-up time can be an issue. It must be remembered that they are only recommended for desktop applications. Both of these tools are not recommended for ASP.net and WCF applications because of their dynamic deployment model and significantly less importance of start-up time for those application types. MPGO is not supported for Windows Store apps.

MPGO & Application Deployment
As we have discussed MPGO requires Visual Studio 2012 Ultimate license. Our clients definitely wouldn't have this license on their machines. Actually they don't need to and this tool is also not part of .net framework. It's rather a Visual Studio tool. We can create IL assemblies optimized through MPGO. We can include these assemblies in our deployment package. While installation we can run NGEN to copy these assemblies to the native image cache for manual native images creation.

If your clients are Windows 8 machines, then these native images can also be automatically created when the application is running for future application startup performance gain.

Using MPGO:
Let's use some existing code for doing some cool stuff with [MPGO]. We can use the project from one of our previous posts. This is the project we developed while discussing Portable Class Libraries in the context of MVVM based designs. You can find the project here. You can also download the code.

We will be setting the output folder for all the projects in the solution to [././ILAssemblies]. This would create a folder on the same level as [bin] folder.

This is where all the assemblies would get copied once you build the project.

Let's create another folder where we you want the optimized image of the assemblies is copied to. The optimized assemblies would be generated by [MPGO].

Running MPGO is easy. We need to specify the required switches to help the tool. Here we are running it for the assemblies (including DLLs and EXEs) in [ILAssemblies folder). We are generating the optimized images in [OptimizedILImages] folder. [Scenario] specifies how we want to run our application including the start-up arguments. The other option is to use the profile data from some other assemblies. In that case, we don't need to run the application. This is supported through another switch [-Import]. Please remember that [-Scenario] and [-Import] are used exclusively.

When we run this, MPGO launches the application. We need to run the application through the pre-thought scenarios and close the application. As soon as we close it, MPGO has all the data to optimize the application (alternatively it also supports Timeout where it would just profile it for the specified duration and uses the profile data). So it runs our [Coffee House App] as follows:

As soon as we close the application, we have our optimized assemblies in the [OutDir] folder.

[MPGO] doesn't override the optimized assemblies in the [OutDir] folder. It just appends them with the next available integer. In this case, I have run the tool again and look at how it has appended "-1" to the name of the assemblies.

[Note] I don't want you to miss one important detail. Two of these assemblies are portable class libraries. It is really great to see that the tool supports them for generation of optimized images.

Mechanism of Optimized Assembly Generation
This should be enough to discuss about the tool. But we need to see what is happening under the hood. Let's discuss briefly what is actually happening under the cover. When we run MPGO it basically compiles our assemblies and their dependencies into native code using NGEN and copies only the native images of source assemblies in the [OutDir] folder.

We can use Process Explorer to see how MPGO is using NGEN to created native images. Remember we discussed about MSCORSVW.exe in the previous post?

You can see the details about the assemblies in registry as follows:

When compilation of these assemblies finishes, [MPGO] adds instrumentation information to those assemblies.

After instrumentation of the assemblies, MPGO launches the application as specified in [-Scenario] switch. It is launched from the source directory. It is done for assemblies individually as they are individually compiled they are picked up for instrumentation.

It then runs the application with the scenario. It waits until the application finishes or timeout expires. During this time, MPGO profiles the application and keeps the details in a temporary storage. The tool creates IBC data files in the [OutDir] folder. They are saved with [IBC] extension as follows:

As soon as the scenario finishes or timout expires, the instrumentation assemblies are uninstalled.

Since MPGO has profiled the application while running the scenario, it just copies the assemblies from the source folder and merges the profile data with the assemblies. It then removes the IBC data files. Now we have the assemblies with the profile data.

Using Optimized data from Previously optimized Assemblies
Once we have the optimized assemblies with the profile information, we don't need to run the scenario every time there is a new version of the assembly is created. We can just source the profile information from previously optimized assemblies and create newer version of the assemblies for native image generation. Let's create another image, called [ReOptimizedILImages], in the same folder as follows:

We can specify the new versions of assemblies with -assemblyList or -assemblyListFile switch. The ones to use the scenarios from could be provided with -Import switch. MPGO processes it and generates the newer versions of assemblies in -OutDir folder.

Here is how we can specify the same in Command Prompt.

Finally we have the merged assemblies in the [-OutDir] folder.

It must be remembered that if an assembly (specified in -assemblyList / -assemblyListFile) has no corresponding assembly in the assemblies containing the profile data (specified as -Import) then it would just fail for the particular assembly. The rest of the assemblies still get merged successfully. In order to see that let's just remove [CoffeeHouseApp.PortableViewModels] assembly from the [-Import] folder. Now execute the same command as above and notice the output in command prompt.

MPGO In Enterprise Eco System
In the enterprise world, MPGO might be made part of the process. As you know we first need to run scenarios. This can be done on a sample machine. Running the scenarios on an application would created assemblies with the profile data. These assemblies can be uploaded to some central location (Or checked into a source control).

Now Build servers (like TeamCity / Cruise Control) come into action. We can make this as one of the last build steps. We can use the -assemblyList from the repository assemblies from current build and provide the -Import details as the checked-in assemblies. Now when MPGO is run, it would create the merged assemblies in the specified folder. It would be tricky though in case of Build Triggers and dependencies if this build would be triggering some other builds on the build server.

[Note] If we are using Visual Studio, we can include a post-build event with MPGO command and have the merged assemblies in the specified folder.