.net framework 4.5 Compression : DeflateStream & GZipSream

.Net framework includes DeflateStream and GZipStream to convert a stream into a compressed stream. Both of them were introduced first in .net framework 2.0. Starting from .net framework 4.5, they use zlib library for compression. DeflateStream and GZipStream are available in System.IO.Compression namespace in System assembly.

DeflateStream and GZipStream directly inherit Stream class.

Since they work on streams, they read the data byte by byte. So they cannot determine the best compression based on the overall data. Since they are based on DEFLATE algorithm which works like a dictionary encoder, they are not good with already compressed data. Neither of them supports adding individual files to the compressed stream. Both of these types are supported in metro style (.net for Windows Store) apps and Portable Class Library projects.

DeflateStream and GZipStream were introduced in .net framework 2.0 to compress a stream. In .net framework 4.5, they are updated to use zlib library. Because of the changes in Stream, they also support asynchronous behavior.

Compression using DeflateStream
DeflateStream is based on DEFLATE algorithm. Compressing a stream is just about copying another stream to DeflateStream instance with the proper compression mode set. In the following code, the path of source and destination file is specified as parameters to a method. It just reads the source file using FileStream and compresses it using DeflateStream to destination path.


Decompression is also as easier as compression. Here we are reading a source file using FileStream and passing it to DeflateStream with Decompression mode set. Then it is just a matter of copying it to a destination FileStream object. The term Inflate is used for decompression.


GZipStream Compression
GZip uses an industry standards data format for compression. The compressed stream can be saved with a *.gz extension. This compressed file can be decompressed with commonly available decompression tools and utilities. Using GZipStream is similar to using DeflateStream. The following code reads a file using FileStream and compresses it using a GZipStream object.


And the following is the code which can be used for decompressing a file using GZipStream. The file must have been compressed with GZip format.


Async compression and decompression
GZipStream and DeflateStream supports two Microsoft Async patterns. They are as follows:

1. Asynchronous Programming Model [APM]
The compression streams support APM by providing BeginRead / EndRead and BeginWrite / EndWrite method pairs. This is a legacy asynchronous pattern by Microsoft and is no longer recommended for new development. The pattern is also sometimes known as ASynchResult pattern. The BeginX / EndX method pair starts and ends the asynchronous operations. The BeginX method returns an ASyncResult object which can be used with the corresponding EndX method to block until the operation started with BeginX is completed. The IASyncResult can also be used to wait on the asynchronous operation to complete. The exceptions are also rethrown in the calling thread when EndX operation is called. Although BeginRead / EndRead and BeginWrite / EndWrite method pairs are inherited from Stream. They do get overridden in GZipStream and DeflateStream but since APM is no more recommended after .net framework 4.5, these methods are not recommended to be used for new development.


2. Task-based Asynchronous Pattern [TAP]
.net framework 4.5 introduced new XAsync methods to support another Microsoft asynchronous pattern TAP [Task based Asynchronous Pattern]. There are following methods provided in .net framework 4.5:


1. FlushAsync
2. ReadAsync
3. WriteAsync
4. CopyToAsync

These methods, including their overrides, are inherited from Stream. There are overrides for these methods which support cancellation.

Word of Caution
As we discussed, DeflateStream is based on LZ77 and Huffman coding. LZ77 work on dictionary based encoding. Based on the extra bits created for dictionary keys, they might result in bigger compressed data, even bigger than the source data. So if the data size is too small, then it is better not to use them at all.

Download Code

.Net Framework 4.5 and Compression

In this post we are going to discuss the new features in System.IO.Compression introduced in .net framework 4.5. We are going to specially discuss how easy is to create zip archives using those new features. We are also going to discuss how to use the zip packages and extract from it using a simple project.

ZipArchive in .net 4.5
Let's create a new .net 4.5 project AppCompression. It's a console application based project.

Let's add the references of System.IO.Compression and System.IO.Compression.FileSystem assemblies to the project. Here System.IO.Compression has additional types and extensions to ease working with compression / decompression.

Let's also add a test folder with some files as follows:

The folder also has a sub-folder. The contents of the sub-folder are shown below:

ZipArchive is introduced in .net framework 4.5. It allows us to create a zipped package / compressed file. All individual files are added as entries to the package. The confirms to universal zip format. This zip file can be transmitted and decompressed using a regular decompression utility including Winzip. The following code uses ZipArchive to compress a folder. We are using the main folder discussed above as the source folder. As you can see, we are creating a ZipArchive instance and adding individual files as entries from the folder.


As expected, when executed, the above creates a zip file in the specified path. We notice that it hasn't maintained the folder hierarchy in the compressed file. So we would not be able to decompress this file with the same folder structure as the original uncompressed folder.

As a matter of fact, ZipArchive does support maintaining the folder hierarchy. We just need to specify the file name including the relative path. This should be enough information for ZipArchive to create the intended folder structure. In the code below, we are using Uri.MakeRelativeUri to determine the relative path of each entry using the base folder's path. Now this relative path is used to create entries as follows:


When we look at the compressed folder in Windows Explorer, we do see the folder hierarchy being maintained. It creates the sub folder and adds the intended files to the sub folder. The folder looks as follows:

If we don't want to add System.IO.Compression.FileSystem assembly reference then we can create a simple ZipArchiveEntry from the ZipArchive instance. We can then use stream to copy the file's contents to the ZipArchiveEntry. The following code can be used to replace ZipArchive.CreateEntryFromFile method provided as an extension method to ZipArchive type in System.IO.Compression.FileSystem assembly.


ZipArchive can also be used to decompress a zip file in a directory. First let us see the easier way to do it. There are extension methods for ZipArchive available in System.IO.Compression.FileSystem assembly to do the same. The below code ExtractToDirectory extension method to decompress a zip file to the specified folder. The method extracts the contents of the compressed file while still maintaining the folder hierarchy. As you can see, we can use ZipArchiveMode.Read as we don't need to write any new entries to the package.


We can also enumerate through the entries in a ZipArchive and decompress each entry. In order to maintain the folder hierarchy, ZipArchiveEntry's FullName method can be used. The ExtractToDirectory method can be replaced with the following code. The code would take care of creating the expected folder structure for the compressed file:


ZipFile in .net 4.5
ZipFile is an easier alternative in .net 4.5 for working with compression / decompression. It also supports opening a ZipArchive, we can use it to add entries to a package. The package can later be used to create and save a new / same compressed file.


How simple is that?? Now let's see how we can use ZipFile to extract and compressed file into a folder. The following code extracts the contents of a zip file into the specified folder. We don't even have to make any extra effort to maintain the folder hierarchy during the extraction, ZipFile automatically takes care of that.


As discussed earlier, ZipFile can also be used to Open a compressed file. There are two overloads of Open() which allow us to load a compressed file using either of Read / Create or Update ZipArchiveMode. There is another method, OpenRead(), which opens the compressed package in Read mode. These modes apply restrictions on what we can do with the returned ZipArchive instance. The following sample code uses ZipFile to open a compressed package and writing the names of the entries on the console.


It results in the following output:

Download Code

.Net Portable Subset & Portable Class Libraries

While developing portable class library projects, you might have seen the term portable subset at different places in Visual Studio. Ever wondered what that is? In this post, we are trying to explore this. Just to refresh, you can find an example in the assembly references section of your portable calss library project. If you look at its properties, you find that it is SDK file type. It has .Net Portable v4.0 identity and it refers to a path in the profiles folder.

If we look at the Profiles folder in assembly refernces, we realize that we can find two versions under .Net Portable. They are v4.0 and v4.5.

Basically both of these folders contain profiles. Most of them are contained by v4.0. There is only one profile in v4.5 folder. This is Profile 7.

As discussed in a previous post, we can easily determine the selection of target frameworks for a profile through the supported frameworks folder under profile. Profile7 should be selected for .net framework 4.5 and .net for Windows Store Apps platforms selection.

This profile is used when we target .Net framework 4.5 and .Net for Windows Store Apps. We can easily verify that by retargeting one of our portable class library projects to these versions.

After updating this, we can find that the profile is updated as follows:

So what is the Portable Subset?
Basically Object Browser can be used to see the list of assemblies for a framework or profile (including .net framework client profiles). With Portable Class Libraries, suddenly so many profiles are introduced that it might have seemed that the IDE would clutter up making the life difficult for the developers. This might have pushed the development team at Microsoft to call it sub-set instead. Basically this sub set includes the assemblies referenced in any combination of the target frameworks for the portable class library. So if you see an assembly there, you still might not be sure that it would be available by your selected framework combination. Remember the overlapping area concept we discussed in this post?

Actually this is the same issue when we look at msdn and find that the portable class library support icon with a method / property. Which profile exactly does this mean? We are on our own on this. The feature set chart that we discussed, might help.

These portable versions [v4.0 and v4.5] seem to be referred as .Net Portable subsets in Visual studio. Here v4.5 seems to be referred as .Net Portable Subset and v4.0 seems to appear as .Net Portable Subset (Legacy).

What's Legacy Portable Subset?
As we have seen there is only one frameworks combination that v4.5 provides. It is based on newer frameworks including .net framework 4.5 and .net for Windows Store Apps. Let's try to see what's cooking inside now. Let's try to see if there is a difference in the assemblies for legacy Vs newer profiles. Below is the profile of mscorlib as in Profile 24 for v4.0 Portable folder. As we have discussed in one of our previous post, this is a retargetable assembly.

Now let's look at the mscorlib as in the only profile [Profile 7] in v4.5. Amazingly this is not retargetable. So definitely, there is something cooking inside.

We might verify the same by looking at the other assemblies in a profile for Legacy and New .net portable versions. They all seem to follow the same i.e. all legacy profiles have retargetable assemblies and Profile7 in v4.5 has non-retargetable assemblies. The answer can be found in the reply of David Kean for this question in StackOverflow.

For new platforms (.NET 4.5, Windows Store, Phone 8), we took a step back and designed portability from the get go. Rather than attempt portability as an afterthought, we designed what we call contracts (basically assemblies) that represent a self-contained, unit of code that either a platform supports all of, or none of. This means that when you see "System.IO 4.0.0.0" on .NET Framework 4.5, it supports exactly the same APIs that you will see when you see it on Windows Phone 8. This makes portability very easy, rather than needing to generate custom assemblies to represent the intersection of a platform, we simply subset at the assembly boundary. For example, given a platform 1 that supports System.Runtime.dll, System.Reflection.dll and System.Reflection.Emit.dll and platform 2 that supports System.Runtime.dll and System.Reflection.dll. When you target these platforms in portable, we simply choose System.Runtime.dll and System.Reflection.dll. Long term this makes portability a lot easier to understand, as you can think in terms of assemblies instead of individual APIs.

Assembly References For a Profile In a Portable Class Library Project

In this post we will be discussing one more common question people ask regarding Portable Class Library projects. This is about how to find out the list of assemblies referenced by the project. The question is raised because the references section is displayed as follows:

Object Browser
The first option is to use the Object Browser [View -> Object Browser]. Just select "My Solution". You might need to unload the other projects if you need to just see the assembly references caused by the portable class library project in the solution.

Class View
We can also see the referenced assemblies in the class explorer. This is also available in View Menu [View -> Class View].

Profile Folder
One of the option is to directly go to the folder for the specific profile and see the assemblies in the folder. The profile information can be found in the solution explorer as follows:

And here are the available assemblies for the profile.

Portable Class Library & Selected Profile

How can we determine the profile used by a portable class library project?

I have seen the question frequently thrown at me in some of my talks. This question is generally raised as soon as I explain how profiles are used by portable libraries. So I thought for the interest it would be beneficial for the Microsoft developer community if I create a separate post about this.

Profiles Info from Visual Studio 2012
Here we are using Visual Studio 2012 RC for this example. The release version should have the same behavior. Let's open the portable class library project, named MyPortableLibrary, created in the previous example in Visual Studio.

In order to see the selected profile for the project, let's open the solution explorer and expand References section. You should only see .Net Portable Subset if you haven't added any other assembly references manually. Right click .Net Portable Subset and select Properties. The Properties should be opened with the following display.

Here you can see that the project is targeted for Profile4. You can see the details of the profiles by following the profile path. In Visual Studio, you can also see the referenced assemblies for the target profile. Just open Object Browser window and select My Solution.

Profile Info outside Visual Studio
Is it possible to find out which profile the project is referencing on a non-developer machine? Generally, you wouldn't find Visual Studio installed on such machines. These profiles are standard profiles provided by Microsoft. Actually the profile info is part of project definition. You can open the project file [*.csproj for c# projects] in any text editor and find the profile info. Here we have opened the project file in Notepad.

Obviously you can see the same in Visual Studio as well. Just unload the project in Solution Explorer and Edit Project file. You should be able to see this.

Profile Info from Assembly Manifest
Profile info is also available in assembly manifest. Just look for TargetFrameworkAttribute and you should be able to find the actual profile that the assembly is targeting. Here we have used the manifest from one of our previous posts.

Obviously we can check the profile of the running assembly:

The code results in the following output for one of my portable library:

The same information is certainly available for a debug session as well:

Portable Vs Retargetable Assemblies

As we have been discussing that Portable Class Libraries allow us to target the same assembly for multiple platforms. Hence the name "portable" as they can be ported to the platforms and frameworks specified at design time. And as we have discussed before, that this sharing can be called Assembly Portability. Although you can create a portable library which is also retargetable but being portable is not the same as being retargetable. We can just use this manifest from one of our previous discussions about portable libraries. You can see that it is referencing a few retargetable libraries.

Are you wondering what actually is being retargetable? Basically portable Class Libraries is a recent introduction to developer's toolkit but retargetability has been there for a long time. As we have discussed in one of our previous posts that the assemblies in profile folder are just there for reference. They are to make sure that only certain features and types are used from them based on the automatic profile selection by the Visual Studio IDE. [The selected profile is part of the definition of portable class library project. You can easily find this in the project definition].

The actual runtime assemblies [e.g. System.dll] wouldn't be used from the profile. They would be used directly from the host framework. So if your portable library has Silverlight 4 as one of the target framework then it would be using the System assembly as provided by Silverlight 4 runtime. Portable Library Developers at Microsoft has made sure that there are no runtime surprises when your portable assembly is loaded by the host framework by only supporting the features provided by all of the selected target frameworks. That is exactly what Retargetability is. For an assembly, being retargetable means that the actual assembly will be used as provided at runtime. Let's see the manifest of a retargetable assembly [System.dll] from Profile1 using ILDASM.

Now you might be wondering if we can create a retargetable library. Actually you can do it. You can provide flags when you are developing your assembly. These flags are used from AssemblyNameFlags enumeration available in System.Reflection.

http://msdn.microsoft.com/en-us/library/system.reflection.assemblynameflags.aspx

The combination of these flags can be accessed through Flags property of your required assembly.

http://msdn.microsoft.com/en-us/library/system.reflection.assemblyname.flags.aspx

Just because you can do this, it doesn't necessarily mean you should. There doesn't seem to be any reason why we would be needing a custom retargetable library. This is from the documentation of the member of the enumeration from MSDN.

"Specifies that the assembly can be retargeted at runtime to an assembly from a different publisher. This value supports the .NET Framework infrastructure and is not intended to be used directly from your code."
This member is also available for your Portable Class Library project. Remember the special icon that shows the Portable Library support in msdn?

To see how this works, let's create a simple portable class library project, named MyPortableLibrary.

Let's build the project and see the manifest in ILDASM tool. This is definitely not a retargetable library.

Now let's add the flag in the assembly info file for the project as follows:

Now let's build the project again and see the updated manifest. You can notice that the assembly is now marked as retargetable.

Code camp Tallahassee - Nov 3rd 2012

I had the amazing opportunity to speak at Tallahassee code camp this weekend [http://www.tallycodecamp.org/]. The event took place in Shores building in Florida State University.

I spoke about Portable Class Library Tools and MVVM.

Tallahassee is a beautiful small university town.

Like all codecamps, the event was followed by after party and prizes.

You can download the event's presentation here:

Portable Class Libraries [PCL] : Targeting Additional Frameworks

In this post, we will continue our discussion on Portable Class Library in Visual Studio. This allows us to target the same assembly to multiple frameworks, which allows to use the same assembly without even a need of recompilation.

Previously we have discussed about the following:

1. An Introduction : http://www.shujaat.net/2012/07/multi-plaftorm-targeting-using-portable.html
2. Portable Class Libraries & MVVM : http://www.shujaat.net/2012/08/portable-class-library-mvvm.html
3. Portable Class Libraries & Features Support using Profiles : http://www.shujaat.net/2012/08/portable-library-tools-supported.html
4. Portable Class Libraries & Type Forwarding : http://www.shujaat.net/2012/08/portable-library-tools-type-forwarding.html

As we know that the portable class library tools now support targeting the library to multiple frameworks including .net framework versions, silverlight versions, Windows Phone versions, Windows Store App and Xbox 360. What if we want to target it to a .net platform which is not listed including mono platforms?

Targeting additional Framework & Usage of Profiles
In order to determine how we would be supporting the framework in Portable Library Tools, we first need to determine the feature set supported by the target framework. We must also consider how those features are supported. During our type forwarding related discussion for portable libraries, we tried to ponder on why it is important to know how the features are provided and that is why MVVM types are not supported with portable class libraries for .net framework 4.0 as they are differently provided than rest of the platforms.

These profiles allow Visual Studio to provide framework lists for portable class library projects. Based on this idea, we should be able to add remove the frameworks and their versions by manipulating these profile folders. To get started, let's move all the profiles from the .NetPortable folder.

Now create a Portable Library Project and see the empty target framework lists. Now it should make sense how profiles affect the target frameworks list available for portable class library projects. I am using Visual Studio 2012 RC here.

Skeleton of a Profile
As sample profile folder for a portable class library project consists of some dlls & their xml for documentation of containing types, RedistList folder and SupportedFrameworks folder.

Here all the dlls are retargetable. This means that the portable library would use the versions of these dlls from the host framework. So if the portable library is being used in a Silverlight project then it would be using the Silverlight version of these assemblies. Similarly, if the same assembly is being used in a .net framework based project, it would be using the assembly provided by .net framework version. Let's look at the manifest of System.Core assembly in Profile1. You can also notice that the other assembly references are also from retargetable versions of these assemblies.

Al the profiles seem to share the code documentation. So the xml files contained in the profile folder is just for redirection to the main folder containing all the profiles.

This is just referring to this documentation xml file. This is super smart idea as the main folder would have the complete documentation and all the containing profiles would just be a subset of the main assembly.

Please remember to reload visual studio i.e. kill it and open it again for the changes in profiles to take effect.

Now we come to RedistList/FrameworkList. This is to describe the list of assemblies available for the target frameworks. For Profile1, the list is as follows:


From the xml file, I am sure that you can guess what assemblies would be looked for for the list of the target frameworks. The corresponding assemblies are looked for in the folder containing the Redist folder. There seems to be no exception if an expected assembly is not there in the folder. Also there doesn't seem to any issue if an unexpected assembly is found in the profile folder.

Now which part of profile is important to show frameworks and their versions in framework selection dialog for portable class library project. This is the SupportedFrameworks folder. You will find a list of xml files in the folder.

The text of the framework for the selection comes for the xml content of these files. Display name shows up for the selection. If we just change Xbox 360 to Xbox 360 2 , that is what is shown on the dialog for selection.

This is how the framework selection is shown after the change:

And that is exactly how this is shown in the Library Tab of Portable Class Library project.

Now on Additional Frameworks
After the above discussion, we have found out that the frameworks available for selection are picked up from the xml files from the SupportedFrameworks folder. So in order to target to an additional framework, we need to find out if the new target framework has features that are provided by any of the available profiles. And also the way the profiles are provided.

This is how the framework is available in the selection dialog:

If there are more than one versions of your framework in different profiles then they are displayed in a selection control. Let's copy a new profile [Profile 2] and a new framework file Additional Framework 2.0 with the following contents.


This should result in displaying the framework selection dialog as follows: