I’ve previously written about Internal .Net Framework Data Provider error 6 and how to obtain the hot fix to fix this issue.

It appears this hot fix never made it to .NET Framework 2.0 SP1 and there is a special fix (and KB article) to apply because the hotfix mentioned in the previous post will simply not work.

To get the hotfix for a .NET Framework 2.0 SP1 installation you’ll need to request a fix for KB948815 in the hot fix request web submission form.

I’ve just been asked through the “Ask” widget on the side of this blog on Yedda (my day job) this question:

Yedda � People. Sharing. Knowledge.Where is sos.dll for .net 3.5?

I installed Visual Studio 2008, but I don’t see sos.dll in the Microsoft.NET\Framework\v3.5 directory? What’s up with that?

Topics: , ,

Asked by dannyR on January 03, 2008

View the entire discussion on Yedda

To which I replied:

Yedda � People. Sharing. Knowledge.Where is sos.dll for .net 3.5?

There is a bit of confusion as to the .NET Framework versions as opposed to the CLR (Common Language Runtime) versions.

Up until .NET Framework 2.0, the CLR versions advanced side by side with the versions of the framework.

Starting from .NET Framework v3.0, the version of the CLR stated at 2.0 (there was no significant change in the CLR itself) and the framework’s version kept on going forward.

.NET Framework v3.5 actually runs on CLR 2.0 SP1.

This means that you can should be able to use the SOS.dll located at Microsoft.NET\Framework\v2.0.50727 with either Visual Studio (as I’ve explained on this post) or with WinDbg.

Dave Broman of Microsoft has written a post about the mapping of the versions of the .NET Framework to the version of the CLR. Do mind that as Dave states in his post, its the way he figures things up in his head and not the official documentation on this matter.

Topics: , ,

Answered by Eran on January 03, 2008

View the entire discussion on Yedda

I know its a bit confusing and I’m not really sure why Microsoft detached the versions of the Framework from the versions of CLR (well, I can understand some of the thoughts leading to this but I don’t think it was that good of a move) but Dave’s excellent post does put things in perspective and allows figuring out in what version of the CLR you are actually using.

I know I haven’t been posting much lately, but I’m a bit caught up in a startup I’m involved in called Yedda. It has taken most of my time, but I do find, once in a while, something to post like the “Ambiguous match found” problem, so bear with me 🙂

Today, I’ll give a small tip for !clrstack.

If you use !clrstack to show a managed call stack and you want to see the addresses of parameters that were passed to the functions without doing a lot of fancy shmancy WinDbg tricks simply run:

!clrstack -p

That’s all you need to do. This will list each function in the call stack and below it the name of the parameter and its address. From there on, you can use !do ( !do == !dumpobject – which shows a managed object) and see the content of the object.

Quite simple and very effective.

As a general rule of thumb, most SOS command have a very descriptive help. To use it, simply run:

!help <command>

i.e.

!help !clrstack

According to this post in Jochen Kalmach’s blog, the new (beta) version of WinDbg – 6.6.3.5 that I’ve mentioned earlier in a couple of posts support mini real dumps.

This means that instead of taking a full memory dump (its not actually full because you don’t take the kernel space) which can be up to the size of the current process (and in server applications it can reach to a few hundreds of MB), you can take a small, light weight mini dump that contains only minimal information such as call stacks.

This previously worked in unmanaged environments but from this version it also works on managed and mixed environments.

I’m still trying to find the original source of this information (Jochen, can you help on this? 🙂 ), but after tracking Jochen’s blog for a while he does know what he is doing.

Remeber a few posts ago that I talked about CorDbg, the managed debugger?
According to this post in Mike Stall’s blog (a fine blog, btw) its out of the SDK.

CorDbg was the first managed debugger that was written. It was written in native code since Interop in those days was still being written and was very unstable.
CorDbg also lacks features that are common in WinDbg such as extending it with extensions and instrumeting with it in a nice way.

Its hard (or even impossible according to this at Mike Stall’s blog) to implement a managed debugger in managed code for v1.1 (although I know there are a few people messing around with it as we speak, me included 😉 ). That is why CorDbg is the only viable managed debugger for production debugging for v1.1

To fill all of these gaps and to make things a lot more easier and more powerful here comes MDbg (ta-da).

MDbg is a managed deubugger fully written in C#. Its extendable by writing managed code to extend it, can be used to instrument code and is just so damn cool 😉

It gives you all the things you ever wanted from a production debugger and can also debug both .NET 1.1 and 2.0 code (to use it to debug 1.1 code you will still have to have the .NET 2.0 Runtime installed).

You can find out more information about MDbg here.
I am in the process of collecting some more information for a more elaborate post on MDbg and some of its features (probably a series of posts), so stay tuned.

WinDbg is a native debugger and although using it combined with SOS to perform managed debugging is great, it sometimes lack the finesse that only a managed debugger can give.

CorDbg is a managed debugger that ships with the .NET Framework SDK. It was not written in managed code since it was the first managed debugger written in the early days of .NET when you couldn’t yet rely on managed code to produce a debugger that will debug it.

CorDbg knows and understand .NET better than WinDbg+SOS can. That is why you can better debug .NET applications with it.
For example, A simple scenario that is hard to do with WinDbg+SOS and very easy to do in CorDbg is to break only when specific .NET exceptions happen.
When I say specific I mean SPECIFIC, i.e., I can specifically specify that I want to ignore System.ThreadAbortException but not System.FileNotFoundException.

To do that all I need to do is run the following commands:

catch exception System.FileNotFoundException
ignore exception System.ThreadAbortException

WinDbg can only break (without adding sophisticated scripts to further analyze an exception) on CLR exceptions since all CLR exception has the same exception code.

Some more cool and useful things CorDbg can do includes:

  • Easily set a breakpoint in managed code (and we all know its not that nice to set a managed breakpoint in WinDbg using SOS) using the b[reak] command:b foo.cs:42 – will break in file foo.cs in line 42.
    b MyClass::MyFunction – will break at the start of MyClass::MyFunction
  • Disassembling to native or IL code using the dis[assemble] command.
  • Evaluate a function and using the f[unceval] command:f MyNamespace.MyClass::MyMethod – for static method
    f MyClass::MyMethod param1 param2 $result – for instance method. The result is stored in $result for further evaluations
  • Printing variables to see their content using their names (for local variables, only if debug symbols are available) using the p[rint] command instead of finding out their addresses to print them in WinDbg:
    p obj1.var1 – for variables inside objects
    p MyClass::MyStaticVar – for static variables
    p arr[2] – for print a specific cell in an array
  • Changing a variable value using the set command.
  • Setting the next line of execution by simply invoking setip [line number] (like you do with right-clicking in Visual Studio .NET)

As you can see, using a managed debugger like CorDbg can be a lot nicer and easier instead of WinDbg.

I heared unsubstantiated rumores that when Microsoft developers have problems with a certain build of Visual Studio they use CorDbg to figure out the problem or to debug instead of finding what the problem with Visual Studio is.

Visual Studio .NET 2005 will contain a new managed debugger that was written in managed code called MDbg. You can get it here (it required .NET Framework 2.0). MDbg has an extensions API so you can extened it just as you can extened WinDbg (but its even easier). It also features a nice simple GUI for those of you who doesn’t like to use the keyboard too much.

You can read more about MDbg in Mike Stall’s blog

I will dedicate another post to MDbg so stay tuned.

Yesterday my RSS client, RSSBandit, decided to stop working.
Luckly RSSBandit is a .NET application with source code, so I’ve decided to start debugging it.
It kept on failing when it tried to deserialize one of its configuration classes from Xml using the XmlSerializer class.

The XmlSerializer class generates a temporary assembly in its constructor according to the XmlMapping it receives, compiles it and loads it.
Sometimes there is a problem at compile time. The code is not actually compiled and you will get a FileNotFoundException exception just for calling “new XmlSerializer(typeof(MyClass))“.

While trying to resolve this issue I’ve stumbled upon two tools that can help you better understand the cause of the problem and at some scenarios might suggest a resolution.

The first tools is a command line too called XmlSerializerPreCompiler written by Chris Sells. You just specify the full path to the assembly and the type name to check and it will try to create an XmlSerializer passing the type to it and analyzing the error.

The second one is a GUI tool that was based on XmlSerializerPreCompiler called XmlPreCompiler written by Mathew Nolton.

Both tools comes with full source code.

While I didn’t have time to check their source code I did see one of the things they use to get more information about the temporary assembly that is being compiled on the fly when creating the XmlSerializer.
It seems that there is a configuration flag that you put in the app.config file (if you don’t have one, just create one that is named myprogram.exe.config, where myprogram is your executeable name).

Just add the following to your app.config file:

<configuration>

<system.diagnostics>

<switches>

<add value=”4″ name=”XmlSerialization.Compilation”>

</switches>

</system.diagnostics>

</configuration>
This option tells the XmlSerializer not to delete the temporary files created to generate the temp assembly. It will leave the .cs file it was trying to compile, a .cmdline file which has the command line parameters that are sent to the compiler and a .out file which has the full command line that was used to compile the .cs file.

All of these files are generated it your %USERPROFILE%\Local Settings\Temp folder (which is usually equivalent to %TEMP%) and by just adding the above flag in your app.config you can get the input that is used in the programs I’ve mentioned above and investigate on your own.

While all of these programs couldn’t help me in resolving what is wrong with RSSBandit (and for that matter, any thing that has the “new XmlSerializer(typeof(MyClass))” code in it that runs on my machine) I’d thought it would be nice to share this information with everyone.

Since I don’t have time to further analyze this and I need my daily RSS feed dose, I did the following things:

  1. Ran WinDbg and started RSSBandit through it.
  2. Set a breakpoint (we have discussed about setting a managed breakpoint before) on System.CodeDom.Compiler.CompilerResults.get_CompiledAssembly (this function is the one that tries to load the temp compiled assembly and will throw the exception I’m getting in RSSBandit since the temp assembly wasn’t compiled).
  3. When the breakpoint hit (and it does so a few times until RSSBandit finishes loading) I ran !dumpheap -type CompilerResults
  4. I dumped the CompilerResults object using !dumpobject
  5. I then took the address of the pathToAssembly member and used !dumpobject on it as well.
  6. I then took the path, went there, opened the file named [strange random chars].out (where [strange random chars] is a unique name for each such assembly), opened it, copied the command line and ran it to generate the assembly.
  7. Continued doing this until we never reach the breakpoint again.

What I did learn from this workaround is that the main problem is executing the compiler command to compile the assembly and that will be the point from which I will continue to troubleshoot this issue.

At least for now RSSBandit is working (though only through WinDbg 😉 ) and I can read my favorite RSS feeds.

Thanks to Matt Adamson for pointing out to me that DebugDiag is now in release candidate stage (RC1) and is available for testing.

In a few words, DebugDiag is a very sophisiticated scriptable and extendable tool that enables you to debug server applications (mainly ones that are related to IIS, but not only limited to them).

Some of its features include scheduled dump taking using DbgSvc (or whatever action you required) as well as advanced memory leak checks using LeakTrack that eventually generates web reports.

I used it when I was in a Microsoft lab a while back mainly to use its scheduled dump taking. I configured it to take dumps every X minutes and it was great.
One of the SIE engineers also used its LeakTrack to find some memory problems in our application.

It a cool tool and a great addition to your debugging tools arsenal.

You can find all the necessary instructions on how to get it from here.
You will find there the tool itself and a PowerPoint presentation about the tool and how to use it.

In a server application even the smallest leak can grow fast to become a major issue.

In a pure unmanaged world, finding these leaks can be challanging but there are more than a few ways of doing so (perhaps I shall discuss this in a later post), but in .NET, with the help of our faithful friends WinDbg and the SOS extension, we can easily find and eliminate these leaks in no time if we stick to a proven and useful methodology.

As you already know, in pure managed code there are no memory leaks in the traditional manner, you can’t forget to free something since we are in a garbage collected environment. What you can do, however, is to forget to unreference an object in which case the GC will never consider it garbage and will never collect it.

Common situations include having some static variable (Shared variable for our VB.NET audiance) that is a collection of some type (usually a Hashtable) that you add various objects to it.
Since its a static variable it is rooted and will never get unreferenced and if we don’t explicitally remember to remove the objects we have added to that collection we have a managed memory leak.

So how can we overcome this problem?

Well… there are three ways.

  1. You can go the money way which involves buying one of the various .NET memory profilers available today (I can personally recommed on the fine and very easy to use .NET Memory Profiler by SciTech. Its an excellent tool and you get a 14 days fully functional evaluation and the new version also have a command line and an instrumentation API that lets you intiate the profiler in certain places in the code without a user intervention).
  2. You can go the programming way by implementing your own memory profiler using the .NET profiling API which involves implementing a COM interface and handling all the plumbing yourself (yet another post material for future posts 😉 and its even very interesting to do).
  3. Using WinDbg and the SOS extension in a certain methodology that will guarentee success.

Obviously, we are going to talk about option #3 in this post.

The tools and commands we are going to use are:

  • adplus.vbs – a sophisticated VBScript that was written by some fine SIE engineers at Microsoft Support that instruments the use of cdb and the process of taking memory dumps automatically. It comes with the Debugging Tools for Windows installation (the WinDbg installation).
  • WinDbg (duh!)
  • !dumpheap – An SOS extension command used to list the objects allocated on the managed heap.
  • !gcroot – An SOS extension command to find who is the object that is referncing the object we are currently checking.
  • !dumpobject – An SOS extension command to investigate a certain object and see what are its fields and who they reference.

Before we can start to work on WinDbg we need a set of dumps taken in a predefined and continous interval. If the memory leak is small we will use a bigger interval (i.e. 20min) if the leak is big we will use small interval (i.e. 1min).

To take the dump we will use the following adplus.vbs command line:

([UPDATE] 6/16/2005: Thanks to some anonymous reader that pointed my mistake. I WAS reffering for the use of -hang and NOT -crash option, which is good for other situations)

adplus.vbs -hang -p [process Id] -quiet

  • -hang tells adplus.vbs to attach the debugged process, take a memory dump and detach (this will only work in Windows 2000 and above, since the detach option is only availabe in Windows 2000 and above).
  • Replace [process Id] with the PID of the process we want to take its dump
  • -quiet will silence unnecessay message boxes that adplus might pop in certain cases.

After we have a series of dumps (usually 3 will suffice to show a trend) we can get down and dirty with WinDbg.

TIP: To save your WinDbg session that will contain all the commands you have executed and all the outputs you have received use the .logopen [file name] command after openning the memory dump and before starting to run any other command

Since our problem is memory increasing and its a pure managed application it means we are allocating objects and some object that is still alive is holding a reference to them making the GC not consider them garbage and therefore not collecting them.

To find them we will use the following set of commands:

  1. Run !dumpheap -stat to get a statistical view of all objects currently allocated in the heap. If the case is indeed objects that are allocated and never gets free you will find that certain classes will have a steady increasing amount of live instances over the course of the series of dump taken and this should point to the objects that we should take a closer look at. Click here to see a Sample Output.
    In our case, we see that we have 4500 instances of type MemoryLeakSample.MyObject which should turn a red light in our head since this is a good candidate of a leaking object.
  2. Run !dumpheap -type MemoryLeakSample.MyObject where “MemoryLeakSample.MyObject” is the full namespace and class name of the class type that we in the step above. This should give us a list of all the instances sorted by their addresses. Since we are in a GC environment and the .NET GC compacts the heap, the instance with the smallest address will be our oldest instance and we should focus on it. Click here to see a Sample Output.
  3. Run !gcroot 0x04ab8348 where 0x04ab8348 is the address of the oldest object (the first one in the list) that we saw earlier and we will get a full list of references showing who is holding who. In our case, since we have a static hashtable we will see something like “HANDLE(Strong):931c0:Root ….” which means that the parent (and true parent that is at the top of the reference list) is rooted, meaning it will never get freed unless the AppDomain will unload (in a multiple AppDomain scenario) or the process will die. Click here to see a Sample Output.

Now all we have left to do, is to find out where in the code we have defined this static variable and either NOT use a static variable (the best option, but not always possible) or make sure we clean this collection.

This methodology can be used to find not only rooted objects, but a big increase in instances of a ceratin type over a period of time and see who is holding them.

The best thing about this methodology is that you can do it at a customer’s site without a problem and finding 80% of the problem without having your symbols and/or your code.

Happy leak hunt!

Setting a break point in native code using WinDbg is easy.
You just run the bp command with the address of the place in memory where you want to place the breakpoint.


Setting a breakpoint in Managed code is a bit trickier.


First of all, WinDbg is a native debugger. Until a method is JITed (JIT as in Just In Time Compilation) it doesn’t even have native code.
The managed assembly might be loaded and mapped, but only after the function in JITed and its native code is generated and loaded it has a place in memory that WinDbg can set a breakpoint on to.

An exception to this are Ahead of Time (AOT) compiled assemblies (assemblies which have a native image generated by ngen.exe).

The steps for setting a breakpoint in Managed code are very simple:

  1. Find the method table handle of the class.
  2. Find the method description handle of the method
    we want to put a breakpoint on.
  3. Place a breakpoint on the virtual address of method we want
    to break into (this will only work on already JITed method whether its from AOT
    or they were called once to actually get JITed).

OK, now that we understand the logical steps on how to do it, what do we actually do? (I’m using the same executable from the previous post).

  1. !name2ee [Assembly name (including extension] [Class Full Namespace]. For example: !name2ee SyncBlkDeadLock.exe SyncBlkDeadLock.Form1. That is the class on which we want to place a breakpoint in one of its methods. The output will look like this.
  2. !dumpmt -md [MethodTable handle that we got from the previous command]. For Example: !dumpmt -md 0x00a8543c. The output will look like this.
  3. !dumpmd [MethodDesc handle that we got from the previous command]. For Example: !dumpmd 0x00a853d8. This is the handle for the method
    SyncBlkDeadLock.Form1.Thread1Handler().The output will look like this.
  4. In the field “Method VA” we now have th method Virtual Address and we can set a breakpoint on that address.

Quite easy, when you know what you are looking for 🙂

Remember, that this only sets a breakpoint on the function entry address, so whenever this function is called it will break (unless you point some kind of a conditional breakpoint).

Enjoy!