Archive for the 'WinDbg' Category

I usualy try to stay away from posts that have only links to other blog posts but, Tess has a great post about various issues causing .NET memory leaks.
I’ve talked about these issues before here and here and here.

It’s nice to see that others from Microsoft are sharing their knowledge and expertise on the subject of debugging with the rest of the world (specifically, when it comes to production debugging, hard .NET debugging problems and WinDbg :-) ).

Enjoy!

I’ve discovered something that might be a bug in some way.

In one of our web applications, one of the developers wrote a user control (ASCX). The ASCX file contained a DIV with the “runat=server” attribute named (just of the sake of the sample) “X”. The ASCX.cs file contained another property called “x” (mind the case change!).

Since the compiler is case sensitive, there was no problem with this and the whole thing compiled and run, specifically it run from Visual Studio while debugging, the problem started when we precompiled the application.

In our build process, we precompile the web applications using the ASP.NET Compiler (through MSBuild, of course ;-) ) and since there was no compilation problem, everything went fine and the build succeeded.

We put the new build in a dev machine and started working with it. When we reached a page that used that control we got the “Ambiguous match found” exception. After searching for it in the newsgroups I found that it is a reflection related problem, so I fired up my faithful WinDbg and dived in.

I attached to the ASP.NET worker process and it broke in when the exception was thrown. I looked at the call stack (using !clrstack) and saw a call to “System.RuntimeType.GetField”, a reflection function used to get a field of an object.

OS Thread Id: 0xf38 (14)
ESP EIP
019eecb0 77e55dea [HelperMethodFrame: 019eecb0]
019eed54 79617ef1 System.RuntimeType.GetField(System.String, System.Reflection.BindingFlags
.
.
.

I run “dds 019eed54″ (this address is the address of the stack frame) to see what’s on the stack and get the addresses/values of the two parameters in the function.

019eed54 0263588c
019eed58 00000001

The first two lines (shown above) contains the addresses/values (depending on the type and number of parameters passed to the function, in our case we have only two) of the function. The second column contains the addresses/values. In our case, since the first parameter is a string, the value 0263588c is actually the address of a string, so I used !dumpobject to see its value and saw that it was “x”.

The second parameter is an enumeration, so the value represents the integer value of that enum flag. I looked in Reflector and saw that the value 1 belongs to BindingFlags.IgnoreCase.

At that point I still didn’t figure out the problem, so I went back to reflector and took a look at the precompiled assembly that this code was compiled into and then saw that I had two protected fields, one named “X” and one named “x”.

Bingo! that’s the problem, staring me back from the screen.

To sum things up, while this code is correct compiler-wise (two parameters with the same characters but different casing are two different parameters), the implementation that tries to access some of it uses a case insensitive reflection search, which causes the bug.

Until I get more information from Microsoft, I suggest making sure in any ASP.NET application that is precompiled to avoid having the same names with different casing to variables in the ASPX/ASCX page and its code behind.

Yesterday I got a question to my Email from David Douglass, a reader of this blog.
David asked me how can he see the threads waiting on a certain managed lock and that he was trying to accomplish that using WinDbg and the SOS that shipped with .NET Framework 2.0.

I told David that he should use the !SyncBlk command and that I talked about it in a previous post but never actually used it in .NET Framework 2.0. I also suggested that if he is still having problems I’ll look into it a bit more using the sample I provided in the previous post.

Before I had a chance to do anything, David replied saying that he checked this issue with Tess Ferrandez (Tess is an Escalation Engineer in the PSS (Product Support Services) at Microsoft) and that the answer he got was that the version of SOS shipped with .NET Framework 2.0 does not have the ability to show on each managed lock which threads are waiting on it like the SOS that shipped in newer version of WinDbg.

When typing !help SyncBlk in WinDbg (after load the SOS extension of .NET 2.0 of course) the help states that there is a way to verify that, but it requires a bit of disassembling.

I think I found a better way of doing that and I’ll present it here. The code is based on the sample from the post about identifying dead lock in managed code and you can download the source code from here.

The sample I use here basically deadlocks two thread which are both waiting on resources held by the other thread.

Runing !SyncBlk shows me this:

Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
12 0017a6a4 3 1 0018a1b0 efc 5 01371c38 System.Object
13 0017a6d4 3 1 00189df0 19f4 4 01371c2c System.Object
—————————–
Total 13
CCW 0
RCW 0
ComClassFactory 0
Free 1

We see here that thread “efc” is the owner of the lock on SyncBlock 01371c38 and thread “19f4″ is the owner of the lock on 01371c2c.
If we want to check which lock thread “efc” is trying to acquire all we need to do is run “~~[efc]kv” (do mind that I use kv to see the arguments being passed to the methods) and we should look for the method “mscorwks!JIT_MonEnterWorker_Portable“. If I’m not mistaken (and I do hope I’m not ;-) ), the first argument (the number showing on the “Args” column) should be the SyncBlock we are trying to acquire.

So in our case, on thread “efc” we should see that the first args of is “mscorwks!JIT_MonEnterWorker_Portable” 01371c2c.

I have not contacted anyone from Microsoft to validate my claims here (yet) but I will update the post if necessary.

Just a reminder: Do mind that when you will run the sample the addresses and thread number will probably be different.

If you are stuck in a problem or have a question, feel free to Email me to eran AT sandler . co . il and I’ll do my best to supply you with an answer.

According to this post in Jochen Kalmach’s blog, the new (beta) version of WinDbg – 6.6.3.5 that I’ve mentioned earlier in a couple of posts support mini real dumps.

This means that instead of taking a full memory dump (its not actually full because you don’t take the kernel space) which can be up to the size of the current process (and in server applications it can reach to a few hundreds of MB), you can take a small, light weight mini dump that contains only minimal information such as call stacks.

This previously worked in unmanaged environments but from this version it also works on managed and mixed environments.

I’m still trying to find the original source of this information (Jochen, can you help on this? :-) ), but after tracking Jochen’s blog for a while he does know what he is doing.

Jochen was kind enough (again) to post a comment in the previous post that there are at least 3 major issues with the new version of WinDbg.

You can find some of the issues here, here and here.

It seems that all problems stem from the fact that this version was not marked as beta.

I personally didn’t have time to test the new version with some custom extensions, but the responses I’ve been seeing in the newsgroups are a bit too aggressive.

I would like to send a bit kudos to all of the WinDbg team and congratulate them on their work.
If you know (or not), WinDbg is the most used debugger within Microsoft and with such a small team they are doing a heck of a job.

There is no need to panic too much and be too blunt about it. I’m sure each and every one of the team members of WinDbg will be more than happy to address any issues that may be caused in the new beta version.

This is a product like any other product and perhaps by moving to a releases model that is similar to the CTPs of the new parts in Vista would be a bit better.

Release more and release often to a small community that will help the team test and achieve a better test matrix coverage. No need to bitch about it if each and every one of us can lend a hand/computer/head/OS or two.

I just hope some of the WinDbg team members see this post… :-)

Jochen Kalmbach was kind enough (thanks Jochen!) to point in a comment to my previous post that release 6.6.3.5 is a BETA version of WinDbg.
He also pointed out this post in the WinDbg newsgroup that confirms this information.

Sorry for not finding this out just now, but it seems the only place that the word “beta” was written was in the “redist.txt” file in the installation folder.

It seems that some code Jochen wrote doesn’t work with the new version of DbgHelp.dll so there are a few things you should watch out in betas…
I personally like to live on the bleeding edge when it comes to WinDbg so I usually do care if a version is a beta or not. On the other hand, I can sympathize with Jochen’s frustration about using a new beta version and finding out something is broken…

Oh well… I’m sure this will get fixed very soon.

I just wish WinDbg releases were a bit more often than every 6 to 9 months.

A new version of WinDbg (version 6.6.3.5) was released a couple of days ago.
Download it from here.

You can read some of the highlights of the new version here.

For some reason, the only SOS.dll that comes with this release (as in previous releases) is for .NET Framework 1.1 only, so there is no new SOS.dll for .NET Framework 2.0 yet.

Since .NET 2.0 is released for about a month, I think its time for me to cover a few new commands that exists in the SOS.dll WinDbg extension that comes with .NET 2.0.

First of all, the current version of WinDbg – 6.5.3.8 – has an SOS.dll that can only work with .NET 1.1, so even if you do load it and try it out it will not find any .NET data in the processes since the internal .NET structures in .NET 2.0 have changed.

To start using the SOS.dll that comes with .NET 2.0 issues the following commands:

.load C:\<Windows Home Directory>\Microsoft.NET\Framework\v2.0.50727\sos.dll

If you are performing a live debug (you have attached to the process), in newer WinDbg versions you can use the following commands

.loadby sos mscorwks

This command will load the sos.dll near the mscorwks.dll that is currently loaded in the process.

Usually newer versions of WinDbg (I always recommend running the latest version of WinDbg ;-) ) contains a newer version of SOS.dll, so you should prefer to use the SOS.dll provided with WinDbg, although currently, the only SOS.dll that supports .NET 2.0 is located in the Framework’s directory, near the mscorwks.dll.

New and interesting commands

  • !dumparray – Dumps the content of an array.
    The .NET 1.1 version of SOS.dll had a !dumpcollection method which usually didn’t work for me, so it nice to have something that will dump arrays.
  • !printexception – If a parameter is given, it will dump that exception, if no parameter is given, it will dump the last exception on the stack.
    In .NET 1.1 version of SOS.dll one had to dump the stack objects, find one that is an exception and print it using !dumpobj. Now, all one has to do is call !dumpexception and the last managed exception on the current thread’s stack will be printed out.In addition to that, !dumpexception can now parse the _stackTrace field, which is a binary array if the _stackTraceString is not filled.

    !printexception will also notify you if there are nested exceptions (exceptions being thrown inside a catch handler). If you want to see those as well, use the -nested parameter.

  • !bpmd – Sets a breakpoint on managed code.
    Setting a breakpoint on a managed code prior to .NET 2.0 was very tricky. I covered this issue in a previous post.
    In that older post I only covered settings a breakpoint on an already JITed method. Setting a breakpoint on a non-JITed method is even harder. !bpmd will cover all of these aspects.To set the breakpoint all you need to do is provide the module name and method name, or the MethodDesc handle (if you have it).
    !bpmd also works on generic types. If a breakpoint is set on a generic type it will break on all derived types.

    The only way to set a breakpoint on managed code with ease before .NET 2.0 was to use CorDbg (which I covered in an previous post as well).

  • !StopOnException – stops on a particular managed exception.
    This functionality was only supported in CorDbg prior to .NET 2.0. Now, instead of breaking on every CLR exception you can specify a specific .NET exception type, such as System.ApplicationException.
  • !DumpIL – Dumps the IL code of a method.
    This was previously very hard to do. In .NET 2.0 one can always dump the IL code of a method (even at runtime).
    The main advantage here is to debug dynamic methods which are built on the fly. Its also useful for dynamically generated types (using Reflection.Emit).

As in previous version of SOS.dll, you can type !help <CommandName> to get detailed help and sometimes examples on that specific commands.

Though I’ve briefly mentioned in a previous post that WinDbg has a built-in Semi-Scripting language to instrument it a bit, I didn’t explained much about it, so here is a nice little trick that you can do with it.

As part of this semi-scripting language there are a few flow controls that are very useful for performing a specific opeartions on multiple items such as object addresses.
One such useful flow control is “.foreach” which enables one to run a command that returns a result and for each line that returns invoke another command on that specific line.

Let’s say you want to view the full content of all of the string objects in your application that are in Generation 2.

The long way to perform this is to run dump all Generation 2 string objects using “!dumpheap“, save the list of addresses on the side and run “!dumpobj” on each and every one of the addresses we got.

The quick way to do it would be to use the “.foreach” command like this:
.foreach (obj { !dh -type System.String -gen 2 -short }) { !do ${obj} }

What we are doing here is running “!dh -type System.String -gen 2 -short” which returns all of the Generation 2 String objects currently alive on the heap (!dh is the short form of !dumpheap).

Mind the “-short” parameter. It is essential for this to work since “-short” tells the “!dumpheap” command to return the object’s address, one in each new line, instead of the normal output of “!dumpheap“.

NOTE: “-short” was only introduced in newer version of the SOS.dll that comes with WinDbg, so be sure to catch the latest version if you want to use it.

The second part of the “.foreach” command expands the obj parameter which contains each new line that we recieved from the “!dumpheap” command and we expand it by using the “${obj}” syntax.

This will run “!do” (the short form command of “!dumpobj“) on each and everyone of the addresses that returned from the “!dumpheap” command.

Just a small and useful tip for debugging… Enjoy!

Dan McKinley Emailed me to tell me about a WinDbg Extension he wrote that interogates CLR information. In the process of writing it he also documented the process of its creation, thus producing a tutorial of how to create WinDbg extension.

It’s all written in a nice and tidy post on his blog.

What is most interesting about his extension is that it interogates CLR information, so it should be a good basis for people writing WinDbg extensions that perform operations on CLR information.

If you are in the process of thinking about writing a WinDbg extension that’s a good place to start.