Archive for the 'SOS' Category

Debug.Assert lets you check various things during the debugging process or when you are a debug build.

One of the problems with Debug.Assert is that it pops up a MessageBox dialog. UI dialogs are a big no-no in server applications since there are scenarios in which this UI dialog will never be show and your server application will simply get stuck waiting for someone to push the buttons of the dialog.

One of the most common cases in which you will not see the dialog is when your server application runs under different credentials than your currently logged on user. In that case the UI dialog will hide somewhere in a different sessions without any chance of being seen by any other user.

.NET employs various huristics to figure out when to display the assert dialog. In most cases, in a server application it will not show the assert dialog at all (even in debug builds). As far as I could tell (and its currently just an calculated guess), if you have a debugger installed and registered as a debugger, .NET WILL display the dialog and that may pose a problem if your server application runs under different credentials than the currently logged on user.

It seems there is a way to overcome this by using a configuration flag.

Add the following section to your web.config (or app.config for applications other than ASP.NET) and it will disable poping up a UI window in your application which make cause it to deadlock:

<system.diagnostics>
<assert assertuienabled=”false” />
</system.diagnostics>

If you have any trace listeners that might popup a UI window you can also add the following configuration in your web.config that will clear all trace listeners:

<system.diagnostics>
<trace>
<listeners>
<clear/>
</listeners>
</trace>
</system.diagnostics>

If you are not sure this is really your problem, its very easy figuring it out using WinDbg, by attaching to the process, loading the SOS.dll and running the command:

~*e!clrstack

This command will show the managed call stack of each thread in the application (some of these threads are not managed so they might return an error and you can safely disregard them).

If you’ll see in the call stack a call to “Debug.Assert” followed by “MessageBox.Show” you can be certain that this is the problem you are facing.

I’ve just been asked through the “Ask” widget on the side of this blog on Yedda (my day job) this question:

Yedda � People. Sharing. Knowledge.Where is sos.dll for .net 3.5?

I installed Visual Studio 2008, but I don’t see sos.dll in the Microsoft.NET\Framework\v3.5 directory? What’s up with that?

Topics: , ,

Asked by dannyR on January 03, 2008

View the entire discussion on Yedda

To which I replied:

Yedda � People. Sharing. Knowledge.Where is sos.dll for .net 3.5?

There is a bit of confusion as to the .NET Framework versions as opposed to the CLR (Common Language Runtime) versions.

Up until .NET Framework 2.0, the CLR versions advanced side by side with the versions of the framework.

Starting from .NET Framework v3.0, the version of the CLR stated at 2.0 (there was no significant change in the CLR itself) and the framework’s version kept on going forward.

.NET Framework v3.5 actually runs on CLR 2.0 SP1.

This means that you can should be able to use the SOS.dll located at Microsoft.NET\Framework\v2.0.50727 with either Visual Studio (as I’ve explained on this post) or with WinDbg.

Dave Broman of Microsoft has written a post about the mapping of the versions of the .NET Framework to the version of the CLR. Do mind that as Dave states in his post, its the way he figures things up in his head and not the official documentation on this matter.

Topics: , ,

Answered by Eran on January 03, 2008

View the entire discussion on Yedda

I know its a bit confusing and I’m not really sure why Microsoft detached the versions of the Framework from the versions of CLR (well, I can understand some of the thoughts leading to this but I don’t think it was that good of a move) but Dave’s excellent post does put things in perspective and allows figuring out in what version of the CLR you are actually using.

I received a question today through my Yedda Ask Widget (that’s the nice little box on the right that you are more than welcome to use to ask me questions directly and have all the knowledge available on Yedda. Disclosure: Yedda is my day work :-) )

greenaj asked why the “-gen” parameter of the “!dumpheap” is missing from the SOS.dll that shipped with .NET Framework 2.0 (I’ve previously talked about the parameters that one can use with “!dumpheap” here, but its for the SOS that comes with WinDbg and is only good for .NET 1.1)

It seems that the SOS.dll that shipped in .NET 2.0 has, in some areas, less functionality than its sibling – SOS of .NET 1.1 – that is being updated regularly with every WinDbg version.

I did offer greenaj another way of doing that same thing without the “-gen” parameter.
It is based upon combining “!eeheap -gc” (which shows all the addresses of the various memory segments of each generation) and using the “start” and “end” command parameters of “!dumpheap“.

First, run “!eeheap -gc” and get the following output:
0:014> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x013f694c
generation 1 starts at 0x013cb21c
generation 2 starts at 0x01391000
ephemeral segment allocation context: (0x01406de8, 0x01408970)
segment begin allocated size
001b2da0 7a721784 7a74248c 0x00020d08(134408)
00197dc8 7b451688 7b467f9c 0x00016914(92436)
001847b0 790d6358 790f5800 0x0001f4a8(128168)
01390000 01391000 01408970 0x00077970(489840)
Large object heap starts at 0x02391000
segment begin allocated size
02390000 02391000 0239a130 0x00009130(37168)
Total Size 0xd7564(882020)
——————————
GC Heap Size 0xd7564(882020)

The bold and underlined number, for example, shows that generation 2 start at the address of 01391000. At the list of segments below we can see that we have a segment who’s address starts at 01391000 and ends at 01408970.

Now we can run the command “!dumpheap 01391000 01408970” and see all the objects in this segments which are, in fact, all generation 2 objects (at least in this sample application).

I just hope newer versions of SOS for .NET 2.0 will be released more frequently with newer version of WinDbg, like they did with the SOS for .NET 1.1.

Mike Taulty (which, by the way, has a great blog) wrote a short tutorial on how to figure out what the .NET Framework code does when you suddenly get an exception and you don’t really know why it is being thrown from the framework.

He also shows how to load our favorite WinDbg Extension – SOS – into Visual Studio and use it while debugging as well as use Reflector to disassemble the framework’s code into readable C# or VB.NET code.
I also briefly talked about loading SOS into Visual Studio a while back and mentioned the use of Reflector all over the place (because its such a great and useful utility!).

Mike, if you are reading this – Yes, I also do what you do and I suppose almost everyone who actually wants to know what is happening under the hood in the .NET framework :-)

On my previous post about the dubious “Ambiguous match found” exception, I talked about the fact that the ASP.NET runtime is case sensitive and if you are writing in C# you better be careful about how you name your private, protected and public variables.

Not so long ago, I’ve encountered another problem which is relatively similar. It manifests itself in the runtime but it is actually related to the ASP.NET compiler.

When first accessing an ASPX page in an application after the server was started, the ASP.NET runtime will try to compile that page. One of the first steps in doing so is to parse the ASPX page and figure out what controls were placed on it. It will then create these controls and the Page object behind the page and render the page outputting the result back to the client.

When the ASP.NET runtime tries to create the various controls placed on the ASPX page, it will use the GetType function, but for some reason, it will only pass the name of the control and not its full namespace.

In my case, we had a custom user control named HtmlAnchor which was suppose to overcome a few quirks with HTML anchors in some browsers. ASP.NET already contains a control named HtmlAnchor which its full namespace is System.Web.UI.HtmlControls.HtmlAnchor.
Our control, had no namespace, since we simply created a user control in our application.

Even though we didn’t use even a single System.Web.UI.HtmlControls.HtmlAnchor, but only our custom HtmlAnchor control we got a strange error about not being able to load a certain .NET assembly which was suppose to be generated when the page first renders to produce the page’s code cache.

I fired up WinDbg and when the exception occurs, search for objects of type CompilerError by issuing:

!dumpheap -type CompilerError

This will list all CompilerError objects in the heap. After that I issued:

!do 0xXXXXXX
Where 0xXXXX is the address of the CompilerError object that I’m inspecting (!do = !dumpobject)

And inside there is a private member that contains the error text which I also dumped (using !do).

The error that I saw was that .NET cannot cast HtmlAnchor (my HtmlAnchor) to System.Web.UI.HtmlControls.HtmlAnchor.

I issued “!clrstack -p” to see the call stack and the parameters that were passed to each function and along the way there was call to GetType with the name parameter that was equal to “HtmlAnchor”.

This was the problem. For some reason .NET figured out that my HtmlAnchor is suppose to be the HtmlAnchor the resides in System.UI.HtmlControls.

As it turns out, the namespace System.UI.HtmlControls is ALWAYS included when the ASP.NET compiler compiles an ASPX page and, for some reason, request the type of the object WITHOUT its full namespace and find the wrong HtmlAnchor even though I have a different TagPrefix for my HtmlAnchor in the ASPX page.

The is a combination of a number of problems.
First, when adding a user control to an ASP.NET web application, there is no specific namespace in the file. It will eventually get the namespace of ASP.XXXX when compiled using the ASP.NET compiler like the rest of the ASCX and ASPX pages.

I haven’t checked it for user controls that are not ASCX, meaning, someone wrote a control in some DLL and does get a namespace. The difference is that when including that control on an ASPX page we do provide the exact namespace and assembly, so it might works without a problem.

Second, if the user control has the same name as any of the control that are in the System.Web.UI.HtmlControls and System.Web.UI.WebControls namespaces, they will, for some reason, get loaded in GetType instead of your controls.

Third, in the @Register tag at the top of the page, where we declare and include our user control (we include the ASCX file), there is no additional information such as namespace to help the ASP.NET compiler resolve this issue.

Conclusion:

Be careful when you name your User Controls (ASCX controls) and make sure they do not have the same name as other controls in System.Web.UI.*.

I know I haven’t been posting much lately, but I’m a bit caught up in a startup I’m involved in called Yedda. It has taken most of my time, but I do find, once in a while, something to post like the “Ambiguous match found” problem, so bear with me :-)

Today, I’ll give a small tip for !clrstack.

If you use !clrstack to show a managed call stack and you want to see the addresses of parameters that were passed to the functions without doing a lot of fancy shmancy WinDbg tricks simply run:

!clrstack -p

That’s all you need to do. This will list each function in the call stack and below it the name of the parameter and its address. From there on, you can use !do ( !do == !dumpobject – which shows a managed object) and see the content of the object.

Quite simple and very effective.

As a general rule of thumb, most SOS command have a very descriptive help. To use it, simply run:

!help <command>

i.e.

!help !clrstack

I usualy try to stay away from posts that have only links to other blog posts but, Tess has a great post about various issues causing .NET memory leaks.
I’ve talked about these issues before here and here and here.

It’s nice to see that others from Microsoft are sharing their knowledge and expertise on the subject of debugging with the rest of the world (specifically, when it comes to production debugging, hard .NET debugging problems and WinDbg :-) ).

Enjoy!

I’ve discovered something that might be a bug in some way.

In one of our web applications, one of the developers wrote a user control (ASCX). The ASCX file contained a DIV with the “runat=server” attribute named (just of the sake of the sample) “X”. The ASCX.cs file contained another property called “x” (mind the case change!).

Since the compiler is case sensitive, there was no problem with this and the whole thing compiled and run, specifically it run from Visual Studio while debugging, the problem started when we precompiled the application.

In our build process, we precompile the web applications using the ASP.NET Compiler (through MSBuild, of course ;-) ) and since there was no compilation problem, everything went fine and the build succeeded.

We put the new build in a dev machine and started working with it. When we reached a page that used that control we got the “Ambiguous match found” exception. After searching for it in the newsgroups I found that it is a reflection related problem, so I fired up my faithful WinDbg and dived in.

I attached to the ASP.NET worker process and it broke in when the exception was thrown. I looked at the call stack (using !clrstack) and saw a call to “System.RuntimeType.GetField”, a reflection function used to get a field of an object.

OS Thread Id: 0xf38 (14)
ESP EIP
019eecb0 77e55dea [HelperMethodFrame: 019eecb0]
019eed54 79617ef1 System.RuntimeType.GetField(System.String, System.Reflection.BindingFlags
.
.
.

I run “dds 019eed54″ (this address is the address of the stack frame) to see what’s on the stack and get the addresses/values of the two parameters in the function.

019eed54 0263588c
019eed58 00000001

The first two lines (shown above) contains the addresses/values (depending on the type and number of parameters passed to the function, in our case we have only two) of the function. The second column contains the addresses/values. In our case, since the first parameter is a string, the value 0263588c is actually the address of a string, so I used !dumpobject to see its value and saw that it was “x”.

The second parameter is an enumeration, so the value represents the integer value of that enum flag. I looked in Reflector and saw that the value 1 belongs to BindingFlags.IgnoreCase.

At that point I still didn’t figure out the problem, so I went back to reflector and took a look at the precompiled assembly that this code was compiled into and then saw that I had two protected fields, one named “X” and one named “x”.

Bingo! that’s the problem, staring me back from the screen.

To sum things up, while this code is correct compiler-wise (two parameters with the same characters but different casing are two different parameters), the implementation that tries to access some of it uses a case insensitive reflection search, which causes the bug.

Until I get more information from Microsoft, I suggest making sure in any ASP.NET application that is precompiled to avoid having the same names with different casing to variables in the ASPX/ASCX page and its code behind.

Did you know that you can use SOS from within Visual Studio 2005, not just from WinDbg?
It has been previously discussed in a couple of places but most people don’t know it is possible to do that.

“Why should I use the SOS extension from within Visual Studio?”, you are probably asking.
The answer is quite simple.

Visual Studio is a rich debugger that shows us most of the information we want to know but it doesn’t show us a few important things like the actual size of an object and all of its references and who is referencing our object. While these things are not straight forward in Visual Studio, they are very easy to accomplish in SOS. For example, you can run the following command in SOS:

  • !objsize – to find out the actual size of the object and all of its referenced objects
  • !gcroot – to find who is referencing our object
  • !dumpheap – to see what’s all the stuff that we have on the heap. By using the -gen [generation number] parameter with !dumpheap, we can see all the objects that are currently in that generation
  • !SyncBlk – find out which managed locks are locked at the moment (I’ve talked about it before)

So, how do we actually do that?

Well, the first thing you need to do is enable “unmanaged debugging” by going to the “Project Properties” -> “Debugging” -> “Enable Unmanaged Debugging”.

After doing that, you can stop somewhere in your program, open the “Immediate” window (Debug -> Windows -> Immediate) and type:

.load sos

This will load the SOS extension. From there on you have most of the commands of the SOS extension at your disposal.
You might need to set the symbols path by calling .sympath, just like you would do in WinDbg.

An added bonus to this whole thing is that it also works on Visual Studio 2003, so you get to use it even with .NET Framework 1.1.

Yesterday I got a question to my Email from David Douglass, a reader of this blog.
David asked me how can he see the threads waiting on a certain managed lock and that he was trying to accomplish that using WinDbg and the SOS that shipped with .NET Framework 2.0.

I told David that he should use the !SyncBlk command and that I talked about it in a previous post but never actually used it in .NET Framework 2.0. I also suggested that if he is still having problems I’ll look into it a bit more using the sample I provided in the previous post.

Before I had a chance to do anything, David replied saying that he checked this issue with Tess Ferrandez (Tess is an Escalation Engineer in the PSS (Product Support Services) at Microsoft) and that the answer he got was that the version of SOS shipped with .NET Framework 2.0 does not have the ability to show on each managed lock which threads are waiting on it like the SOS that shipped in newer version of WinDbg.

When typing !help SyncBlk in WinDbg (after load the SOS extension of .NET 2.0 of course) the help states that there is a way to verify that, but it requires a bit of disassembling.

I think I found a better way of doing that and I’ll present it here. The code is based on the sample from the post about identifying dead lock in managed code and you can download the source code from here.

The sample I use here basically deadlocks two thread which are both waiting on resources held by the other thread.

Runing !SyncBlk shows me this:

Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
12 0017a6a4 3 1 0018a1b0 efc 5 01371c38 System.Object
13 0017a6d4 3 1 00189df0 19f4 4 01371c2c System.Object
—————————–
Total 13
CCW 0
RCW 0
ComClassFactory 0
Free 1

We see here that thread “efc” is the owner of the lock on SyncBlock 01371c38 and thread “19f4″ is the owner of the lock on 01371c2c.
If we want to check which lock thread “efc” is trying to acquire all we need to do is run “~~[efc]kv” (do mind that I use kv to see the arguments being passed to the methods) and we should look for the method “mscorwks!JIT_MonEnterWorker_Portable“. If I’m not mistaken (and I do hope I’m not ;-) ), the first argument (the number showing on the “Args” column) should be the SyncBlock we are trying to acquire.

So in our case, on thread “efc” we should see that the first args of is “mscorwks!JIT_MonEnterWorker_Portable” 01371c2c.

I have not contacted anyone from Microsoft to validate my claims here (yet) but I will update the post if necessary.

Just a reminder: Do mind that when you will run the sample the addresses and thread number will probably be different.

If you are stuck in a problem or have a question, feel free to Email me to eran AT sandler . co . il and I’ll do my best to supply you with an answer.