Archive for the 'WinDbg' Category

Debug.Assert lets you check various things during the debugging process or when you are a debug build.

One of the problems with Debug.Assert is that it pops up a MessageBox dialog. UI dialogs are a big no-no in server applications since there are scenarios in which this UI dialog will never be show and your server application will simply get stuck waiting for someone to push the buttons of the dialog.

One of the most common cases in which you will not see the dialog is when your server application runs under different credentials than your currently logged on user. In that case the UI dialog will hide somewhere in a different sessions without any chance of being seen by any other user.

.NET employs various huristics to figure out when to display the assert dialog. In most cases, in a server application it will not show the assert dialog at all (even in debug builds). As far as I could tell (and its currently just an calculated guess), if you have a debugger installed and registered as a debugger, .NET WILL display the dialog and that may pose a problem if your server application runs under different credentials than the currently logged on user.

It seems there is a way to overcome this by using a configuration flag.

Add the following section to your web.config (or app.config for applications other than ASP.NET) and it will disable poping up a UI window in your application which make cause it to deadlock:

<system.diagnostics>
<assert assertuienabled=”false” />
</system.diagnostics>

If you have any trace listeners that might popup a UI window you can also add the following configuration in your web.config that will clear all trace listeners:

<system.diagnostics>
<trace>
<listeners>
<clear/>
</listeners>
</trace>
</system.diagnostics>

If you are not sure this is really your problem, its very easy figuring it out using WinDbg, by attaching to the process, loading the SOS.dll and running the command:

~*e!clrstack

This command will show the managed call stack of each thread in the application (some of these threads are not managed so they might return an error and you can safely disregard them).

If you’ll see in the call stack a call to “Debug.Assert” followed by “MessageBox.Show” you can be certain that this is the problem you are facing.

I’ve just been asked through the “Ask” widget on the side of this blog on Yedda (my day job) this question:

Yedda � People. Sharing. Knowledge.Where is sos.dll for .net 3.5?

I installed Visual Studio 2008, but I don’t see sos.dll in the Microsoft.NET\Framework\v3.5 directory? What’s up with that?

Topics: , ,

Asked by dannyR on January 03, 2008

View the entire discussion on Yedda

To which I replied:

Yedda � People. Sharing. Knowledge.Where is sos.dll for .net 3.5?

There is a bit of confusion as to the .NET Framework versions as opposed to the CLR (Common Language Runtime) versions.

Up until .NET Framework 2.0, the CLR versions advanced side by side with the versions of the framework.

Starting from .NET Framework v3.0, the version of the CLR stated at 2.0 (there was no significant change in the CLR itself) and the framework’s version kept on going forward.

.NET Framework v3.5 actually runs on CLR 2.0 SP1.

This means that you can should be able to use the SOS.dll located at Microsoft.NET\Framework\v2.0.50727 with either Visual Studio (as I’ve explained on this post) or with WinDbg.

Dave Broman of Microsoft has written a post about the mapping of the versions of the .NET Framework to the version of the CLR. Do mind that as Dave states in his post, its the way he figures things up in his head and not the official documentation on this matter.

Topics: , ,

Answered by Eran on January 03, 2008

View the entire discussion on Yedda

I know its a bit confusing and I’m not really sure why Microsoft detached the versions of the Framework from the versions of CLR (well, I can understand some of the thoughts leading to this but I don’t think it was that good of a move) but Dave’s excellent post does put things in perspective and allows figuring out in what version of the CLR you are actually using.

Sometimes in ASP.NET when you are trying to access a certain page you might get the following error:

External component has thrown an exception

In some other situations in .NET you might get this error as well (not directly related to ASP.NET, but sometimes to Interoping code or COM code), but this is beyond the scope of this post.

In the cases that I’ve stumbled upon, the reason is usually related to on-the-fly compilations ASP.NET performs when first accessing a certain page on an application.

The main problem with this exception that it is less than informative 🙂 . The only way of actually figuring out what is wrong (if this is indeed a case related to compilation error like I’m talking about) is to dig deep using WinDbg.

Luckily, every compilation that runs inside ASP.NET uses an API that upon encountering an error will create an object in the heap called CompilerError.

To debug this issue perform the following steps:

  1. Attach WinDbg to the ASP.NET worker process (w3wp.exe in Windows 2003, aspnet_wp.exe in Windows XP/2000)
  2. Make sure you have CLR exceptions enabled (see this previous post about how to enable CLR exceptions in WinDbg).
  3. Try to access the page that is not working, it should throw an exception and WinDbg should break in.
  4. Run the following command:

    !dumpheap -type CompilerError

    This should produce a list of all CompilerError objects which should allow us to inspect them and see the exact compilation error.

  5. Now dump each of the objects using the following command:

    !dumpobject 0xXXXX

    Where 0xXXXX is the address of the objects listed in the list you got from the !dumpheap command.

  6. At that point you should see the CompilerError object itself and you can dump its errorMessageText member (it’s address is near its name) using !dumpobject which should show you the exact compilation error.

In my case, the problem was that in the web.config file under the <compilation> / <assemblies> element there was no reference to one of the AJAX.NET assemblies (System.Web.Extensions) which caused the page to fail during on-the-fly compilation.

Do remember that this exception sometimes has nothing to do with compilation. If you don’t have any object of type CompilerError or there are a few but they make no sense in regards to the page you are accessing it might be something else.

If you get the following error:

System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.

Make sure that you are closing your connection. Specifically closing connections that you opened transactions from.

It seems the Connection Pool implemented in ADO.NET has two lists. One for connections that you did not open transaction from and the other for connections that a transaction was opened (and did not close, yet).

The two lists share the same amount of connections set in the “Max Pool Size” parameter passed in the connection string (the default is 100).

Be sure to use one of the following patterns and make sure to call connection.Close() when you are done with running your SQL statements, otherwise, it will take until the next run of the Finalizer thread to call Dispose on your connection object which will return it to the pool.

If your application is loaded it might take quite a while until the Finalizer thread runs and by then you will exhaust your connection pool and start getting mentioned above.

The two patterns used in this scenario are:

using (IDbConnection = new SqlConnection()) {

DoSomething();

}

The “using” statement is only good for C# and it will call Dispose (which in turn will call Close) on your connection object.

Another pattern which is similar and gives you more control is this:

IDbConnection = new SqlConnection();

try {

DoSomething();

}

finally {

connection.Close();

}

The finally block gives you better control and can also be combined with a “Catch” block that you can gracefully handle various errors such as “Unique Key Violation” (if you application logic needs this kind of treatment).

If you really want to know that you have exhausted your connection pool and you forgot to run PerfMon (see my previous post about PerfMon) you can take a memory dump using adplus (you can read a little bit about adplus here) and use WinDbg and the SOS extension to look at the Connection Pool object.

Search for “ConnectionPool” and you’ll get various objects, one of them will be “DbConnectionPoolGroup” which has a property named “_poolCount” which can tell you the count of connections used on your pool. This will give you a rough estimate as to what is the status of your connection pool.

 

I received a question today through my Yedda Ask Widget (that’s the nice little box on the right that you are more than welcome to use to ask me questions directly and have all the knowledge available on Yedda. Disclosure: Yedda is my day work 🙂 )

greenaj asked why the “-gen” parameter of the “!dumpheap” is missing from the SOS.dll that shipped with .NET Framework 2.0 (I’ve previously talked about the parameters that one can use with “!dumpheap” here, but its for the SOS that comes with WinDbg and is only good for .NET 1.1)

It seems that the SOS.dll that shipped in .NET 2.0 has, in some areas, less functionality than its sibling – SOS of .NET 1.1 – that is being updated regularly with every WinDbg version.

I did offer greenaj another way of doing that same thing without the “-gen” parameter.
It is based upon combining “!eeheap -gc” (which shows all the addresses of the various memory segments of each generation) and using the “start” and “end” command parameters of “!dumpheap“.

First, run “!eeheap -gc” and get the following output:
0:014> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x013f694c
generation 1 starts at 0x013cb21c
generation 2 starts at 0x01391000
ephemeral segment allocation context: (0x01406de8, 0x01408970)
segment begin allocated size
001b2da0 7a721784 7a74248c 0x00020d08(134408)
00197dc8 7b451688 7b467f9c 0x00016914(92436)
001847b0 790d6358 790f5800 0x0001f4a8(128168)
01390000 01391000 01408970 0x00077970(489840)
Large object heap starts at 0x02391000
segment begin allocated size
02390000 02391000 0239a130 0x00009130(37168)
Total Size 0xd7564(882020)
——————————
GC Heap Size 0xd7564(882020)

The bold and underlined number, for example, shows that generation 2 start at the address of 01391000. At the list of segments below we can see that we have a segment who’s address starts at 01391000 and ends at 01408970.

Now we can run the command “!dumpheap 01391000 01408970” and see all the objects in this segments which are, in fact, all generation 2 objects (at least in this sample application).

I just hope newer versions of SOS for .NET 2.0 will be released more frequently with newer version of WinDbg, like they did with the SOS for .NET 1.1.

Mike Taulty (which, by the way, has a great blog) wrote a short tutorial on how to figure out what the .NET Framework code does when you suddenly get an exception and you don’t really know why it is being thrown from the framework.

He also shows how to load our favorite WinDbg Extension – SOS – into Visual Studio and use it while debugging as well as use Reflector to disassemble the framework’s code into readable C# or VB.NET code.
I also briefly talked about loading SOS into Visual Studio a while back and mentioned the use of Reflector all over the place (because its such a great and useful utility!).

Mike, if you are reading this – Yes, I also do what you do and I suppose almost everyone who actually wants to know what is happening under the hood in the .NET framework 🙂

First of all, let me apologize for the lack of activity in the blog. I’ve been abroad for about a month and that’s why I was slow on responding to comments (which I did for most of them, so go back and check it out) as well as posting here.

OK… Now that the sorry execuse is behind us, let’s get back to business from now on…. 😉

A reader named Kipp left some comments on my “Ambiguous match found” post. He had some problems making WinDbg break on CLR exceptions (like the ones generated by the compiler during the ASP.NET compilation of the page that eventually causes the “Ambiguous match found” error).

Kipp also asked the question on Yedda (disclosure: Yedda is my day time job) and I answered him.

I think this is an important piece of information that should be shared with everyone, so here it goes – How to enable and disable CLR exceptions breaking in WinDbg.

First, a little background about Structured Exception Handling (SEH) in Windows and the CLR.

CLR Exceptions in SEH World
In the regular world of Structured Exception Handling (SEH) each exception has its own specific exception code. All CLR exceptions are seen in the regular Windows world as the same type of exception since they all have the same error code.

The additional information of the exception is stored in the managed world and that is why you need the SOS extension to print it our, read it and see it.

WinDbg has a built in ability to filter in or out various exception codes and sometimes, the CLR exception code is not enabled causing CLR exception not to break in WinDbg.

Enabling/Disabling CLR exceptions in WinDbg

  1. Open WinDbg and make sure you are breaked into a program that you attached or run (if you need to break when attached simply press Ctrl-Break).
  2. Go to the “Debug” menu and select “Event Filters…”.
  3. Select “CLR exception” and click on the “Enabled” radio button in the “Exception” area.

UPDATE: Dan McKinley was kind enough to remind me that there are shortcuts that you can type in the WinDbg prompt to do what I’ve written above. To enable CLR exceptions type “sxe clr” and to disable type “sxd clr“. To view the list of exception types and their current status type “sx“. Thanks Dan!

That’s all, you are now good to go and any .NET exception that will be thrown (even ones that are caught eventually in the code) will break in WinDbg.
To disable, simply selected “Disabled” instead of “Enabled”.

I hope it will help you all!

There is a new WinDbg version (6.6.7.5) available here (it was released yesterday – July 18th).
You can check out some of the new stuff here.

Be careful though, the previous version had more than a few bugs and problems. Hopefully, this version fixed some (if not all) of them.

I still haven’t had much time to check it more thoroughly, but you are more then welcome to try it out, just don’t forget to report back! 🙂

Update: I’ve checked whether there is a new verison of SOS for .NET Framework and there isn’t any. There is only a new SOS version for .NET Framework 1.1 (which was compiled about 11 days ago).
Anyhow, this is kind of annoying. I’m really looking forward to get a new version of the .NET 2.0 SOS with at least all the features that were in the 1.1 version and are missing in the 2.0 version.

On my previous post about the dubious “Ambiguous match found” exception, I talked about the fact that the ASP.NET runtime is case sensitive and if you are writing in C# you better be careful about how you name your private, protected and public variables.

Not so long ago, I’ve encountered another problem which is relatively similar. It manifests itself in the runtime but it is actually related to the ASP.NET compiler.

When first accessing an ASPX page in an application after the server was started, the ASP.NET runtime will try to compile that page. One of the first steps in doing so is to parse the ASPX page and figure out what controls were placed on it. It will then create these controls and the Page object behind the page and render the page outputting the result back to the client.

When the ASP.NET runtime tries to create the various controls placed on the ASPX page, it will use the GetType function, but for some reason, it will only pass the name of the control and not its full namespace.

In my case, we had a custom user control named HtmlAnchor which was suppose to overcome a few quirks with HTML anchors in some browsers. ASP.NET already contains a control named HtmlAnchor which its full namespace is System.Web.UI.HtmlControls.HtmlAnchor.
Our control, had no namespace, since we simply created a user control in our application.

Even though we didn’t use even a single System.Web.UI.HtmlControls.HtmlAnchor, but only our custom HtmlAnchor control we got a strange error about not being able to load a certain .NET assembly which was suppose to be generated when the page first renders to produce the page’s code cache.

I fired up WinDbg and when the exception occurs, search for objects of type CompilerError by issuing:

!dumpheap -type CompilerError

This will list all CompilerError objects in the heap. After that I issued:

!do 0xXXXXXX
Where 0xXXXX is the address of the CompilerError object that I’m inspecting (!do = !dumpobject)

And inside there is a private member that contains the error text which I also dumped (using !do).

The error that I saw was that .NET cannot cast HtmlAnchor (my HtmlAnchor) to System.Web.UI.HtmlControls.HtmlAnchor.

I issued “!clrstack -p” to see the call stack and the parameters that were passed to each function and along the way there was call to GetType with the name parameter that was equal to “HtmlAnchor”.

This was the problem. For some reason .NET figured out that my HtmlAnchor is suppose to be the HtmlAnchor the resides in System.UI.HtmlControls.

As it turns out, the namespace System.UI.HtmlControls is ALWAYS included when the ASP.NET compiler compiles an ASPX page and, for some reason, request the type of the object WITHOUT its full namespace and find the wrong HtmlAnchor even though I have a different TagPrefix for my HtmlAnchor in the ASPX page.

The is a combination of a number of problems.
First, when adding a user control to an ASP.NET web application, there is no specific namespace in the file. It will eventually get the namespace of ASP.XXXX when compiled using the ASP.NET compiler like the rest of the ASCX and ASPX pages.

I haven’t checked it for user controls that are not ASCX, meaning, someone wrote a control in some DLL and does get a namespace. The difference is that when including that control on an ASPX page we do provide the exact namespace and assembly, so it might works without a problem.

Second, if the user control has the same name as any of the control that are in the System.Web.UI.HtmlControls and System.Web.UI.WebControls namespaces, they will, for some reason, get loaded in GetType instead of your controls.

Third, in the @Register tag at the top of the page, where we declare and include our user control (we include the ASCX file), there is no additional information such as namespace to help the ASP.NET compiler resolve this issue.

Conclusion:

Be careful when you name your User Controls (ASCX controls) and make sure they do not have the same name as other controls in System.Web.UI.*.

I know I haven’t been posting much lately, but I’m a bit caught up in a startup I’m involved in called Yedda. It has taken most of my time, but I do find, once in a while, something to post like the “Ambiguous match found” problem, so bear with me 🙂

Today, I’ll give a small tip for !clrstack.

If you use !clrstack to show a managed call stack and you want to see the addresses of parameters that were passed to the functions without doing a lot of fancy shmancy WinDbg tricks simply run:

!clrstack -p

That’s all you need to do. This will list each function in the call stack and below it the name of the parameter and its address. From there on, you can use !do ( !do == !dumpobject – which shows a managed object) and see the content of the object.

Quite simple and very effective.

As a general rule of thumb, most SOS command have a very descriptive help. To use it, simply run:

!help <command>

i.e.

!help !clrstack