Archive for July, 2005

[UPDATE – 9:30AM]
[Added some of the links from Rico’s post and some more input on them]

This is a bit off topic, but I’d thought I’ll still give it a post since, as we all know, the best way to debug is to not debug at all. Avoiding common pitfalls and understanding how things work can save hours of debugging work at a later time.

Anyhow, I’ve stumbled upon a post by Rico Mariani in his blog about Whidbey Performance Opportunities.

(SIDE NOTE: his blog is full of interesting things since his current position is in the CLR performance team. If there is someone that knows how to squeeze extra performance from the CLR that’s him)

This post talks about some of the new things Whidbey (Visual Studio 2005) has to offer as well as the new .NET CLR 2.0.

The post offer links to each subject with additional information such as:

  • Review by Jason Zander of NGen – Its a little old but give a good insight as to what this tool does and how and when to use it. Keep in mind, as stated in the post, that NGen in v1.0 and v1.1 was mainly created for usage of the runtime itself. It got revamped a bit in Whidbey so that it is much better.
  • NGen is only good for a specific version of the CLR. If a new service pack was installed, there is a need to re-NGen your assemblies, otherwise their native image will be ignored and they will be considered as standard .NET assemblies that were not NGen’ed. As part of WinFX (which will be released in Windows codenamed Longhorn, now named Windows Vista) there is a new service called Native Image Service that does this tedious task of figuring out that certain assemblies needs to be re-NGen’ed and does that. You can use ceratin attributes in an assembly to give it better understanding of what to actually do with your assembly. You can read more about it here.
  • Another great post by Rico Mariani titled Six Questions About Generics and Performance.
  • An excellent two pieces post by Maoni about using GC Efficiently, Part1 and Part2.
  • Dodge Common Performance Pitfalls to Craft Speedy Applications using Reflection by Joel Pobar on this month’s (July) MSDN Magazine issue. It talks about imporvments of the reflection implementation in .NET 2.0 as well as the usage of some new APIs that lets you do things more quickly.Read, understand, let it sink in and by the time you’ll start working with it you’ll be better prepared.

Yesterday my RSS client, RSSBandit, decided to stop working.
Luckly RSSBandit is a .NET application with source code, so I’ve decided to start debugging it.
It kept on failing when it tried to deserialize one of its configuration classes from Xml using the XmlSerializer class.

The XmlSerializer class generates a temporary assembly in its constructor according to the XmlMapping it receives, compiles it and loads it.
Sometimes there is a problem at compile time. The code is not actually compiled and you will get a FileNotFoundException exception just for calling “new XmlSerializer(typeof(MyClass))“.

While trying to resolve this issue I’ve stumbled upon two tools that can help you better understand the cause of the problem and at some scenarios might suggest a resolution.

The first tools is a command line too called XmlSerializerPreCompiler written by Chris Sells. You just specify the full path to the assembly and the type name to check and it will try to create an XmlSerializer passing the type to it and analyzing the error.

The second one is a GUI tool that was based on XmlSerializerPreCompiler called XmlPreCompiler written by Mathew Nolton.

Both tools comes with full source code.

While I didn’t have time to check their source code I did see one of the things they use to get more information about the temporary assembly that is being compiled on the fly when creating the XmlSerializer.
It seems that there is a configuration flag that you put in the app.config file (if you don’t have one, just create one that is named myprogram.exe.config, where myprogram is your executeable name).

Just add the following to your app.config file:

<configuration>

<system.diagnostics>

<switches>

<add value=”4″ name=”XmlSerialization.Compilation”>

</switches>

</system.diagnostics>

</configuration>
This option tells the XmlSerializer not to delete the temporary files created to generate the temp assembly. It will leave the .cs file it was trying to compile, a .cmdline file which has the command line parameters that are sent to the compiler and a .out file which has the full command line that was used to compile the .cs file.

All of these files are generated it your %USERPROFILE%\Local Settings\Temp folder (which is usually equivalent to %TEMP%) and by just adding the above flag in your app.config you can get the input that is used in the programs I’ve mentioned above and investigate on your own.

While all of these programs couldn’t help me in resolving what is wrong with RSSBandit (and for that matter, any thing that has the “new XmlSerializer(typeof(MyClass))” code in it that runs on my machine) I’d thought it would be nice to share this information with everyone.

Since I don’t have time to further analyze this and I need my daily RSS feed dose, I did the following things:

  1. Ran WinDbg and started RSSBandit through it.
  2. Set a breakpoint (we have discussed about setting a managed breakpoint before) on System.CodeDom.Compiler.CompilerResults.get_CompiledAssembly (this function is the one that tries to load the temp compiled assembly and will throw the exception I’m getting in RSSBandit since the temp assembly wasn’t compiled).
  3. When the breakpoint hit (and it does so a few times until RSSBandit finishes loading) I ran !dumpheap -type CompilerResults
  4. I dumped the CompilerResults object using !dumpobject
  5. I then took the address of the pathToAssembly member and used !dumpobject on it as well.
  6. I then took the path, went there, opened the file named [strange random chars].out (where [strange random chars] is a unique name for each such assembly), opened it, copied the command line and ran it to generate the assembly.
  7. Continued doing this until we never reach the breakpoint again.

What I did learn from this workaround is that the main problem is executing the compiler command to compile the assembly and that will be the point from which I will continue to troubleshoot this issue.

At least for now RSSBandit is working (though only through WinDbg 😉 ) and I can read my favorite RSS feeds.

The Monitor class in .NET is a pure managed implementation of a Critical Section. It is used to synchronize multiple threads’ access to a code block.

C#’s lock statement and VB.NET’s SyncLock statement are actually evaluated into a Try..Finally block that has Monitor.Enter at the start and Monitor.Exit inside the Finally block.

For example:

[C#]
lock(myObject)
{
// Some code requiring synchronization
}

[VB.NET]
SyncLock MyObject
' Some code requiring synchronization
End SyncLock


The actual code produced is:

[C#]
Monitor.Enter(myObject);
try
{
// Some code requiring synchronization
}
finally
{
Monitor.Exit(myObject);
}

[VB.NET]
Monitor.Enter MyObject
Try
' Some code requiring synchronization
Finally
Monitor.Exit MyObject
End Try

There are two major issues that should be considered when using the Monitor class (or C#’s Lock and VB.NET’s SyncLock):

  1. Monitor only works on Reference objects and NOT value types. This means you cannot use an Integer as the object to lock when entering a monitor because it wil be boxed and releaed right away, effectivly not synchronizing the code.
  2. Locking should always be performed on private or internal objects.

There are a great deal of samples floating around that locks the “this” (“Me” in VB.NET) object, which means you are using the current instance as the object that is used for locking.

If this object is a public one and is used or can be used by others (either inside your organization or outside, if you work in an organization to begin with 😉 ), if you lock this instance and that instance is also available to someone else, they might also use it for locking and essentially create a potential deadlock.

So, what should we do about that?

ALWAYS USE PRIVATE OR INTERNAL OBJECTS FOR LOCKS!

The best way to avoid a deadlock is to ALWAYS use private object for locking.

For example:

[C#]
public class LockClass
{
private static object myLock = new Object()
public function void DoIt()
{
lock(myLock)
{
// Some code block that is in
// need of a sync
}
}
}

[VB.NET]
Public Class LockClass
Private Shared MyLock As Object = New Object()

Public Sub DoIt()
SyncLock MyLock
' Some code block that is in
' need of a sync
End SyncLock
End Sub
End Class

Remeber these two guidelines mentioned above and you will save yourself a lot of time trying to find an elusive managed deadlock that is usually hard to reproduce.

This is a bit off topic, but since I’ve already mentioned the Performance Monitor (perfmon) I’d thought I’ll address a behavior that some find intriguing.

When collecting performance information one must define a new counter log that will collect data from all relevant counters. When adding .NET counters to this log and openning the log file after running it for some time, sometimes you will see that it does not log .NET counters data and you will see them as if there is no data available on them.

This happens due to the credentials of the user running the performance monitor service.

So, how do we resolve it?

The solution is simple but is different between Windows 2000 and Windows XP/2003.

In Windows 2000, the only way of doing so is to change the user running the “Performance Logs and Alerts” service from the Services console.

In Windows XP and Windows 2003 a bit more thought was given to this issue and you can set the specific user that will run this counter log (its per counter log!).

To do so, just add the user at the “Run as” section and it will also question you for the password (as you can see by the screen shot below).

Hope this helps, Enjoy!

This is a bit off-topic, but I thought I would recommend some books on .NET debugging and general issues that are related to that.

I’ve setup a page with links to Amazon (so you will get a full description of the book).

Some of the books I recommend personally (and I’ve written bit below the link). Some were recommended to me by others.

I’ll try to update this page with recommendations that I will find/get but if you feel there are some additional books that you know are useful, please contact me and I will add them to the list.

The link also appears on the right in the “Links” area.
Enjoy!

From day 1, .NET presented an easy to use ThreadPool that will handle all of your queuing/dispatching needs.

While ThreadPool is a cool thing that saves you a lot of time dealing with multi threaded issues, one must be careful when using it and understand how it works and who else is using it to avoid common pitfalls.

Who Uses the ThreadPool?

  • Any asynchronous delegate invocation (BeginInvoke/EndInvoke pattern)
  • HttpWebRequest uses it internally to queue and dispatch web requests.
  • WebRequest uses it to dispatch the request
  • Some timers (like System.Threading.Timer).
  • The end user that uses Thread Pool directly.

The Problem

Consider a situation in which you have one of the above filling up the Thread Pool with queued items. Each queue item was implemeneted in a way that also uses the ThreadPool (either directly or by an async invocation). We then reach to a point where the ThreadPool is full and all of its requests are blocking and waiting to get some work done using the ThreadPool.

Bang! You are deadlocked! You have circular dependency that code that runs inside a ThreadPool worker thread is blocking because it runs a code that also needs the ThreadPool, but the ThreadPool is already full and it is also waiting to get a worker Thread to run on to of it.

Reaching this situation is quite easy under high load since by default, the ThreadPool is set to have 25 threads per CPU (A Hyper Threaded CPU is considered as two CPUs by .NET and the Operating System and therefore a ThreadPool running on that CPU will have 50 threads, 25 per CPU).

The Solution

There is actually more than one way of avoiding this situation:

  • Query the ThreadPool using ThreadPool.GetAvailableThreads to see if you should queue the item or not. If you can’t, either throw an exception or wait until you can use it (just don’t wait if you are on a Thread Pool worker thread, i.e. you are running inside an ASP.NET page 😉 We are trying to avoid that). This is a similar approach that .NET takes internally.
  • Avoidance – Don’t call this code from within a place that uses the ThreadPool such as from within ASP.NET or from within an Async call.
  • Implement your own ThreadPool instead of using .NET’s ThreadPool – This is not as easy as it sounds, though.

There is an additional solution but its ONLY applicable in the coming .NET 2.0 which is to specifically call ThreadPool.SetMaxThreads. By doing so you will be able to increase the amount of threads available to the ThreadPool.

NOTE: Do mind that having too many threads per CPU is not recommended since the Operating System will spend a lot of time just making sure they are not starved.

As we can see, ThreadPool is a great thing that saves us the time and effort of implementing a multi threaded pool with all of its synchronization and other thread related issues but we still must understand that it has its limitation and we need to understand how it works and what it do to avoid unnecessary deadlocks and/or strange behavior.

A few posts ago I talked about a situation in which the finalizer thread is being blocked by a switch to an STA apartment when we are disposing an STA COM object without calling Marshal.ReleaseComObject.

In that post, I suggested using the sieextpub.dll debugger extension and call its !comcalls command that shows us which COM call is headed to which thread.

I just got an Email from Herve Chapalain (Herve, please let me know if you have a link that you want me to put here, or your Email address) about another way of getting the STA thread to which we are going to without using sieextpub.dll.

The method involves understanding the memory structure of a structure called OXIDEntry which, as far as I know (please correct me if I’m not) is an internal struct and does not appear in public symbols.

Fortunately, Herve is familiar with its memory layout and gave us the necessary offsets to use.

We first need to see the native stack of the finalizer thread (or in a case that we are not talking about a blocked finalizer thread and just want to find a deadlock regadless of .NET, the thread that is blocked) using the one of k commands.

The output should look something like this.
Notice that the call stack contains the GetToSTA call which means we do want to switch to the right STA thread that our object was created in.

We then take the 3rd parameter of the function “ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall” (you will see it in the call stack).
In Herve’s sample the address is 0016f6f4.

Next, we use the dd command to dump the memory at that address and we should be getting this:

0:005> dd 0016f6f4

0016f6f4 774a2328 774a22f8 00000003 0000002a
0016f704 001f5790 00000000 0016eec0 0016f978
0016f714 0017a120 01cca3e8 774a1930 00070005
0016f724 00000003 00000e54 00000000 004c0002
0016f734 000c01e0 0016f4d8 00150178 000201f5
0016f744 000801ee 00000000 0000000c 00000005
0016f754 00000000 00000000 0016f3b8 00171040
0016f764 0016eec0 0000b400 0e54086c c6a4049a

We need to take the 3rd value on the second line (its mared in bold) and run dd on it as well (this is, probably, a pointer inside the OXIDEntry structure):

0:005> dd 0016eec0
0016eec0 0016ef38 775c6fa0 0000086c 00000e54
0016eed0 d3b9b1cc ca2488ab b5ad2644 88cf2210
0016eee0 b5ad2644 88cf2210 0000b400 0e54086c
0016eef0 c6a4049a 49cd531e 00000103 004a02e6
0016ef00 0015a580 00000000 00000000 00000000
0016ef10 00000001 ffffffff 0016ee08 0150803c
0016ef20 00000006 00000000 00000000 00000000
0016ef30 00000000 00070005 0016efb0 0016eec0

Now we got the process id which is 86c and the thread id which is e54 (they are both marked in bold).

This means we are trying to move to thread e54 because that is where our STA COM object was created. In this sample, e54 is the main thread.

It’s a bit trickier, but if you know your offsets you can find it in no time and it’s very useful if you forgot to download sieextpub.dll or you are not able to install it at a customer site due to various limitations.

Thanks again for Herve for sending this information.

I’ve stumbled upon this article at MSDN titled “Registration-Free Activation of .NET-Based Components: A Walkthrough” which explains how to write a .NET component that is exposed through COM and when deployed it allows you to NOT register your .NET Component in the regsitry to make it visible to the COM code accessing it.

This is specifically useful for a couple of things:

  1. Deployments at which the registry cannot be modified due the organization’s security policy (I’ve met a few of those in the past and its not nice. Believe me.).
  2. Better Side-By-Side support. This is a better way of supporting side by side without using the registry. Everything is embedded in the .NET assembly and just by dropping it into the GAC you get everything you need.
  3. No dependecy on the registry which is always a good thing :-).

The only down side of this technology is that it is only supported in Windows XP SP2 and Windows 2003.

There are a lot of problems that can occur due to misusing the .NET registration like:

  • Using regasm with the /codebase parameter that fixes the exact assembly (and its location) that will be used when creating this object. This is like writing in .NET and deploying in a non side-by-side supported environment like COM.
  • Garbage left in the Registry by ncorrect uninstallation which can lead to unnecessary debugging until coming to the conclusion that the problem was a dirty registry (and that CAN be annoying, believe me).

You can spend a good couple of hours debugging in a production environment just to realize that the problem is due to stupid mistakes like I’ve mentioned above.

Anything that can alleviate this situation is blessed.

I’m such a statistics junkie that I found another interesting Google query that return my site and someone clicked on the link and got in (thanks again ShinyStat guys for that cool feature).

Anyhow, the query that got my attention today was “marshal.releasecomobject spinning“.

Now this is interesting and I thought I would elaborate a bit on Marshal.ReleaseComObject and what problems that can happen due to calling it (or not calling it).

First, we need to understand what Marshal.ReleaseComObject actually does and then we will cover the two assumptions mentioned above.

What does Marshal.ReleaseComObject actually do?
.NET can access COM objects through something called a Runtime Callable Wrapper (RCW).

The RCW is the wrapper for a COM object. There is one per a COM object instance and what it does is to provide the interface between .NET and COM.
COM uses reference counting to know when to release and destroy an object. .NET, on the other hand, is a garbage collected environment and these two approaches are opposite to each other.

COM will release an object the minute the minute the object’s ref count reached zero.
.NET will only release the COM object when no one is referencing the RCW and it is ready to be finalized.

RCW encapsulates the COM object and ALWAYS keeps a ref count of 1 to that object. An exception to this would be when the RCW is marshaled to another AppDomain, in which case it will increase the ref count of the COM object.

The RCW is a managed object and therefore have all the properties of a managed object. It will not be considered garbage as long as it is reachable from another object (and that object is either rooted or is referenced by a rooted object… You get the idea).

RCW has a finalizer in which the RCW will de-reference the COM object, get it destroyed and its memory freed (assuming it is not marshaled to another AppDomain). This means that it takes at least two garbage collections until a COM object that is being referenced from .NET will actually get released after no one references it.

So I did call Marshal.ReleaseComObject, so what?
If you did call Marshal.ReleaseComObject when you no longer need a certain COM object its great. It means that you release the memory of that object right away and you are not holding it until the next GC.

It can, however, cause a few problems:

  • If, for some reason the COM object is an STA object, call Marshal.ReleaseComObject means you will have to context switch to the STA thread that created that object (unless you are in the STA thread that created the object). This means that if you are not careful enough there is a possibility for a deadlock (for more information about deadlocks, refer to my previous posts about deadlocks).
  • You are passing the reference to that RCW around and you call Marshal.ReleaseComObject on it but you still have a reference to the RCW itself (which is still alive because its a manged object and will only die when it has no references to it and get Garbage Collected) in another place.
    In this case (which is an easy one), the RCW will throw an exception which states that you actaully disconnected the underlying COM object so you have a valid RCW object but it points to a dead COM object and you have nothing to do with it now.

I didn’t call Marshal.ReleaseComObject, so sue me

If you didn’t call Marshal.ReleaseComObject you might get into a different set of problems:

  • If you have STA objects you might get to a blocked finalizer thread situation.
  • It will just take more time for this memory to get released because, as I’ve said, the RCW has a finalizer and it takes at least 2 garbage collections to actually clean it and the memory its underlying COM object takes.

As you can see by this short post, Marshal.ReleaseComObject is an important thing that should not be taken lightly when used. You need to know when to use it and be aware of the consequences.

Happy COM interoping and memory releasing 😉

In the post about perfmon I’ve briefly mentioned the fact that you can utilize GC.Collect to find memory leaks.

By combining some custom code, WinDbg and SOS you can track most (if not all) of your managed memory leaks without buying a memory profiler.

The methodology is very simple and very easy to implement and use.
It involves taking serveral memory dumps at specific points and anaylzing the objects on the managed heap.

Preparing the data for analysis
Before we can start analyzing we need to prepare the data by doing the following things:

  • Prepare a call to GC.Collect. The exact code should look like this:
    GC.Collect();
    GC.WaitForPendingFinalizers();
    Just to be sure that everything is clean, call this code 3 times. This will ensure that everything, including all pending finalizers (objects that awaits for their finalizer to be called), are really gone.

    If this is a WinForms application, just add a button that will call the above code.
    If this is a Web application (ASP.NET or ASP.NET WebService) add a page called Collect.aspx with this code (If you don’t want to compile this into a code behind assembly, just download this page and place it in the root folder of the application).

  • Run the application and reach to a point before the area of code you wish analyze and take a memory dump using adplus.vbs -hang -p [process id].
    This dump will be used to show us what objects were already on the heap before we started the operation.
  • Run the operation we want to check for memory leaks and after it ends take another dump using the same command line as we used above.
    This dump will be used to show us what objects were added to the heap during the operation we are checking.
  • Run the Collect code (either by pressing the button you have prepared in the WinForms application or access the Collect.aspx page if this is a web application) and take another memory dump.
    This dump will be used to show us what objects are left on the heap after the operation has ended and after we called GC.Collect a few times to make sure all garbage was collected. All objects that cannot be accounted for (explained why they are still alive) are leaks and should be handled.

How to analyze the data

We first need to get some statistics on each one of the dumps to get a hold of the number of objects and their types that we currently have in the heap. To do that, we need to run the command: !dumpheap -stat on each one of the dumps and get a statistical view of all types of objects and the number of living instances.

We then need to compare between the 2nd and 3rd dumps to see if some of the objects did not decrease in number. If they haven’t (and we know they should have) these are our leaking objects.

We can find who is referencing them by using the !dumpheap -type [Object Type] (where [Object Type] is the namesapce and class name of the objects we want to check).

Then, we need to take one of the addresses (take the upper ones, they are the oldest ones) and run !gcroot [Object Address] on them to see who is referencing them.

Continue until you can account for all the objects in the 3rd dump and check if they are really supposed to be there after we called GC.Collect & GC.WaitForPendingFinalizers 3 times. Objects that should not be there should be traced to find who is referencing them and find out why they are being referenced.

That’s it. Quite simple, saves money but doesn’t produce nice graphs like the other tools 🙂