Archive for November, 2005

I got the following question from a reader named “Bysza” in the form of a comment on post about “How to tackle memory issues in a server application”:

“I’m using this approach for some time now, but I not always succeed in finding roots to objects.
I’m looking for a method to find objects that hold reference to my live object which sould be already gone.

Could you help me in that matter?”

If I understand the question correctly, you want to be able to find who is holding your live objects.

If that is the case, its quite simple.
What you need to do is either take a memory dump (I explained a bit on how to use it in this post) or attach WinDbg to the relevant process.

After you loaded the dump or attached to the process perform the following operations:

  1. Run !dumpheap -type [Part of the full namespace of the object] for example:
    !dumpheap -type MyNamespace.MyObjectThis will get you a list of the live objects of that type.
  2. Take the address from the first colum (that’s the address of the object) of the relevant object and run the command !gcroot [Object Address], for example:
    !gcroot 0x235025aThis will show you the chain of references up to the root object that is holding your referencing your object.
    This chain contains the addresses of each object in the chain so you will be able to know exactly who is holding who and why.

If a rooted object is directly holding your object you need to check one of the following things:

  • Do you have some kind of a Static (Shared in VB.NET) object in the application? If you do, this object is rooted and therefore every object it referecnces will never die until the application’s process is closed.Usually people have some kind of a static collection in which they store various objects and in a lot of the cases they forget to remove the objects from the collection.
  • Do you specifically pin objects? Do you call GCHandle.Alloc ? If so, be sure to release those handles or make sure that the objects being pinned do not hold references to objects you want to release.

I hope this helps a bit. If not, post a comment and we will continue the discussion there.

If any of you have various questions as to issues raised in this blog or from personal experience, please write me an email to eran.sandler at gmail.com or leave a comment on one of the posts. Also, if any of you have some other comments about how did you implemet/use some of the things we have talked about here, I would love to hear about it.

I’m going to be helping out in the Visual Studio 2005 Launch Event here in Israel in a side session titled (and it’s a translation from Hebrew): “Error detection in off-road terrain and combining your application with Windows Error Reporting”.

Sounds a bit long for a title, isn’t it? 😉

There are a few such side sessions which will be going on at the same time as other “Main” sessions. The main idea behind the side sessions is to have a smaller more intimate session (~50+ peoplee) that will allow better interaction between the lecturers and the people coming to hear the lecture.

We will be talking about how to do some production debugging at a customer site and how to handle memory dumps for post mortem debugging when you can’t have access to the client’s site. Gadi Meir of Idag (who is leading the session) will also talk about Windows Error Reporting (WER) and how to make your program work with it.
I will be talking about the managed areas of production and post mortem debugging as well as some samples on how to use, instrument and extend MDbg (which I talked about quite a bit in previous posts). Mainly these are topics that I’m usually covering here on the blog.

For some reason, since I’m a second chair in the presentation (and there are a lot of other second chairs for the other side sessions) I was not mentioned in the invitation.

That’s not nice, isn’t it? (remind me to have a small chat with the organizers. The second chairs should also be mentioned!)

Oh well… at least it’s going to be a cool event.
All of you in Israel – be sure to register here.

See you there and “Get Ready to Rock” (god, these marketing guys needs someone to help them a bit… 😉 )

If you want to catch me for a quick talk/question/whatever I will be hanging around the place where the session will be conducted and probably all around the place, so be sure to catch me!

While I usually don’t speak too much about performance, I’ve decided to write about this subject since I’ve stumbled upon it one time too many.

This issue was at the bottom of my post ideas list, but after seeing this article in ZDNet, I’ve decide to pay a bit more attention to this issue.

What’s Hyper-Threading?
Hyper-threading is Intel’s implementation for simultaneous multithreading, a technology that enables the processor to utilize empty cycles that are wasted when the currently running thread is waiting for a long operation such as RAM access or disk access.

While on paper this technology should speed up certain opeartions, in certain workload and certain scenarios of intensive server applications performance actually decreases.

A Sample Scenario
Let’s imagine the following scenario (which is very common to a lot of server applications).
Let’s say we have a few worker threads that are handling requests from clients. If too many requests are coming they are being queued.

Since this queue is shared among all threads and is accessed frequently, it will also be in the L1 or L2 cache of the CPU.

Now consider another different thread that is also running the background. It belongs to the application but it is not a worker thread that handles requests. It’s a thread that is being awakened periodically or by a trigger.
This thread runs and scans large chunk of the memory / objects / cache / (put your memory intensive task here) for changes. While it is running, sometimes getting some of those free cycles while one or more of the other worker threads are waiting for that operation, it will trash the L1 and L2 cache on the CPU due to the various tasks that its performing.

In that case, when the worker threads returns to work they try to access the memory that was previously in the L1 and L2 cache, but since the other non-worker thread trashed the cache we will get a cache miss that will cause us to fetch things from the RAM, or even worse, from the page file.
These operations will take instead of 2-4 cycles some where between 10-100 or even more cycles.

What actually happens is that in the L2 and L3 caches in the processor are being trashed whenever they switch to a different thread while the current one is waiting for some I/O operation.
This is specifically bad for applications such as ASP.NET and SQL Server.

ASP.NET and SQL Server
ASP.NET and SQL Server are two common server applications that use .NET (SQL Server 2005 hosts the CLR for stored procedures).

These two server applications are heavy on thread usage and since they both use the CLR (probably with the server GC) it means that they will have a GC thread per heap (and we have heap per Logical CPU) that the GC is being performed on them.
GC threads are memory intensive since they scan through all the memory of the generations being collected and traverse the various pointer to determine which objects are garbage or not.

In addition to the GC threads, SQL server has additional system threads running in the background that can also lead, in certain situations, to a decrease in performance.

While I’m not aware of system threads in ASP.NET, I know that some applications have additional threads running inside an ASP.NET which may cause them, in certain situations (of course), to suffer a decrease in performance.

You can read more about the effect Hyper-Threading has on SQL Server in this blog post by an MS developer.

So what should you do?
As I’ve said numerous times during this post, this behavior only happens in certain situations.
This means, that the best way of handling these issues is to stress test your application under high load with both Hyper-Threading enable and disabled.

Only then you will be able to determine if under the tested load Hyper-Threading is working with you or against you.

Server GC pre-allocation in Hyper-Threaded enabled environments
There is another benefit for disabling Hyper-Threading regarding the whole cache misses issues that we talked about above.

The Server version of the GC allocates a separate heap and a GC thread per Logical CPU. Hyper-Threading causes Windows to see a single physical Hyper-Threaded enabled CPU and 2 Logical CPUs (that’s one of the tricks that it uses to reschedule other threads for executions while others are waiting for their costly operations).
This means that, if you have a 2-CPU machine with Hyper-Threading enabled, upon starting your ASP.NET application (or any application that uses the Server GC) the GC will pre-reserve 64Mb x 4 CPUs = 256Mb of your virtual address space (which is 2Gb per process or 3Gb if you set the /3Gb flag in boot.ini).

If you have a memory intensive application that might be more than you need. By disabling Hyper-Threading you will be able to reduce that to 128Mb.

This is another factor one should consider when enabling or disabling Hyper-Threading.

Summary
The funny thing about Hyper-Threading and .NET is that I’ve found this post in MSDN saying that it can boost .NET and everything will be great and fine. The problem is that they neglected to mention some of the issues we’ve talked about here.

I’m often being asked to resolve various problems in which the .NET Assembly loader does not find the right assembly to load and therefore doesn’t load it at all.

While this is not hard core debugging that you would usually expect using WinDbg, CorDbg, MDbg and others, this is a very common case that with a little more information about how .NET works and one simple tool, can be easily resolved in both development and production environments.

Did you see the error “File or assembly name XXX, or one of its dependencies, was not found” error (replace XXX with the relevant assembly name) before?

That’s exactly the error I’m talking about.

Before we will go into common problems and how to resolve them, let us first understand how the .NET assembly loader actually works and how and where it searches for the assembly we want to load.

Strongly Signed Assembly
A strongly signed assembly is an assembly that was signed using a key pair during compilation (or after it).
The key pair is generated using the “sn.exe” utility and can be unique per assembly, per product or per publisher (depending on your needs).

Strong signing an assembly gives the assembly a unique identifer which is composed out of the following parameters:

  1. Assembly Name – the file name of the assembly minus the “.dll” suffix.
  2. Assembly Version – The version of the assembly as defined in the AssemblyInfo file. It can contain a fixed number which the developer / build machine will advance manually or have the “*” sign at the end so each build will increment it automatically.
  3. Culture – The deafult value for this parameter in most assembly is “neutral” meaning, that this assembly is not culture specific. It is mainly used to create assembly (or satellite assemblies) that are culture specific such as a specific assembly which has resource string that contains German text instead of English text. In which case, the culture will be “de”.
  4. Public Key Token – A token generate from the public key that you have in your key pair. It is used to verify the signature of the assembly to check that it was not tempered with after it was signed.

All of these parameters together create, what is referred to as, a “Fully Qualified Name” of a .NET assembly and looks like this:

MyAssembly, Version=1.0.2.3, Culture=neutral, PublicKeyToken=b77a5c561934d069

A fully qualified name for a specific type will look like this:

MyType, MyAssembly, Version=1.0.2.3, Culture=neutral, PublicKeyToken=b77a5c561934d069

If there is a program that needs to load some type and create it dynamically and the assembly of that type is strongly signed you can use the following piece of code to load and create that specific type given its fully qualified name:

string typeFullQualifiedName = “fully qualified name of type here”;
Type type = Type.GetType(typeFullQualifiedName);
if (type != null) {
object obj = Activator.CreateInstance(type);
}

This piece of code will try to load the relevant assembly (we will talk about the .NET assembly loader logic next) and if it succeeds, it will get the Type object which will allow us to create an instance of that type.

If you don’t have a full qualified name of the type, you can always use System.Reflection.Assembly.Load, give it the full path (or relative path) to the assembly and call assembly.GetType with the type you want to get and continue from there on.

.NET Assembly Loader – The real story

The first thing the .NET assembly loader checks is whether the assembly is strongly signed. If it is, it will start its search in the Global Assembly Cache (a.k.a the GAC. You can read more about it here).

The loader will search for a policy file named in the format of:
policy.AssemblyMajorVersion.AssembyMinorVersion.AssemblyName

For example:
Policy.1.2.MyAssembly

If such a file exists it will look inside of it if the version of the assembly that we are trying to load matches the version/versions range written in the policy file. If it does, it will try to load the assembly with the version specified there. If no such policy file exists, it will try to load the assembly from the GAC.

If it will fail to find it in the GAC, it will start to serach in the system’s search path.
In web applications it will also include the application’s Bin directory in the search path.

You can manully add folders to an AppDomain’s search path by using the “AppendPrivatePath” method.

So, what should I do if it can’t find the assembly?
The best method I’ve found to check if there is a policy problem or a path/version problem is to use FileMon from SysInternals.

FileMon tracks all activities that relates to accessing files in your operating system including read, write, access denieds on files and so on.

I would suggest adding a filter with the name (or partial name) of your application and start the application.

It will start showing you all file access that your appliction does. When you’ll receive the assembly load error, search for “NOT FOUND” or “ACCESS DENIED”.

If you have policies for that specified assembly you will see that the application accessed the policy file and directly after it will try to load the assembly with the version specified in the policy file.

If this is not a strongly signed assembly or the file is not in the GAC you will see all the places the application searched for to load the file.

Summary
This is a very easy trick that saves long hours of debugging, specifically in a production or semi-production environment.

If you are using the various abilities of .NET for side-by-side versioning, policy files and the GAC, do mind that you WILL run into these simple problems that consumes a lot of time and effort.

If this post helped you, comment on this post or just Email me. I would like to know that I’m not the only one fighting with the GAC and policy files 🙂

Following some of the samples posted in Mike Stall’s blog and after actually needing something like this at work, I’ve decided to write a small program called ExceptionDbg which uses MDbg’s APIs and tracks all exceptions being thrown in an application (including those that are being swallowed by try..catch blocks).

We had a case of exceptions being thrown in the finalization code (this is C++ code that has destructors and therefore has finalizers. I’ve talked about C++ destructor in this previous post).

ExceptionDbg let you attach to a process and dump all of the exceptions being thrown live to your console and/or log file.

There are a few filters on the exception types being logged including:

  • Show only exceptions being caught by try..catch blocks
  • Show full Call Stack and if debug symbols are present, show line numbers as well as source file names.

So what is it good for, you ask?

  • Check exceptions in the finalizer thread – These exceptions are ALWAYS being caught so the finalizer thread will not die. It can mean that if you have some code that frees some unmanaged resources they will not get freed and this is a potential memory leak.
  • Got a performance problem? Check how what are all the exceptions being thrown and caught (try..catch blocks) to verify that some pieces of code doesn’t use exception throwing as a logic to handle things other than failure.

As you can see, it’s useful for a number of scenarios and it help me a lot in finding some of these problems.

You can get the binaries here and the source code here.

You need the .NET Framework 2.0 for this which can be downloaded from here.
If you want to compile it you’ll need the .NET Framework 2.0 SDK which you can download from here.

Enjoy and send me feedback!

After further experimentation it looks writing a WinDbg extension that will let one write WinDbg extensions in .NET is actually feasible. If you remember, I’ve talked about it in my previous post on the subject.

I could have gone the easy way and written some code in .NET exposed as a COM object and make my WinDbg extension call that managed code which will do all the rest, but that would be too easy and a lot less fun 🙂

Therefore, I’ve decide to gain some experience and knowledge in the .NET CLR hosting API.

I’m using the hosting API to bind and start the CLR, call the rest of my .NET code and pass the various interfaces that WinDbg is passing me back to the .NET code.

The challenge is to convert the interface definitions of the various WinDbg interface used to interact with WinDbg to .NET. It’s a bit of a tedious job, but someone has to do it… don’t you think?! 🙂

Anyhow, once I’ll finish the part about the CLR hosting APIs I’d write a long post about this subject and what I’ve learned while using this API so stay tuned!

Oh, one last thing. In that previous post tinkered with the idea of using some other way of instrumenting WinDbg instead of using its archaic and strange scripting language (actually that’s WinDbg’s command line interface turned into a scripting language).

Anyhow, technically I can enable any language, compile the script at runtime and run the assembly, but its a bit annoying. There is the ability of using IronPython or some other .NET dynamic language (maybe JScript .NET).

What do you think? would you like to be able to script WinDbg in your favorite .NET scripting language?

I don’t know how many of you are writing WinDbg extensions but its not that trivial to most people and especially not that robust and nice such as writing a plugin to MDbg or instrumenting it.

For managed debugging MDbg (which can help you debug both 1.1 and 2.0 frameworks) should be more than enough for all of your managed debugging needs but, AFAIK, MDbg does not handle dump files which are VERY important for post mortem debugging or for performance tuning/debugging in an environment that simply cannot be slowed down due to an attached debugger.

The only way of writing WinDbg extensions, at the moment, is to write C/C++ code. There are a lot of samples that comes with the WinDbg installation but it is not as robust or easy as writing it, say, in C#.

In addition to that, WinDbg has a built-in proprietary mini scripting language that you can use to instrument it a bit but its not standard, lack documentation (at least public one) and is not that easy to use (at least to most people).

What if you could write a WinDbg plugin in C# or in any one of your favorite .NET languages?
What if you could use JavaScript or a Monad like script? Or perhaps some C# and/or VB.NET as your “scripting language” for WinDbg?

I’m playing with the idea of writing a WinDbg extension that will enable writing WinDbg extensions in .NET.
Perhaps it will have a feature to run a special script that is not written in WinDbg’s strange scripting language.

What do you think?

Have you encountered a situation in which you needed to instrument WinDbg but didn’t know its “scripting language” or simply couldn’t do that certain thing you wanted to do?

Did you ever need some kind of functionality that is missing in WinDbg and didn’t want to spend too much time playing around with C/C++ to write it down?

I’ve started writing a prototype to see if this is fesible and it seems it is (at least for the moment 😉 ). I’ll try to find a bit more time to finish it and I’ll post my findings (and hopefully the code if it actually works).