If you are getting the error “Internal .Net Framework Data Provider error 6” and you are using SQL Server 2005 with database mirroring know that its a bug in the SQL managed client code.

Check out KB article 944099 for more info.

At the time of writing this post the only solution for this problem is a private hot fix that you need to obtain from Microsoft.

The links around the web for the Hotfix Request Web Submission form seems to be broken. I’ve contacted Microsoft and got this replacement link.

Use this link to contact Microsoft, state that you require the hotfix for KB article 944099 and submit the form. You will be contacted shortly by a Microsoft representative that will give you the download details or request further information to better understand your needs.

I’ve just been asked through the “Ask” widget on the side of this blog on Yedda (my day job) this question:

Yedda � People. Sharing. Knowledge.Where is sos.dll for .net 3.5?

I installed Visual Studio 2008, but I don’t see sos.dll in the Microsoft.NET\Framework\v3.5 directory? What’s up with that?

Topics: , ,

Asked by dannyR on January 03, 2008

View the entire discussion on Yedda

To which I replied:

Yedda � People. Sharing. Knowledge.Where is sos.dll for .net 3.5?

There is a bit of confusion as to the .NET Framework versions as opposed to the CLR (Common Language Runtime) versions.

Up until .NET Framework 2.0, the CLR versions advanced side by side with the versions of the framework.

Starting from .NET Framework v3.0, the version of the CLR stated at 2.0 (there was no significant change in the CLR itself) and the framework’s version kept on going forward.

.NET Framework v3.5 actually runs on CLR 2.0 SP1.

This means that you can should be able to use the SOS.dll located at Microsoft.NET\Framework\v2.0.50727 with either Visual Studio (as I’ve explained on this post) or with WinDbg.

Dave Broman of Microsoft has written a post about the mapping of the versions of the .NET Framework to the version of the CLR. Do mind that as Dave states in his post, its the way he figures things up in his head and not the official documentation on this matter.

Topics: , ,

Answered by Eran on January 03, 2008

View the entire discussion on Yedda

I know its a bit confusing and I’m not really sure why Microsoft detached the versions of the Framework from the versions of CLR (well, I can understand some of the thoughts leading to this but I don’t think it was that good of a move) but Dave’s excellent post does put things in perspective and allows figuring out in what version of the CLR you are actually using.

After further experimentation it looks writing a WinDbg extension that will let one write WinDbg extensions in .NET is actually feasible. If you remember, I’ve talked about it in my previous post on the subject.

I could have gone the easy way and written some code in .NET exposed as a COM object and make my WinDbg extension call that managed code which will do all the rest, but that would be too easy and a lot less fun 🙂

Therefore, I’ve decide to gain some experience and knowledge in the .NET CLR hosting API.

I’m using the hosting API to bind and start the CLR, call the rest of my .NET code and pass the various interfaces that WinDbg is passing me back to the .NET code.

The challenge is to convert the interface definitions of the various WinDbg interface used to interact with WinDbg to .NET. It’s a bit of a tedious job, but someone has to do it… don’t you think?! 🙂

Anyhow, once I’ll finish the part about the CLR hosting APIs I’d write a long post about this subject and what I’ve learned while using this API so stay tuned!

Oh, one last thing. In that previous post tinkered with the idea of using some other way of instrumenting WinDbg instead of using its archaic and strange scripting language (actually that’s WinDbg’s command line interface turned into a scripting language).

Anyhow, technically I can enable any language, compile the script at runtime and run the assembly, but its a bit annoying. There is the ability of using IronPython or some other .NET dynamic language (maybe JScript .NET).

What do you think? would you like to be able to script WinDbg in your favorite .NET scripting language?

I don’t know how many of you are writing WinDbg extensions but its not that trivial to most people and especially not that robust and nice such as writing a plugin to MDbg or instrumenting it.

For managed debugging MDbg (which can help you debug both 1.1 and 2.0 frameworks) should be more than enough for all of your managed debugging needs but, AFAIK, MDbg does not handle dump files which are VERY important for post mortem debugging or for performance tuning/debugging in an environment that simply cannot be slowed down due to an attached debugger.

The only way of writing WinDbg extensions, at the moment, is to write C/C++ code. There are a lot of samples that comes with the WinDbg installation but it is not as robust or easy as writing it, say, in C#.

In addition to that, WinDbg has a built-in proprietary mini scripting language that you can use to instrument it a bit but its not standard, lack documentation (at least public one) and is not that easy to use (at least to most people).

What if you could write a WinDbg plugin in C# or in any one of your favorite .NET languages?
What if you could use JavaScript or a Monad like script? Or perhaps some C# and/or VB.NET as your “scripting language” for WinDbg?

I’m playing with the idea of writing a WinDbg extension that will enable writing WinDbg extensions in .NET.
Perhaps it will have a feature to run a special script that is not written in WinDbg’s strange scripting language.

What do you think?

Have you encountered a situation in which you needed to instrument WinDbg but didn’t know its “scripting language” or simply couldn’t do that certain thing you wanted to do?

Did you ever need some kind of functionality that is missing in WinDbg and didn’t want to spend too much time playing around with C/C++ to write it down?

I’ve started writing a prototype to see if this is fesible and it seems it is (at least for the moment 😉 ). I’ll try to find a bit more time to finish it and I’ll post my findings (and hopefully the code if it actually works).

What is a deadlock?

(from the first entry when running the google query “define:deadlock“)
“This is an inter-blocking that occurs when two processes want to access at shared variables mutually locked. For example, let A and B two locks and P1 and P2 two processes: P1: lock A P2: lock B P1: lock B (so P1 is blocked by P2) P2: lock A (so P2 is blocked by P1) Process P1 is blocked because it is waiting for the unlocking of B variable by P2. However P2 also needs the A variable to finish its computation and free B. So we have a deadlock”

What are the sympotms of a deadlock?
Usually you will see that the application stops responding for some reason and the CPU (if you’ll open up the Task Manager) is at 0% most of the time (if this is the only major application running on this machien).

What to do?
So… how can we find a deadlock, specifically in a production environment without a debugger or development envrionment? (just the kind of problem we like 🙂 )

What we want to find out is who is holding the locks and who is waiting on them.

Below are a few easy steps to figure this out.

  1. Attach WinDbg to the relevant process (F6 and select the process).
  2. make sure the symbols paths are OK by calling .sympath. If you don’t have the symbols for .NET and the rest of windows just type “.symfix c:\symbols” this will put a path to Microsoft’s public symbols server and will download all relevant symbols into c:\symbols. If you have symbols of your own in some folder you can use .sympath with the += flag (like this .sympath += c:\mypath) to add the additional path.
  3. type “.load clr10\sos.dll” to load the SOS extension (it might already be loaded, but it won’t do any harm calling it again).
  4. Run the command “!SyncBlk.

You can see a sample output here.

As you can see, we have two locks, since we have 2 resources that are locked. Each resource is locked by a different thread and each one of these thread is trying to acquire a lock on the other resource.

In the “ThreadID” column we can see the ID (and near it its Index in the current debug session) of the thread that is the owner of this lock. Under each lock we can see the list the indices of the waiting threads.

Now all we have left is to use the “!clrstack command on each of the waiting threads and the locking threads to see where are they in the code and determine why is the deadlock happening and figure out a way to avoid this situation.

To run the !clrstack command on a specific thread just use the following command “~3e!clrstack“. This command displays the CLR stack of the thread whose index is 3.

You can see a sample output of !clrstack (specifically ~4e!clrstack) here

While debugging this I had the debug symbols available, that is why WinDbg was able to give me an exact line number of where this is happening. If you don’t have the debug symbols of your code you will only see the function name which is good enough in most cases.

Can’t do a live debug? The problem is at a remote customer’s site?

If you are not able to live debug this situation because this happens at a customer site and you cannot get there or unable to have direct access to the machine, you can perform all the steps above on a dump taken using the “adplus.vbs” script.

Just tell the customer to download WinDbg from here and tell him to reproduce the problem. When he does reproduce it tell him to run the following command line:

adplus.vbs -p [process Id] -hang

(Replace [process Id] with the process Id of the application).

And tell him to send the dump back to you. You will be able to run the same commands as above and figure out where the deadlock happens.

Download sample code used in this post.

Happy Deadlock Hunt!

Hello Everyone.
My name is Eran Sandler and I’ve been messing around with .NET for the past 5+ years (a little bit after the first PDC it was introduced to the world).

I wanted to share with the world some of the experiences I’ve gathered in the last couple of years in regards to .NET debugging, specifically to Production debugging and Post Mortem debugging since there is little to no information available on the web (not to mention not a whole lot of tools to do that).

So what is Production Debugging?
In my own terms, production debugging is the ability to debug your application in a production environment without the ability to install a fully feldged development environment. This means, NO Visual Studio or some other IDE at your disposal, NO code compilation on the machine and a lot of other NOs that you can figure yourself.

This type of debugging is very suited to situations where you need to debug on a production environment some tricky problem and there is no way (or ability) to install a development environment.
A good example of this situtation is deubgging on a customer’s production environment.

OK, so what is Post Mortem Debugging?
I’m not sure its the right term for it (and maybe I mixed it up with some other term), but basically its the ability to investigate a problem after it occured.
We are capable of doing that using a tool that actually takes a snapshot of all the memory that is being used at the moment in a given process. This gives us a current image of the process at a certain point in time enabling us to investigate various things like the objects alive at this momemnt, what are the various threads doing and so on.

Post Mortem debugging is very useful if you can’t reach a cutomer’s site (or don’t want to) and you need to get more information to better understand and solve a problem.
Another good use of post mortem debugging is the ability to take a few sequencial snapshots of a ceratin process and examine some information as it changes over time, for example, checking the virtual memory fragmentation that is happening over time in a ceratin process.

By now you are probably wonder if this is actually possible. Well… It is!
Microsoft even supplies the tools (Debugging Tools for Windows – contains WinDbg and friends, and there is also CorDbg which is the .NET managed Debugger that comes with the .NET Framework SDK) and its even quite easy but the main problem is lack of documentation and places on the net to actually learn this stuff.

I will try to give some tips, pointers and tutorials on how to do various things I’ve picked up a long the way and hopefully we will have enough information here to help everyone.

In the next couple of posts I’m going to show the basic things of how to use WinDebug (that’s the main tool we’ll use) with .NET and we will learn some cool stuff.

So… hold on to your hourses because we are going on a crazy debugging ride 🙂