What is a deadlock?

(from the first entry when running the google query “define:deadlock“)
“This is an inter-blocking that occurs when two processes want to access at shared variables mutually locked. For example, let A and B two locks and P1 and P2 two processes: P1: lock A P2: lock B P1: lock B (so P1 is blocked by P2) P2: lock A (so P2 is blocked by P1) Process P1 is blocked because it is waiting for the unlocking of B variable by P2. However P2 also needs the A variable to finish its computation and free B. So we have a deadlock”

What are the sympotms of a deadlock?
Usually you will see that the application stops responding for some reason and the CPU (if you’ll open up the Task Manager) is at 0% most of the time (if this is the only major application running on this machien).

What to do?
So… how can we find a deadlock, specifically in a production environment without a debugger or development envrionment? (just the kind of problem we like 🙂 )

What we want to find out is who is holding the locks and who is waiting on them.

Below are a few easy steps to figure this out.

  1. Attach WinDbg to the relevant process (F6 and select the process).
  2. make sure the symbols paths are OK by calling .sympath. If you don’t have the symbols for .NET and the rest of windows just type “.symfix c:\symbols” this will put a path to Microsoft’s public symbols server and will download all relevant symbols into c:\symbols. If you have symbols of your own in some folder you can use .sympath with the += flag (like this .sympath += c:\mypath) to add the additional path.
  3. type “.load clr10\sos.dll” to load the SOS extension (it might already be loaded, but it won’t do any harm calling it again).
  4. Run the command “!SyncBlk.

You can see a sample output here.

As you can see, we have two locks, since we have 2 resources that are locked. Each resource is locked by a different thread and each one of these thread is trying to acquire a lock on the other resource.

In the “ThreadID” column we can see the ID (and near it its Index in the current debug session) of the thread that is the owner of this lock. Under each lock we can see the list the indices of the waiting threads.

Now all we have left is to use the “!clrstack command on each of the waiting threads and the locking threads to see where are they in the code and determine why is the deadlock happening and figure out a way to avoid this situation.

To run the !clrstack command on a specific thread just use the following command “~3e!clrstack“. This command displays the CLR stack of the thread whose index is 3.

You can see a sample output of !clrstack (specifically ~4e!clrstack) here

While debugging this I had the debug symbols available, that is why WinDbg was able to give me an exact line number of where this is happening. If you don’t have the debug symbols of your code you will only see the function name which is good enough in most cases.

Can’t do a live debug? The problem is at a remote customer’s site?

If you are not able to live debug this situation because this happens at a customer site and you cannot get there or unable to have direct access to the machine, you can perform all the steps above on a dump taken using the “adplus.vbs” script.

Just tell the customer to download WinDbg from here and tell him to reproduce the problem. When he does reproduce it tell him to run the following command line:

adplus.vbs -p [process Id] -hang

(Replace [process Id] with the process Id of the application).

And tell him to send the dump back to you. You will be able to run the same commands as above and figure out where the deadlock happens.

Download sample code used in this post.

Happy Deadlock Hunt!

  • phil

    Excellent, thnak you very much, you’ve helped me no end.

    I really appreciate the clear and concise example.

    One thing that you dont mention is how to load the symbols for the application you’re debugging. Can you point me to something i can read?

    Many thanks again.

  • Actually, I’ve explained a bit on how to load the symbols in this very post.

    make sure the symbols paths are OK by calling .sympath. If you don’t have the symbols for .NET and the rest of windows just type “.symfix c:\symbols” this will put a path to Microsoft’s public symbols server and will download all relevant symbols into c:\symbols. If you have symbols of your own in some folder you can use .sympath with the += flag (like this .sympath += c:\mypath) to add the additional path.

    It’s in the “What to do?” part of the post, point #2.

  • phil

    Hey Again, Eran.

    I looked again at you advice and you do indeed talk about loading “your own” symbols so appolgies for not reading carefully enough.

    I’ve tried the .sympath += flag to set the directory with my symbols in it but it keeps coming back with the following;

    WARNING: Inaccessible path: ‘+= c:\symbols\MySymbols’

    The path is correct but it just wont take it… Any idea’s.

    Cheers.

  • Try doing this:

    Run “.symfix c:\symbols”

    This will set the .sympath to Microsoft’s symbols server and will download all of the symbols to c:\symbols (you can feel free to change c:\symbols to whatever you want to).

    Then, write “.sympath”.
    This will display the path set by the previous call to “.symfix”.
    Copy the value and write:
    “.sympath C:\Path to your symbols;[Paste here what you have copied]”

    So it will first look in your application’s symbols path and then use the Microsoft Symbols server.

    Hope this helps.

  • Pingback: !SyncBlk in SOS for .NET Framework 2.0 » Advanced .NET Debugging()

  • Pingback: Marshal.ReleaseComObject and CPU Spinning » Advanced .NET Debugging()

  • Pingback: Blocked Finalizer Thread » Advanced .NET Debugging()

  • Pingback: Setting a breakpoint in WinDbg for Managed Code » Advanced .NET Debugging()

  • Luke

    FYI for anybody else that might be having trouble — if you get “Doesn’t work with 2.x” error when you try to run !syncblk, try the following advice (worked for me):

    http://groups.google.com.au/group/microsoft.public.windbg/browse_thread/thread/f4d539d98f2f817f/13d7f7dcd825002b?lnk=raot

  • Luke, thanks for sharing.

  • Doug

    To add a directory to the symbol path (or several directories delimited with semicolons (;), use .sympath+

    For example:

    .sympath+ c:\code\myproject\bin

    or

    .sympath+ c:\code\myproject\bin;c:\mylibrary\bin

    Where the path is the location of the pdb file

  • Coder

    unable to download the code

  • Simon

    Your external links are broken. Just FYI.

  • Michal

    thanx man, great article, helped to solve my pain in the ass

  • Neither the txt files nor the code is getting downloaded on this page.

  • Sorry about that. I switched servers a long time ago and it seems the conversion lost these files. I’ll try to find them and place them back.

  • nike mercurial vapor iii

    diligent blog ! thank you !

  • nike Total 90

    good

  • Mac Cosmetics

    wonderful!

  • alpinestars boots

    What we want to find out is who is holding the locks and who is waiting on them.

  • Oh, and then there’s the Studebaker bus, the ’55 Chevy George Barris custom (see what I mean?)….

  • What we want to find out is who is holding the locks and who is waiting on them.

  • supreme

    Supreme gives us a preview of an exciting new development for Spring/Summer 2012 as the iconic

    New York street brand T-shirt, comfortable fashion design, no matter where, put on.
    supreme T shirtswill be a beautiful scenery.