While I usually don’t speak too much about performance, I’ve decided to write about this subject since I’ve stumbled upon it one time too many.
This issue was at the bottom of my post ideas list, but after seeing this article in ZDNet, I’ve decide to pay a bit more attention to this issue.
Hyper-threading is Intel’s implementation for simultaneous multithreading, a technology that enables the processor to utilize empty cycles that are wasted when the currently running thread is waiting for a long operation such as RAM access or disk access.
While on paper this technology should speed up certain opeartions, in certain workload and certain scenarios of intensive server applications performance actually decreases.
A Sample Scenario
Let’s imagine the following scenario (which is very common to a lot of server applications).
Let’s say we have a few worker threads that are handling requests from clients. If too many requests are coming they are being queued.
Since this queue is shared among all threads and is accessed frequently, it will also be in the L1 or L2 cache of the CPU.
Now consider another different thread that is also running the background. It belongs to the application but it is not a worker thread that handles requests. It’s a thread that is being awakened periodically or by a trigger.
This thread runs and scans large chunk of the memory / objects / cache / (put your memory intensive task here) for changes. While it is running, sometimes getting some of those free cycles while one or more of the other worker threads are waiting for that operation, it will trash the L1 and L2 cache on the CPU due to the various tasks that its performing.
In that case, when the worker threads returns to work they try to access the memory that was previously in the L1 and L2 cache, but since the other non-worker thread trashed the cache we will get a cache miss that will cause us to fetch things from the RAM, or even worse, from the page file.
These operations will take instead of 2-4 cycles some where between 10-100 or even more cycles.
What actually happens is that in the L2 and L3 caches in the processor are being trashed whenever they switch to a different thread while the current one is waiting for some I/O operation.
This is specifically bad for applications such as ASP.NET and SQL Server.
ASP.NET and SQL Server
ASP.NET and SQL Server are two common server applications that use .NET (SQL Server 2005 hosts the CLR for stored procedures).
These two server applications are heavy on thread usage and since they both use the CLR (probably with the server GC) it means that they will have a GC thread per heap (and we have heap per Logical CPU) that the GC is being performed on them.
GC threads are memory intensive since they scan through all the memory of the generations being collected and traverse the various pointer to determine which objects are garbage or not.
In addition to the GC threads, SQL server has additional system threads running in the background that can also lead, in certain situations, to a decrease in performance.
While I’m not aware of system threads in ASP.NET, I know that some applications have additional threads running inside an ASP.NET which may cause them, in certain situations (of course), to suffer a decrease in performance.
You can read more about the effect Hyper-Threading has on SQL Server in this blog post by an MS developer.
So what should you do?
As I’ve said numerous times during this post, this behavior only happens in certain situations.
This means, that the best way of handling these issues is to stress test your application under high load with both Hyper-Threading enable and disabled.
Only then you will be able to determine if under the tested load Hyper-Threading is working with you or against you.
Server GC pre-allocation in Hyper-Threaded enabled environments
There is another benefit for disabling Hyper-Threading regarding the whole cache misses issues that we talked about above.
The Server version of the GC allocates a separate heap and a GC thread per Logical CPU. Hyper-Threading causes Windows to see a single physical Hyper-Threaded enabled CPU and 2 Logical CPUs (that’s one of the tricks that it uses to reschedule other threads for executions while others are waiting for their costly operations).
This means that, if you have a 2-CPU machine with Hyper-Threading enabled, upon starting your ASP.NET application (or any application that uses the Server GC) the GC will pre-reserve 64Mb x 4 CPUs = 256Mb of your virtual address space (which is 2Gb per process or 3Gb if you set the /3Gb flag in boot.ini).
If you have a memory intensive application that might be more than you need. By disabling Hyper-Threading you will be able to reduce that to 128Mb.
This is another factor one should consider when enabling or disabling Hyper-Threading.
The funny thing about Hyper-Threading and .NET is that I’ve found this post in MSDN saying that it can boost .NET and everything will be great and fine. The problem is that they neglected to mention some of the issues we’ve talked about here.