Bradley Kuszmaul of TokuDB fame pointed out that my benchmark for Green Mutexes used usleep to implement lock hold times of a few microseconds and that usleep(1) would sleep for much more than 1 microsecond. So I changed the test code to use gettimeofday to implement the lock hold time and repeated the tests.

There are a few differences with the new results. First, the lock hold time is now accurate when less than 50 microseconds. Second, the thread that holds the lock uses the CPU while waiting to unlock so CPU utilization should increase a small amount. I also report two results for the pthread mutex. One when using PTHREAD_ADAPTIVE_MUTEX_NP and one without it. Finally, a different server was used. It has the same specs as the previous server used but tests run a bit faster on it. The test server has 24 CPUs with HT enabled.

For the results below I use vmstat to measure the context switch rate and CPU utilization (sum of user + sys).

My summary of these results is that the InnoDB mutex uses too much CPU when the lock hold time is more than a few microseconds. Additional work is needed to determine whether this is useful. How frequently are locks held more than a few microseconds? The other point is that the lock hold time for which the InnoDB mutex has an advantage is reduced at higher concurrency levels. This a hard and interesting problem to solve. Hopefully the InnoDB team will enjoy working on it.One change that I tried is to limit the number of threads that can do the busy-wait loop concurrently. I don't think it is useful to have 200 threads doing that on a 24-core server. That change reduce the CPU utilization for the InnoDB mutex without hurting latency. But I won't post the numbers here. 

Results at 256 threads

1 microsecond lock hold time:

• InnoDB mutex: 1.3 usecs/loop, 580k context switches/second, 57% CPU (user=30, sys=27)
• adaptive pthread: 1.5 usecs/loop, 235k context switches/second, 50% CPU (user=5, sys=45)
• non-adaptive pthread: 2.0 usecs/loop, 225k context switches/second, 47% CPU (user=2, sys=45)

10 microsecond lock hold time:

• InnoDB mutex: 10.6 usecs/loop, 655k context switches/second, 50% CPU (user=20, sys=30)
• adaptive pthread: 10.5 usecs/loop, 190k context switches/second, 5% CPU (user=4, sys=1)
• non-adaptive pthread: 10.6 usecs/loop, 190k context switches/second, 5% CPU (user=4, sys=1)

40 microsecond lock hold time:

• InnoDB mutex: 41.8 usecs/loop, 690k context switches/second, 45% CPU (user=19, sys=26)
• adaptive pthread: 40.1 usecs/loop, 51k context switches/second, 4% CPU (user=4, sys=0)
• non-adaptive pthread: 40.1 usecs/loop, 51k context switches/second, 4% CPU (user=4, sys=0)

Results at 64 threads

1 microsecond lock hold time:

• InnoDB mutex: 1.2 usecs/loop, 620k context switches/second, 56% CPU (user=30, sys=26)
• adaptive pthread: 1.4 usecs/loop, 425k context switches/second, 46% CPU (user=6, sys=40)
• non-adaptive pthread: 1.7 usecs/loop, 425k context switches/second, 43% CPU (user=3, sys=40)

10 microsecond lock hold time:

• InnoDB mutex: 10.4 usecs/loop, 645k context switches/second, 52% CPU (user=20, sys=32)
• adaptive pthread: 10.1 usecs/loop, 190k context switches/second, 5% CPU (user=4, sys=1)
• non-adaptive pthread: 10.1 usecs/loop, 190k context switches/second, 5% CPU (user=4, sys=1)

40 microsecond lock hold time:

• InnoDB mutex: 41.4 usecs/loop, 665k context switches/second, 47% CPU (user=19, sys=28)
• adaptive pthread: 40.1 usecs/loop, 51k context switches/second, 4% CPU (user=4, sys=0)
• non-adaptive pthread: 40.1 usecs/loop, 51k context switches/second, 4% CPU (user=4, sys=0)

Results at 32 threads

1 microsecond lock hold time:

• InnoDB mutex: 1.2 usecs/loop, 600k context switches/second, 56% CPU (user=30, sys=26)
• adaptive mutex: 1.3 usecs/loop, 490k context switches/second, 45% CPU (user=8, sys=37)
• non-adaptive mutex: 1.7 usecs/loop, 490k context switches/second, 39% CPU (user=3, sys=36)

10 microsecond lock hold time:

• InnoDB mutex: 10.4 usecs/loop, 670k context switches/second, 50% CPU (user=22, sys=28)
• adaptive mutex: 10.1 usecs/loop, 190k context switches/second, 5% CPU (user=4, sys=1)
• non-adaptive mutex: 10.1 usecs/loop, 190k context switches/second, 5% CPU (user=4, sys=1)

40 microsecond lock hold time:

• InnoDB mutex: 41.3 usecs/loop, 700k context switches/second, 44% CPU (user=20, sys=24)
• adaptive mutex: 40.1 usecs/loop, 51k context switches/second, 4% CPU (user=4, sys=0)
• non-adaptive mutex: 40.1 usecs/loop, 51k context switches/second, 4% CPU (user=4, sys=0)

Results at 10 threads

1 microsecond lock hold time:

• InnoDB mutex: 1.0 usecs/loop, 180k context switches/second, 49% CPU (user=35, sys=4)
• adaptive mutex: 1.2 usecs/loop, 700k context switches/second, 34% CPU (user=9, sys=25)
• non-adaptive mutex: 1.7 usecs/loop, 700k context switches/second, 29% CPU (user=3, sys=26)

10 microsecond lock hold time:

• InnoDB mutex: 10.1 usecs/loop, 440k context switches/second, 33% CPU (user=25, sys=8)
• adaptive mutex: 10.0 usecs/loop, 200k context switches/second, 5% CPU (user=4, sys=1)
• non-adaptive mutex: 10.0 usecs/loop, 200k context switches/second, 5% CPU (user=4, sys=1)

40 microsecond lock hold time:

• InnoDB mutex: 41.6 usecs/loop, 350k context switches/second, 24% CPU (user=20, sys=4)
• adaptive mutex: 40.0 usecs/loop, 51k context switches/second, 4% CPU (user=4, sys=0)
• non-adaptive mutex: 40.1 usecs/loop, 51k context switches/second, 4% CPU (user=4, sys=0)