Three ways that the poor man’s profiler can hurt MySQL

Over the last few years, Domas’s technique of using GDB as a profiler has become a key tool in helping us analyze MySQL when customers are having trouble. We have our own implementation of it in Percona Toolkit (pt-pmp) and we gather GDB backtraces from pt-stalk and pt-collect.

Although it’s helped us figure out a lot of problems, sometimes it doesn’t go well. Getting GDB backtraces is pretty intrusive. Here are three things that I’ve witnessed:

1. The server freezes for the duration of the process. This is the most obvious impact on the running server: GDB forklifts the process and gets a stack trace from every thread, then lets it go on working. But this can take a while. It’s usually a couple of seconds, but on big servers with a lot of memory and many threads, it can take much longer (I’ve seen tens of seconds, but heard reports of minutes).
2. The server can crash. I haven’t seen this myself, but I’ve heard reports from others.
3. The server can be left in an unstable state. Sometimes when GDB detaches, the process’s state isn’t quite as it was before. I have rarely seen this, but the other day I had a customer experience a very slow-running server that was using tons of CPU time and lots of system CPU, and exhibiting the classic signs of InnoDB kernel_mutex contention. I was not able to find anything wrong, and was still trying to determine whether the sudden slowness was due to some cause such as an increase in traffic or just InnoDB having trouble, when the customer Skyped me to say that they’d restarted the server and it resolved the problems. This server had previously been analyzed with GDB around the time the problems began.

So although it’s extremely useful, it does have risks.

Domas, and others at Facebook and beyond, have developed a variety of tools that can help them get stack traces less intrusively. I think we need to investigate some of those and see whether there is something that would work well a broad variety of cases for more users, and how much less intrusive they can be.

Of course, I’m really waiting for MySQL 5.6 and the improved Performance Schema that will be included with it. A built-in solution will be much better than a technique such as using GDB. We’ll probably never stop using GDB completely, but hopefully we’ll be able to use such tools much less frequently after 5.6 is released.