How expensive is USER_STATISTICS?

One of our customers asked me whether it’s safe to enable the so-called USER_STATISTICS features of Percona Server in a heavy-use production server with many tens of thousands of tables.

If you’re not familiar with this feature, it creates some new INFORMATION_SCHEMA tables that add counters for activity on users, hosts, tables, indexes, and more. This is very helpful information. It lets you run queries to see which tables get the most reads and writes, for example. Another popular use for it is to find unused indexes.

I knew that some of our customers were using the feature enabled in production all the time, and I knew that Google and Facebook and others (the original developers) did also. But I didn’t know the real answer about how expensive it is.

There are two dimensions to the question. One is how much slower the server becomes, and the other is how much more memory it uses. The second question is important on servers with many tables. Actually, “many indexes” is an even better metric. If you have tens of thousands of tables and they have a handful of indexes each, you have many more indexes than tables.

Our performance engineering team ran a variety of benchmark scenarios with the feature disabled versus enabled. In brief, we started the server on a database with 25,000 sysbench tables, performed a warmup, ran a benchmark for 900 seconds, collected a snapshot of system state, enabled userstats, repeated the warmup and benchmark, and captured another snapshot. We ran this in a couple of configurations. This allowed us to see what the impact of the feature is versus the behavior of the server itself. We used tools such as the gperf heap profiler and /proc/pmap to help examine memory usage.

The results? After running the benchmark on 25,000 tables with indexes, the performance change was indistinguishable from measurement error (53 transactions per second). And the feature used less than 10MB of memory to store all the statistics.

It might be time for us to consider making this feature enabled by default instead of disabled by default. It seems that it’s been adequately tested and the benefit is much larger than the cost (possibly infinitely, if you consider that the cost is indistinguishable from zero).