MySQL Adventures: How max_prepared_stmt_count can bring down production

We recently moved an On-Prem environment to GCP for better scalability and availability. The customer’s main database is MySQL. Due to the nature of customer’s business, it’s a highly transactional workload (one of the hot startups in APAC). To deal with the scale and meet availability requirements, we have deployed MySQL behind ProxySQL — which takes care of routing some of the resource intensive SELECTs to chosen replicas. The setup consists of:

• One Master
• Two slaves
• One Archive database server

Post migration to GCP, everything was nice and calm for a couple of weeks, until MySQL decided to start misbehaving and leading to an outage. We were able to quickly resolve and bring the system back online and what follows are lessons from this experience.

The configuration of the Database:

• CentOS 7.
• MySQL 5.6
• 32 Core CPU
• 120GB Memory
• 1 TB SSD for MySQL data volume.
• The total database size is 40GB. (yeah, it is small in size, but highly transactional)
• my.cnf is configured using Percona’s configuration wizard.
• All tables are InnoDB Engine
• No SWAP partitions.

The Problem

It all started with an alert that said MySQL process was killed by Linux’s OOM Killer. Apparently MySQL was rapidly consuming all the memory (about 120G) and OOM killer perceived it as a threat to the stability of the system and killed the process. We were perplexed and started investigating.

Sep 11 06:56:39 mysql-master-node kernel: Out of memory: Kill process 4234 (mysqld) score 980 or sacrifice child

Sep 11 06:56:39 mysql-master-node kernel: Killed process 4234 (mysqld) total-vm:199923400kB, anon-rss:120910528kB, file-rss:0kB, shmem-rss:0kB

Sep 11 06:57:00 mysql-master-node mysqld: /usr/bin/mysqld_safe: line 183:  4234 Killed   nohup /usr/sbin/mysqld --basedir=/usr --datadir=/mysqldata --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --open-files-limit=65535 --pid-file=/var/run/mysqld/mysqld.pid --socket=/mysqldata/mysql.sock < /dev/null > /dev/null 2>&1

Naturally, we started looking at mysql configuration to see if something is off.

InnoDB Parameters:

innodb-flush-method            = O_DIRECT
innodb-log-files-in-group      = 2
innodb-log-file-size           = 512M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table          = 1
innodb-buffer-pool-size        = 100G

Other Caching Parameters:

tmp-table-size                 = 32M
max-heap-table-size            = 32M
query-cache-type               = 0
query-cache-size               = 0
thread-cache-size              = 50
open-files-limit               = 65535
table-definition-cache         = 4096
table-open-cache               = 50

We are not really using query cache and one of the heavy front end service is PHP Laravel.

Here is the memory utilization graph.

The three highlighted areas are the points at which we had issues in production. The second issue happened very shortly, so we reduced the innodb-buffer-pool-size to 90GB. But even though the memory utilization never came down. So we scheduled a cronjob to flush OS Cache at least to give some addition memory to the Operating system by using the following command. This was a temporary measure till we found the actual problem.

sync; echo 3 > /proc/sys/vm/drop_cache

But This didn’t help really. The memory was still growing and we had to look at what’s really inside the OS Cache?

Fincore:

There is a tool called fincore helped me find out what’s actually the OS cache held. Its actually using Perl modules. use the below commands to install this.

yum install perl-Inline

rpm -ivh http://fr2.rpmfind.net/linux/dag/redhat/el6/en/x86_64/dag/RPMS/fincore-1.9-1.el6.rf.x86_64.rpm

It never directly shows what files are inside the buffer/cache. We instead have to manually give the path and it’ll check what files are in the cache for that location. I wanted to check about Cached files for the mysql data directory.

cd /mysql-data-directory

fincore -summary * > /tmp/cache_results

Here is the sample output of the cached files results.

page size: 4096 bytes
auto.cnf: 1 incore page: 0
dbadmin: no incore pages.
Eztaxi: no incore pages.
ibdata1: no incore pages.
ib_logfile0: 131072 incore pages: 0 1 2 3 4 5 6 7 8 9 10......
ib_logfile1: 131072 incore pages: 0 1 2 3 4 5 6 7 8 9 10......
mysql: no incore pages.
mysql-bin.000599: 8 incore pages: 0 1 2 3 4 5 6 7
mysql-bin.000600: no incore pages.
mysql-bin.000601: no incore pages.
mysql-bin.000602: no incore pages.
mysql-bin.000858: 232336 incore pages: 0 1 2 3 4 5 6 7 8 9 10......
mysqld-relay-bin.000001: no incore pages.
mysqld-relay-bin.index: no incore pages.
mysql-error.log: 4 incore pages: 0 1 2 3
mysql-general.log: no incore pages.
mysql.pid: no incore pages.
mysql-slow.log: no incore pages.
mysql.sock: no incore pages.
ON: no incore pages.
performance_schema: no incore pages.
mysql-production.pid: 1 incore page: 0

6621994 pages, 25.3 Gbytes in core for 305 files; 21711.46 pages, 4.8 Mbytes per file.

How we investigated this issue:

The first document that everyone refers is How mysql uses the memory from MySQL’s documentation. So we started with where are all the places that mysql needs memory. I’ll explain this about in a different blog. Lets continue with the steps which we did.

Make sure MySQL is the culprit:

Run the below command and this will give you the exact memory consumption about MySQL.

ps --no-headers -o "rss,cmd" -C mysqld | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"M") }'

119808M

Additional Tips:

If you want to know each mysql’s threads memory utilization, run the below command.

# Get the PID of MySQL:
ps aux | grep mysqld

mysql     4378 41.1 76.7 56670968 47314448 ?   Sl   Sep12 6955:40 /usr/sbin/mysqld --basedir=/usr --datadir=/mysqldata --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --open-files-limit=65535 --pid-file=/var/run/mysqld/mysqld.pid --socket=/mysqldata/mysql.sock

# Get all threads memory usage:
pmap -x 4378

4378:   /usr/sbin/mysqld --basedir=/usr --datadir=/mysqldata --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --open-files-limit=65535 --pid-file=/var/run/mysqld/mysqld.pid --socket=/mysqldata/mysql.sock
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000   11828    4712       0 r-x-- mysqld
000000000118d000    1720     760     476 rw--- mysqld
000000000133b000     336     312     312 rw---   [ anon ]
0000000002b62000 1282388 1282384 1282384 rw---   [ anon ]
00007fd4b4000000   47816   37348   37348 rw---   [ anon ]
00007fd4b6eb2000   17720       0       0 -----   [ anon ]
00007fd4bc000000   48612   35364   35364 rw---   [ anon ]
.........
.........
.........
00007fe1f0075000    2044       0       0 ----- libpthread-2.17.so
00007fe1f0274000       4       4       4 r---- libpthread-2.17.so
00007fe1f0275000       4       4       4 rw--- libpthread-2.17.so
00007fe1f0276000      16       4       4 rw---   [ anon ]
00007fe1f027a000     136     120       0 r-x-- ld-2.17.so
00007fe1f029c000    2012    2008    2008 rw---   [ anon ]
00007fe1f0493000      12       4       0 rw-s- [aio] (deleted)
00007fe1f0496000      12       4       0 rw-s- [aio] (deleted)
00007fe1f0499000       4       0       0 rw-s- [aio] (deleted)
00007fe1f049a000       4       4       4 rw---   [ anon ]
00007fe1f049b000       4       4       4 r---- ld-2.17.so
00007fe1f049c000       4       4       4 rw--- ld-2.17.so
00007fe1f049d000       4       4       4 rw---   [ anon ]
00007ffecc0f1000     132      72      72 rw---   [ stack ]
00007ffecc163000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- ------- 
total kB         122683392 47326976 47320388

InnoDB Buffer Pool:

Initially we suspected the InnoDB. We have checked the innoDB usage from the monitoring system. But the result was negative. It never utilized more than 40GB. That thickens the plot. If buffer pool only has 40 GB, who is eating all that memory?

Is this correct? Does Buffer Pool only hold 40GB?

What’s Inside the BufferPool and whats its size?

SELECT
    page_type AS page_type,
    sum(data_size) / 1024 / 1024 AS size_in_mb
FROM
    information_schema.innodb_buffer_page
GROUP BY
    page_type
ORDER BY
    size_in_mb DESC;

+-------------------+----------------+
| Page_Type         | Size_in_MB     |
+-------------------+----------------+
| INDEX             | 39147.63660717 |
| IBUF_INDEX        |     0.74043560 |
| UNDO_LOG          |     0.00000000 |
| TRX_SYSTEM        |     0.00000000 |
| ALLOCATED         |     0.00000000 |
| INODE             |     0.00000000 |
| BLOB              |     0.00000000 |
| IBUF_BITMAP       |     0.00000000 |
| EXTENT_DESCRIPTOR |     0.00000000 |
| SYSTEM            |     0.00000000 |
| UNKNOWN           |     0.00000000 |
| FILE_SPACE_HEADER |     0.00000000 |
+-------------------+----------------+

A quick guide about this query.

• INDEX: B-Tree index
• IBUF_INDEX: Insert buffer index
• UNKNOWN: not allocated / unknown state
• TRX_SYSTEM: transaction system data

Bonus:

To get the buffer pool usage by index

SELECT
    table_name AS table_name,
    index_name AS index_name,
    count(*) AS page_count,
    sum(data_size) / 1024 / 1024 AS size_in_mb
FROM
    information_schema.innodb_buffer_page
GROUP BY
    table_name, index_name
ORDER BY
    size_in_mb DESC;

Then where mysql was holding the Memory?

We checked all of the mysql parts where its utilizing memory. Here is a rough calculation for the memory utilization during the mysql crash.

BufferPool: 40GB
Cache/Buffer: 8GB
Performance_schema: 2GB
tmp_table_size: 32M
Open tables cache for 50 tables: 5GB
Connections, thread_cache and others: 10GB

Almost it reached 65GB, we can round it as 70GB out of 120GB. But still its approximate only. Something is wrong right? My DBA mind started to think where is the remaining?

Till now,

1. MySQL is the culprit who is consuming all of the memory.
2. Clearing OS cache never helped. Its fine.
3. Buffer Pool is also in healthy state.
4. Other memory consuming parameters are looks good.

It’s time to Dive into the MySQL.

Lets see what kind of queries are running into the mysql.

show global status like 'Com_%';
+---------------------------+-----------+
| Variable_name             | Value     |
+---------------------------+-----------+
| Com_admin_commands        | 531242406 |
| Com_stmt_execute          | 324240859 |
| Com_stmt_prepare          | 308163476 |
| Com_select                | 689078298 |
| Com_set_option            | 108828057 |
| Com_begin                 | 4457256   |
| Com_change_db             | 600       |
| Com_commit                | 8170518   |
| Com_delete                | 1164939   |
| Com_flush                 | 80        |
| Com_insert                | 73050955  |
| Com_insert_select         | 571272    |
| Com_kill                  | 660       |
| Com_rollback              | 136654    |
| Com_show_binlogs          | 2604      |
| Com_show_slave_status     | 31245     |
| Com_show_status           | 162247    |
| Com_show_tables           | 1105      |
| Com_show_variables        | 10428     |
| Com_update                | 74384469  |
+---------------------------+-----------+

Select, Insert, Update these counters are fine. But a huge amount of prepared statements were running into the mysql.

One more Tip: Valgrind

Valgrind is a powerful open source tool to profile any process’s memory consumption by threads and child processes.

Install Valgrind:

# You need C compilers, so install gcc

wget ftp://sourceware.org/pub/valgrind/valgrind-3.13.0.tar.bz2
tar -xf valgrind-3.13.0.tar.bz2 
cd valgrind-3.13.0
./configure 
make
make install

Note: Its for troubleshooting purpose, you should stop MySQL and Run with Valgrind.

• Create an log file to Capture

touch /tmp/massif.out
chown mysql:mysql /tmp/massif.out
chmod 777 /tmp/massif.out

• Run mysql with Valgrind

/usr/local/bin/valgrind --tool=massif --massif-out-file=/tmp/massif.out /usr/sbin/mysqld –default-file=/etc/my.cnf

• Lets wait for 30mins (or till the mysql takes the whole memory). Then kill the Valgranid and start mysql as normal.

Analyze the Log:

/usr/local/bin/ms_print /tmp/massif.out

We’ll explain mysql memory debugging using valgrind in an another blog.

Memory Leak:

We have verified all the mysql parameters and OS level things for the memory consumption. But no luck. So I started to think and search about mysql’s memory leak parts. Then I found this awesome blog by Todd.

Yes, the only parameter I didn’t check is max_prepared_stmt_count.

What is this?

From MySQL’s Doc,

This variable limits the total number of prepared statements in the server. It can be used in environments where there is the potential for denial-of-service attacks based on running the server out of memory by preparing huge numbers of statements.

• Whenever we prepared a statement, we should close in the end. Else it’ll not the release the memory which is allocated to it.
• For executing a single query, it’ll do three executions (Prepare, Run the query and close).
• There is no visibility that how much memory is consumed by a prepared statement.

Is this the real root cause?

Run this query to check how many prepared statements are running in mysql server.

mysql> show global status like 'com_stmt%';

+-------------------------+-----------+
| Variable_name           | Value     |
+-------------------------+-----------+
| Com_stmt_close          | 0         |
| Com_stmt_execute        | 210741581 |
| Com_stmt_fetch          | 0         |
| Com_stmt_prepare        | 199559955 |
| Com_stmt_reprepare      | 1045729   |
| Com_stmt_reset          | 0         |
| Com_stmt_send_long_data | 0         |
+-------------------------+-----------+

You can see there are 1045729 prepared statements are running and the Com_stmt_close variables is showing none of the statements are closed.

This query will return the max count for the preparements.

mysql> show variables like 'max_prepared_stmt_count';
+-------------------------+---------+
| Variable_name           | Value   |
+-------------------------+---------+
| max_prepared_stmt_count | 1048576 |
+-------------------------+---------+

Oh, its the maximum value for this parameter. Then we immediately reduced it to 2000.

mysql> set global max_prepared_stmt_count=2000;

-- Add this to my.cnf
vi /etc/my.cnf

[mysqld]
max_prepared_stmt_count = 2000

Now, the mysql is running fine and the memory leak is fixed. Till now the memory utilization is normal. In Laravel framework, its almost using this prepared statement. We can see so many laravel + prepare statements questions in StackOverflow.

Conclusion:

The very important lesson as a DBA I learned is, before setting up any parameter value check the consequences of modifying it and make sure it should not affect the production anymore. Now the mysql side is fine, but the application was throwing the below error.

Can't create more than max_prepared_stmt_count statements (current value: 20000)

To continue about this series, the next blog post will explain how we fixed the above error using multiplexing and how it helped to dramatically reduce the mysql’s memory utilization.

How max_prepared_stmt_count bring down the production MySQL system was originally published in Searce Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

MySQL Adventures: Reduce MySQL Memory Utilization With ProxySQL Multiplexing

In our previous post, we explained about how max_prepared_statement_count can bring production down . This blog is the continuity of that post. If you can read that blog from the below link.

How max_prepared_stmt_count bring down the production MySQL system

We had set the max_prepared_stmt_count to 20000. But after that, we were facing the below error continuously.

Can't create more than max_prepared_stmt_count statements (current value: 20000)

We tried to increase it to 25000, 30000 and finally 50000. But unfortunately, we can’t fix it and increasing this value will lead to a memory leak which we explained in our previous blog.

We are using ProxySQL to access the database servers. And the architecture looks like below.

Multiplexing in ProxySQL:

The main purpose of the multiplexing is to reduce the connections to MySQL servers. So we can send thousands of connections to only a hundred backend connections.

We enabled multiplexing while setting up the Database environment. But multiplexing will not be work in all the times. ProxySQL has some sense to track the transactions which are executing in that connection. If any transactions are not committed or rollback then, it’ll never use that connection for the next request. It’ll pick another free connection from the connection pool.

From the ProxySQL Doc, there are few scenarios where multiplexing is disabled.

• active transaction
• Tables are locked.
• Set queries (like SET FOREIGN_KEY_CHECKS)

In our case, the most of the errors are due to prepare statement count. Believe it, this issue made us to reduce the memory utilization also.

Get the currently active prepared statements:

Run the below query which will give tell us the number of active prepare statements in the backend.

SELECT
    *
FROM
    stats_mysql_global
WHERE
    variable_name LIKE '%stmt%';

+---------------------------+----------------+
| Variable_Name             | Variable_Value |
+---------------------------+----------------+
| Com_backend_stmt_prepare  | 168318911      |
| Com_backend_stmt_execute  | 143165882      |
| Com_backend_stmt_close    | 0              |
| Com_frontend_stmt_prepare | 171609010      |
| Com_frontend_stmt_execute | 171234713      |
| Com_frontend_stmt_close   | 171234006      |
| Stmt_Client_Active_Total  | 19983          |
| Stmt_Client_Active_Unique | 4              |
| Stmt_Server_Active_Total  | 84             |
| Stmt_Server_Active_Unique | 23             |
| Stmt_Max_Stmt_id          | 10002          |
| Stmt_Cached               | 2186           |
+---------------------------+----------------+

# You can get the same results in MySQL also

MySQL> show status like 'Prepared_stmt_count';
+---------------------+-------+
| Variable_name       | Value |
+---------------------+-------+
| Prepared_stmt_count | 19983 |
+---------------------+-------+

You can see the number of active prepare statements are 19983. while running the query again and again, I could see a random count but those all are more than 19000. And you can see the Com_backend_stmt_close is 0.

Yes, ProxySQL will never close the prepared statements in the backend. But there is a mechanism in ProxySQL which allocates 20 prepared statements(20 is the default value) to each connection. Once its executed all 20 statements then the connection will come back to the connection pool and close all 20 statements in one shot.

Run the below query to get the default statement count for a connection.

proxysql> show variables like "%stmt%";

+--------------------------------+-------+
| Variable_name                  | Value |
+--------------------------------+-------+
| mysql-max_stmts_per_connection | 20    |
| mysql-max_stmts_cache          | 10000 |
+--------------------------------+-------+

There is a great explanation about this variable by René Cannaò who is the founder of ProxtSQL. You can read about that here.

Why this much prepared statements are running?

As mentioned earlier, proxysql will never close the prepared statements in the backend. We realize that we are getting heavy traffic on both ProxySQL servers and its send it to one Master node. And also the Laravel has all the queries with prepared statement format. That's why we are getting this much prepared statements.

Get where the most of the prepared statements are used:

Run the below query in proxySQL and this will give you the total count of executed prepared statements on all the databases and the usernames.

SELECT
    username, schemaname, count(*)
FROM
    stats_mysql_prepared_statements_info
GROUP BY
    1, 2
ORDER BY
    3 DESC;

+----------+---------------+----------+
| username | schemaname    | count(*) |
+----------+---------------+----------+
| DBname   |    user1      | 2850     |
| DBname   |    user2      | 257      |
| DBname   |    user3      | 108      |
| DBname   |    user1      | 33       |
| DBname   |    user2      | 33       |
| DBname   |    user1      | 16       |
| DBname   |    user1      | 15       |
+----------+---------------+----------+

#There is a Bug in this view. The Username column is actually showing the schemaname and the schemaname column is showing usernames. I have reported this bug in proxySQL's github repo.

https://github.com/sysown/proxysql/issues/1720

Force ProxySQL to use multiplex:

There are few cases proxysql will disable the multiplexing. Particularly all queries that have @ in their query_digest will disable multiplexing.

Collect the queries which has @

SELECT
    DISTINCT digest, digest_text
FROM
    stats_mysql_query_digest
WHERE
    digest_text LIKE '%@%' \G;

*************************** 1. row ***************************
     digest: 0xA8F2FFB14312850C
digest_text: SELECT @@max_allowed_packet

*************************** 2. row ***************************
     digest: 0x7CDEEF2FF695B7F8
digest_text: SELECT @@session.tx_isolation

*************************** 3. row ***************************
     digest: 0x2B838C3B5DE79958
digest_text: SELECT @@session.auto_increment_increment AS auto_increment_increment, @@character_set_client AS character_set_client, @@character_set_connection AS character_set_connection, @@character_set_results AS character_set_results, @@character_set_server AS character_set_server, @@init_connect AS init_connect, @@interactive_timeout AS interactive_timeout, @@license AS license, @@lower_case_table_names AS lower_case_table_names, @@max_allowed_packet AS max_allowed_packet, @@net_buffer_length AS net_buffer_length, @@net_write_timeout AS net_write_timeout, @@query_cache_size AS query_cache_size, @@query_cache_type AS query_cache_type, @@sql_mode AS sql_mode, @@system_time_zone AS system_time_zone, @@time_zone AS time_zone, @@tx_isolation AS tx_isolation, @@wait_timeout AS wait_timeout

Finally, the above statements are executed by prepared statements. These queries are by default disabled the multiplexing. But ProxySQL has another cool feature that we can allow these queries (which has @ ) to use multiplexing.

Run the below query to set proxysql to use multiplexing for these queries. We can insert it by Query pattern or query digest.

# Add multiplexing to a query using query_text

INSERT INTO mysql_query_rules 
(active, match_digest, multiplex)
VALUES
('1', '^SELECT @@session.tx_isolation', 2);

INSERT INTO mysql_query_rules 
(active, match_digest, multiplex)
VALUES
('1', '^SELECT @@max_allowed_packet', 2);

# Add multiplexing to a query using query Diagest

INSERT INTO mysql_query_rules 
(active, digest, multiplex)
VALUES
('1', '0x2B838C3B5DE79958', 2);

# Save it to Runtime and Disk

LOAD MYSQL QUERY RULES TO RUNTIME;
SAVE MYSQL QUERY RULES TO DISK;

Restart ProxySQL:

If we enabled multiplexing in proxySQL then its mandatory to restart ProxySQL service or close all the existing connections and open it as a new one.

service proxysql restart

Lets check the active prepared statements in both ProxySQL and MySQL.

#proxySQL

SELECT
    *
FROM
    stats_mysql_global
WHERE
    variable_name LIKE '%stmt%';

+--------------------------+----------------+
| Variable_Name            | Variable_Value |
+--------------------------+----------------+
| Stmt_Client_Active_Total | 6              |
+--------------------------+----------------+

#mysql
show status like 'Prepared_stmt_count';
+---------------------+-------+
| Variable_name       | Value |
+---------------------+-------+
| Prepared_stmt_count | 166   |
+---------------------+-------+

The number of Prepared_stmt_count is dramatically reduced from 20000 to 200. But why there is a different count between proxySQL and mysql?

Again refer to the ProxySQL’s doc. Once whenever a connection executed 20 statements, then only it’ll Close that prepared statement. In ProxySQL, its showing active statements. But MySQL is showing active + executed statements count.

Number of Backend connections:

After this change, there is a sudden drop in a number of backend connections in the ProxySQL. This proves that the statements which disable the multiplexing will create more backend connections.

MySQL’s Memory:

Now we can see the explanation for this blog title. See the below graph which is showing the high memory drop.

The two main parts where mysql used more memory:

1. mysql connections.
2. Prepared statements.

We all already knows that each mysql connection requires some amount of memory. And for prepared statements, I have explained about its memory consumption in my previous blog.

Conclusion:

ProxySQL has a lot of great features. Multiplexing is good. But after this incident only we realize that multiplexing will help to reduce the number of backend connections and MySQL’s memory utilization as well.

I hope if you are a DBA you will read this and implement it in your environment as well. If it helped for you then feel free to give your claps.

Reduce MySQL Memory Utilization With ProxySQL Multiplexing was originally published in Searce Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introducing the MySQL 8 Document Store

MySQL and JSON A Practical Programming Guide

MySQL Connector/Python Revealed

Pro MySQL NDB Cluster

Recommendation

The FOSDEM organization just confirmed that again this year the ecosystem of your favorite database will have its Devroom !

More info to come soon, but save the day: 2 & 3rd February 2019 in Brussels !

It seems the MySQL & Friends Devroom  (MariaDB, Percona, Oracle, and all tools in the ecosystem) will be held on Saturday (to be confirmed).

Stay tuned !

The MySQL Development team is very happy to announce that MySQL 8.0.13, the second 8.0 Maintenance Release, is now available for download at dev.mysql.com. In addition to bug fixes there are a few new features added in this release.  Please download 8.0.13 from dev.mysql.com or from the MySQL  Yum,  APT, or SUSE repositories.…

Probably not well known but quite an important optimization was introduced in MySQL 5.6 – reduced overhead for “read only transactions”. While usually by a “transaction” we mean a query or a group of queries that change data, with transaction engines like InnoDB, every data read or write operation is a transaction.

Now, as a non-locking read operation obviously has less impact on the data, it does not need all the instrumenting overhead a write transaction has. The main thing that can be avoided, as described by documentation, is the transaction ID. So, since MySQL 5.6, a read only transaction does not have a transaction ID. Moreover, such a transaction is not visible in the SHOW ENGINE INNODB STATUS output, though I will not go deeper on what really that means under the hood in this article. The fact is that this optimization allows for better scaling of workloads with many RO threads. An example RO benchmark, where 5.5 vs 5.6/5.7 difference is well seen, may be found here: https://www.percona.com/blog/2016/04/07/mysql-5-7-sysbench-oltp-read-results-really-faster/

To benefit from this optimization in MySQL 5.6, either a transaction has to start with the explicit START TRANSACTION READ ONLY clause or it must be an autocommit, non-locking SELECT statement. In version 5.7 and newer, it goes further, as a new transaction is treated as read-only until a locking read or write is executed, at which point it gets “upgraded” to a read-write one.

Information Schema Instrumentation

Let’s see how it looks like (on MySQL 8.0.12) by looking at information_schema.innodb_trx and information_schema.innodb_metrics tables. The second of these, by default, has transaction counters disabled, so before the test we have to enable it with:

SET GLOBAL innodb_monitor_enable = 'trx%comm%';

or by adding a parameter to the

[mysqld]

innodb_monitor_enable = "trx_%"

Now, let’s start a transaction which should be read only according to the rules:

mysql [localhost] {msandbox} (db1) > START TRANSACTION; SELECT count(*) FROM db1.t1;
Query OK, 0 rows affected (0.00 sec)
+----------+
| count(*) |
+----------+
|        3 |
+----------+
1 row in set (0.00 sec
mysql [localhost] {msandbox} (db1) > SELECT trx_id,trx_weight,trx_rows_locked,trx_rows_modified,trx_is_read_only,trx_autocommit_non_locking
FROM information_schema.innodb_trx\G
*************************** 1. row ***************************
                    trx_id: 421988493944672
                trx_weight: 0
           trx_rows_locked: 0
         trx_rows_modified: 0
          trx_is_read_only: 0
trx_autocommit_non_locking: 0
1 row in set (0.00 sec)

Transaction started as above, did not appear in SHOW ENGINE INNODB STATUS, and its trx_id looks strangely high. And first surprise—for some reason, trx_is_read_only is 0. Now, what if we commit such a transaction—how do the counters change? (I reset them before the test):

mysql [localhost] {msandbox} (db1) > commit;
Query OK, 0 rows affected (0.00 sec)
mysql [localhost] {msandbox} (db1) > SELECT name, comment, status, count
FROM information_schema.innodb_metrics   WHERE name like 'trx%comm%';
+---------------------------+--------------------------------------------------------------------+---------+-------+
| name                      | comment                                                            | status  | count |
+---------------------------+--------------------------------------------------------------------+---------+-------+
| trx_rw_commits            | Number of read-write transactions  committed                       | enabled |     0 |
| trx_ro_commits            | Number of read-only transactions committed                         | enabled |     1 |
| trx_nl_ro_commits         | Number of non-locking auto-commit read-only transactions committed | enabled |     0 |
| trx_commits_insert_update | Number of transactions committed with inserts and updates          | enabled |     0 |
+---------------------------+--------------------------------------------------------------------+---------+-------+
4 rows in set (0.01 sec)

OK, so clearly it was a read-only transaction overall, just the trx_is_read_only property wasn’t set as expected. I had to report this problem here: https://bugs.mysql.com/bug.php?id=92558

What about an explicit RO transaction:

mysql [localhost] {msandbox} (db1) > START TRANSACTION READ ONLY; SELECT count(*) FROM db1.t1;
Query OK, 0 rows affected (0.00 sec)
+----------+
| count(*) |
+----------+
|        3 |
+----------+
1 row in set (0.00 sec
mysql [localhost] {msandbox} (db1) > SELECT trx_id,trx_weight,trx_rows_locked,trx_rows_modified,trx_is_read_only,trx_autocommit_non_locking
FROM information_schema.innodb_trx\G
*************************** 1. row ***************************
                    trx_id: 421988493944672
                trx_weight: 0
           trx_rows_locked: 0
         trx_rows_modified: 0
          trx_is_read_only: 1
trx_autocommit_non_locking: 0
1 row in set (0.00 sec)
mysql [localhost] {msandbox} (db1) > commit;
Query OK, 0 rows affected (0.00 sec)
mysql [localhost] {msandbox} (db1) > SELECT name, comment, status, count
FROM information_schema.innodb_metrics   WHERE name like 'trx%comm%';
+---------------------------+--------------------------------------------------------------------+---------+-------+
| name                      | comment                                                            | status  | count |
+---------------------------+--------------------------------------------------------------------+---------+-------+
| trx_rw_commits            | Number of read-write transactions  committed                       | enabled |     0 |
| trx_ro_commits            | Number of read-only transactions committed                         | enabled |     2 |
| trx_nl_ro_commits         | Number of non-locking auto-commit read-only transactions committed | enabled |     0 |
| trx_commits_insert_update | Number of transactions committed with inserts and updates          | enabled |     0 |
+---------------------------+--------------------------------------------------------------------+---------+-------+
4 rows in set (0.01 sec)

OK, both transactions are counted as the same type. Moreover, the two transactions shared the same strange trx_id, which appears to be a fake one. For a simple read executed in autocommit mode, the counters increase as expected too:

mysql [localhost] {msandbox} (db1) > select @@autocommit; SELECT count(*) FROM db1.t1;
+--------------+
| @@autocommit |
+--------------+
|            1 |
+--------------+
1 row in set (0.00 sec)
+----------+
| count(*) |
+----------+
|        3 |
+----------+
1 row in set (0.00 sec)
mysql [localhost] {msandbox} (db1) > SELECT name, comment, status, count
FROM information_schema.innodb_metrics   WHERE name like 'trx%comm%';
+---------------------------+--------------------------------------------------------------------+---------+-------+
| name                      | comment                                                            | status  | count |
+---------------------------+--------------------------------------------------------------------+---------+-------+
| trx_rw_commits            | Number of read-write transactions  committed                       | enabled |     0 |
| trx_ro_commits            | Number of read-only transactions committed                         | enabled |     2 |
| trx_nl_ro_commits         | Number of non-locking auto-commit read-only transactions committed | enabled |     1 |
| trx_commits_insert_update | Number of transactions committed with inserts and updates          | enabled |     0 |
+---------------------------+--------------------------------------------------------------------+---------+-------+
4 rows in set (0.00 sec)

Now, let’s test how a transaction looks when we upgrade it to RW later:

mysql [localhost] {msandbox} (db1) > START TRANSACTION; SELECT count(*) FROM db1.t1;
Query OK, 0 rows affected (0.00 sec)
+----------+
| count(*) |
+----------+
|        3 |
+----------+
1 row in set (0.00 sec)
mysql [localhost] {msandbox} (db1) > SELECT trx_id,trx_weight,trx_rows_locked,trx_rows_modified,trx_is_read_only,trx_autocommit_non_locking
FROM information_schema.innodb_trx\G
*************************** 1. row ***************************
                    trx_id: 421988493944672
                trx_weight: 0
           trx_rows_locked: 0
         trx_rows_modified: 0
          trx_is_read_only: 0
trx_autocommit_non_locking: 0
1 row in set (0.00 sec)
mysql [localhost] {msandbox} (db1) > SELECT count(*) FROM db1.t1 FOR UPDATE;
+----------+
| count(*) |
+----------+
|        3 |
+----------+
1 row in set (0.00 sec)
mysql [localhost] {msandbox} (db1) > SELECT trx_id,trx_weight,trx_rows_locked,trx_rows_modified,trx_is_read_only,trx_autocommit_non_locking
FROM information_schema.innodb_trx\G
*************************** 1. row ***************************
                    trx_id: 4106
                trx_weight: 2
           trx_rows_locked: 4
         trx_rows_modified: 0
          trx_is_read_only: 0
trx_autocommit_non_locking: 0
1 row in set (0.00 sec)
mysql [localhost] {msandbox} (db1) > commit;
Query OK, 0 rows affected (0.00 sec)
mysql [localhost] {msandbox} (db1) > SELECT name, comment, status, count
FROM information_schema.innodb_metrics   WHERE name like 'trx%comm%';
+---------------------------+--------------------------------------------------------------------+---------+-------+
| name                      | comment                                                            | status  | count |
+---------------------------+--------------------------------------------------------------------+---------+-------+
| trx_rw_commits            | Number of read-write transactions  committed                       | enabled |     1 |
| trx_ro_commits            | Number of read-only transactions committed                         | enabled |     2 |
| trx_nl_ro_commits         | Number of non-locking auto-commit read-only transactions committed | enabled |     1 |
| trx_commits_insert_update | Number of transactions committed with inserts and updates          | enabled |     0 |
+---------------------------+--------------------------------------------------------------------+---------+-------+
4 rows in set (0.00 sec)

OK, as seen above, after a locking read was done, our transaction has transformed: it got a real, unique trx_id assigned. Then, when committed, the RW counter increased.

Performance Schema Problem

Nowadays it may feel natural to use performance_schema for monitoring everything. And, indeed, we can monitor types of transactions with it as well. Let’s enable the needed consumers and instruments:

mysql [localhost] {msandbox} (db1) > UPDATE performance_schema.setup_consumers SET ENABLED = 'YES' WHERE NAME LIKE '%transactions%';
Query OK, 0 rows affected (0.00 sec)
Rows matched: 3  Changed: 0  Warnings: 0
mysql [localhost] {msandbox} (db1) > UPDATE performance_schema.setup_instruments SET ENABLED = 'YES', TIMED = 'YES' WHERE NAME = 'transaction';
Query OK, 0 rows affected (0.01 sec)
Rows matched: 1  Changed: 0  Warnings: 0
mysql [localhost] {msandbox} (db1) > SELECT * FROM performance_schema.setup_instruments WHERE NAME = 'transaction';
+-------------+---------+-------+------------+------------+---------------+
| NAME        | ENABLED | TIMED | PROPERTIES | VOLATILITY | DOCUMENTATION |
+-------------+---------+-------+------------+------------+---------------+
| transaction | YES     | YES   |            |          0 | NULL          |
+-------------+---------+-------+------------+------------+---------------+
1 row in set (0.00 sec)
mysql [localhost] {msandbox} (db1) > SELECT * FROM performance_schema.setup_consumers WHERE NAME LIKE '%transactions%';
+----------------------------------+---------+
| NAME                             | ENABLED |
+----------------------------------+---------+
| events_transactions_current      | YES     |
| events_transactions_history      | YES     |
| events_transactions_history_long | YES     |
+----------------------------------+---------+
3 rows in set (0.01 sec)
mysql [localhost] {msandbox} (db1) > SELECT COUNT_STAR,COUNT_READ_WRITE,COUNT_READ_ONLY
FROM performance_schema.events_transactions_summary_global_by_event_name\G
*************************** 1. row ***************************
      COUNT_STAR: 0
COUNT_READ_WRITE: 0
 COUNT_READ_ONLY: 0
1 row in set (0.00 sec)

And let’s do some simple tests:

mysql [localhost] {msandbox} (db1) > START TRANSACTION; COMMIT;
Query OK, 0 rows affected (0.01 sec)
Query OK, 0 rows affected (0.00 sec)
mysql [localhost] {msandbox} (db1) > SELECT COUNT_STAR,COUNT_READ_WRITE,COUNT_READ_ONLY
FROM performance_schema.events_transactions_summary_global_by_event_name\G
*************************** 1. row ***************************
      COUNT_STAR: 1
COUNT_READ_WRITE: 1
 COUNT_READ_ONLY: 0
1 row in set (0.00 sec)
mysql [localhost] {msandbox} (db1) > SELECT name, comment, status, count
FROM information_schema.innodb_metrics   WHERE name like 'trx%comm%';
+---------------------------+--------------------------------------------------------------------+---------+-------+
| name                      | comment                                                            | status  | count |
+---------------------------+--------------------------------------------------------------------+---------+-------+
| trx_rw_commits            | Number of read-write transactions  committed                       | enabled |     0 |
| trx_ro_commits            | Number of read-only transactions committed                         | enabled |     0 |
| trx_nl_ro_commits         | Number of non-locking auto-commit read-only transactions committed | enabled |     0 |
| trx_commits_insert_update | Number of transactions committed with inserts and updates          | enabled |     0 |
+---------------------------+--------------------------------------------------------------------+---------+-------+
4 rows in set (0.00 sec)

A void transaction caused an increase to this RW counter in Performance Schema view! Moreover, a simple autocommit select increases it too:

mysql [localhost] {msandbox} (db1) > SELECT count(*) FROM db1.t1;
+----------+
| count(*) |
+----------+
|        3 |
+----------+
1 row in set (0.01 sec)
mysql [localhost] {msandbox} (db1) > SELECT COUNT_STAR,COUNT_READ_WRITE,COUNT_READ_ONLY
FROM performance_schema.events_transactions_summary_global_by_event_name\G
*************************** 1. row ***************************
      COUNT_STAR: 2
COUNT_READ_WRITE: 2
 COUNT_READ_ONLY: 0
1 row in set (0.00 sec)
mysql [localhost] {msandbox} (db1) > START TRANSACTION READ ONLY; COMMIT;
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
mysql [localhost] {msandbox} (db1) > SELECT COUNT_STAR,COUNT_READ_WRITE,COUNT_READ_ONLY
FROM performance_schema.events_transactions_summary_global_by_event_name\G
*************************** 1. row ***************************
      COUNT_STAR: 3
COUNT_READ_WRITE: 2
 COUNT_READ_ONLY: 1
1 row in set (0.00 sec)
mysql [localhost] {msandbox} (db1) > SELECT name, comment, status, count
FROM information_schema.innodb_metrics   WHERE name like 'trx%comm%';
+---------------------------+--------------------------------------------------------------------+---------+-------+
| name                      | comment                                                            | status  | count |
+---------------------------+--------------------------------------------------------------------+---------+-------+
| trx_rw_commits            | Number of read-write transactions  committed                       | enabled |     0 |
| trx_ro_commits            | Number of read-only transactions committed                         | enabled |     0 |
| trx_nl_ro_commits         | Number of non-locking auto-commit read-only transactions committed | enabled |     1 |
| trx_commits_insert_update | Number of transactions committed with inserts and updates          | enabled |     0 |
+---------------------------+--------------------------------------------------------------------+---------+-------+
4 rows in set (0.01 sec)

As seen above, with regard to monitoring transactions via Performance Schema, everything seems completely broken, empty transactions increase counters, and the only way to increase RO counter is to call a read-only transaction explicitly, but again, it should not count when no real read was done from a table. For this reason I filed another bug report: https://bugs.mysql.com/bug.php?id=92364

PMM Dashboard

We implemented a transactions information view in PMM, based on Information_schema.innodb_metrics, which—as presented above—is reliable and shows the correct counters. Therefore, I encourage everyone to use the innodb_monitor_enable setting to enable it and have the PMM graph it. It will look something like this:

The post Instrumenting Read Only Transactions in InnoDB appeared first on Percona Database Performance Blog.

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and monitoring MySQL® and MongoDB® performance. You can run PMM in your own environment for maximum security and reliability. It provides thorough time-based analysis for MySQL® and MongoDB® servers to ensure that your data works as efficiently as possible.

This release offers two new features for both the MySQL Community and Percona Customers:

• MySQL Custom Queries – Turn a SELECT into a dashboard!
• Server and Client logs – Collect troubleshooting logs for Percona Support

We addressed 17 new features and improvements, and fixed 17 bugs.

MySQL Custom Queries

In 1.15 we are introducing the ability to take a SQL SELECT statement and turn the result set into metric series in PMM.  The queries are executed at the LOW RESOLUTION level, which by default is every 60 seconds.  A key advantage is that you can extend PMM to profile metrics unique to your environment (see users table example), or to introduce support for a table that isn’t part of PMM yet. This feature is on by default and only requires that you edit the configuration file and use vaild YAML syntax.  The configuration file is in /usr/local/percona/pmm-client/queries-mysqld.yml.

Example – Application users table

We’re going to take a fictional MySQL users table that also tracks the number of upvotes and downvotes, and we’ll convert this into two metric series, with a set of seven labels, where each label can also store a value.

Browsing metrics series using Advanced Data Exploration Dashboard

Lets look at the output so we understand the goal – take data from a MySQL table and store in PMM, then display as a metric series.  Using the Advanced Data Exploration Dashboard you can review your metric series. Exploring the metric series  app1_users_metrics_downvotes we see the following:

MySQL table

Lets assume you have the following users table that includes true/false, string, and integer types.

SELECT * FROM `users`
+----+------+--------------+-----------+------------+-----------+---------------------+--------+---------+-----------+
| id | app  | user_type    | last_name | first_name | logged_in | active_subscription | banned | upvotes | downvotes |
+----+------+--------------+-----------+------------+-----------+---------------------+--------+---------+-----------+
|  1 | app2 | unprivileged | Marley    | Bob        |         1 |                   1 |      0 |     100 |        25 |
|  2 | app3 | moderator    | Young     | Neil       |         1 |                   1 |      1 |     150 |        10 |
|  3 | app4 | unprivileged | OConnor   | Sinead     |         1 |                   1 |      0 |      25 |        50 |
|  4 | app1 | unprivileged | Yorke     | Thom       |         0 |                   1 |      0 |     100 |       100 |
|  5 | app5 | admin        | Buckley   | Jeff       |         1 |                   1 |      0 |     175 |         0 |
+----+------+--------------+-----------+------------+-----------+---------------------+--------+---------+-----------+

Explaining the YAML syntax

We’ll go through a simple example and mention what’s required for each line.  The metric series is constructed based on the first line and appends the column name to form metric series.  Therefore the number of metric series per table will be the count of columns that are of type GAUGE or COUNTER.  This metric series will be called app1_users_metrics_downvotes:

app1_users_metrics:                                 ## leading section of your metric series.
  query: "SELECT * FROM app1.users"                 ## Your query. Don't forget the schema name.
  metrics:                                          ## Required line to start the list of metric items
    - downvotes:                                    ## Name of the column returned by the query. Will be appended to the metric series.
        usage: "COUNTER"                            ## Column value type.  COUNTER will make this a metric series.
        description: "Number of upvotes"            ## Helpful description of the column.

Full queries-mysqld.yml example

Each column in the SELECT is named in this example, but that isn’t required, you can use a SELECT * as well.  Notice the format of schema.table for the query is included.

---
app1_users_metrics:
  query: "SELECT app,first_name,last_name,logged_in,active_subscription,banned,upvotes,downvotes FROM app1.users"
  metrics:
    - app:
        usage: "LABEL"
        description: "Name of the Application"
    - user_type:
        usage: "LABEL"
        description: "User's privilege level within the Application"
    - first_name:
        usage: "LABEL"
        description: "User's First Name"
    - last_name:
        usage: "LABEL"
        description: "User's Last Name"
    - logged_in:
        usage: "LABEL"
        description: "User's logged in or out status"
    - active_subscription:
        usage: "LABEL"
        description: "Whether User has an active subscription or not"
    - banned:
        usage: "LABEL"
        description: "Whether user is banned or not"
    - upvotes:
        usage: "COUNTER"
        description: "Count of upvotes the User has earned.  Upvotes once granted cannot be revoked, so the number can only increase."
    - downvotes:
        usage: "GAUGE"
        description: "Count of downvotes the User has earned.  Downvotes can be revoked so the number can increase as well as decrease."
...

We hope you enjoy this feature, and we welcome your feedback via the Percona forums!

Server and Client logs

We’ve enhanced the volume of data collected from both the Server and Client perspectives.  Each service provides a set of files designed to be shared with Percona Support while you work on an issue.

Server

From the Server, we’ve improved the logs.zip service to include:

• Prometheus targets
• Consul nodes, QAN API instances
• Amazon RDS and Aurora instances
• Version
• Server configuration
• Percona Toolkit commands

You retrieve the link from your PMM server using this format:   https://pmmdemo.percona.com/managed/logs.zip

Client

On the Client side we’ve added a new action called summary which fetches logs, network, and Percona Toolkit output in order to share with Percona Support. To initiate a Client side collection, execute:

pmm-admin summary

The output will be a file you can use to attach to your Support ticket.  The single file will look something like this:

summary__2018_10_10_16_20_00.tar.gz

New Features and Improvements

• PMM-2913 – Provide ability to execute Custom Queries against MySQL – Credit to wrouesnel for the framework of this feature in wrouesnel/postgres_exporter!
• PMM-2904 – Improve PMM Server Diagnostics for Support
• PMM-2860 – Improve pmm-client Diagnostics for Support
• PMM-1754 – Provide functionality to easily select query and copy it to clipboard in QAN
• PMM-1855 – Add swap to AMI
• PMM-3013 – Rename PXC Overview graph Sequence numbers of transactions to IST Progress
• PMM-2726 – Abort data collection in Exporters based on Prometheus Timeout – MySQLd Exporter
• PMM-3003 – PostgreSQL Overview Dashboard Tooltip fixes
• PMM-2936 – Some improvements for Query Analytics Settings screen
  
• PMM-3029 – PostgreSQL Dashboard Improvements

Fixed Bugs

• PMM-2976 – Upgrading to PMM 1.14.x fails if dashboards from Grafana 4.x are present on an installation
• PMM-2969 – rds_exporter becomes throttled by CloudWatch API
• PMM-1443 – The credentials for a secured server are exposed without explicit request
• PMM-3006 – Monitoring over 1000 instances is displayed imperfectly on the label
• PMM-3011 – PMM’s default MongoDB DSN is localhost, which is not resolved to IPv4 on modern systems
• PMM-2211 – Bad display when using old range in QAN
• PMM-1664 – Infinite loading with wrong queryID
• PMM-2715 – Since pmm-client-1.9.0, pmm-admin detects CentOS/RHEL 6 installations using linux-upstart as service manager and ignores SysV scripts
• PMM-2839 – Tablestats safety precaution does not work for RDS/Aurora instances
• PMM-2845 – pmm-admin purge causes client to panic
• PMM-2968 – pmm-admin list shows empty data source column for mysql:metrics
• PMM-3043 – Total Time percentage is incorrectly shown as a decimal fraction
• PMM-3082 – Prometheus Scrape Interval Variance chart doesn’t display data

How to get PMM Server

PMM is available for installation using three methods:

• On Docker Hub – docker pull percona/pmm-server – Documentation
• AWS Marketplace – Documentation
• Open Virtualization Format (OVF) – Documentation

Help us improve our software quality by reporting any Percona Monitoring and Management bugs you encounter using our bug tracking system.

The post Percona Monitoring and Management (PMM) 1.15.0 Is Now Available appeared first on Percona Database Performance Blog.

In a DBA’s day to day activities, we are doing Archive operation on our transnational database servers to improve your queries and control the Disk space. The archive is a most expensive operation since its involved a huge number of Read and Write will be performed. So its mandatory to run the archive queries in …

The post Archive MySQL Data In Chunks Using Stored Procedure appeared first on SQLgossip.

Good news ! The MySQL, MariaDB & Friends Devroom has been accepted for FOSDEM’19 ‘s edition as announced earlier!

This event is a real success story for the MySQL ecosystem; the content, the speakers and the attendees are growing every year.

The first big change for this 2019 edition is that the MariaDB Foundation (Ian) is joining my efforts to build this Devroom. Don’t forget that FOSDEM takes place in Belgium, and our motto is “l’Union fait la Force” [“Unity is Strength”].

FOSDEM 2019’s edition will take place 2nd & 3rd February in Brussels and our MySQL, MariaDB & Friends devroom will run on Saturday 2nd (may change). FOSDEM & MySQL/MariaDB is a love story started 19 years ago !

The committee selecting the content for our devroom is not yet created and if you want to be part of this experience, just send me an email (candidate at mysqlmariadbandfriends dot eu) before Oct 29th.

If you want to join the Committee you have to align with the following conditions:

• planning to be present at FOSDEM
• having a link with MySQL & MariaDB Ecosystem
• have some time to review and rate talks
• be an ambassador for the event by promoting it

The Call for Paper is now offcialy open and ends November 15th. You can submit now() your proposal using FOSDEM’s submission tool.

Marketing and Sales speeches are not welcome, focus on the engineering, the operations and of course the developers.

Don’t forget to specify the track (MySQL, MariaDB and Friends devroom) and set the duration to 20 mins (short but intense ;-) )!

Thank you, and see you soon in Brussels !

lefred

After much speculation following the announcement in Santa Clara earlier this year, we are delighted to announce Percona Live 2019 will be taking place in Austin, Texas.

Save the dates in your diary for May, 28-30 2019!

The conference will take place just after Memorial Day at The Hyatt Regency, Austin on the shores of Lady Bird Lake.

This is also an ideal central location for those who wish to extend their stay and explore what Austin has to offer! Call for papers, ticket sales and sponsorship opportunities will be announced soon, so stay tuned!

In other Percona Live news, we’re less than 4 weeks away from this year’s European conference taking place in Frankfurt, Germany on 5-7 November. The tutorials and breakout sessions have been announced, and you can view the full schedule here. Tickets are still on sale so don’t miss out, book yours here today!

The post Percona Live 2019 – Save the Date! appeared first on Percona Database Performance Blog.

So, if we’re applying GDPR to our system, and we’re already making use of MySQL Transparent Data Encryption / keyring, then here’s an example on how to migrate from filed-based keyring to the encrypted keyring. Online.

If you’re looking to go deeper into the TDE then I suggest reading the MySQL Server Team’s InnoDB Transparent Tablespace Encryption blog.

You’d already have your environment running, whereas I have to create one.. give me a minute please, 8.0.12 here we come:

mysqld --defaults-file=my_okv.cnf --initialize-insecure --user=khollman
mysqld --defaults-file=my_okv.cnf --user=khollman &

mysql --defaults-file=my_okv.cnf -uroot
show plugins;
show variables like 'keyring%';
alter user 'root'@'localhost' identified by 'oracle';

create database nexus;
create table nexus.replicant (id INT(11) NOT NULL AUTO_INCREMENT ,
`First name` varchar(40) not null default '',
`Last name` varchar(40) not null default '',
`Replicant` enum('Yes','No') not null default 'Yes',
PRIMARY KEY (id)) engine=InnoDB row_format=COMPACT ENCRYPTION = 'Y';

INSERT INTO nexus.`replicant` (`First name`,`Last name`,`Replicant`)
VALUES 
('Roy','Hauer','Yes'), ('Rutger','Batty','Yes'), ('Voight','Kampff','Yes'), ('Pris','Hannah','Yes'), ('Daryl','Stratton','Yes'), ('Rachael','Young','Yes'), ('Sean','Tyrell','Yes'), ('Rick','Ford','No'), ('Harrison','Deckard','Yes');

select * from nexus.replicant;

Now we have an environment using the keyring file-based TDE.

Before migrating the key store, there are a few things we need to be aware of, as well as reading the manual on this topic:

• mysqld. Yes, we start up another mysqld process, but it’s not a fully functioning server, far from it. It is just a means to migrate the keys from the old file-based  to the new encrypted file. So don’t worry about the defaults-file, the innodb_xxxx params nor anything else. We actually need to reuse the existing datadir.
• datadir. As just mentioned, don’t try and use another datadir as it won’t find any files there to encrypt with the new key and the process won’t be successful. Use the existing online server datadir. (of course, I recommend this process be run in a non-production test environment first!)
• -source & -destination. I think this is quite obvious.  The plugin we’re coming from, and going to.
• keyring_file_data is the existing file-based keyring being used.
• keyring_encrypted_file_data & _password is the new encrypted password being stored in its file in this location.
• keyring-migration- params. We need to connect to the existing instance with super user privs. As it’s locally to the instance, we can use -socket.

mysqld --basedir=/usr/local/mysql/mysql-commercial-8.0.12-linux-glibc2.12-x86_64 \
 --plugin-dir=/usr/local/mysql/mysql-commercial-8.0.12-linux-glibc2.12-x86_64/lib/plugin \
 --lc_messages_dir=/usr/local/mysql/mysql-commercial-8.0.12-linux-glibc2.12-x86_64/share \
 --datadir=/opt/mysql/okv/data \
 --keyring-migration-source=keyring_file.so \
 --keyring_file_data=/opt/mysql/okv/keyring \
 --keyring-migration-destination=keyring_encrypted_file.so \
 --keyring_encrypted_file_data=/opt/mysql/okv/keyring_enc \
 --keyring_encrypted_file_password=oracle2018 \
 --keyring-migration-socket=/opt/mysql/okv/mysql.socket \
 --keyring-migration-user=root \
 --keyring-migration-password=oracle

And if, only if, the migration is successful, you should see output like the following. Anything else, i.e. if no output comes back, or some of the lines don’t appear in your scenario, double check the parameters in the previous command as it’s more than likely impeding a successful key migration somewhere:

2018-10-08T11:26:22.227161Z 0 [Note] [MY-010098] [Server] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2018-10-08T11:26:22.227219Z 0 [Note] [MY-010949] [Server] Basedir set to /usr/local/mysql/mysql-commercial-8.0.12-linux-glibc2.12-x86_64/.
2018-10-08T11:26:22.227226Z 0 [System] [MY-010116] [Server] mysqld (mysqld 8.0.12-commercial) starting as process 13758
2018-10-08T11:26:22.254234Z 0 [Note] [MY-011085] [Server] Keyring migration successful.
2018-10-08T11:26:22.254381Z 0 [Note] [MY-010120] [Server] Binlog end
2018-10-08T11:26:22.254465Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'keyring_encrypted_file'
2018-10-08T11:26:22.254642Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'keyring_file'
2018-10-08T11:26:22.255757Z 0 [System] [MY-010910] [Server] mysqld: Shutdown complete (mysqld 8.0.12-commercial) MySQL Enterprise Server - Commercial.

Migrated.

To make sure the instance has the new parameters in the defaults file, and before any risk of restarting the instance, we’ll need to add the new ‘encrypted’ params to the my.cnf:

[mysqld]
plugin_dir =/usr/local/mysql/mysql-commercial-8.0.12-linux-glibc2.12-x86_64/lib/plugin
#early-plugin-load =keyring_file.so
#keyring_file_data =/opt/mysql/okv/keyring
early-plugin-load =keyring_encrypted_file.so
keyring_encrypted_file_data =/opt/mysql/okv/keyring_enc
keyring_encrypted_file_password =oracle2018
...

And upon the next most convenient / least inconvenient moment, restart the instance:

mysqladmin --defaults-file=/usr/local/mysql/mysql-commercial-8.0.12-linux-glibc2.12-x86_64/my_okv.cnf -uroot -poracle shutdown
mysqld --defaults-file=/usr/local/mysql/mysql-commercial-8.0.12-linux-glibc2.12-x86_64/my_okv.cnf --user=khollman &

And let’s double check which keyring plugin we’re using:

select * from information_schema.plugins where plugin_name like '%keyring%' \G
*************************** 1. row ***************************
PLUGIN_NAME: keyring_encrypted_file
PLUGIN_VERSION: 1.0
PLUGIN_STATUS: ACTIVE
PLUGIN_TYPE: KEYRING
PLUGIN_TYPE_VERSION: 1.1
PLUGIN_LIBRARY: keyring_encrypted_file.so
PLUGIN_LIBRARY_VERSION: 1.9
PLUGIN_AUTHOR: Oracle Corporation
PLUGIN_DESCRIPTION: store/fetch authentication data to/from an encrypted file
PLUGIN_LICENSE: PROPRIETARY
LOAD_OPTION: ON
1 row in set (0,00 sec)

And also that we can select the data from the encrypted tablespace:

select * from nexus.replicant;
+----+------------+-----------+-----------+
| id | First name | Last name | Replicant |
+----+------------+-----------+-----------+
| 1 | Roy         | Hauer     | Yes       |
| 2 | Rutger      | Batty     | Yes       |
| 3 | Voight      | Kampff    | Yes       |
| 4 | Pris        | Hannah    | Yes       |
| 5 | Daryl       | Stratton  | Yes       |
| 6 | Rachael     | Young     | Yes       |
| 7 | Sean        | Tyrell    | Yes       |
| 8 | Rick        | Ford      | No        |
| 9 | Harrison    | Deckard   | Yes       |
+----+------------+-----------+-----------+
9 rows in set (0,00 sec)

Seems quite straight forward.

Well how about, in a test environment, changing the keyring_encrypted_file_password value to something different now, and restart the instance, and run the same select on the same table.

Hey presto:

select * from nexus.replicant;
ERROR 3185 (HY000): Can't find master key from keyring, please check in the server log if a keyring plugin is loaded and initialized successfully.
Error (Code 3185): Can't find master key from keyring, please check in the server log if a keyring plugin is loaded and initialized successfully.
Error (Code 1877): Operation cannot be performed. The table 'nexus.replicant' is missing, corrupt or contains bad data.

Hope this helps someone out there. Enjoy encrypting!

Now we can run encrypted backups safely and not worry about moving those files around different systems now.

With the exception of the three configuration variables described here, ProxySQL will only parse the configuration files the first time it is started, or if the proxysql.db file is missing for some other reason.

If we want to change any of this data we need to do so via ProxySQL’s admin interface and then save them to disk. That’s fine if ProxySQL is running, but what if it won’t start because of these values?

For example, perhaps we accidentally configured ProxySQL to run on port 3306 and restarted it, but there’s already a production MySQL instance running on this port. ProxySQL won’t start, so we can’t edit the value that way:

2018-10-02 09:18:33 network.cpp:53:listen_on_port(): [ERROR] bind(): Address already in use

We could delete proxysql.db and have it reload the configuration files, but that would mean any changes we didn’t mirror into the configuration files will be lost.

Another option is to edit ProxySQL’s database file using sqlite3:

[root@centos7-pxc57-4 ~]# cd /var/lib/proxysql/
[root@centos7-pxc57-4 proxysql]# sqlite3 proxysql.db
sqlite> SELECT * FROM global_variables WHERE variable_name='mysql-interfaces';
mysql-interfaces|127.0.0.1:3306
sqlite> UPDATE global_variables SET variable_value='127.0.0.1:6033' WHERE variable_name='mysql-interfaces';
sqlite> SELECT * FROM global_variables WHERE variable_name='mysql-interfaces';
mysql-interfaces|127.0.0.1:6033

Or if we have a few edits to make we may prefer to do so with a text editor:

[root@centos7-pxc57-4 ~]# cd /var/lib/proxysql/
[root@centos7-pxc57-4 proxysql]# sqlite3 proxysql.db
sqlite> .output /tmp/global_variables
sqlite> .dump global_variables
sqlite> .exit

The above commands will dump the global_variables table into a file in SQL format, which we can then edit:

[root@centos7-pxc57-4 proxysql]# grep mysql-interfaces /tmp/global_variables
INSERT INTO “global_variables” VALUES(‘mysql-interfaces’,’127.0.0.1:3306’);
[root@centos7-pxc57-4 proxysql]# vim /tmp/global_variables
[root@centos7-pxc57-4 proxysql]# grep mysql-interfaces /tmp/global_variables
INSERT INTO “global_variables” VALUES(‘mysql-interfaces’,’127.0.0.1:6033’);

Now we need to restore this data. We’ll use the restore command to empty the table (as we’re restoring from a missing backup):

[root@centos7-pxc57-4 proxysql]# sqlite3 proxysql.db
sqlite> .restore global_variables
sqlite> .read /tmp/global_variables
sqlite> .exit

Once we’ve made the change, we should be able to start ProxySQL again:

[root@centos7-pxc57-4 proxysql]# /etc/init.d/proxysql start
Starting ProxySQL: DONE!
[root@centos7-pxc57-4 proxysql]# lsof -I | grep proxysql
proxysql 15171 proxysql 19u IPv4 265881 0t0 TCP localhost:6033 (LISTEN)
proxysql 15171 proxysql 20u IPv4 265882 0t0 TCP localhost:6033 (LISTEN)
proxysql 15171 proxysql 21u IPv4 265883 0t0 TCP localhost:6033 (LISTEN)
proxysql 15171 proxysql 22u IPv4 265884 0t0 TCP localhost:6033 (LISTEN)
proxysql 15171 proxysql 23u IPv4 266635 0t0 TCP *:6032 (LISTEN)

While you are here

You might enjoy my recent post Using ProxySQL to connect to IPV6-only databases over IPV4

You can download ProxySQL from Percona repositories, and you might also want to check out our recorded webinars that feature ProxySQL too.

The post How to Fix ProxySQL Configuration When it Won’t Start appeared first on Percona Database Performance Blog.

In the context of providing managed WordPress hosting services, at Presslabs we operate with lots of small to medium-sized databases, in a DB-per-service model, as we call it. The workloads are mostly reads, so we need to efficiently scale that. The MySQL® asynchronous replication model fits the bill very well, allowing us to scale horizontally from one server—with the obvious availability pitfalls—to tens of nodes. The next release of the stack is going to be open-sourced.

As we were already using Kubernetes, we were looking for an operator that could automate our DB deployments and auto-scaling. Those available were doing synchronous replication using MySQL group replication or Galera-based replication. Therefore, we decided to write our own operator.

Solution architecture

The MySQL operator, released under Apache 2.0 license, is based on Percona Server for MySQL for its operational improvements —like utility user and backup locks—and relies on the tried and tested Orchestrator to do the automatic failovers. We’ve been using Percona Server in production for about four years, with very good results, thus encouraging us to continue implementing it in the operator as well.

The MySQL Operator-Orchestrator integration is highly important for topology, as well as for cluster healing and system failover. Orchestrator is a MySQL high availability and replication management tool that was coded and opened by GitHub.

As we’re writing this, the operator is undergoing a full rewrite to implement the operator using the Kubebuilder framework, which is a pretty logical step to simplify and standardize the operator to make it more readable to contributors and users.

Aims for the project

We’ve built the MySQL operator with several considerations in mind, generated by the needs that no other operator could satisfy at the time we started working on it, last year.

Here are some of them:

• Easily deployable MySQL clusters in Kubernetes, following the cluster-per-service model
• DevOps-friendly, critical to basic operations such as monitoring, availability, scalability, and backup stories
• Out-of-the-box backups, scheduled or on-demand, and point-in-time recovery
• Support for cloning, both inside a cluster and across clusters

It’s good to know that the MySQL operator is now in beta version, and can be tested in production workloads. However, you can take a spin and decide for yourself—we’re already successfully using it for a part of our production workloads at Presslabs, for our customer dashboard services.

Going further to some more practical info, we’ve successfully installed and tested the operator on AWS, Google Cloud Platform, and Microsoft Azure and covered the step by step process in three tutorials here.

Set up and configuration

It’s fairly simple to use the operator. Prerequisites would be the ubiquitous Helm and Kubectl.

The first step is to install the controller. Two commands should be run, to make use of the Helm chart bundled in the operator:

$ helm repo add presslabs https://presslabs.github.io/charts
$ helm install presslabs/mysql-operator --name mysql-operator

These commands will deploy the controller together with an Orchestrator cluster.

The configuration parameters of the Helm chart for the operator and its default values are as follows:

replicaCount

1

image

quay.io/presslabs/mysql-operator:v0.1.5

imagePullPolicy

IfNotPresent

helperImage

quay.io/presslabs/mysql-helper:v0.1.5

installCRDs

true

resources

nodeSelector

tolerations

affinity

extraArgs

rbac.create

true

rbac.serviceAccountName

default

orchestrator.replicas

3

orchestrator.image

quay.io/presslabs/orchestrator:latest

Further Orchestrator values can be tuned by checking the values.yaml config file.

Cluster deployment

The next step is to deploy a cluster. For this, you need to create a Kubernetes secret that contains MySQL credentials (root password, database name, user name, user password), to initialize the cluster and a custom resource MySQL cluster as you can see below:

An example of a secret (example-cluster-secret.yaml):

apiVersion: v1
kind: Secret
metadata:
 name: my-secret
type: Opaque
data:
 ROOT_PASSWORD: # root password, base_64 encoded

An example of simple cluster (example-cluster.yaml):

apiVersion: mysql.presslabs.org/v1alpha1
kind: MysqlCluster
metadata:
 name: my-cluster
spec:
 replicas: 2
 secretName: my-secret

The usual kubectl commands can be used to do various operations, such as a basic listing:

$ kubectl get mysql

or detailed cluster information:

$ kubectl describe mysql my-cluster

Backups

A further step could be setting up the backups on an object storage service. To create a backup is as simple as creating a MySQL Backup resource that can be seen in this example (example-backup.yaml):

apiVersion: mysql.presslabs.org/v1alpha1
kind: MysqlBackup
metadata:
 name: my-cluster-backup
spec:
 clusterName: my-cluster
 backupUri: gs://bucket_name/path/to/backup.xtrabackup.gz
 backupSecretName: my-cluster-backup-secret

To provide credentials for a storage service, you have to create a secret and specify your credentials to your provider; we currently support AWS, GCS or HTTP as in this example (example-backup-secret.yaml):

apiVersion: v1
kind: Secret
metadata:
 name: my-cluster-backup-secret
type: Opaque
Data:
 # AWS
 AWS_ACCESS_KEY_ID: #add here your key, base_64 encoded
 AWS_SECRET_KEY: #and your secret, base_64 encoded
 # or Google Cloud base_64 encoded
 # GCS_SERVICE_ACCOUNT_JSON_KEY: #your key, base_64 encoded
 # GCS_PROJECT_ID: #your ID, base_64 encoded

Also, recurrent cluster backups and cluster initialization from a backup are some additional operations you can opt for. For more details head for our documentation page.

Further operations and new usage information are kept up-to-date on the project homepage.

Our future plans include developing the MySQL operator and integrating it with Percona Management & Monitoring for better exposing the internals of the Kubernetes DB cluster.

Open source community

Community contributions are highly appreciated; we should mention the pull requests from Platform9, so far, but also the sharp questions on the channel we’ve opened on Gitter, for which we do the best to answer in detail, as well as issue reports from early users of the operator.

Come and talk to us about the project

Along with my colleague Calin Don, I’ll be talking about this at Percona Live Europe in November. It would be great to have the chance to meet other enthusiasts and talk about what we’ve discovered so far!

The content in this blog is provided in good faith by members of the open source community. The content is not edited or tested by Percona, and views expressed are the authors’ own. When using the advice from this or any other online resource test ideas before applying them to your production systems, and always secure a working back up.

The post Deploying MySQL on Kubernetes with a Percona-based Operator appeared first on Percona Community Blog.

Mydbops has partnered with OSI days for the second consecutive year. OSI days is one of the Asia’s leading  open source conference.

Presentations on MySQL 

Topic        : Evolution of DBA’s in Cloud

Presenter : Manosh Malai ,Senior Devops / DB Consultant Mydbops

Kabilesh P R. Co-Founder / DB Consultant Mydbops

As Cloud is more widely adopted by industry now DBA’s should focus on ramping up their Skills on core optimisation and designing more scalable database. Our consultants emphasis the role of DBA in cloud environment and share their experience in handling large scale systems.

Topic : MySQL 8.0 = NoSQL + SQL

Presenter : Tomas Ulin, Vice President MySQL Engineering Oracle

Topic : High Availability framework for MySQL wth Semi-Synchronous replication

Presenter : Prasad Nagaraj,VP, Engineering Scalegrid

Credit: RunDeck Rundeck is one of my favorite Automation tools. Here we are going to see how can we install and configure rundek on a CentOS server with mysql as a backend. Even I like Jenkins, but as a SYSadmin, I like the Rundeck a lot.You may think like both can do automation. But as …

The post Rundeck Series: Install And Configure RunDeck 3.0 On CentOS 7 appeared first on SQLgossip.

There are a number of options for generating ID values for your tables. In this post, Alexey Mikotkin of Devart explores your choices for generating identifiers with a look at auto_increment, triggers, UUID and sequences.

AUTO_INCREMENT

Frequently, we happen to need to fill tables with unique identifiers. Naturally, the first example of such identifiers is PRIMARY KEY data. These are usually integer values hidden from the user since their specific values are unimportant.

When adding a row to a table, you need to take this new key value from somewhere. You can set up your own process of generating a new identifier, but MySQL comes to the aid of the user with the AUTO_INCREMENT column setting. It is set as a column attribute and allows you to generate unique integer identifiers. As an example, consider the users table, the primary key includes an id column of type INT:

CREATE TABLE users (
  id int NOT NULL AUTO_INCREMENT,
  first_name varchar(100) NOT NULL,
  last_name varchar(100) NOT NULL,
  email varchar(254) NOT NULL,
PRIMARY KEY (id)
);

Inserting a NULL value into the id field leads to the generation of a unique value; inserting 0 value is also possible unless the NO_AUTO_VALUE_ON_ZERO Server SQL Mode is enabled::

INSERT INTO users(id, first_name, last_name, email) VALUES (NULL, 'Simon', 'Wood', 'simon@testhost.com');
INSERT INTO users(id, first_name, last_name, email) VALUES (0, 'Peter', 'Hopper', 'peter@testhost.com');

It is possible to omit the id column. The same result is obtained with:

INSERT INTO users(first_name, last_name, email) VALUES ('Simon', 'Wood', 'simon@testhost.com');
INSERT INTO users(first_name, last_name, email) VALUES ('Peter', 'Hopper', 'peter@testhost.com');

The selection will provide the following result:

You can get the automatically generated value using the LAST_INSERT_ID() session function. This value can be used to insert a new row into a related table.

There are aspects to consider when using AUTO_INCREMENT, here are some:

• In the case of rollback of a data insertion transaction, no data will be added to a table. However, the AUTO_INCREMENT counter will increase, and the next time you insert a row in the table, holes will appear in the table.
• In the case of multiple data inserts with a single INSERT command, the LAST_INSERT_ID() function will return an automatically generated value for the first row.
• The problem with the AUTO_INCREMENT counter value is described in Bug #199 – Innodb autoincrement stats los on restart.

For example, let’s consider several cases of using AUTO_INCREMENT for table1:

CREATE TABLE table1 (
  id int NOT NULL AUTO_INCREMENT,
  PRIMARY KEY (id)
)
ENGINE = INNODB; -- transactional table
-- Insert operations.
INSERT INTO table1 VALUES (NULL); -- 1
INSERT INTO table1 VALUES (NULL); -- 2
INSERT INTO table1 VALUES (NULL); -- 3
SELECT LAST_INSERT_ID() INTO @p1; -- 3
-- Insert operations within commited transaction.
START TRANSACTION;
INSERT INTO table1 VALUES (NULL); -- 4
INSERT INTO table1 VALUES (NULL); -- 5
INSERT INTO table1 VALUES (NULL); -- 6
COMMIT;
SELECT LAST_INSERT_ID() INTO @p3; -- 6
-- Insert operations within rolled back transaction.
START TRANSACTION;
INSERT INTO table1 VALUES (NULL); -- 7 won't be inserted (hole)
INSERT INTO table1 VALUES (NULL); -- 8 won't be inserted (hole)
INSERT INTO table1 VALUES (NULL); -- 9 won't be inserted (hole)
ROLLBACK;
SELECT LAST_INSERT_ID() INTO @p2; -- 9
-- Insert multiple rows operation.
INSERT INTO table1 VALUES (NULL), (NULL), (NULL); -- 10, 11, 12
SELECT LAST_INSERT_ID() INTO @p4; -- 10
-- Let’s check which LAST_INSERT_ID() values were at different stages of the script execution:
SELECT @p1, @p2, @p3, @p4;
+------+------+------+------+
| @p1  | @p2  | @p3  | @p4  |
+------+------+------+------+
|    3 |    9 |    6 |   10 |
+------+------+------+------+
-- The data selection from the table shows that there are holes in the table in the values of identifiers:
SELECT * FROM table1;
+----+
| id |
+----+
|  1 |
|  2 |
|  3 |
|  4 |
|  5 |
|  6 |
| 10 |
| 11 |
| 12 |
+----+

Note: The next AUTO_INCREMENT value for the table can be parsed from the SHOW CREATE TABLE result or read from the AUTO_INCREMENT field of the INFORMATION_SCHEMA TABLES table.

The rarer case is when the primary key is surrogate — it consists of two columns. The MyISAM engine has an interesting solution that provides the possibility of generating values for such keys. Let’s consider the example:

CREATE TABLE roomdetails (
  room char(30) NOT NULL,
  id int NOT NULL AUTO_INCREMENT,
PRIMARY KEY (room, id)
)
ENGINE = MYISAM;
INSERT INTO roomdetails VALUES ('ManClothing', NULL);
INSERT INTO roomdetails VALUES ('WomanClothing', NULL);
INSERT INTO roomdetails VALUES ('WomanClothing', NULL);
INSERT INTO roomdetails VALUES ('WomanClothing', NULL);
INSERT INTO roomdetails VALUES ('Fitting', NULL);
INSERT INTO roomdetails VALUES ('ManClothing', NULL);

It is quite a convenient solution:

Special values auto generation

The possibilities of the AUTO_INCREMENT attribute are limited because it can be used only for generating simple integer values. But what about complex identifier values? For example, depending on the date/time or [A0001, A0002, B0150…]). To be sure, such values should not be used in primary keys, but they might be used for some auxiliary identifiers.

The generation of such unique values can be automated, but it will be necessary to write code for such purposes. We can use the BEFORE INSERT trigger to perform the actions we need.

Let’s consider a simple example. We have the sensors table for sensors registration. Each sensor in the table has its own name, location, and type: 1 –analog, 2 –discrete, 3 –valve. Moreover, each sensor should be marked with a unique label like [symbolic representation of the sensor type + a unique 4-digit number] where the symbolic representation corresponds to such values [AN, DS, VL].

In our case, it is necessary to form values like these [DS0001, DS0002…] and insert them into the label column.

When the trigger is executed, it is necessary to understand if any sensors of this type exist in the table. It is enough to assign number “1” to the first sensor of a certain type when it is added to the table.

In case such sensors already exist, it is necessary to find the maximum value of the identifier in this group and form a new one by incrementing the value by 1. Naturally, it is necessary to take into account that the label should start with the desired symbol and the number should be 4-digit.

So, here is the table and the trigger creation script:

CREATE TABLE sensors (
  id int NOT NULL AUTO_INCREMENT,
  type int NOT NULL,
  name varchar(255) DEFAULT NULL,
  `position` int DEFAULT NULL,
  label char(6) NOT NULL,
PRIMARY KEY (id)
);
DELIMITER $$
CREATE TRIGGER trigger_sensors
BEFORE INSERT
ON sensors
FOR EACH ROW
BEGIN
  IF (NEW.label IS NULL) THEN
    -- Find max existed label for specified sensor type
    SELECT
      MAX(label) INTO @max_label
    FROM
      sensors
    WHERE
      type = NEW.type;
    IF (@max_label IS NULL) THEN
      SET @label =
        CASE NEW.type
        WHEN 1 THEN 'AN'
        WHEN 2 THEN 'DS'
        WHEN 3 THEN 'VL'
        ELSE 'UNKNOWN'
      END;
      -- Set first sensor label
      SET NEW.label = CONCAT(@label, '0001');
    ELSE
      -- Set next sensor label
      SET NEW.label = CONCAT(SUBSTR(@max_label, 1, 2), LPAD(SUBSTR(@max_label, 3) + 1, 4, '0'));
    END IF;
  END IF;
END$$
DELIMITER;

The code for generating a new identifier can, of course, be more complex. In this case, it is desirable to implement some of the code as a stored procedure/function. Let’s try to add several sensors to the table and look at the result of the labels generation:

INSERT INTO sensors (id, type, name, `position`, label) VALUES (NULL, 1, 'temperature 1', 10, 'AN0025'); -- Set exact label value 'AN0025'
INSERT INTO sensors (id, type, name, `position`, label) VALUES (NULL, 1, 'temperature 2', 11, NULL);
INSERT INTO sensors (id, type, name, `position`, label) VALUES (NULL, 1, 'pressure 1', 15, NULL);
INSERT INTO sensors (id, type, name, `position`, label) VALUES (NULL, 2, 'door 1', 10, NULL);
INSERT INTO sensors (id, type, name, `position`, label) VALUES (NULL, 2, 'door 2', 11, NULL);
INSERT INTO sensors (id, type, name, `position`, label) VALUES (NULL, 3, 'valve 1', 20, NULL);
INSERT INTO sensors (id, type, name, `position`, label) VALUES (NULL, 3, 'valve 2', 21, NULL);

Using UUID

Another version of the identification data is worth mentioning – Universal Unique Identifier (UUID), also known as GUID. This is a 128-bit number suitable for use in primary keys.

A UUUI value can be represented as a string – CHAR(36)/VARCHAR(36) or a binary value – BINARY(16).

Benefits:

• Ability to generate values ​​from the outside, for example from an application.
• UUID values ​​are unique across tables and databases since the standard assumes uniqueness in space and time.
• There is a specification – A Universally Unique IDentifier (UUID) URN Namespace.

Disadvantages:

• Possible performance problems.
• Data increase.
• More complex data analysis (debugging).

To generate this value, MySQL function UUID() is used. New functions have been added to Oracle MySQL 8.0 server to work with UUID values ​​- UUID_TO_BIN, BIN_TO_UUID, IS_UUID. Learn more about it at the Oracle MySQL website – UUID()

The code shows the use of UUID values:

CREATE TABLE table_uuid (id binary(16) PRIMARY KEY);
INSERT INTO table_uuid VALUES(UUID_TO_BIN(UUID()));
INSERT INTO table_uuid VALUES(UUID_TO_BIN(UUID()));
INSERT INTO table_uuid VALUES(UUID_TO_BIN(UUID()));
SELECT BIN_TO_UUID(id) FROM table_uuid;
+--------------------------------------+
| BIN_TO_UUID(id)                      |
+--------------------------------------+
| d9008d47-cdf4-11e8-8d6f-0242ac11001b |
| d900e2b2-cdf4-11e8-8d6f-0242ac11001b |
| d9015ce9-cdf4-11e8-8d6f-0242ac11001b |
+--------------------------------------+

You may also find useful the following article – Store UUID in an optimized way.

Using sequences

Some databases support the object type called Sequence that allows generating sequences of numbers. The Oracle MySQL server does not support this object type yet but the MariaDB 10.3 server has the Sequence engine that allows working with the Sequence object.

The Sequence engine provides DDL commands for creating and modifying sequences as well as several auxiliary functions for working with the values. It is possible to specify the following parameters while creating a named sequence: START – a start value, INCREMENT – a step, MINVALUE/MAXVALUE – the minimum and maximum value; CACHE – the size of the cache values; CYCLE/NOCYCLE – the sequence cyclicity. For more information, see the CREATE SEQUENCE documentation.

Moreover, the sequence can be used to generate unique numeric values.  This possibility can be considered as an alternative to AUTO_INCREMENT but the sequence additionally provides an opportunity to specify a step of the values. Let’s take a look at this example by using the users table. The sequence object users_seq will be used to fill the values of the primary key. It is enough to specify the NEXT VALUE FOR function in the DEFAULT property of the column:

CREATE SEQUENCE users_seq;
CREATE TABLE users (
  id int NOT NULL DEFAULT (NEXT VALUE FOR users_seq),
  first_name varchar(100) NOT NULL,
  last_name varchar(100) NOT NULL,
  email varchar(254) NOT NULL,
PRIMARY KEY (id)
);
INSERT INTO users (first_name, last_name, email) VALUES ('Simon', 'Wood', 'simon@testhost.com');
INSERT INTO users (first_name, last_name, email) VALUES ('Peter', 'Hopper', 'peter@testhost.com');

Table content output:

Information

The images for this article were produced while using dbForge Studio for MySQL Express Edition, a download is available from https://www.devart.com/dbforge/mysql/studio/dbforgemysql80exp.exe

It’s free!

Thank you to community reviewer Jean-François Gagné for his review and suggestions for this post.

The content in this blog is provided in good faith by members of the open source community. The content is not edited or tested by Percona, and views expressed are the authors’ own. When using the advice from this or any other online resource test ideas before applying them to your production systems, and always secure a working back up.

The post Generating Identifiers – from AUTO_INCREMENT to Sequence appeared first on Percona Community Blog.

We will have a kiosk at Moscone Center, south exhibition hall, Oracle’s Data Management area number 123, close to high availability area and exits 16 and 18. Our kiosk number is DBA-P1.

Our CEO and Co-Founder Seppo Jaakola will host a presentation highlighting the main features of our upcoming Galera Cluster 4.0. The presentation will take place Monday, Oct 22, 1:00 p.m. – 1:20 p.m at the Exchange @ Moscone South – Theater 2. Seats are limited!

Come and meet us! Let’s discuss your MySQL high availability plan or your Galera Cluster deployment. If you want to set up a meeting with us please email to ínfo@galeracluster.com for a meeting request.