The Southeast Linuxfest will be virtual this year and I am happy to announce that I have two sessions.
Friday, June 12th
(All times US EDT, GMT-4. Time between talks is used for Q&A, discussion, wanton displays of old hardware and boxed Linux software hoarding, and breaks.)
6:00 - 7:00 PM: Dave Stokes - MySQL JSON Document Validation
JSON is a popular data interchange format and MySQL has had a native JSON Data Type for a few years. But until recently there was no way to make sure JSON data included required fields, data type checking, or range checking. But thanks to the JSON-Schema.org's designs, there is now a way to ensure that the JSON data going into your MySQL Database is correct. You will learn how to use constraint checks with the new JSON validation functions to catch bad data before it gets into your server.
Saturday, June 13th
3:30 - 4:30 PM: David Stokes - What’s New With MySQL
MySQL has been releasing new features each quarter for the past few years but do you know what is included in those releases? Well, this session will cover in reverse order the new features such as hash joins, dual password, multi-valued indexes, derived tables, and more. There is some really cool stuff that can save you a lot of grief if you know about them.
Even though Percona Monitoring and Management 2 (PMM) comes with a lot of dashboards and metrics out of the box, sometimes we need to extend the default metrics by running custom queries.
For example, suppose you want to have information about cached indexes from Innodb tables from innodb_cached_indexes table. That metric is not being captured by any default dashboard, but it is possible to extend PMM and make it capture the result of custom queries.
Getting Started With Custom Queries
Custom queries can be added to mysqld_exporter by adding them to the appropriate config file in /usr/local/percona/pmm2/collectors/custom-queries/mysql. There are three subdirectories inside it: high-resolution, low-resolution, and medium-resolution. PMM allows three resolutions for MySQL exporter: five seconds, five seconds (it is not an error), and 60 seconds.
Suppose we want to add metrics from the innodb_cached_indexes table in MySQL and we want to run this query every five seconds (high resolution). To achieve that, let’s edit the file high-resolution/queries-mysqld.yml located in the exporter’s custom queries config mentioned above, and add this:
mysql_innodb_index_stats: ## The namespace (prefix of the metric name) for the custom query.
query: "select database_name as schema_name,table_name,index_name,stat_value*@@innodb_page_size as size_bytes from mysql.innodb_index_stats where stat_name='size';"
metrics:
- table_name:
usage: "LABEL"
description: "Table name"
- size_bytes:
usage: "GAUGE"
description: "Index Size in Bytes"
- schema_name:
usage: "LABEL"
description: "Schema Name"
- index_name:
usage: "LABEL"
description: "Index name"
As we can see, this query will return four columns and the column size_bytes will be used to plot the data graph (usage gauge). The metric name in Grafana will be a composition of the name we defined, mysql_innodb_index_stats plus the column name: mysql_innodb_index_stats_size_bytes.
Yaml description:
query: The query itself. It can be split into several lines as long as you keep it enclosed in quotes. There is no limit for the query size and special chars can be escaped using regular \ escape sequences or Unicode escapes like
\u0022
for a double quote.
metrics: This is where the query fields are being described. Is the list of the query fields with their attributes.
usage: The metric type. It should be the one from the following: LABEL, COUNTER, GAUGE, DURATION, DISCARD. If usage is LABEL this value will be used as Prometheus dimension. description: The field description.
Where Can I See the Dashboard?
Go to the search box located at the upper left corner and start writing Advanced. When it appears, click on Advanced Data Exploration.
Then, look for the metric name. Remember the name is a composition between the name we gave to the metric + the column name.
and here we have the graph:
It is possible to run multiple queries from the same yml file by adding more queries. There’s just the one in the example, but consider this:
All queries are going to be executed sequentially
Query execution should fit into the selected resolution. If the resolution is 5 seconds, all queries should run in less than 5 seconds.
Because of these limits, remember that if a query runs for more than four seconds, no data will be collected. Also, there is no need to set a maximum number of connections to the database since all queries run sequentially, so, only one connection at a time will be used.
As an example, if you add a query that takes more than four seconds to run (in this example, just a SELECT SLEEP), the graph might look like this:
All the gaps are the result of the timeout in the queries execution.
Error handling
Query error
If a query fails, you first will notice it because there won’t be any metric for that query but, why is it failing? All the errors from the mysqld exporter are logged into
syslog
. A tail to
/var/log/syslog
combined with a grep, it will show the queries with errors, if there are any.
tail /var/log/syslog | grep mysqld_exporter
May 9 20:05:18 karl-OMEN pmm-agent[1044]: #033[36mINFO#033[0m[2020-05-09T20:05:18.662-03:00] time="2020-05-09T20:05:18-03:00"
level=error msg="Error scraping for collect.custom_query.mr: mysql_performance_schema_blog:error running query on database: mysql_performance_schema_blog,
Error 1356: View 'sys.memory_global_by_current_bytes' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them"
source="exporter.go:116" #033[36magentID#033[0m=/agent_id/4d308ab4-4791-4b8b-b7cb-24b05c012be8 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=mysqld_exporter
Notice that if a query fails, it won’t affect the results for all the other queries before or after it.
Field definitions errors:
Every row in the query result must have a unique combination of names and labels. Following the previous examples, what happens if I make the index name constant? All rows will have different numeric values but the label’s combination will be the same for all rows. Even when this is a coarse example, it lets me show you a Prometheus metrics collector error.
In the following example, notice that the
index_name
is now enclosed in single quotes so, it is a constant.
mysql_performance_index_stats_blog2:
query: "select database_name as schema_name,table_name, 'index_name',stat_value*@@innodb_page_size as size_bytes from mysql.innodb_index_stats where stat_name='size';"
metrics:
- schema_name:
usage: "LABEL"
description: "Performance Schema Event Name"
- table_name:
usage: "LABEL"
description: "Performance Schema Event Name"
- index_name:
usage: "LABEL"
description: "Performance Schema Event Name"
- size_bytes:
usage: "GAUGE"
description: "Memory currently allocated to the Event"
Since all the metrics will have duplicated labels, only the first one will be used and all the rest will be skipped by Prometheus because label combinations should be unique. In the error log, you can find a message like this:
pmm-agent[1044]: #033[36mINFO#033[0m[2020-05-10T10:21:59.202-03:00] time="2020-05-10T10:21:59-03:00" level=error
msg="error gathering metrics: 24 error(s) occurred: [from Gatherer #1] collected metric "mysql_performance_index_stats_blog2_size_bytes"
{ label:<name:"index_name" value:"index_name" > label:<name:"schema_name" value:"sakila" >
label:<name:"table_name" value:"actor" > gauge:<value:16384 > } was collected before with the same name and label values
And in the graph, there will be only one metric.
Another example of possible error with labels, is what happens is a label duplicates an inherited label, for example, the instance name?
Well, let’s try it. Let’s change the query to make it return a new constant field name “instance”. It will duplicate the inherited instance name label.
mysql_performance_index_stats_blog:
query: "select database_name as schema_name,table_name,index_name,stat_value*@@innodb_page_size as leaf_pages_size_bytes, 'abc001' AS instance from mysql.innodb_index_stats where stat_name='n_leaf_pages';"
metrics:
- schema_name:
usage: "LABEL"
description: "Performance Schema Event Name"
- instance:
usage: "LABEL"
description: "fake instance"
- table_name:
usage: "LABEL"
description: "Performance Schema Event Name"
- index_name:
usage: "LABEL"
description: "Performance Schema Event Name"
- leaf_pages_size_bytes:
usage: "GAUGE"
description: "Memory currently allocated to the Event"
As we can see, duplicated labels will be ignored in favor of the already set labels:
There is a repository at PerconaLab with some query examples located here. Feel free to contribute with your custom queries.
MySQL Backup Strategies – What you should know before considering MySQL DR solutions ?
MySQL powers all the major internet properties, Which include Google, Facebook, Twitter, LinkedIn, Uber etc. So how do we plan for MySQL disaster recovery and what are the most common MySQL DR tools used today for building highly reliable database infrastructure operations on MySQL ? There can be several reasons for a MySQL database outage: hardware failure, power outage, human error, natural disaster etc. We may not be able prevent all the disaster from happening but investing on a robust disaster recovery plan is very important for building fault-tolerant database infrastructure operations on MySQL. Every MySQL DBA is accountable for developing a disaster recovery plan addressing data sensitivity, data loss tolerance and data security. Functionally you have several database backup strategies available with MySQL:
Full backup – Full backup backs up the whole database, This also include transaction log so that the full database can be recovered after a full database backup is restored. Full database backups represent the database at the time the backup finished. Full backups are storage resource intensive and takes more time to finish. If you have for a large database, we strongly recommend to supplement a full database backup with a series of differential database backups.
Differential backup – A differential backup is based on the most recent, previous full data backup. A differential backup captures only the data that has changed since that full backup. The full backup upon which a differential backup is based is known as the base of the differential. Full backups, except for copy-only backups, can serve as the base for a series of differential backups, including database backups, partial backups, and file backups. The base backup for a file differential backup can be contained within a full backup, a file backup, or a partial backup. The differential backups are most recommended when the subset of a database is modified more frequently than the rest of the database.
Incremental backup – A incremental backup contains all changes to the data since the last backup. Both differential and incremental backup does only backing up changed files. But they differ significantly in how they do it, and how useful the result is.while an incremental backup only includes the data that has changed since the previous backup, a differential backup contains all of the data that has changed since the last full backup. The advantage that differential backup offers over incremental backups is a shorter restore time. Because, the backup has to be reconstituted from the last full backup and all the incremental backups since.
MySQL Backup tools
The following are the list of MySQL backup tools (logical and physical) discussed in this blog:
mysqldump
mysqldump is a MySQL client utility which can be used to perform logical backups , The mysqldump generate output in SQL ( default and most commonly used to reproduce MySQL schema objects and data), CSV, other delimited text or XML format. We have copied below the restrictions of mysqldump:
mysqldump does not dump performance_schema or sys schema be default. To enforce dumping or logical backup of any of these schema objects, You have to explicitly mention them –databases option or if you want to just dump performance_schema use –skip-lock-tables option.
mysqldump does not dump the INFORMATION_SCHEMA schema.
mysqldump does not dump the InnoDB CREATE TABLESPACE statements.
mysqldump does not dump the NDB Cluster ndbinfo information database.
mysqldump includes statements required to recreate the general_log and slow_query_log tables for dumps of the mysql database. But, Log table contents are not dumped
The mysqlpump is another client utility for logical backup of MySQL database like mysqldump which is capable of parallel processing of databases and other schema objects with databases to perform high performance dumping process, We listed below mysqlpump most compelling features:
MySQL dump parallel processing for databases and other objects within databases.
MySQL user accounts will be dumped as account-management statements (CREATE USER, GRANT) rather than as inserts into the mysql system database.
By using mysqlpump you can create a compressed output.
Much faster compared to mysqldump. Because, The InnoDB secondary indexes are created after rows are inserted to the table.
P.S. – We have blogged about “How to use mysqlpump for faster MySQL logical backup ? ” here
MySQL Enterprise Backup
MySQL Enterprise Backup is a hot / online backup tool for MySQL ( optimized for InnoDB only though capable of backup and restore of tables created on other storage engines supported by MySQL ) capable of performing full, incremental and differential backup. MySQL Enterprise Backup also support cloud storage backup, backup encryption and compression. We have explained most compelling MySQL Enterprise Backup 8.0 features below:
Transparent page compression for InnoDB.
Backup history available for all members of Group Replication by making sure backup_history table is updated on primary node after each mysqlbackup operation.
Storage engine of the mysql.backup_history table on a backed-up server has switched from CSV to InnoDB.
mysqlbackup now supports high performance incremental backup by setting page tracking functionality on MySQL (set –incremental=page-track).
Much better MySQL Enterprise Backup 8.0 troubleshooting with now mysqlbackup prints a stack trace after being terminated by a signal.
Selective restores of tables or schema from full backup for Table-Level Recovery (TLR)
Percona XtraBackup
Percona XtraBackup is an open source MySQL hot backup solution from Percona addressing incremental, fast, compressed and secured backup for InnoDB optimally. Most of our customers users Percona XtraBackup for DR of their MySQL infrastructure, The following features makes Percona XtraBackup obvious choice for MySQL Backup and DR:
Hot backup solution for InnoDB without blocking / locking transaction processing.
High performance streaming backup support for InnoDB.
Parallel backup and copy-back support for faster backup and restoration.
Secondary indexes defragmentation support for InnoDB.
Percona XtraBackup support rsync to minimize locking.
Track Percona XtraBackup history with Backup history table.
Percona XtraBackup supports offline backup.
Conclusion
We always recommend a combination of multiple backup strategies / tools for maximum data reliability and optimal restoration. We cannot have a common backup strategy for all the customers, It depends on factors like infrastructure, MySQL distribution, database size, SLA etc. Backups are most import components in database infrastructure operations and we follow zero tolerance DR for building highly available and fault-tolerant MySQL infrastructure operations.
Do you want to engage MinervaDB Remote DBA for MySQL Disaster Recovery (DR) and Database Reliability Engineering (Data SRE) ?
Your favourite monitoring software throws an alert or sends an email after it finds the following in the mysql error log: InnoDB: page [page id: space=8858, page number=206777] (95 records, index id 29361) How do you find which index is … Continue reading →
Just recently, I have been asked to look into what a Disaster Recovery site for InnoDB Cluster would look like.
If you’re reading this, then I assume you’re familiar with what MySQL InnoDB Cluster is, and how it is configured, components, etc.
Reminder: InnoDB Cluster (Group Replication, Shell & Router) in version 8.0 has had serious improvements from 5.7. Please try it out.
So, given that, and given that we want to consider how best to fulfill the need, i.e. create a DR site for our InnoDB Cluster, let’s get started.
Basically I’ll be looking at the following scenario:
InnoDB Cluster Master site with a Group Replication Disaster Recovery Site.
Now, just before we get into the nitty-gritty, here’s the scope.
Life is already hard enough, so we want as much automated as possible, so, yes, InnoDB Cluster gets some of that done, but there are other parts we will still have to assume control over, so here’s the idea:
Master Site: – InnoDB Cluster x3 – Router – Accesses Master (Dynamic) & DR (Static) – Full Automation
DR Site: – Group Replication x2 – Router – Accesses Master (Dynamic) & DR (Static) – Configured Asynchronous Replication from Primary node.
External / LBR 3rd Site: – MySQL Router (static) routing connections to Master Router – Allowing for 2 instance automatic failovers at Master site. – Manually reroute connections to DR site once Master Site outage confirmed or use different Router port. – Redundancy recommended to reduce SPOF.
Let’s get it all started then.
First things first, if you’re not quite sure what you’re doing, then I highly suggest looking at the following:
(And thanks to Andrew & Tony for their continuous help!)
Let’s set up our Master Site
3x CentOS 7 servers.
8.0.20: Server, Shell & Router.
:: All servers have had their SELinux, firewalls, ports, /etc/hosts and so on checked and validated, haven’t they? We will want to be using a private IP between all hosts, for obvious reasons..
I downloaded all the MySQL Enterprise rpm’s from https://edelivery.oracle.com and run the following on all 3 servers (getting mystic here: “centos01”, “centos02” & “centos03”):
sudo grep 'A temporary password is generated for root@localhost' /var/log/mysqld.log |tail -1 mysql -uroot -p
Once we’re in, we want to control what command get into the binlogs, and the ”alter user’ and create user’ will cause us issues later on, hence “sql_log_bin=OFF” here:
SET sql_log_bin = OFF;
alter user 'root'@'localhost' identified by 'passwd';
create user 'ic'@'%' identified by 'passwd';
grant all on . to 'ic'@'%' with grant option;
flush privileges;
SET sql_log_bin = ON;
This “ic@%” user will be used throughout, as we just don’t know when and which instance will end up being the master primary and/or a clone so by using this single user setup, I keep things easier for myself. For your production setup, please look deeper into specific privs ‘n’ perms.
Now, you will see that, although we have installed the binaries and started up the instances on centos02 & centos03, “addInstance” goes and clones our primary. This makes life soooo much easier.
HINT: Before cloning, if you need to provision data, do it prior and we kill a couple of proverbial birdies here.
Now we have a 3 instance Single Primary InnoDB Cluster.
Setting up the local “InnoDB Cluster aware” Router:
mkdir -p /opt/mysql/myrouter
chown -R mysql:mysql /opt/mysql/myrouter
cd /opt/mysql
mysqlrouter --bootstrap ic@centos02:3306 -d /opt/mysql/myrouter -u mysql
./myrouter/start.sh
Now Router is up and running.
Let’s set up our Group Replication Disaster Recovery site now.
Here, I’m doing a naughty, and just setting up a 2 node Group Replication group. If you want to see how you should be doing it, then let me reference Tony again:
:: Again, ports, firewalls, SELinux, /etc/hosts n similar have been validated again? Please?
The environment:
2x Oracle Linux 7 servers.
Version 8.0.20: MySQL Server, Router & Shell
Do the following on both slave servers (olslave01 & olslave02):
sudo yum install -y mysql-8.0.20rpm sudo systemctl start mysqld.service sudo systemctl enable mysqld.service sudo grep 'A temporary password is generated for root@localhost' /var/log/mysqld.log |tail -1 mysql -uroot -p
SET sql_log_bin = OFF;
alter user 'root'@'localhost' identified by 'passwd';
create user 'ic'@'%' identified by 'passwd';
grant all on . to 'ic'@'%' with grant option;
flush privileges;
SET sql_log_bin = ON;
No we have basic servers created, we need to clone the data from the Primary instance in our InnoDB Cluster.
SET sql_log_bin = OFF; INSTALL PLUGIN CLONE SONAME "mysql_clone.so"; GRANT BACKUP_ADMIN ON . to 'ic'@'%'; GRANT SELECT ON performance_schema.* TO 'ic'@'%'; GRANT EXECUTE ON . to 'ic'@'%'; SET sql_log_bin = ON;
Now on each slave:
INSTALL PLUGIN CLONE SONAME "mysql_clone.so";
INSTALL PLUGIN group_replication SONAME 'group_replication.so';
SET GLOBAL clone_valid_donor_list = 'centos02:3306';
GRANT CLONE_ADMIN ON . to 'ic'@'%';
# Keep this, if we ever want to clone from any of the slaves
GRANT BACKUP_ADMIN ON . to 'ic'@'%';
GRANT SELECT ON performance_schema.* TO 'ic'@'%';
GRANT EXECUTE ON . to 'ic'@'%';
Now to execute the clone operation:
set global log_error_verbosity=3;
CLONE INSTANCE FROM 'ic'@'centos02':3306 IDENTIFIED BY 'passwd';
View how the clone process is going:
select STATE, ERROR_NO, BINLOG_FILE, BINLOG_POSITION, GTID_EXECUTED, CAST(BEGIN_TIME AS DATETIME) as "START TIME", CAST(END_TIME AS DATETIME) as "FINISH TIME", sys.format_time(POWER(10,12) * (UNIX_TIMESTAMP(END_TIME) - UNIX_TIMESTAMP(BEGIN_TIME))) as DURATION from performance_schema.clone_status \G
As we’re cloning, we might run into the duplicate UUID issue, so, on both of the slaves, force the server to have a new UUID:
vi /etc/my.cnf
# GR setup
server-id =11
log-bin =mysql-bin
gtid-mode =ON
enforce-gtid-consistency =TRUE
log_slave_updates =ON
binlog_checksum =NONE
master_info_repository =TABLE
relay_log_info_repository =TABLE
transaction_write_set_extraction=XXHASH64
plugin_load_add ="group_replication.so"
group_replication = FORCE_PLUS_PERMANENT
group_replication_bootstrap_group = OFF
#group_replication_start_on_boot = ON
group_replication_group_name = 8E2F4761-C55C-422F-8684-D086F6A1DB0E
group_replication_local_address = '10.0.0.41:33061'
# Adjust the following according to IP's and numbers of hosts in group:
group_replication_group_seeds = '10.0.0.41:33061,10.0.0.42:33061'
On 2nd node:
server-id =22
log-bin =mysql-bin
gtid-mode =ON
enforce-gtid-consistency =TRUE
log_slave_updates =ON
binlog_checksum =NONE
master_info_repository =TABLE
relay_log_info_repository =TABLE
transaction_write_set_extraction=XXHASH64
plugin_load_add ="group_replication.so"
group_replication = FORCE_PLUS_PERMANENT
group_replication_bootstrap_group = OFF
#group_replication_start_on_boot = ON
group_replication_group_name = 8E2F4761-C55C-422F-8684-D086F6A1DB0E
group_replication_local_address = '10.0.0.42:33061'
# Adjust the following according to IP's and numbers of hosts in group:
group_replication_group_seeds = '10.0.0.41:33061,10.0.0.42:33061'
Restart both servers:
systemctl restart mysqld
Check they’re in a GR group and the plugin is ok:
mysql -uroot
SELECT * FROM performance_schema.replication_group_members;
SELECT * FROM performance_schema.replication_group_members\G
select * from information_schema.plugins where PLUGIN_NAME = 'group_replication'\G
Now to create the recovery replication channel on all servers (although this is for single primary setup, the master could fail and then come back as a Read-Only slave, so we need to set this up):
CHANGE MASTER TO MASTER_USER='ic', MASTER_PASSWORD='passwd' FOR CHANNEL 'group_replication_recovery';
On Server 1:
SET GLOBAL group_replication_bootstrap_group=ON; START GROUP_REPLICATION; SET GLOBAL group_replication_bootstrap_group=OFF;
On Server 2:
START GROUP_REPLICATION;
Check all the servers super_read_only mode and if they’re in a group:
select @@super_read_only;
SELECT * FROM performance_schema.replication_group_members;
Set up Router on one of the slaves bootstrapping against the InnoDB Cluster master:
mkdir -p /opt/mysql/myrouter chown -R mysql:mysql /opt/mysql/myrouter mysqlrouter --bootstrap ic@centos02:3306 -d /opt/mysql/myrouter -u mysql cd /opt/mysql/myrouter ./start.sh
Check connectivity:
mysql -uic -P6446 -h olslave01 -N -e "select @@hostname, @@port"
mysql -uic -P6446 -h centos02 -N -e "select @@hostname, @@port"
Replication config for slaves
Setting up replication for the slave from the primary means that we MUST be prepared to script the following for when the primary fails in the group replication setup, or keep the following command at hand, as it isn’t automatic so we’ll need to control this ourselves.
Also, albeit a Group Replication group, when the primary fails n the DR site, we will also need to check all connections and sessions to the GR DR site to make sure we can set this up safely.
So, from the PRIMARY instance only, execute the following:
As you can see, this is replicating via the GR DR site Router that is bootstrapped against the InnoDB Cluster. Hence when the primary on the cluster moves, Router will take us to the new primary without having to re-create replication or similar.
Now let’s start it all up:
start slave ;
And have a look at our InnoDB Cluster master site replicating data to our DR Group Replication site:
show slave status for channel 'idc_gr_replication'\G
Application Connectivity
Now we have a HA solution for our data, what about connecting to one and the other.
We have 2 routers, each local to each data-saving site, fine. But, if we’re being realistic, we have 1 data entry point, the InnoDB Cluster Primary instance on centos01, and then 4 copies of the data, syncing nicely between a (dedicated? please?) local network to ensure we’re as safe as possible.
As you’ll probably have noticed, upon bootstrapping Router on both sides, it chooses the routing_strategy=first_available on the default port 6446 which is fine. And yes, this port can be changed if we want to, so feel free to adjust as required.
But, depending on where the application is, and the user entry point, you’ll have a VIP, floating IP or something similar halping to load balance maybe. But here is where we start to ask more questions:
What happens when the Master site goes down?
eg:
Well, there are a number of things to keep in mind. The local Routers will take care of instance failures, and here, I feel obliged to forewarn you all about what happens when we loose 2 out of 3 instances on the master site. Here, the last instance could be configured to have group_replication_exit_state_action=’OFFLINE_MODE’ or ABORT for example. That way, it doesn’t get used as a valid node via Router.
Now, once that has happened, i.e. the master site goes down, we need to manually intervene, and make sure that everything is really down, and kick into effect the DR site. Now both Routers on Master & DR site that were bootstrapped to the InnoDB Cluster will fail of course.
We are left with the Group Replication DR site.
So what do we do here? Well, create an additional Router, configured with a static (non-dynamic) setup, as we don’t have the metadata that the IdC-aware Router had via the mysql_innodb_cluster_metadata schema that gets stored in “dynamic_state=/opt/mysql/myrouter/data/state.json”.
What I did was create a new Router process on one of the other servers (instead of centos02 & olslave01, on centos01 & olslave02). Now this isn’t something you’d do in Production as we want Router close to the app, not the MySQL instances. So in effect you could just use a new path ,eg.
mkdir -p /opt/mysql/myrouter7001 chown -R mysql:mysql /opt/mysql/myrouter7001 cd /opt/mysql/myrouter7001
mysql -uic -P7001 -h olslave02 -N -e "select @@hostname, @@port"
mysql -uic -P7001 -h centos01 -N -e "select @@hostname, @@port"
And this will give us back olslave01, then when that fails, olslave02.
When the InnoDB Cluster-bootstrapped-Router that listens on 6446 fails, we know to redirect all requests to port 7001 on our production IP’s (in my case, olslave02 & centos01).
Summing up, we’ve now got 4 routers running, 2 are InnoDB Cluster aware, listening on 6446, and the static GR DR Routers are listening on 7001.
That’s a lot of ports and a long connect string. Maybe we could make this simpler?
Ok, well I added yet another Router process, high level. This Router process, again, has a hard-coded static configuration, because we don’t have InnoDB Cluster-aware Routers on both sides, which means having a mysql_innodb_cluster_metadata schema on both sides, fully aware of both InnoDB Clusters, and also fully aware that metadata changes on the master are NOT to be replicated across. Who knows.. maybe in the future…
So in my case I created another Router process configuration on centos03:
mkdir -p /opt/mysql/myrouter
chown -R mysql:mysql /opt/mysql/myrouter
cd /opt/mysql/myrouter
(You might want to use something different, but I did this. I could have all of these on the same server, using different paths. There isn’t any limit to how many Router processes we have running on the same server. It’s more a “port availability” or should I say “network security” concern)
Just in case I’ve lost you a little, what I’m doing here is providing a Router access point via centos03:8008 to the InnoDB-Cluster-aware Routers on centos02:6446 and olslave01:6446. And then an additional / backup access point to the static Routers on centos01: 7001 and olslave02:7001. The app would then have a connect string similar to “centos03:8008,centos03:8009”.
Now, I know to send Master site connections to centos03:8008 always. And when I get errors and notifications from monitoring that this is giving problems, I can use my back up connection to centos03:8009.
Side note: I originally setup the single Router process on each side, but with a mix of InnoDB Cluster aware entry and an additional static configuration. This didn’t work well as really, having bootstrapped against the IdC, the dynamic_state entry only allows for the members taken from the mysql_innodb_cluster_metadata schema. So it won’t work. Hence, the additional router processes. But if you think about it, making each Router more modular and dedicated, it makes for a more atomic solution. Albeit a little more to administer.
However, maybe instead of this high-level Router process listening on 8008 & 8009, you want to reuse your keepalived solution. Have a look at Lefred’s “MySQL Router HA with Keepalived” solution for that.
And there you have it!
This was all done on Oracle Cloud Infrastructure compute instances, opening ports internally within the Security List and not much else.
In this webinar, Sveta Smirnova, MySQL Engineer at Percona, will uncover nuances of Percona XtraDB Cluster (PXC) schema upgrades and point out details you need to give extra attention to.
Percona XtraDB Cluster (PXC) is a 100% synchronized cluster in regards to DML operations. It is ensured by the optimistic locking model and ability to rollback transaction which cannot be applied on all nodes. However, DDL operations are not transactional in MySQL. This adds complexity when you need to change the schema of the database. Changes made by DDL may affect the results of the queries. Therefore all modifications must replicate on all nodes prior to next data access. For operations that run momentarily, it can be easily achieved, but schema changes may take hours to apply. Therefore in addition to the safest synchronous blocking schema upgrade method: TOI, – PXC supports more relaxed, though not safe, method RSU. RSU: Rolling Schema Upgrade is advertised to be non-blocking. But you still need to take care of updates, running while you are performing such an upgrade. Surprisingly, even updates on not related tables and schema can cause RSU operation to fail.
Please join the Sveta Smirnova on Thursday, June 25 at 12 pm EDT for her webinar “How to Avoid Pitfalls in Schema Upgrade with Percona XtraDB Cluster“.
Would you play a multiplayer game if you discovered other people are cheating? According to a survey by Irdeto, 60% of online games were negatively impacted by cheaters, and 77% of players said they would stop playing a multiplayer game if they think opponents are cheating. Player churn grows as cheating grows.
Stopping this is therefore essential if you want to build and develop your community, which is essential to success for today’s gaming companies. This session at Percona Live ONLINE was presented by Yahav Biran, specialist solutions architect, gaming technologies at Amazon Web Services, and Yoav Eilat, Senior Product Manager at Amazon Web Services, presented a talk and demonstration about anti-cheating tools in gaming based on using automation and machine learning (ML).
Yoav notes that while people might think of ML in terms of text or images, but: “There’s a considerable percentage of the world’s data sitting in relational databases. How can your application use it to get results and make predictions?”
Six steps for adding Machine Learning to an Application
Traditionally there are a lot of steps for adding ML to an application with considerable expertise required and manual work, with the efforts of an application developer, database user and some help from a machine learning database scientist:
Select and train database models
Write application code to read data from the database
Format the data for the ML model
Call a machine learning service to run the ML model on the formatted data
Format the output for the application
Load the results to the application
The result is most machine learning is done offline by a data scientist in a desktop tool. “We would like to be able to add some code to your game and use the models directly from there,” explained Yahav.
With multiple databases such as the customer service database or order management system, or in the instance of gaming, this would all be a lot of work to do manually. “So, we want to see how we can do that in an easier and automated way,” continued Yahav.
Examples where cheating can occur
The duo provided some examples of common cheating behaviour that can occur in games:
Authentication: player authentication in the game, to prove they are who they say they are and that they have the right account
Transactional data: what the players purchase inside the game, so they either don’t spend funds they don’t have or don’t lose items they purchased legitimately
Player moves: for example where players in cahoots are walking in front of each other like a human shield
“Where you have a player that’s walking in one direction, shooting in the other direction and doing five other things at the same time, then it’s probably a bot,” said Yahav.
Demonstrating ML in action
The demo was built on Amazon Aurora, a relational database offered by AWS and that is compatible with MySQL and PostgreSQL. The database includes some optimizations and performance improvements, plus a few additional features. It has pay as you go pricing.
As Yahav explains: “The machine learning capabilities added in 2019 allow you to do a query in your Aurora database and then transfer it to a machine learning service for making a prediction. There’s integration with Amazon SageMaker and Amazon Comprehend, which are two machine learning services offered by AWS. The whole thing was done using SQL queries.
Thus, you don’t need to call API’s; there’s no need to write additional code, you’re doing a step, you can just write a statement where you’re selecting from the results of the machine learning call. You can just use the results like you would use any other data from your database.”
Shortening the process from six steps to three
Using this approach, the process is now made much simpler:
(Optional) select and configure the ML model with Amazon SageMaker Autopilot
Run a SQL query to invoke the ML service
Use the results in the application
This article focuses on gaming; however, the presentation also provides details about fraud detection in financial transactions, sentiment analysis in the text (such a customer review written on a website), and a classification example to sort customers by predicted spend.
ML queries in gaming scenarios
Yahav and Yoav trained a SageMaker model to recognize anomalous user authentication activities such as the wrong password. You can dig deep into the code for the demonstration over at GitHub, so we’ll only walk through some of the code.
The model can also use the function auth_cheat_score to find players with a significant cheat score during authentication.
Introducing EmuStarOne
The game was developed initially in 2018 and is a massively multiplayer online (MMO) game that enables players to fight, build, explore and trade goods with each other.
Players authenticate from supporting clients, suc as a PC or game console.
Five personality traits and game events define Emulants: they can move, forge, dodge, etc. and they can transact with virtual goods.
What does cheating look like in the data?
To understand what cheating looks like within games, we have to understand what good and bad behaviour looks like in our game data over time:
Players can cheat as they make illegal trades or run bots that manipulate game moves on behalf of other players.
Cheating can manifest in different ways, such as player move anomalies and consecutive failed login attempts from two different sources.
In general, ML solutions work very well with problems that are evolving and are not static.
How can we stop cheating in the game?
To stop cheating requires a plan and some decisions to be made before creating the data model or ML approach:
We can form an anti-cheat team.
Take action against cheaters e.g., force logout with a hard captcha as a warning.
Escalate the anti-cheating actions as needed.
Eventually, cheaters learn the system behavior, so there is also the consideration of false positives.
Continuously redefine our cheating algorithms.
What we want to enable by forming this anti-cheat team is to stop those that cheat and continuously refine the algorithm.
EmuStar One game data authentication
Yoav explained:
“In the first data set, we have the player authentication; this is the authentication transaction. There is a timestamp that the player came, and in this case, the authentication method was the Xbox Live token.”
It means that the user logged through to the Xbox Authentication Service. It includes the playerGuid, the user agent, which in this case, is an Xbox device. You can see the source IP and the cidir and the geo-location.
Player transaction
The player moves
The player moves (in this case is the X and Z coordination) include the timestamp and the player. There are three more properties – the quadrant, the sector, and the events can be traversing, user traversing from one place to another, or forging or dodging or other events that the game allowed.
The three ML models used for game data
For authentication: IP insights is an unsupervised learning algorithm for detecting anomalous behavior and usage patterns of IP addresses
For transactions: Supervised linear regression – this is because most transactions are already classified by Customer care and player surveys
For player moves: “Random cut forest (RCF), assuming most player moves are legit so anomalous moves indicate potential cheaters,” explained Yoav.
Data preparation
The game involves a mammoth amount of data. Yoav shared: “We have 700,000 authentication events, 1 million transactions and 65 million player moves. For the supervised data, we classified data between 0.1 to 3% to allow the model to distinguish between legit transactions. Move authentication and other Models were built using Jupyter notebooks hosted by SageMaker. Data was stored on s3.
“Once that we were able to distill the data and train the model, we deployed the model with hosted inference endpoints using SageMaker as the service. We used Aurora to invoke the endpoints.”
Data encoding and transformation
In general, ML models like numbers – interest, doubles, or floats. So the String attributes were encoded. The same encoding was on the Aurora side, covering for example player move events such as TraverseSector or Travel.Explore
The notebook is open source so you can see how encoding strings of the player moves was achieved.
Yoav explained: ” I took the quadrant, encoded the sector. encoded the event, and encoded it using the pandas, in the end, and the OneHot encoder.”
The code for an alternative method for achieving this was also shared:
The demo
Based on the characteristics of cheating in our game, cheaters are found via:
Looking for suspicious transactions
Looking for suspicious authentication by the players who executed these transactions
Then seeing if the player moves were suspicious
Yahav shared code for the materialized view for authentication, querying the parameters and filtering only the suspicious ones that are mentioned as cls>zero classified as fraudulent.
An anomaly score cls>2 indicates a suspicious move – the tools are very flexible!
Yahav then executed a query for “the timestamp and the player Guids that are basically are suspicious.”
The live demo presented worked to filter suspicious transactions. Then the authentication cheat was joined with the transaction cheat. Subsequently, 13 suspicious cases were revealed based on timestamps. The suspicious moves were then queried based on the timestamps.
The demo included lots of movements, and transactions from all directions.
Through exploring the player timestamp, playerGuid, quadrant, and sector of all the suspicious cases, it revealed where suspicious behavior occurred so that monitoring could occur in that specific area.
Day-to-day database operation requires, from an administrator, deep knowledge of db internals and security issues, in particular things like SQL injections. In order to prevent such kind of an attack, we have included go-sql-driver into our code for secure placeholder escaping.
Unfortunately, not all cases are secured by the driver.
In case we are using the standard driver for working with MySQL, if we need to pass a variable to the database query, we use a placeholder “?” in order for the server to understand that it needs to process the incoming variable to avoid injection. It works fine with just regular SELECT/INSERT/UPDATE statements, but, unfortunately, MySQL server is not able to process all types of queries.
For example:
db.Exec("CREATE USER ?@? IDENTIFIED BY ?", name, host, pass)
This query will return an error from the server, so we have to conduct all operations on the client-side.
For such cases in the “go-sql-driver/mysql” there is an option called “interpolateParams”. If you set it to “true”, the driver will escape the variables for any requests and send ready-for-use queries to the server.
Don’t forget that interpolateParams cannot be used together with the multibyte encodings BIG5, CP932, GB2312, GBK, or SJIS. These are rejected as they may introduce a SQL injection vulnerability!
Now we are ready to use our request.
db.Exec("CREATE USER ?@? IDENTIFIED BY ?", "myuser", "localhost", "password")
The server will receive a single query string with given parameters and will look like this:
CREATE USER 'myuser'@'localhost' IDENTIFIED BY 'password'
So now you are able to not use, for example, SQL variables or/and CONCAT() function for creating queries.
As we see the use of the go-sql-driver allows us to reduce the number of operations with the database, and to facilitate working with it. And, of course, using interpolateParams, we can easily – and most importantly – safely work with any kind of request.
(Thursday, June 18, 2020 – 06:00 PM to 07:00 PM PDT)
There can be several reasons for a MySQL database outage: hardware failure, power outage, human error, natural disaster etc. We may not be able prevent all the disaster from happening but investing on a robust disaster recovery plan is very important for building fault-tolerant database infrastructure operations on MySQL. Every MySQL DBA is accountable for developing a disaster recovery plan addressing data sensitivity, data loss tolerance and data security. Join Shiv Iyer, Founder and Principal of MinervaDB to lean about the best practices for building highly reliable MySQL DR strategy and operations on Thursday, June 18, 2020 – 06:00 PM to 07:00 PM PDT. Building DR for a high traffic MySQL database infrastructure means deep understanding of multiple backup strategies and choosing optimal ones which are best suited for performance and reliability. Most of the data intensive MySQL infrastructure will have a combination of multiple backup methods and tools, In this webinar Shiv talks about his experiences in the past and present on building MySQL DR Ops, tools and zero tolerance data loss methods.
FromDual has the pleasure to announce the release of the new version 1.2.0 of its popular Database Performance Monitor for MySQL fpmmm.
The FromDual Performance Monitor for MySQL (fpmmm) enables DBAs and System Administrators to monitor and understand what is going on inside their MySQL database instances and on the machines where the databases reside.
The new FromDual Performance Monitor for MySQL (fpmmm) can be downloaded from here or you can use our FromDual repositories. How to install and use fpmmm is documented in the fpmmm Installation Guide.
In the inconceivable case that you find a bug in the FromDual Performance Monitor for MySQL please report it to the FromDual Bug-tracker or just send us an email.
Any feedback, statements and testimonials are welcome as well! Please send them to us.
Monitoring as a Service (MaaS)
You do not want to set-up your Database monitoring yourself? No problem: Choose our MySQL Monitoring as a Service (Maas) program to safe time and costs!
Installation of Performance Monitor 1.2.0
A complete guide on how to install FromDual Performance Monitor you can find in the fpmmm Installation Guide.
Upgrade of fpmmm tarball from 1.0.x to 1.2.0
Upgrade with DEB/RPM packages should happen automatically. For tarballs follow this:
shell> cd /opt
shell> tar xf /download/fpmmm-1.2.0.tar.gz
shell> rm -f fpmmm
shell> ln -s fpmmm-1.2.0 fpmmm
Changes in FromDual Performance Monitor for MySQL 1.2.0
This release contains new features and various bug fixes.
You can verify your current FromDual Performance Monitor for MySQL version with the following command:
shell> fpmmm --version
General
MySQL 8.0 problems fixed. Fpmmm supports MySQL 8.0 now!
Naming convention for Type changed from host to machine and mysqld to instance, including downwards compatibility.
New host screens added.
All Screens removed because they are customer specific, we have host screens now.
Zabbix templates adapted to the more flexible trigger URL.
Renamed all files to make it more agnostic.
Server
Code made more robust for cloud databases.
Free file descriptors removed because it is always zero, trigger added for 80% file descriptors used.
Cache file base bug in getIostat fixed.
Server graph for file descriptors improved
I/O queue ymin set to 0.
Server template optimized.
Iostat graphs added to server template.
Integrated iostat data into fpmmm.
All registered devices and bug on svctm fixed.
More info added when server module is called with --debug option.
Disk status items cleaned-up and filesystem names added for creating new items.
Interface eth1 removed but list of all interfaces added.
NUMA trigger added.
Macros for network interfaces added.
Data
Data module added to measure schema and instance size.
Code made ready for cloud databases.
User
Module for per user data added.
Dirty code fixed, found on cloud databases.
User info for transactions added.
Tmp disk tables and sort merge passes per user information added.
Agent
MySQL 8.0 compatibility issue fixed with user privileges.
Output format zabbix, icinga, nagios and centreon should be supported now.
Error messages for connect improved.
Option --debug added, one message was not handled correctly in verbosity level.
Parameters in function goThroughAllSections cleaned-up.
Option -h added, info more clear when wrong options were used.
URLs added to fpmmm template.
fpmmm check and trigger improved.
Made error handling better after test of 1.1.0 on CentOS 7.
fpmmm trigger error message improved.
InnoDB
NUMA information and warning trigger added to InnoDB module.
Trigger for innodb_force_recovery made repeatable.
Alert level for innodb_force_recovery increased, InnoDB non default page size alert added.
InnoDB deadlock detection is alarmed, when disabled.
innodb_metrics only works with SUPER privilege, fixed.
InnoDB Log Buffer much to small trigger added on Innodb_log_waits item.
Items and graphs for InnoDB temporary tables added.
In most cases, MySQL password instructions provide information on changing MySQL user passwords on the production system (e.g., reset root password without restart). It is even recommended to change passwords regularly for security reasons. But still, sometimes DBA duties on legacy systems offer surprises and you need to recover the original password for some old users.
There is no magic: as long as only hashes are stored and not the original passwords, the only way to recover the lost password is to brute force it from the known hash.
Note on Security and mysql-unsha1 Attack
Interestingly, if a hacker has access to password hash, he doesn’t need to recover a plain text password from it. It doesn’t matter how strong the password and how strong the hashing algorithm inside the auth plugin, because due to MySQL protocol design, hash is enough to connect to a database with a patched version of MySQL client. It means, if a hacker has access to a database backup, he automatically receives all needed information (SHAs) for connecting to a running database. See for the attack details.
Since MySQL 8.0, caching_sha2_password auth plugin is used by default, and this plugin brings a stronger sha256 function instead of sha1 used in mysql_native_password plugin. For authentication with caching_sha2_password plugin, it is also enough to have only a hash, see for the implementation details.
So, caching_sha2_password plugin doesn’t add any additional security compared to mysql_native_password plugin: if you have a hash, you don’t need plain text password to be able to connect to the instance with a patched MySQL client.
Still, if you want to have a password that works with an unmodified client, however, you need to do some hacking, see instructions below.
Dump Hash
Let’s return to the password recovery. First of all, we need to dump hashes.
MySQL 5.7 uses the mysql_native_password auth plugin by default and we can dump sha1 hashes with the following command.
% mysql -Ns -uroot -e "SELECT SUBSTR(authentication_string,2) AS hash FROM mysql.user WHERE plugin = 'mysql_native_password' AND authentication_string NOT LIKE '%THISISNOTAVALIDPASSWORD%' AND authentication_string !='';" > sha1_hashes
MySQL 8.0 uses the caching_sha2_password auth plugin by default and we can dump sha256 hashes as follows.
% mysql -Ns -uroot -e "SELECT CONCAT('\$mysql',LEFT(authentication_string,6),'*',INSERT(HEX(SUBSTR(authentication_string,8)),41,0,'*')) AS hash FROM mysql.user WHERE plugin = 'caching_sha2_password' AND authentication_string NOT LIKE '%INVALIDSALTANDPASSWORD%' AND authentication_string !='';" > sha256_hashes
If you need to get the root password hash and don’t have a user who has read access to mysql.user table, you should start mysqld with the --skip-grant-tables option, see the official doc for details.
Run Linode GPU Instance
For password recovery, it is needed to run calculations on some powerful GPUs, and there are not so many cloud providers with GPU instances on the market. Linode is one of the remarkable cloud providers if you need a simple, reliable provider with a really helpful support department. Linode has a powerful CLI tool that simplifies “bash” automation a lot. Also, for more serious automation, the official Terraform provider exists.
128GB GPU Linode instance password recovery speed is 30000 MH/s (million hashes per second), which is very good. It needs only 2 hours to brute-force an 8-characters MySQL 5.7 passwords (upper case, lower case, numbers). Instance price is only 6 USD/Hour.
For example, the other biggest cloud provider (4 x NVIDIA Tesla V100 instance) with the same recovery speed cost two times more expensive – 12.24 USD/Hour.
Prepare Dictionary
The password brute-forcing is done based on dictionaries. We will use a small rockyou dictionary as an example, to show how it goes.
You can find really good dictionaries on the weakpass dot com website.
But it is possible that even the largest dictionary will not be enough for the recovery. In such a case you should check if the validate_password plugin is enabled and prepare a dictionary based on it. Check it as follows:
If the output of this command is empty, it means that the plugin is disabled. You can find some more details about the plugin in one of our previous blog posts about it, Improving MySQL Password Security with Validation Plugin.
The validate_password_policy field is the most important one here. It can have the following values:
Policy
Tests Performed
0 or LOW
Length
1 or MEDIUM
Length; numeric, lowercase/uppercase, and special characters
2 or STRONG
Length; numeric, lowercase/uppercase, and special characters; dictionary file
If validate_password_policy=STRONG and validate_password_dictionary_file is set, we need to exclude passwords from validate_password_dictionary_file:
In the example above: -m 8 is the minimal length of the password, value from validate_password_length variable; -M 32 is the maximal length of the password, for replication passwords the maximal length is 32 characters, see MySQL release nodes; -n password should contain numbers, see validate_password_number_count variable; -l -u password should contain lowercase/uppercase characters, see validate_password_mixed_case_count variable; -p password should contain special characters, see validate_password_special_char_count variable; prohibited.txt is a file from validate_password_dictionary_file variable; huge-dictonary.txt is the initial dictionary; reduced-dictonary.txt is the new dictionary without words from prohibited.txt.
If the dictionary attack failed, you have to create your own dictionary for the brute force. In this case, we recommend using one of the following tools: crunch, maskprocessor or via Hashcat options.
Compile Hashcat
In the case of MySQL 8.0, the latest version of hashcat from the master branch should be compiled due to the fact that code from https://github.com/hashcat/hashcat/issues/2305 wasn’t released in any version right now.
% sudo apt -y install make gcc
% git clone https://github.com/hashcat/hashcat.git
% cd hashcat
% make
% sudo make install
Enable OpenCL for NVIDIA
Update to the latest software, disable the nouveau driver and reboot:
% sudo apt update && sudo apt full-upgrade -y
% echo -e "blacklist nouveau\noptions nouveau modeset=0\nalias nouveau off" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
% sudo update-initramfs -u
% reboot
8-chars password with lower and upper case letters and digits for MySQL 5.7 can be recovered only in 2 hours on the Linode GPU instance. The same password for MySQL 8.0 can be recovered in 2.8 years. But in general, hackers don’t need to recover plain text passwords at all (see “mysql-unsha1 attack” section above). To reduce risks, it is needed to protect the content of mysql.user table, there are a few things that can be done:
Codership is pleased to announce a new Generally Available (GA) release of the multi-master Galera Cluster for MySQL 5.6 and 5.7, consisting of MySQL-wsrep 5.6.48 (release notes, download) and MySQL-wsrep 5.7.30 (release notes, download) with a new Galera Replication library 3.30 (release notes, download), implementing wsrep API version 25. This release incorporates all changes to MySQL 5.6.48 and 5.7.30 respectively, making it a MySQL High Availability solution.
A highlight of this release is that with MySQL 5.7.30, you will now have access to using the Percona audit log plugin, which will help with monitoring and logging connection and query activity that has been performed on specific servers. This implementation is provided as an alternative to the MySQL Enterprise Audit Log Plugin.
The Galera Replication library 3.30 has an enhancement to ensure that upon GCache recovery, all available space will be reclaimed in the ring buffer. Frequent cluster configuration changes handling of errors are also improved.
MySQL-wsrep 5.6.48 is an updated rebase to the 5.6.48 release, but also includes improvements around crash recovery: when binary logs are enabled, there is a more consistent recovery. SSL initialization has seen improvements, and error handling of cluster wide conflicts have been improved when the cluster itself is acting as an asynchronous secondary to a MySQL primary.
MySQL-wsrep 5.7.30 is an updated rebase to the 5.7.30 release, and in addition to what is present in 5.6.48, there is also the audit log plugin as mentioned above. One important note is that for your Galera Cluster, ensure that InnoDB tablespaces are kept within the data directory (if kept outside, they may not be copied over during SST).
Across the board, there is now also support and packages for CentOS 8 and RHEL 8.
You can get the latest release of Galera Cluster from http://www.galeracluster.com. There are package repositories for Debian, Ubuntu, CentOS, RHEL, OpenSUSE and SLES. The latest versions are also available via the FreeBSD Ports Collection.
A common MySQL strategy to perform updates with accumulating functions is to employ user-defined variables, using the UPDATE [...] SET mycol = (@myvar := EXPRESSION(@myvar, mycol)) pattern.
This pattern though doesn’t play well with the optimizer (leading to non-deterministic behavior), so it has been deprecated. This left a sort of void, since the (relatively) sophisticated logic is now harder to reproduce, at least with the same simplicity.
In this article, I’ll have a look at two ways to apply such logic: using, canonically, window functions, and, a bit more creatively, using recursive CTEs.
Although CTEs are fairly intuitive, I advise, to those unfamiliar with the subject, to read my previous post on the subject.
The same principle applies to the window functions principles; I will break the query/concepts down, however, it’s advised to have at least an idea. There is a vast amount of literature about window functions (which is the reason why I haven’t written about them until now); pretty much all the tutorials use as example either corporate budgets, or populations/countries. Here instead, I’ll use a real-world case.
In relation to the software, MySQL 8.0.19 is convenient (but not required). All the statements need to be run in the same console, due to reusing @venue_id.
There is always an architectural dilemma between placing the logic at the application level as opposed as the database level. Although this is an appropriate debate, in this context the underlying assumption is that it’s necessary that the logic stays at the database level; a requirement for this can be, for example, speed, which has actually been our case.
The problem
In this problem, we manage venue (theater) seats.
As a business requirement, we need to assign a “grouping”: an additional number representing each seat.
In order to set the grouping value:
start with grouping 0, and the top left seat;
if there is a space between the previous and current seat, or if it’s a new row, increase the grouping by 2 (unless it’s the first absolute seat), otherwise, increase by 1;
assign the grouping to the seat;
move to the next seat in the same row, or to the next row (if the row is over), and iterate from point 2., until the seats are exhausted.
In pseudocode:
current_grouping = 0
for each row:
for each number:
if (is_there_a_space_after_last_seat or is_a_new_row) and is_not_the_first_seat:
current_grouping += 2
else
current_grouping += 1
seat.grouping = current_grouping
In practice, we want the setup on the left to have the corresponding values on the right:
x→ 0 1 2 0 1 2
y ╭───┬───┬───╮ ╭───┬───┬───╮
↓ 0 │ x │ x │ │ │ 1 │ 2 │ │
├───┼───┼───┤ ├───┼───┼───┤
1 │ x │ │ x │ │ 4 │ │ 6 │
├───┼───┼───┤ ├───┼───┼───┤
2 │ x │ │ │ │ 8 │ │ │
╰───┴───┴───╯ ╰───┴───┴───╯
Setup
Let’s use a minimalist design for the underlying table:
We won’t need the row/number columns, however, on the other hand, we don’t want to use a table whose records are fully contained in an index, in order to be closer to a real-world setting.
Based on the diagram of the previous section, the seat coordinates are, in the form (y, x):
(0, 0), (0, 1)
(1, 0), (1, 2)
(2, 0)
Note that we’re using y as first coordinate, because it makes it easier to reason in terms of rows.
We’re going to load a large enough number of records, in order to make sure the optimizer doesn’t take unexpected shortcuts. We use recursive CTEs, of course 😉:
we’re using the CTEs in a (hopefully!) interesting way - each cycle represents a venue id, but since we want multiple seats to be generated for each venue (cycle), we cross join with a table including the seats data;
we’re using the v8.0.19’s row constructor (VALUES ROW()...) in order to represent a (joinable) table without actually creating it;
we generate random row/number data, as they’re filler;
for simplicity, no tweaks have been applied (e.g. data types are wider than needed, the indexes are added before the records are inserted, etc.).
The old-school approach
The old-school solution is very straightforward:
SET@venue_id=5000;-- arbitrary venue id; any (stored) id will doSET@grouping=-1;SET@y=-1;SET@x=-1;WITHseat_groupings(id,y,x,`grouping`,tmp_y,tmp_x)AS(SELECTid,y,x,@grouping:=@grouping+1+(seats.x>@x+1ORseats.y!=@y),@y:=seats.y,@x:=seats.xFROMseatsWHEREvenue_id=@venue_idORDERBYy,x)UPDATEseatssJOINseat_groupingssgUSING(id)SETs.grouping=sg.grouping;-- Query OK, 5 rows affected, 3 warnings (0,00 sec)
Nice and easy (but keep in mind the warnings)!
A little side note: I’m taking advantage of boolean arithmetic properties here; specifically, the following statements are equivalent:
some people find it intuitive, some don’t - it’s a matter of taste; since it’s clarified now, for compactness purposes, I will use it for the rest of the article.
It has just a “small” defect: it may work… or not.
The reason is that the query optimizer doesn’t necessarily evaluate left to right, so the assignment operations (:=) may be evaluated out of order, causing the result to be wrong. This is a problem typically experienced after MySQL upgrades.
As of MySQL 8.0, this functionality is indeed deprecated:
-- To be run immediately after the UPDATE.--SHOWWARNINGS\G-- *************************** 1. row ***************************-- Level: Warning-- Code: 1287-- Message: Setting user variables within expressions is deprecated and will be removed in a future release. Consider alternatives: 'SET variable=expression, ...', or 'SELECT expression(s) INTO variables(s)'.-- [...]
Let’s fix this!
Modern approach #1: Window functions
Window functions have been a long-awaited functionality in the MySQL world.
Generally speaking, the “rolling” nature of window functions fits very well accumulating functions. However, some complex accumulating functions require the results of the latest expression to be available, which is something window functions don’t support, since they work on a column basis.
This doesn’t mean that the problem can’t be solved, rather, than it needs to be re-thought.
In this case, we split the problem in two concepts; we think the grouping value for each seat as the sum of two values:
the sequence number of each seat, and
the cumulative value of the increments of all the seats up to the current one.
Those familiar with window functions will recognize the patterns here 🙂
The sequence number of each seat is a built-in function:
ROW_NUMBER()OVER<window>
The cumulative value is where things get interesting. In order to accomplish this task, we perform two steps:
we calculate each seat increment, and put it on a table (or CTE),
then, for each seat, we use a window function to sum the increments up to that seat.
calculates the increments for each seat, compared to the previous (more on LAG() later). It works purely on each record and the previous; it’s not cumulative.
Now, in order to calculate the cumulative increments, we just use a window function to compute the sum, for and up to each seat:
The LAG function, in the simplest form (LAG(x)), returns the previous value of the given column. A typical nuisance of window functions is to deal with the first record(s) in the window - since there is no previous record, they return NULL. With LAG, we can specify the value we want as third parameter:
LAG(x,1,x-1)-- defaults to `x -1`LAG(y,1,y)-- defaults to `y`
By specifying the defaults above, we make sure that the very first seat in the window will be treated by the logic as adjacent to the previous one (x - 1) and in the same row (y).
The alternative to defaults is typically IFNULL, which is very intrusive, especially considering the relative complexity of the expression:
-- Both valid. And both ugly!--IFNULL(x>LAG(x)OVERtzw+1ORy!=LAG(y)OVERtzw,0)IFNULL(x>LAG(x)OVERtzw+1,FALSE)ORIFNULL(y!=LAG(y)OVERtzw,FALSE)
The second LAG() parameter is the number of positions to go back in the window; 1 is the previous, which is also the default value.
Technical aspects
Named windows
In this query, we’re using multiple times the same window. The following queries are formally equivalent:
However, the latter may cause a suboptimal plan (which I’ve experienced, at least in the past); the optimizer may treat the windows as independent, and iterate them separately.
For this reason, I advise to always use named windows, at least when there are duplicated ones.
PARTITION BY clause
Typically, window functions are executed over a partition, which in this case would be:
Query result rows are determined from the FROM clause, after WHERE, GROUP BY, and HAVING processing, and windowing execution occurs before ORDER BY, LIMIT, and SELECT DISTINCT.
Considerations
Abstractly speaking, in order to solve this class of problems, instead of representing each entry as as a function of the previous one, we calculate the state change for each entry, then sum the changes up.
Although more complex than the functionality it replaces, this solution is very solid. This approach though, may not be always possible, or at least easy, so that’s where the recursive CTE solution comes into play.
Modern approach #2: Recursive CTE
This approach requires a workaround due to a limitation in MySQL’s CTE functionality, but, on the other hand, it’s a generic, direct, solution, and as such, it doesn’t require any rethinking of the approach.
Let’s start from a the simplified version of the end query:
-- `p_` is for `Previous`, in order to make the conditions a bit more intuitive.--WITHRECURSIVEgroupings(p_id,p_venue_id,p_y,p_x,p_grouping)AS((SELECTid,venue_id,y,x,1FROMseatsWHEREvenue_id=@venue_idORDERBYy,xLIMIT1)UNIONALLSELECTs.id,s.venue_id,s.y,s.x,p_grouping+1+(s.x>p_x+1ORs.y!=p_y)FROMgroupings,seatssWHEREs.venue_id=p_venue_idAND(s.y,s.x)>(p_y,p_x)ORDERBYs.venue_id,s.y,s.xLIMIT1)SELECT*FROMgroupings;
Bingo! This query is (relatively) simple, but most importantly, it expresses the grouping accumulating function in the simplest possible way:
p_grouping+1+(s.x>p_x+1ORs.y!=p_y)-- the above is equivalent to:@grouping:=@grouping+1+(seats.x>@x+1ORseats.y!=@y),@y:=seats.y,@x:=seats.x
Even for those who are not accustomed with CTEs, the logic is simple.
The initial row is the first seat of the venue, in order:
the WHERE condition, along with the ORDER BY and LIMIT clauses, simply find the next seat, that is, the one seat with the same venue id, which, in order of (venue_id, x, y), has greater (x, y) coordinates.
The s.venue_id part of the ordering is crucial! This allows us to use the index.
The SELECT clause takes care of:
performing the accumulation (computation of (p_)grouping),
passing the values of the current seat (s.id, s.venue_id, s.y, s.x) to the next cycle.
We select FROM groupings so that we fulfill the requirements for the CTE to be recursive.
What’s interesting here is that we use the recursive CTE essentially as iterator, via selection from the groupings table in the recursive subquery, while joining with seats, in order to find the data to work on.
The JOIN is formally a cross join, however, only one record is returned, due to the LIMIT clause.
Working version
Unfortunately, the above query doesn’t work because the ORDER BY clause is currently not supported in the recursive subquery; additionally, the semantics of the LIMIT as used here are not the intended ones, as they apply to the outermost query:
LIMIT is now supported […] The effect on the result set is the same as when using LIMIT in the outermost SELECT
However, it’s not a significant problem. Let’s have a look at the working version:
It’s a bit of shame having to use a subquery, but it works, and the boilerplate is minimal, as several clauses are required anyway.
Here, instead of performing the ordering and limiting, in the relation resulting from the join of groupings and seats, we do it in a subquery, and pass it to the outer query, which will consequently select only the target record.
Performance considerations
Let’s have a look at the query plan, using the EXPLAIN ANALYZE functionality:
mysql> EXPLAIN ANALYZE WITH RECURSIVE groupings [...]
-> Table scan on groupings (actual time=0.000..0.001 rows=5 loops=1)
-> Materialize recursive CTE groupings (actual time=0.140..0.141 rows=5 loops=1)
-> Limit: 1 row(s) (actual time=0.019..0.019 rows=1 loops=1)
-> Index lookup on seats using venue_id_y_x (venue_id=(@venue_id)) (cost=0.75 rows=5) (actual time=0.018..0.018 rows=1 loops=1)
-> Repeat until convergence
-> Nested loop inner join (cost=3.43 rows=2) (actual time=0.017..0.053 rows=2 loops=2)
-> Scan new records on groupings (cost=2.73 rows=2) (actual time=0.001..0.001 rows=2 loops=2)
-> Filter: (s.id = (select #5)) (cost=0.30 rows=1) (actual time=0.020..0.020 rows=1 loops=5)
-> Single-row index lookup on s using PRIMARY (id=(select #5)) (cost=0.30 rows=1) (actual time=0.014..0.014 rows=1 loops=5)
-> Select #5 (subquery in condition; dependent)
-> Limit: 1 row(s) (actual time=0.007..0.008 rows=1 loops=9)
-> Filter: ((si.y,si.x) > (groupings.p_y,groupings.p_x)) (cost=0.75 rows=5) (actual time=0.007..0.007 rows=1 loops=9)
-> Index lookup on si using venue_id_y_x (venue_id=groupings.p_venue_id) (cost=0.75 rows=5) (actual time=0.006..0.006 rows=4 loops=9)
The plan is very much as expected. The foundation of an optimal plan for this case, is in the index lookups:
-> Nested loop inner join (cost=3.43 rows=2) (actual time=0.017..0.053 rows=2 loops=2)
-> Single-row index lookup on s using PRIMARY (id=(select #5)) (cost=0.30 rows=1) (actual time=0.014..0.014 rows=1 loops=5)
-> Index lookup on si using venue_id_y_x (venue_id=groupings.p_venue_id) (cost=0.75 rows=5) (actual time=0.006..0.006 rows=4 loops=9)
which are paramount; if even an index scan is performed (in short, when the index entries are scanned linearly, instead of finding directly the desired one), the performance will tank.
Therefore, the requirements for this strategy to work, are that the related indexes are in place and are used by the optimizer very efficiently.
It’s expected that, in the future, if the restrictions are lifted, not having to use the subquery will make the task considerably simpler for the optimizer.
Alternative for suboptimal plans
For particular use cases where an optimal plan can’t be found, just use a temporary table:
Do-it-Yourself (DIY), database migration; what other approaches are there to challenges of MySQL availability, reliability, scalability and application performance?
Continuent Tungsten enables greater application availability and performance especially across WAN, it saves tons of manual labor and headache, it’s backed by premium 24/7/365 support, and it enables all sorts of new freedom and possibilities for data infrastructure including multi-cloud, hybrid-cloud, and advanced, real-time analytics.
Join Peter Zaitsev, CEO at Percona, as he discusses database challenges and the concepts of open source and open core. Over the years, open source companies have tried to bring products to market and maximize their revenue streams. To date, the most popular open source business model remains “Open Core”. But is open core software still open source? Or is it a freemium model designed to separate you from your money? Not all companies follow the same processes, ethics, and rules when building and launching open source products. Let’s talk about how the open core bait and switch works for many companies in the open source space.
Join this webinar to learn more about:
– Open Source and Open core distinction
– Free enterprise-class community versions alternatives
– Use-cases that successfully moved away from vendor lock-in
Please join Peter Zaitsev on Tuesday, June 30 at 11 am EDT for his webinar “Database Challenges: Open Source Vs. Open Core“.
The RED Method (Rate, Errors, Duration) is one of the more popular performance monitoring approaches. It is often applied to Monitoring Microservices though there is nothing that prevents it from being applied to databases like MySQL.
In Percona Monitoring and Management (PMM) v2 we have all the required information stored in the ClickHouse database, and with the built-in ClickHouse datasource it is a matter of creating a dashboard to visualize the data.
While I was editing the dashboard, I added a few other panels, beyond what RED Method requires, in order to show some of the cool things you can do with Grafana + ClickHouse data source and information we store about MySQL query performance.
Let’s explore this dashboard in more detail.
We can see the RED Method Classical panels showing Query Rate, Error Rate, as well as query latency (average and 99 percentile) for all the nodes in the system. The panel below shows the breakdown for different nodes which is very helpful to compare performance between the nodes. If one of the hosts starts to perform differently from the rest of similar hosts in the fleet, it will warrant investigation.
You do not have to look at the whole of your data through the “Filters” functionality at the top of the dashboard to apply any filters you want. For example, I can choose to only look at the queries which hit “sbtest” schema for hosts located in region “datacenter4”:
This Ad-Hoc Filtering is really powerful. You can also match queries you want to look at by regular expression, look at particular QueryID, Queries from particular Client Hosts, etc. You can check out all columns available in ClickHouse and their meaning in the blog post Advanced Query Analysis in Percona Monitoring and Management with Direct ClickHouse Access.
From most of the panels you can quickly jump to Query Analytics to see more details of query performance or if you notice one of the hosts having unusual performance you can use the “Data Links” functionality to see queries for this host only – click on the line on the graph and follow the highlighted link:
You can also see the same RED Metrics you show for the whole system for each of the systems separately. I would keep those rows collapsed by default, especially if you have many hosts under monitoring.
Now we’re done with RED Method Panels, so let’s see what additional panels this dashboard provides and how to use them.
Row Based Efficiency dashboard shows how many rows are traversed for every row which is sent or changed. Values higher than 100 tend to mean you either have poor indexes or run some very complicated queries which crunch through a lot of data to send or modify a few. Both such cases are prime examples of queries to review.
Time-Based Efficiency does the same math but looks at the query execution time rather than the number of scanned rows. This allows us to float problems caused by slow disk or query contention. Generally, for a high-performance system, you should expect it to spend fractions of a millisecond to send a row to the client or modify it. Queries that send a lot of rows or modify a lot of rows will have a lower value.
Queries Per Host is pretty self-explanatory and it is very helpful to see alongside the Query Load Per Host panel which shows which hosts have the highest amount of queries active at the same time. We can see here while MySQL4 does not have the highest query rate in the mix, it has the highest load or highest number of average active queries.
As I was thinking about what other metrics might be helpful, I also added these additional panels:
These panels break down the Query Processing efficiency to READ queries (those which send rows) and WRITE queries (which has row_affected).
QueryTime Based Efficiency is the same as the panel described before, just with a focus on particular kinds of queries.
Data Crunching Efficiency is a bit different spin on the same data; it looks at how many rows are examined by those queries vs query execution time. This, on one extent, shows system processing power; system with many cores having all data in memory may be able to crunch millions of rows per second and do a lot of work. But it does not mean all this work has value. In fact, systems that crunch a lot of data quickly are often doing many full table scans.
Finally, there are few Query Lists on the same page.
Frequent queries, queries which are the slowest on average, queries which cause the most load, and queries which terminated with an error or warning. You can get these in Query Analytics as well, but I wanted to show those as examples.
One of the good traits of this plugin is that it allows authentication using challenge-response mechanism which made it possible to verify identity of the client on an unencrypted channel without having a need to send the actual password.…
In recent years, the use of platform infrastructure has shifted from on premise to cloud computing. This is based on the absence of cost capital costs that must be incurred by the company if used when implementing IT infrastructure. Cloud computing provides flexibility in every line of resources ie. on human resource, energy, time savings.
Cloud computing makes it easy for organizations to do IT planning, executing, maintaining platforms to support business interests.
But both have similarities, we had to think about BCP (Business Continuity Plan) and Disaster Recovery Plan (DRP) when using cloud. Data storage becomes critical when we talk about DRP, how fast we do recovery (Recovery Point Objective) when a disaster occurs. Multi-cloud architecture plays a big role when we want to design and implement an infrastructure in the cloud environment. In this blog, we review the related multi cloud deployment for storing data in MySQL.
Environment Setup in the Cloud
This time we use Amazon Web Service (AWS), which is widely used by companies, and Google Cloud Platform (GCP) as the second cloud provider in a multi cloud database setup. Making instances (the term used in cloud computing for new Virtual Machines) on AWS is very straight forward.
AWS uses the term Amazon EC2 (Elastic Compute Cloud) for their compute instance service. You can login to AWS, then select EC2 service.
Here’s the display of an instance that has been provisioned with EC2.
For security reasons, which is the biggest concern of cloud services, make sure we only enable ports that are needed when deploying ClusterControl, such as SSH port (22), xtrabackup (9999) and database (3306) are secured but reachable across the cloud providers. One way to implement such connectivity would be to create VPN that would connect instances in AWS with instances in GCP. Thanks to such design, we can treat all instances as local, even though they are located in different cloud providers. We will not be describing exactly the process of setting up VPN therefore please keep in mind that the deployment we present is not suitable for real world production. It is only to illustrate possibilities that come with ClusterControl and multi-cloud setups.
After completing the AWS EC2 Setup, continue with setting up the compute instance in GCP, in GCP the compute service is called Compute Engine.
In this example, we will create 1 instance in GCP cloud which will be used as one of the Slaves.
When it is completed, it will be shown in the management console as below:
Make sure you secure and enable port SSH port (22), xtrabackup (9999) and database (3306).
After deploying instances in both AWS and GCP, we should continue with installation of ClusterControl on one of the instances in the cloud provider, where the master will be located. In this example setup we will use one of AWS instances as the Master.
Deployment MySQL Replication pada Amazon Web Service
To install ClusterControl you should follow simple instructions that you can find on Severalnines’ website. Once ClusterControl is up and running in the cloud provider where our master is going to be located (in this example we will use AWS for our master node) we can start the deployment of MySQL Replication using ClusterControl. There are following steps you need to take to install MySQL replication cluster:
Open ClusterControl then select MySQL Replication, you will see three forms need to be filled for the installation purpose
General and SSH Settings
Enter SSH User, Key and Password, SSH Port and the name of the cluster
Then select ‘Continue’
Define MySQL Servers
Select vendor, version number, and root password of MySQL, then click ‘Continue’
Define Topology
As you remember, we have two nodes created in AWS. We can use both of them here. One shall be our master, the other should be added as slave. Then we can proceed with ‘Deploy’
If you want, and if the cross-cloud connectivity is already in place, you can also set the GCP Instance IP address under ‘Add slaves to master A’ then continue with ‘Deploy’. In this way ClusterControl will deploy master and both slaves at the same time.
Once started the deployment you can monitor the progress in the Activity tab. You can see the example of the progress messages below. Now it’s time to wait until the job is completed.
Once it is completed, you can see the newly created Cluster named “Cloud MySQL Replication”.
If you already added GCP node as a second slave in the deployment wizard, you have already completed the Master-Slaves setup between AWS andGCP Instances.
If not, you can add the GCP slave to the running cluster. Please make sure that the connectivity is in place before proceeding further.
Add a New Slave from Google Cloud Platform
After MySQL Replication on AWS has been created, you can continue by adding your node in GCP as a new slave. You can accomplish that by performing following steps::
On the cluster list find your new cluster and then click on and select ‘Add Replication Slave’
Add Replication Slave’ wizardwill appear, as you can see below.
Continue by picking the IP of Master Instance (located in AWS) and entering the IP address and Port of GCP instance that you want to use as a slave in the ‘Slave Hostname / IP’ box. Once you fill everything,you can proceed with clicking on ‘Add Replication Slave’.
As before, you can monitor the progress in the activity tab. Now it’s time to wait until the job is completed.
Once deployment is done we can check the cluster in the topology tab.
You can see the topology of our Master – Slave cluster below.
As you can see, we have a master and one slave in AWS and we have as well a slave in GCP, making it easier for our database to survive any outages that happen in one of our cloud providers.
Conclusion
For high-availability of database services, a multi cloud deployment takes a very important role to make it happen. ClusterControl is created to navigate this process and make it easier for the user to manage the multi-cloud deployments.
One of the critical things to consider when doing Multi-Cloud Deployment are security aspects. As we mentioned earlier, you can setup a VPN Site to Site between the two cloud providers as the best practice that can be applied. There are also other options like SSH tunnels.
MinervaDB Webinar – MySQL Backup Strategies and Tools
Most often Database Systems outages happen due to user error and it is also the biggest reason for data loss / damage or corruption. In these type of failures, It is application modifying or destroying the data on its own or through a user choice. Hardware failure also contributes to database infrastructure crashes and corruption. To address these sort of data reliability issues, you must recover and restore to the point in time before the corruption occurred. Disaster Recover tools returns the data to its original state at the cost of any other changes that were being made to the data since the point the corruption took place. MinervaDB founder and Principal, hosted a webinar (Thursday, June 18, 2020 – 06:00 PM to 07:00 PM PDT) on MySQL backup strategies and tools addressing the topics below:
Even though 
CONTACT GLOBAL SALES (24*7)
(844) 588-7287 (USA)
MINERVADB FAX
MinervaDB Email - General / Sales / Consulting
CORPORATE ADDRESS: CALIFORNIA
In this webinar, Sveta Smirnova, MySQL Engineer at Percona, will uncover nuances of Percona XtraDB Cluster (PXC) schema upgrades and point out details you need to give extra attention to.
Day-to-day database operation requires, from an administrator, deep knowledge of db internals and security issues, in particular things like SQL injections. In order to prevent such kind of an attack, we have included go-sql-driver into our code for secure placeholder escaping.
In most cases, MySQL password instructions provide information on changing MySQL user passwords on the production system (e.g., 
Join Peter Zaitsev, CEO at Percona, as he discusses database challenges and the concepts of open source and open core. Over the years, open source companies have tried to bring products to market and maximize their revenue streams. To date, the most popular open source business model remains “Open Core”. But is open core software still open source? Or is it a freemium model designed to separate you from your money? Not all companies follow the same processes, ethics, and rules when building and launching open source products. Let’s talk about how the open core bait and switch works for many companies in the open source space.
and select ‘Add Replication Slave’
