As a part of Mydbops Consulting we have a below problem statement from one of our client.

“We have a high powered server for reporting which in turn powers our internal dashboard for viewing the logistics status.Even with a high end hardware, we had a heavy CPU usage and which in turn triggers spikes in replication lag and slowness. Below is the hardware configuration.“

OS : Debian 9 (Stretch)

CPU : 40

RAM : 220G (Usable)

Disk : 3T SSD with 80K sustained IOPS.

MySQL : 5.6.43-84.3-log Percona Server (GPL)

Datasize : 2.2TB

Below is the graph on CPU utilisation from Grafana.

Since the work load is purely reporting(OLAP) we could observe a similar type of queries with different ranges. Below is the Execution plan of the query. It is a join query over 6 tables.

Explain Plan:

+----+-------------+-------+--------+--------------------------------------------------------------------------------+---------------+---------+---------------------------------+------+------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+--------------------------------------------------------------------------------+---------------+---------+---------------------------------+------+------------------------------------+
| 1 | SIMPLE | cf | ref | PRIMARY,name_x | name_x | 103 | const | 1 | Using where; Using index |
| 1 | SIMPLE | scf | ref | sip_idx,active_idx,flag_idx | flag_idx | 8 | logis.cf.flagId | 5820 | Using where |
| 1 | SIMPLE | sre | ref | staId,sre_idx1,shelfId_hubId,updateDate_statusId_statId_idx,statId_statusId_x | sre_idx1 | 18 | logis.scf.sipId | 1 | Using index condition; Using where |
| 1 | SIMPLE | scam | eq_ref | mappingId | mappingId | 8 | logis.scf.mapId | 1 | NULL |
| 1 | SIMPLE | ssdm | ref | sipIdIdx,SDetailIdIdx | shipmentIdIdx | 17 | logis.sre.sipId | 1 | NULL |
| 1 | SIMPLE | sd | eq_ref | PRIMARY,mrchIdIdx | PRIMARY | 8 | logis.ssdm.SDId | 1 | Using where |
+----+-------------+-------------------------------------------------

With the initial screening it looks normal as it’s a reporting (OLAP) query and its bound to run longer and this has started to bit our system resources (CPU) and the replication lag cause stale or obsolete data in internal dashboard.

As the execution plan depicts the query is using index and the columns used are being perfectly indexed. Only with table “SCF” we could see a scan of “5820 “ , The index used here has a less cardinality.

Now we should tweak the optimizer to choose the right index. Optimizer chooses the index based on the stats collected for the table and stored under mysql.innodb_table_stats and mysql.innodb_index_stats. .

The default value of innodb_stats_persistent_sample_pages and innodb_stats_transient_sample_pages are 20 and 8 respectively, which is too low for our dataset, This works wells for smaller tables, but in our case tables are in few 100’s of GB. We increased the values below globally since its a dynamic variable by 10X approx.

mysql> set global innodb_stats_persistent_sample_pages=200;
Query OK, 0 rows affected (0.00 sec)

mysql> set global innodb_stats_transient_sample_pages=100;
Query OK, 0 rows affected (0.00 sec)

Below is the definition from the official documentation on these variables,

“The number of index pages to sample when estimating cardinality and other statistics for an indexed column, such as those calculated by ANALYZE TABLE. Increasing the value improves the accuracy of index statistics, which can improve the query execution plan, at the expense of increased I/O during the execution of ANALYZE TABLE for an InnoDB table.”

Now we will have to force the index stats recalculation by running a “Analyze table table_name” on all the table involved in the query or else you can make variables persistent and invoke a DB restart to calculate stats for all the tables , we chose the first method since its less harming.

Let us review the execution plan now, we could see a reduced row scans and better index usage with the optimiser as below:

+----+-------------+-------+--------+--------------------------------------------------------------------------------+---------------+---------+---------------------------------+------+------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+--------------------------------------------------------------------------------+---------------+---------+---------------------------------+------+------------------------------------+
| 1 | SIMPLE | cf | ref | PRIMARY,name_x | name_x | 103 | const | 1 | Using where; Using index |
| 1 | SIMPLE | sre | ref | staId,sre_idx1,shelfId_hubId,updateDate_statusId_statId_idx,statId_statusId_x | shelfId_hubId | 9 | const | 2936 | Using where |
| 1 | SIMPLE | scf | ref | sip_idx,active_idx,flag_idx | sip_idx | 22 | logis.sre.shipmentId | 1 | Using index condition; Using where |
| 1 | SIMPLE | scam | eq_ref | mappingId | mappingId | 8 | logis.scf.mappingId | 1 | NULL |
| 1 | SIMPLE | ssdm | ref | shipmentIdIdx,sellerDetailIdIdx | sipIdIdx | 17 | logis.sre.shipmentId | 1 | NULL |
| 1 | SIMPLE | sd | eq_ref | PRIMARY,merchantIdIdx | PRIMARY | 8 | logis.ssdm.sellerDetailId | 1 | Using where |
+----+-------------+-------+--------+--------------------------------------------------------------------------------+---------------+---------+---------------------------------+------+------------------------------------+

Note the new index plan is applicable for the new incoming queries. Within a short span of time the CPU usage has dropped down drastically there is huge boost in performance, please find the graph below. Now the dashboards are faster than ever before.

Key Takeaways:

Below are points to note while setting this variable.

• Too high value can result in longer time for stats calculation .
• Too low can have inaccurate stats and leads to a situation discussed above
• As the table grows, InnoDB allegedly re-ANALYZEs ie., re-calculates stats after 10% growth, so no manual action needed.
• Most tables have decent stats from sampling 20 pages (Smaller tables)
• Tables with uneven distribution won’t benefit from changing the ’20’, to tackle that we have “Histograms” from MySQL 8.0

Data nodes are the distributed, sharded storage core of MySQL NDB Cluster. Its data is usually accessed by MySQL Servers (also called SQL nodes in NDB parlance). The MySQL servers each have their own transactional Data Dictionary (DD) where all the metadata describing tables, databases, tablespaces, logfile groups, foreign keys, and other objects are stored for use by MySQL server.…

In part 1, we took a brief, high-level look at the various protocols and mechanisms used to keep the Data Dictionary (DD) of MySQL servers connected to a MySQL Cluster in synchronization with each other and with the NDB Dictionary.…

As the second part of the earlier post Give Love to Your SSDs – Reduce innodb_io_capacity_max! we wanted to put together some concepts on how InnoDB flushing works in recent Percona Server for MySQL versions (8.0.x prior to 8.0.19, or 5.7.x). It is important to understand this aspect of InnoDB in order to tune it correctly. This post is a bit long and complex as it goes very deep into some InnoDB internals.

InnoDB internally handles flush operations in the background to remove dirty pages from the buffer pool. A dirty page is a page that is modified in memory but not yet flushed to disk. This is done to lower the write load and the latency of the transactions. Let’s explore the various sources of flushing inside InnoDB.

Idle Flushing

We already discussed the idle flushing in the previous post mentioned above. When there are no write operations, which means the LSN isn’t moving, InnoDB flushes dirty pages at the innodb_io_capacity rate.

Dirty Pages Percentage Flushing

This source of flushing is a slightly modified version of the old InnoDB flushing algorithm used years ago. If you have been around MySQL for a while, you probably don’t like this algorithm. The algorithm is controlled by these variables:

innodb_io_capacity (default value of 200) 
innodb_max_dirty_pages_pct (default value of 75) 
innodb_max_dirty_pages_pct_lwm (default value of 0 in 5.7 and 10 above 8.0.3)

If the ratio of dirty pages over the total number of pages in the buffer pool is higher than the low water mark (lwm), InnoDB flushes pages at a rate proportional to the actual percentage of dirty pages over the value of Innodb_max_dirty_pages_pct multiplied by innodb_io_capacity. If the actual dirty page percentage is higher than Innodb_max_dirty_pages_pct, the flushing rate is capped at innodb_io_capacity.

The main issue with this algorithm is that it is not looking at the right thing. As a result, transaction processing may often freeze for a flush storm because the max checkpoint age is reached. Here’s an example from a post written by Vadmin Tkachenko back in 2011: InnoDB Flushing: a lot of memory and slow disk.

TPCC New order transaction over time

We can see a sharp drop in NOTP (New Order Transaction per second) at time = 265, and that’s because InnoDB reached the maximum checkpoint age and had to furiously flush pages. This is often called a flush storm. These storms freeze the write operations and are extremely bad for the database operation. For more details about flush storms, please see InnoDB Flushing: Theory and solutions.

Free List Flushing

To speed up reads and page creation, InnoDB tries to always have a certain number of free pages in each buffer pool instance. Without some free pages, InnoDB could need to flush a page to disk before it can load a new one.

This behavior is controlled by another poorly understood variable: innodb_lru_scan_depth. This is a pretty bad name for what the variable controls. Although the name makes sense if you look at the code, for a normal DBA the name should be innodb_free_pages_per_pool_target. At regular intervals, the oldest pages in the LRU list of each buffer pool instance is scanned (hence the name), and pages are freed up to the variable value. If one of these pages is dirty, it will be flushed before it is freed.

Adaptive Flushing

The adaptive flushing algorithm was a major improvement to InnoDB and it allowed MySQL to handle much heavier write load in a decent manner. Instead of looking at the number of dirty pages like the old algorithm, the adaptive flushing looks at what matters: the checkpoint age. The first adaptive algorithm that we are aware of came from Yasufumi Kinoshita in 2008 while he was working at Percona. The InnoDB plugin 1.0.4 integrated similar concepts, and eventually Percona removed its flushing algorithm because the upstream one was doing a good job.

The following description is valid for Percona Server for MySQL 8.0.18-. While we were busy writing this post, Oracle released 8.0.19 which introduces significant changes to the adaptive flushing code. It looks like a good opportunity for a follow-up post in the near future…

Some Background

But let’s first take a small step back to put together some concepts. InnoDB stores rows in pages of normally 16KB. These pages are either on disk, in the data files, or in memory in the InnoDB buffer pool. InnoDB only modifies pages in the buffer pool.

Pages in the buffer pool may be modified by queries, and then they become dirty. At commit, the page modifications are written to the redo log, the InnoDB log files. After the write, the LSN (last sequence number) is increased. The dirty pages are not flushed back to disk immediately and are kept dirty for some time. The delayed page flushing is a common performance hack. The way InnoDB flushes its dirty pages is the focus of this post.

Let’s now consider the structure of the InnoDB Redo Log.

The InnoDB redo log files form a ring buffer

The InnoDB log files form a ring buffer containing unflushed modifications. The above figure shows a crude representation of the ring buffer. The Head points to where InnoDB is currently writing transactional data. The Tail points to the oldest unflushed data modification. The distance between the Head and Tail is the checkpoint age. The checkpoint age is expressed in bytes. The size and the number of log files determine the max checkpoint age, and the max checkpoint age is approximately 80% of the combined size of the log files.

Write transactions are moving the Head forward while page flushing is moving the Tail. If the Head moves too fast and there is less than 12.5% available before the Tail, transactions can no longer commit until some space is freed in the log files. InnoDB reacts by flushing at a high rate, an event called a Flush storm. Needless to say, flush storms should be avoided.

How the Adaptive Flushing Algorithm Works

The adaptive flushing algorithm is controlled by the following variables:

innodb_adaptive_flushing  (default ON)
innodb_adaptive_flushing_lwm  (default 10)
innodb_flush_sync (default ON)
innodb_flushing_avg_loops (default 30)
innodb_io_capacity (default value of 200)
innodb_io_capacity_max (default value of at least 2000)
innodb_cleaner_lsn_age_factor (Percona server only, default high_checkpoint)

The goal of the algorithm is to adapt the flushing rate (speed of the Tail) to the evolution of the checkpoint age (speed of the Head). It starts when the checkpoint age is above the adaptive flushing low water mark, by the default 10% of the max checkpoint age. Percona Server for MySQL offers two algorithms: Legacy and High Checkpoint.

The Legacy algorithm is given below. Notice the power of 3/2 on the age factor and the 7.5 denominator.

Legacy age factor

It also offers the High Checkpoint algorithm shown below:

Percona High-checkpoint age factor

This time the age power factor is 5/2 and the denominator is 700.5. It is important to note that in both equations, innodb_io_capacity (ioCap) appears as the denominator in a ratio with innodb_io_capacity_max (ioCapMax). If we plot both equations together, we have:

Flushing pressure for the legacy and high-checkpoint algorithm

The graph has been generated for a ratio ioCapMax/ioCap of 10 like with the default values. The Percona High Checkpoint starts slowly but then it increases rapidly. This allows for more dirty pages (see the post we previously discussed) and it is normally good for performance. The returned percentage value can be much higher than 100.

Average Over Time

So far, only the checkpoint age was used. While this is good, the goal of the algorithm is to flush pages in a way that the tail of the redo log ring buffer moves at about the same speed at the head. Approximately every innodb_flushing_avg_loops second, the rate of pages flushed and the progression of the head of the redo log is measured and the new value is averaged with the previous one. The goal here is to give some inertia to the algorithm, to damp the changes. A higher value of innodb_flushing_avg_loops makes the algorithm slower to react, while a smaller value makes it more reactive. Let’s call these quantities avgPagesFlushed and avgLsnRate.

Pages to Flush for the avgLsnRate

Based on the avgLsnRate value, InnoDB scans the oldest dirty pages in the buffer pool and calculates the number of pages that are at less than avgLsnRate of the tail. Since this is calculated every second, the number of pages returned is what needs to be flushed to maintain the correct rate. Let’s call this number pagesForLsnRate.

Finally…

We now have all the parts we need. The actual number of pages that will be flushed is given by:

This quantity is then capped to ioCapMax. As you can see, pctOfIoCapToUse is multiplied by ioCap. If you look back at the equations giving pctOfIoCapToUse, they have ioCap at the denominator. The ioCap cancels out and the adaptive flushing algorithm is thus independent of innodb_io_capacity, as only innodb_io_capacity_max matters. There could also be more pages flushed if innodb_flush_neighbors is set.

Can InnoDB Flush Pages at a Rate Higher Than innodb_io_capacity_max?

Yes, if innodb_flush_sync is ON, InnoDB is authorized to go beyond if the max checkpoint age is reached or almost reached. If set to OFF, you’ll never go beyond innodb_io_capacity_max. If the latency of your read queries is critical, disabling innodb_flush_sync will prevent an IO storm, but at the expense of stalling the writes.

InnoDB page_cleaner Error Message

Error messages like

[Note] InnoDB: page_cleaner: 1000ms intended loop took 4013ms. The settings might not be optimal. (flushed=1438 and evicted=0, during the time.)

are rather frequent. They basically mean the hardware wasn’t able to flush innodb_io_capacity_max pages per second. In the above example, InnoDB tried to flush 1,438 pages but the spinning disk is able to perform only 360 per second. Thus, the flushing operation which was supposed to take 1 second ended up taking 4 seconds. If you really think the storage is able to deliver the number of write iops stated by innodb_io_capacity_max, then it may be one of these possibilities:

• innodb_io_capacity_max represents a number of pages to be flushed, flushing a page may require more than one IO, especially when the number of tablespaces is large.
• A spike of read IOPs has competed with the flushing.
• The device had a write latency spike. Garbage collection on SSDs can cause that, especially if the SSD is rather full.
• The doublewrite buffer had contention. Try Percona Server for MySQL with the parallel doublewrite buffer feature.
• Do you have enough page cleaners to fully maximize your hardware?

Tuning InnoDB

Now that we understand how InnoDB flushes dirty pages to disk, the next obvious step is to tune it. InnoDB tuning will be covered in a follow-up post, so stay tuned.

From the initial release, one of the biggest complaints I had about Group Replication is that it allowed “stale” reads and there was no way to prevent them or to even know that you read “stale” data. That was a huge limitation. Thankfully, Oracle released features to control the consistency levels, and it was exactly a year ago! I don’t know about you, but I personally was confused by naming it group_replication_consistency=’AFTER’ or ‘BEFORE’.

So now I want to try to make sense of it and share my understanding (even if it is one year later).

Setup:

• 3 Node Percona Server for MySQL 8.0.18 running in Group Replication.

We will start with the default group_replication_consistency=’EVENTUAL’ and work from there. So let’s consider a very simple table:

CREATE TABLE `t1` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `cnt` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`)
)

With over 10mln rows:

select count(*) from t1;
+----------+
| count(*) |
+----------+
| 10485760 |
+----------+

And we will do a very simple action. On Node 1 we will update the table and on Node 2 we will read data.

Node 1:

UPDATE t1 SET cnt=11;
Query OK, 10485760 rows affected (1 min 20.23 sec)
Rows matched: 10485760  Changed: 10485760  Warnings: 0

And immediately after that on the Node 2:

select cnt from t1 where id=10001;
+------+
| cnt  |
+------+
|   10 |
+------+
1 row in set (0.00 sec)

There are a few points to note:

• On Node 1 it took 1min 20 sec for the transaction to execute
• On Node 2 we got the result immediately, but we essentially got a “stale” read in that data was already updated, but we got an old version of it.

How can we get a better outcome:

group_replication_consistency=’BEFORE’;

Let’s look into the consistency level ‘BEFORE’. It says that a transaction on Node 2 will wait until the previously committed transaction on the Node 1 transaction is committed also on Node 2. So let’s change the set session to group_replication_consistency=’BEFORE’; on Node 2 and execute a similar update again.

Node 1 (still group_replication_consistency=’EVENTUAL’;):

UPDATE t1 SET cnt=12;
Query OK, 10485760 rows affected (1 min 18.66 sec)
Rows matched: 10485760  Changed: 10485760  Warnings: 0

And after this, on Node 2 (with set session group_replication_consistency=’BEFORE’;):

select cnt from t1 where id=10001;
+------+
| cnt  |
+------+
|   12 |
+------+
1 row in set (1 min 11.45 sec)

So there are very notable changes:

• The transaction on Node 2 returned the correct result now.
• But now it took 1 min 11 sec to return it (instead of 0sec as in the previous case). Basically, the transaction waited on the transaction from Node 1 to be applied on Node 2
• The execution time on Node 1 did not change.

This mode allowed us to do exactly what we wanted – prevent stale reads. Great outcome! So what about group_replication_consistency=’AFTER’ ?

group_replication_consistency=’AFTER’;

This mode says that a transaction on Node 1 will not return OK until it makes sure that other nodes applied the transaction. To see it in action, we will put Node 1 into group_replication_consistency=’AFTER’ and Node 2 into group_replication_consistency=’EVENTUAL’;.

Node 1 (in group_replication_consistency=’AFTER’):

UPDATE t1 SET cnt=13;
Query OK, 10485760 rows affected (3 min 0.46 sec)
Rows matched: 10485760  Changed: 10485760  Warnings: 0

After that, Node 2 (in group_replication_consistency=’EVENTUAL’):

select cnt from t1 where id=10001;
+------+
| cnt  |
+------+
|   13 |
+------+
1 row in set (0.00 sec)

Here the situation is different. Now the execution time on Node 1 doubled, as the transaction waits to be committed on all nodes and after that on Node 2 the execution is immediate.

This mode still avoids “stale” reads, but in this case, we shifted the wait time from Node 2 to Node 1, and this is how we can view the difference between ‘BEFORE’ and ‘AFTER’ consistency modes in Group Replication.

Both modes provide a consistent view, but:

• In ‘BEFORE’ mode: the readers on secondary nodes will be blocked, waiting on the moment when the consistent view is available, and
• In ‘AFTER’ mode the writers are blocked till other nodes get a consistent view.

So which mode to choose? Actually I think it is good to have an option here. You can choose, if you want, to put wait time on your readers or on your writers; the decision is up to how your application is designed.

We know the MySQL Shell is the advanced client tool for communicate to the MySQL server . MySQL Shell has lot of features like InnoDB Cluster control , InnoDB ReplicaSet, MySQL Shell utilities , MySQL server management etc … Today I came to know, MySQL shell helps lot in monitoring as well ( query, threads, resource consumption , locking ) .

In this blog I am going to explain how to use the MySQL Shell for monitor your server .

MySQL Shell provides two hot commands \show and \watch for monitor the MySQL server and report generating purpose .

\show : Execute the report with the provided options

\watch : Execute the report in loop with provided options

\show with thread example :

\show with query example :

You can execute any query within the double quotes .

\show with threads example :

As I showned in the screenshot there are two types in threads .

• –foreground
• –background

Similarly you can use the \watch command to execute the reports in loop .

All good, now I am going to show some examples,

1. How to find the top three MySQL threads which consuming more memory for the particular user ?

tid : thread id
cid : connection id
memory : the number of bytes allocated by the thread
started : time when thread started to be in its current state
user : the user who issued the statement, or NULL for a background thread

cmd : \show threads –foreground -o tid,cid,user,memory,started –order-by=memory –desc –where “user = ‘app_user'” –limit=3

2. How to find the blocking and blocked threads ?

Consider I started the below transaction in a terminal 1 but not committed ,

At terminal 2, I am trying to update the same value ,

root@localhost:sakthi>update sakthi_j set id=10;

now, lets execute the \show with the required options ,

tidle : the time the thread has been idle
nblocked : the number of other threads blocked by the thread
nblocking : the number of other threads blocking the thread

After commit the transaction there is no blocking transactions . Make sense ?

cmd : \show threads –foreground -o tid,cid,tidle,nblocked,nblocking,digest,digesttxt –where “nblocked=1 or nblocking=1”

3. How to find the top 10 threads, which used huge IO events ?

ioavgltncy : the average wait time per timed I/O event for the thread
ioltncy : the total wait time of timed I/O events for the thread
iomaxltncy : the maximum single wait time of timed I/O events for the thread
iominltncy : the minimum single wait time of timed I/O events for the thread
nio : the total number of I/O events for the thread

cmd : \show threads –foreground -o tid,ioavgltncy,ioltncy,iomaxltncy,iominltncy,nio –order-by=nio –desc  –limit=10

Like this way, you can find more details about query statistics , JOIN informations , system resource utilisation etc …

I hope this blog will helps someone who is looking MySQL Shell for effectively handle the MySQL server . Will come up with my next blog soon …

Thanks !!!

We will have a look at a new feature in MySQL 8.0 called binlog encryption. This feature is available from the MySQL version 8.0.14 or above.

Our previous blogs discussed about table space encryption in MySQL and Percona servers. In Mydbops, we are giving high importance about achieving security compliances.

The binary log records changes made to the databases so that it can be used to replicate the same to the slaves and also for the point in time recovery (PITR). So, it means that if someone has access to the binary logs, they can reproduce our entire database in many forms. As a DBA, we need to make sure that the binary log files are protected from users who are having access to the file system and also, log files need to be encrypted to follow the security compliance requirements by some clients.

These new features came as a rescue to satisfy those requirements. Now we will have a look at how to use and maintain it.

Note: To explore this feature, I have installed MySQL 8.0.18 in our testing environment.

Below are the topics that we are going to discuss in this blog:

1. Enabling Encryption
2. Rotating Key Manually
3. High Level Architecture
4. To access the encrypted binary log
5. To decrypt the encrypted binary log

Enabling Encryption:

Let’s start this by enabling the binlog encryption. Let us list down the current binary logs and their status. 

mysql> show binary logs;
+---------------+-----------+-----------+
| Log_name      | File_size | Encrypted |
+---------------+-----------+-----------+
| binlog.000001 |       474 | No        |
+---------------+-----------+-----------+
1 row in set (0.02 sec)

MySQL Server has one binary log and it is not encrypted it is not encrypted by default.

To proceed further, make sure you are having SUPER or BINLOG_ENCRYPTION_ADMIN privilege for your user to enable the encryption online. 

mysql> show global variables like 'binlog_encryption';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| binlog_encryption | OFF   |
+-------------------+-------+
1 row in set (0.00 sec)

mysql> set global binlog_encryption=on;
 Query OK, 0 rows affected (0.18 sec)

mysql> show global variables like 'binlog_encryption';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| binlog_encryption | ON    |
+-------------------+-------+
1 row in set (0.17 sec)

mysql> show binary logs;
+---------------+-----------+-----------+
| Log_name      | File_size | Encrypted |
+---------------+-----------+-----------+
| binlog.000001 |       497 | No        |
| binlog.000002 |       199 | No        |
| binlog.000003 |       667 | Yes       |
+---------------+-----------+-----------+
3 rows in set (0.03 sec)

Now, we can observe the latest binary log is encrypted. If the encrypted binary logs were the only one to be present in the server we can safely purge(remove) them. 

mysql> purge binary logs to 'binlog.000003';
Query OK, 0 rows affected (0.07 sec)

mysql> show binary logs;
+---------------+-----------+-----------+
| Log_name      | File_size | Encrypted |
+---------------+-----------+-----------+
| binlog.000003 |       667 | Yes       |
+---------------+-----------+-----------+
1 row in set (0.00 sec)

Currently, the server has encrypted binary logs. Also, we can observe the increase in the size of the encrypted binary log (empty) when compared with unencrypted ones. It is because of the 512 bytes encrypted header and the header is never replicated. 

Rotating Encryption Key:

The encryption of binary logs is made and it is time to rotate the encryption key. This must be done periodically to comply with security compliances. We can rotate the encryption key manually. 

mysql> alter instance rotate binlog master key;
Query OK, 0 rows affected (0.36 sec)

mysql> show binary logs;
+---------------+-----------+-----------+
| Log_name      | File_size | Encrypted |
+---------------+-----------+-----------+
| binlog.000003 |       711 | Yes       |
| binlog.000004 |       667 | Yes       |
+---------------+-----------+-----------+
2 rows in set (0.00 sec)

The below process is performed by the server while rotating the key.

1. The binary and relay log files are flushed (rotated).
2. A new binary log encryption key is generated with the next available sequence number, stored on the keyring, and used as the new binary log master key.
3. Binary log encryption keys that are no longer in use for any files after the re-encryption process are removed from the keyring.

Also, we are having one more variable binlog_rotate_encryption_master_key_at_startup which ensures the binary log encryption key has to be rotated at server startup.

High-Level Architecture:

Binlog Encryption is designed to use two-tier encryption. 

1. File Passwords
2. Replication Encryption key 

The File Password is used to encrypt/decrypt binary log content and Replication Encryption Key is used to encrypt/decrypt the file password in the encrypted binary log header.

A single replication encryption key may be used to encrypt/decrypt many binary and relay log files passwords, while a file password is intended to encrypt/decrypt a single binary or relay log file.

Multiple file passwords rely on the single encryption key whereas file password is used for the individual log files.

Okay, what happens if the encryption key REK1 is rotated?

The server generates a new replication encryption key REKj and it iterates over all encrypted log files to re-encrypt their password (Iterate from the last file to the first file ) by overwriting the encrypted file header with the new one. So, my new encryption key can decrypt all the available file passwords.

Compare the encrypted log file with unencrypted one:

The only difference is the binlog header. 

Here is the detailed header of the encrypted binlog:

Magic Number :

It is needed to distinguish from an encrypted to an unencrypted binary log files.

An encrypted binlog file has 0xFD62696E . An unencrypted binlog file : 0xFE62696E

Form an encrypted log file:

[root@testing mysql]# hexdump binlog.000003 -n 4 -e '/1 "%02X"'
FD62696E

Form an unencrypted log file:

[root@gr1 mysql]# hexdump binlog.000001 -n 4 -e '/1 "%02X"'
FE62696E

Replication logs encryption version :

The current value is 1. It might change in the future.

Replication encryption key ID: 

The key ID of the replication master key that encrypted the file password. It is a combination of MySQLReplicationKey, UUID, and SEQ_NO. It can be identified in the following format.

MySQLReplicationKey_{UUID}_{SEQ_NO}
Where MySQLReplicationKey is the prefix
      UUID is the MySQL server’s UUID that generated the key               SEQ_NO is the global replication master key sequence number.

Here is an example:

                                      |<-----Keyword------|<---------------UUID--------------->|SEQ_NO|                        
Keyring key ID for 'binlog.000003' is 'MySQLReplicationKey_d1deace2-24cc-11ea-a1db-0800270a0142_2'

Now Let us try rotating the encryption key.

mysql> alter instance rotate binlog master key;
Query OK, 0 rows affected (0.12 sec)

After successful rotation, the encryption key is 

Keyring key ID for 'binlog.000003' is 'MySQLReplicationKey_d1deace2-24cc-11ea-a1db-0800270a0142_3'

We can observe that My SEQ_NO is incremented by 1. It will keep going on all the encryption key rotations.

Encrypted File Password: 

It is used to generate the key for encrypting/decrypting the binary log content.

IV for encrypting file password:

         It is used with the replication master key to encrypt the file password.

Padding:

        Unused header space will be filled with 0.

How to access the encrypted binary log using mysqlbinlog utility ?

Sometimes, It is necessary to decode the binary log for PITR / to get to know the write pattern. Using mysqlbinlog, we can’t directly decode the encrypted binlog as it is not having access to the keyring file.

For example:

[root@testing mysql]# mysqlbinlog binlog.000003
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
ERROR: Reading encrypted log files directly is not supported.
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# End of log file
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
[root@testing mysql]#

But it is possible to get it by making a request to the MySQL server

[root@testing mysql]# mysqlbinlog -R --host localhost --user root binlog.000003
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#191222 15:28:57 server id 1  end_log_pos 124 CRC32 0xcb35f1c9  Start: binlog v 4, server v 8.0.18 created 191222 15:28:57
BINLOG '
uYv/XQ8BAAAAeAAAAHwAAAAAAAQAOC4wLjE4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAEwANAAgAAAAABAAEAAAAYAAEGggAAAAICAgCAAAACgoKKioAEjQA
CgHJ8TXL
'/*!*/;
# at 124
#191222 15:28:57 server id 1  end_log_pos 155 CRC32 0xb7909c71  Previous-GTIDs
# [empty]
# at 155
#191223  1:55:30 server id 1  end_log_pos 199 CRC32 0x46148c5f  Rotate to binlog.000004  pos: 4
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# End of log file
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

How to decrypt an encrypted binary log file?

The manual decryption of encrypted binary log is possible when the value of the key that encrypted its file password is known. We can use this amazing post https://mysql.wisborg.dk/2019/01/28/automatic-decryption-of-mysql-binary-logs-using-python/ to decrypt the binary log. To use this script, we need only have access to the keyring file.

Note: Even if we do not have enabled binary log on the server, we can still use it to encrypt relay log files on the slave servers. 

]

Introduction In this article, we are going to see how the SQL EXISTS operator works and when you should use it. Although the EXISTS operator has been available since SQL:86, the very first edition of the SQL Standard, I found that there are still many application developers who don’t realize how powerful SQL subquery expressions really are when it comes to filtering a given table based on a condition evaluated on a different table. Database table model Let’s assume we have the following two tables in our database, that form a one-to-many... Read More

The post SQL EXISTS and NOT EXISTS appeared first on Vlad Mihalcea.

As hinted at earlier this month, we’re happy to announce our latest on-demand webinar:
“An Introduction to Database Proxies (for MySQL)”

In this webinar, Gilles Rayrat, our VP of Engineering and database proxies guru, shares some of his knowledge on the world of database proxies, how they work, why they’re important and what to use them for.

Starting with a simple database connectivity scenario, Gilles builds up the content by discussing clustered databases and what happens in the case of a failure through to explaining the important role database proxies play; including a more in-depth look into some advanced database connectivity setups and proxies functionalities.

More specifically, Gilles covers the following:

• A simple database connectivity scenario
• The concept of a clustered database
• Failure in a clustered database: the nightmare scenario
• The solution: use a proxy! Preferably a smart one …
• Advanced database connectivity setups
• Advanced proxy functionalities
• Recap

Watch the database proxies webinar

While this webinar discusses database proxy concepts in general, we do call ourselves the MySQL Availability Company for a reason … and hence the content of this webinar is geared at MySQL (or MariaDB and Percona Server) predominantly.

This topic is not entirely innocent of course, as we’ve been involved with MySQL proxy technology for quite some time … namely, Tungsten Proxy, known to Continuent customers as Tungsten Connector. It is in fact a MySQL proxy; and at Continuent, we like to refer to it as an ‘Intelligent Proxy for MySQL’.

Tungsten Proxy – The Intelligent MySQL Proxy – in a nutshell

• Provides intelligent traffic routing to valid MySQL master(s), locally and globally
• Scales read queries to valid slaves via query inspection and other methods
• Application and active users do not disconnect during MySQL master failover events
• Combined with another intelligent layer of Tungsten Clustering called Tungsten Manager, it provides automatic, rapid master failover for MySQL High Availability and automated cross-site level failover for Disaster Recovery

Tungsten Connector (Tungsten Proxy) has been an important part of the Continuent Tungsten Clustering solution since 2006 and we’re gearing up to make it more widely known in the coming months.

Stay tuned! And watch our webinar on database proxies in the meantime!

MySQL 8.0.19 introduces Binlogless replicas with commit ordering which means you can deploy asynchronous replicas without binary logs enabled, and commit transactions in the same order they are replicated in. Yes, you can disable binlog (skip-log-bin) and the logging of changes done by the applier (log-slave-updates=FALSE) while at the same preserve commit order (slave-preserve-commit-order=TRUE).…