Quantcast
Channel: Planet MySQL
Viewing all 18842 articles
Browse latest View live
↧

S3 is not just for backups

February 9, 2017, 9:30 pm
$
0
0

For a long time, I treated S3 as a backup location for databases. After spending the last few months thinking about cloud architecture and designing Epsilon, I slowly learned that services like S3 can be a core storage component in a cloud architecture.

Here’s the gist of my previous thinking:

On a single instance of a database like MySQL, data are in two places: cached in the buffer pool in memory or on disk.

Buffer pool and disk

The disk is considered to be durable storage. For backups, depending on the strategy you use, you either end up with a single snapshot of the data or incremental pieces. Everything ends up going to S3 (or some equivalent) and stays untouched until you actually need to use the backups (maybe for disaster recovery).

Now, I believe that S3 can play a way more active role. What you can also do is add S3 as another storage tier.

Buffer pool, disk, and S3

Instead of treating S3 as some continuous backup destination, you can actually treat it like the system of record, i.e. what your storage system actually is. At that point, the disk starts to look like a cache, just like the buffer pool. But unlike the buffer pool, it’s a durable cache. But does it need to be?

At VividCortex, all of our “big” data are stored in Kafka before reaching MySQL. If we had continuous backups to S3, we actually wouldn’t require MySQL to be fully durable. There would be data that are in MySQL that haven’t reached S3 yet, but those are already durable in Kafka.

In MySQL terms, a MySQL instance just holds dirty data before checkpointing to S3. If we have a crash or instance failure, we recover using the Kafka log. Just like how pages can be read into the buffer pool on-demand from the disk, you can do the same using S3.

Redshift apparently does that already:

With this approach, you can scale compute independently of storage, which I think is a huge win.

If you want to learn more, I suggest watching Netflix’s talk from re:Invent 2016:

“Using Amazon S3 as the fabric of our big data ecosystem.”

YouTube: https://www.youtube.com/watch?v=o52vMQ4Ey9I

Slides:

↧
Search

FromDual.en: FromDual Performance Monitor for MySQL and MariaDB 1.0.1 has been released

February 10, 2017, 6:13 am
$
0
0

FromDual has the pleasure to announce the release of the new version 1.0.1 of its popular Database Performance Monitor for MySQL, MariaDB, Galera Cluster and Percona Server fpmmm.

You can download fpmmm from here.

In the inconceivable case that you find a bug in the fpmmm please report it to our Bug-tracker.

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

This release contains various bug fixes.

Changes in fpmmm v1.0.1

fpmmm agent

  • Fpmmm suppresses server has gone away message to stdout.
  • Fpmmm should behave correctly now when database was started after fpmmm agent.
  • MyEnv library synced from MyEnv project.
  • Added LaunchDaemon configuration for fpmmmm on Max OSX (darwin).

fpmmm agent installer

  • Added some more support for Mac OSX (darwin).

For subscriptions of commercial use of fpmmm please get in contact with us.

Taxonomy upgrade extras: 
mysql
performance
monitor
monitoring
fpmmm
maas
performance monitor
mpm
release
↧
↧

Treating NULLs as not less than zero in ORDER BY Revisited

February 10, 2017, 9:09 am
$
0
0

In my post yesterday, I shared a little known trick for sorting NULLs last when using ORDER BY ASC.

To summarize briefly, NULLs are treated as less than 0 when used in ORDER BY, However, sometimes you do not want that behavior, and you need the NULLs listed last, even though you want your numbers in ascending order.

So a query like the following returns the NULLs first (expected behavior):

SELECT * FROM t1 ORDER BY col1 ASC, col2 ASC;
+--------+------+
| col1   | col2 |
+--------+------+
| apple  | NULL |
| apple  |    5 |
| apple  |   10 |
| banana | NULL |
| banana |    5 |
| banana |   10 |
+--------+------+

The trick I mentioned in my post is to rewrite the query like:

SELECT * FROM t1 ORDER BY col1 ASC, -col2 DESC;

The difference is that we added a minus sign (-) in front of the column we want sorted and we change ASC to DESC.

Now this query returns what we’re looking for:

+--------+------+
| col1   | col2 |
+--------+------+
| apple  |    5 |
| apple  |   10 |
| apple  | NULL |
| banana |    5 |
| banana |   10 |
| banana | NULL |
+--------+------+

I could not really find this behavior documented at the time, and thus did some more digging to find out if this is intended behavior and if it should continue to work in the future (i.e., can we rely on it).

The answer is yes to both, it is intended, and it will continue to work.

Now, why does this work this way?

  • It is known that sorting in ascending (ASC) order NULLs are listed first, and if descending (DESC) they are listed last.
  • It is known that minus sign (-) preceding the column followed by DESC is the same as ASC (essentially the inverse). This is because if a > b, then -a
  • And the last bit of the puzzle is that NULL == -NULL.

So since -NULL == NULL, and we are now using DESC, the NULLs will be last. And then the remainder of the INT values will be in ASC order, since we effectively converted them to negative values and the changed order to DESC, effectively putting those INTs in ASC order.

Hope this helps clarify for anyone out there interested.

↧

An Introduction to MariaDB’s Data at Rest Encryption (DARE) – Part 1

February 10, 2017, 10:56 am
$
0
0

Encryption is becoming more and more prevalent and increasingly necessary in today’s world, so I wanted to provide a good overall “getting started” article on using MariaDB’s data at rest encryption (DARE) for anyone out there interested in setting this up in their environment.

MariaDB’s data encryption at rest manual page covers a lot of the specifics, but I wanted to create a quick start guide and also note a few items that might not be immediately obvious.

And due to the number of my examples, I’m splitting this into two posts. The first will focus solely on setting up encryption so you can use it. The second will focus on using it with a number of examples and common use cases.

Also, I feel that I should mention from the outset that, currently, this data at rest encryption only applies to InnoDB/XtraDB tables and Aria tables (that are created with ROW_FORMAT=PAGE, the default), and other limitations are listed on the above page.

1. First off, you will need to create a keys.txt file. The example on the above page works fine for this, but you may want to eventually create your own:

1;770A8A65DA156D24EE2A093277530142
18;F5502320F8429037B8DAEF761B189D12F5502320F8429037B8DAEF761B189D12

Save that as keys.txt (or whatever you want) and I’ll place it in the datadir for the sake of this example.

The first number is the encryption key identifier (a 32-bit number), and the latter the actual hex-encoded encryption key (which can be 128, 192, or 256-bit). The two values are to be separated by a semi-colon. You only need the first key (i.e., #1), but you can add as many keys as you like, and use various keys for various tables if you desire. The number 1 key is also the default.

2. Add the following lines to your configuration file (these are the basics from the manual page above):

plugin-load-add=file_key_management.dll
file-key-management
file-key-management-filename = "C:/Program Files/MySQL/mariadb-10.1.21/data/keys.txt"
innodb-encrypt-tables
innodb-encrypt-log
innodb-encryption-threads=4

Of course if on Linux, change the “.dll” to “.so”. Or you can always load it dynamically, but when you also need to set related variables, I prefer this way.

  • The first three lines load the key_management plugin, enable file-key-management, and point to the file-key file you created.
  • The fourth is what enables encryption. You can set to 0 (off), 1 (on), or FORCE (to always force every single table to be encrypted).
  • The fifth tells the server to enable encryption for the log files. From what I understand, this is for the InnoDB log files. There is a separate option to encrypt binary logs (- -encrypt-binlog). The manual recommends to enable this if you are encrypting tables since the data in the logs would not be encrypted otherwise, and thus viewable.
  • The sixth specifies how many background encryption threads to startup and use. These threads perform background key rotation and scrubbing. Note that this will add some overhead, and these background threads can cause higher CPU utilization since they are running and checking tables all of the time. More on this topic can be found here: https://jira.mariadb.org/browse/MDEV-11581

3. Restart your instance, and check that all looks normal.

Verify that the plugin loaded properly:

MariaDB> SHOW PLUGINS;
+-------------------------------+----------+--------------------+-------------------------+---------+
| Name                          | Status   | Type               | Library                 | License |
+-------------------------------+----------+--------------------+-------------------------+---------+
...
| file_key_management           | ACTIVE   | ENCRYPTION         | file_key_management.dll | GPL     |
+-------------------------------+----------+--------------------+-------------------------+---------+

And if using the above, you will see the 4 encryption threads start up in the error log:

2017-02-08 16:41:44 55840 [Note] InnoDB: Creating #1 thread id 33908 total threads 4.
2017-02-08 16:41:44 55840 [Note] InnoDB: Creating #2 thread id 33924 total threads 4.
2017-02-08 16:41:44 55840 [Note] InnoDB: Creating #3 thread id 38532 total threads 4.
2017-02-08 16:41:44 55840 [Note] InnoDB: Creating #4 thread id 29060 total threads 4.

4. Now you are ready to create an encrypted table. More on this in part #2 of this blog post.

Click here to read An Introduction to MariaDB’s Data at Rest Encryption (DARE) – Part 2.

Hope this helps.

↧

An Introduction to MariaDB’s Data at Rest Encryption (DARE) – Part 2

February 10, 2017, 10:57 am
$
0
0

Okay, so you’ve read the first post on enabling MariaDB’s data at rest encryption, and now you are ready to create an encrypted table.

And just to get it out of the way for those interested, you can always check your encrypted (and non-encrypted) table stats via:

SELECT * FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION;

ENCRYPTION_SCHEME=1 means the table is encrypted and ENCRYPTION_SCHEME=0 means they are not.

But let’s get into some specific examples.

I find the following 4 tables interesting, as the first 3 essentially all create the same table, and the 4th shows how to create a non-encrypted table once you have encryption enabled.

CREATE TABLE t10 (id int) ENGINE=INNODB;
CREATE TABLE t11 (id int) ENGINE=INNODB ENCRYPTED=YES;
CREATE TABLE t12 (id int) ENGINE=INNODB ENCRYPTED=YES ENCRYPTION_KEY_ID=18;
CREATE TABLE t13 (id int) ENGINE=INNODB ENCRYPTED=NO;
MariaDB> SELECT * FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE NAME LIKE 'dare/t1%';
+-------+----------+-------------------+--------------------+-----------------+---------------------+--------------------------+------------------------------+----------------+
| SPACE | NAME     | ENCRYPTION_SCHEME | KEYSERVER_REQUESTS | MIN_KEY_VERSION | CURRENT_KEY_VERSION | KEY_ROTATION_PAGE_NUMBER | KEY_ROTATION_MAX_PAGE_NUMBER | CURRENT_KEY_ID |
+-------+----------+-------------------+--------------------+-----------------+---------------------+--------------------------+------------------------------+----------------+
|    48 | dare/t10 |                 1 |                  1 |               1 |                   1 |                     NULL |                         NULL |              1 |
|    49 | dare/t11 |                 1 |                  1 |               1 |                   1 |                     NULL |                         NULL |              1 |
|    50 | dare/t13 |                 0 |                  0 |               0 |                   1 |                     NULL |                         NULL |              1 |
|    51 | dare/t12 |                 1 |                  1 |               1 |                   1 |                     NULL |                         NULL |             18 |
+-------+----------+-------------------+--------------------+-----------------+---------------------+--------------------------+------------------------------+----------------+

So when configuered as above, then we see t10, t11, and t12 were all encrypted no matter whether we specified ENCRYPTED=YES in the CREATE TABLE. We also see t10 and t11 used KEY_ID #1 since we did not specify ENCRYPTION_KEY_ID in the CREATE TABLE, whereas t12 did use KEY_ID #18 since we specified ENCRYPTION_KEY_ID=18 in the CREATE TABLE.

We see that t13 was not encrypted, since we did specify ENCRYPTED=NO in the CREATE TABLE. This is possible when you use innodb_file_per_table, the default.

If we attempt to create a table specifying ENCRYPTION_KEY_ID to a value that is not in the keys.txt file, then you’ll see an error like the following:

MariaDB> CREATE TABLE t15 (id int) ENGINE=INNODB ENCRYPTED=YES ENCRYPTION_KEY_ID=17;
ERROR 1005 (HY000): Can't create table `dare`.`t15` (errno: 140 "Wrong create options")

You will see the same if you try to ALTER an existing table from one ENCRYPTION_KEY_ID to another that does not exist.

Examples with above settings:

Let’s encrypt a non-encrypted table (t13) to be encrypted:

MariaDB> SELECT NAME, ENCRYPTION_SCHEME FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE NAME LIKE 'dare/t13';
+----------+-------------------+
| NAME     | ENCRYPTION_SCHEME |
+----------+-------------------+
| dare/t13 |                 0 |
+----------+-------------------+

MariaDB> ALTER TABLE t13 ENCRYPTED=YES;
Query OK, 0 rows affected (0.16 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB> SELECT NAME, ENCRYPTION_SCHEME FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE NAME LIKE 'dare/t13';
+----------+-------------------+
| NAME     | ENCRYPTION_SCHEME |
+----------+-------------------+
| dare/t13 |                 1 |
+----------+-------------------+

Now let’s perform the reverse, un-encrpyt an encrypted table:

MariaDB> SELECT NAME, ENCRYPTION_SCHEME FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE NAME LIKE 'dare/t13';
+----------+-------------------+
| NAME     | ENCRYPTION_SCHEME |
+----------+-------------------+
| dare/t13 |                 1 |
+----------+-------------------+

MariaDB> ALTER TABLE t13 ENCRYPTED=NO;
Query OK, 0 rows affected (0.19 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB> SELECT NAME, ENCRYPTION_SCHEME FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE NAME LIKE 'dare/t13';
+----------+-------------------+
| NAME     | ENCRYPTION_SCHEME |
+----------+-------------------+
| dare/t13 |                 0 |
+----------+-------------------+

Let’s convert an ENCRYPTION_KEY_ID:

MariaDB> SELECT NAME, CURRENT_KEY_ID FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE NAME LIKE 'dare/t11';
+----------+----------------+
| NAME     | CURRENT_KEY_ID |
+----------+----------------+
| dare/t11 |              1 |
+----------+----------------+

MariaDB [dare]> ALTER TABLE t11 ENCRYPTION_KEY_ID=18;
Query OK, 0 rows affected (0.13 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB> SELECT NAME, CURRENT_KEY_ID FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE NAME LIKE 'dare/t11';
+----------+----------------+
| NAME     | CURRENT_KEY_ID |
+----------+----------------+
| dare/t11 |             18 |
+----------+----------------+

Now change innodb-encrypt-tables=FORCE:

With this setting, if you attempt to create a table with ENCRYPTED=NO, then the command will fail:

MariaDB [dare]> CREATE TABLE t23 (id int) ENGINE=INNODB ENCRYPTED=NO;
ERROR 1005 (HY000): Can't create table `dare`.`t23` (errno: 140 "Wrong create options")

Similarly, if you attempt to change an encrypted table to a non-encrypted table with FORCE in effect, you’ll see an error like the below:

MariaDB> CREATE TABLE t20 (id int) ENGINE=INNODB;
Query OK, 0 rows affected (0.04 sec)

MariaDB> ALTER TABLE t20 ENCRYPTED=NO;
ERROR 1005 (HY000): Can't create table `dare`.`#sql-29b4_2` (errno: 140 "Wrong create options")

Now revert the FORCE back to 1 for innodb-encrypt-tables, and drop innodb-encryption-threads from 4 to 0 because of high CPU usage issue – if you need to avoid it (and also revert innodb-encrypt-tables back to 1 instead of FORCE)

MariaDB> CREATE TABLE t30 (id int) ENGINE=INNODB;
Query OK, 0 rows affected (0.14 sec)

MariaDB> SELECT * FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE NAME LIKE 'dare/t3%';
+-------+----------+-------------------+--------------------+-----------------+---------------------+--------------------------+------------------------------+----------------+
| SPACE | NAME     | ENCRYPTION_SCHEME | KEYSERVER_REQUESTS | MIN_KEY_VERSION | CURRENT_KEY_VERSION | KEY_ROTATION_PAGE_NUMBER | KEY_ROTATION_MAX_PAGE_NUMBER | CURRENT_KEY_ID |
+-------+----------+-------------------+--------------------+-----------------+---------------------+--------------------------+------------------------------+----------------+
|    63 | dare/t30 |                 1 |                  1 |               1 |                   1 |                     NULL |                         NULL |              1 |
+-------+----------+-------------------+--------------------+-----------------+---------------------+--------------------------+------------------------------+----------------+

I did this test since it could be necessary that to set innodb-encryption-threads=0 if the high CPU utilzation issue requires you to do so, and you do not need the background threads to perform key rotation (discussed more on the manual page listed above).

Also, I did this because I’ve heard that if you set innodb-encryption-threads=0, then newly created tables where you do not explicitly set ENCRYPTED=YES will not be encrypted. I do not see that behavior, as you can see above that the table is encrypted. So I do want to dispell this notion whilst I’m at it.

Misc:

I did notice that one you set an ENCRYPTION_KEY_ID for a table, it remains with that CREATE TABLE output forever, even if you un-encrypt the table. Even an null ALTER TABLE does not remove it. It is harmless, but soemthing to be aware of:

MariaDB> ALTER TABLE t3 ENCRYPTED=NO;
Query OK, 0 rows affected, 1 warning (0.05 sec)
Records: 0  Duplicates: 0  Warnings: 1

MariaDB> SHOW WARNINGS;
+---------+------+------------------------------------------------------------------+
| Level   | Code | Message                                                          |
+---------+------+------------------------------------------------------------------+
| Warning |  140 | InnoDB: Ignored ENCRYPTION_KEY_ID 18 when encryption is disabled |
+---------+------+------------------------------------------------------------------+

MariaDB> ALTER TABLE t3 ENGINE=INNODB;
Query OK, 0 rows affected, 1 warning (0.30 sec)
Records: 0  Duplicates: 0  Warnings: 1

MariaDB> SHOW WARNINGS;
+---------+------+------------------------------------------------------------------+
| Level   | Code | Message                                                          |
+---------+------+------------------------------------------------------------------+
| Warning |  140 | InnoDB: Ignored ENCRYPTION_KEY_ID 18 when encryption is disabled |
+---------+------+------------------------------------------------------------------+

MariaDB> SHOW CREATE TABLE t3\G
*************************** 1. row ***************************
       Table: t3
Create Table: CREATE TABLE `t3` (
  `id` int(11) DEFAULT NULL,
  `value` varchar(255) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1 `encryption_key_id`=18 `encrypted`=NO

Lastly, I should mention that if you want to export a certain table to another instance, if it is encrypted, it will not work as-is. You will need to un-encrypt the table first, then perform the discard/import tablespace on the destination server, and then re-encrypt the table on the source server.

I hope this has helped covered the basics and a number of use cases to help you get your data at rest encrypted.

↧
↧

With 500+ VividCortex Users, Shopify Eliminates High Latency Queries From Reddit and MySQL

February 10, 2017, 12:42 pm
$
0
0

As intuitive and streamlined as ecommerce technology might seem from the user's perspective, it involves so much data that engineering ingenuity and smart database management must constantly deliver in order to keep up. At organizations like Shopify—responsible for the easy and reliable transactions at top brands around the world—that excellence of performance involves deep monitoring of their MySQL core and their Redis caching infrastructure, plus insightful query profiling, packet captures, and the admittance of developers to platforms that measure database performance.

shopify-logo.jpg

Shopify’s motto is “Make commerce better for everyone.” That mantra applies whether the shopping's done online, on mobile, or in-store. For Shopify's engineering team, better means a fast, reliable application that delivers a positive customer experience, particularly during periods of high traffic volume. Shopify is a top ecommerce solution provider, delivering online and point-of-sale services for over 325,000 businesses in 150 countries, from your local coffee shop to trusted brands such as Tesla Motors, Red Bull, Nestlé, GE, and Kylie Cosmetics.

Shipping Code Fast and Often

For Shopify’s developers and DBAs, working in a high-availability environment means delivering at high velocity. It’s not unusual for them to deploy new software 20 times per day, and during peak periods—such as Black Friday and Cyber Monday—they have deployed new code over 80 times in a single day.

Sergio Roysen, Shopify's MySQL DBA Team lead has told us that "Shopify deploys new software twenty times a day, sometimes. After Black Friday, after the code freeze, we deployed eighty-six times in the same day. Each of these new deploys can introduce new queries. Tracking those queries without VividCortex would take 24 hours. It could be too late. We could be down for 24 hours until we come up with a solution. Real-time monitoring is the only way to keep our wheels running."

You can find a four-minute video case study here, from a recent trip we made to Shopify's offices; it includes highlights from our interviews with DBAs, developers, and engineers. Below is a summary of the key monitoring methods and practices they shared with us.

Monitoring What Matters

The Shopify team maintain a laser focus on anything that gets in the way of performance. Their approach demands extreme precision and, for that precision, clear visibility into how the system behaves. Here’s how:

  1. Tracking fast query effects. One part of Shopify’s existing monitoring process included maintaining a digest of slow queries. The problem was obtaining visibility into high-frequency, really fast queries. Though each query executed in just microseconds--and low latency is great--there were potentially millions firing every hour, which all together consumed significant time in the application, plus potentially made it hard to see if there was any issue within one of these fast queries. Once Shopify started using VividCortex, they were able to zoom to one-second granularity, which allowed them to spot and cache every query, pinpointing any that were causing performance issues.  

  2.  Getting rid of bad queries. The team constantly look for patterns that could reveal issues, such as a particular shop generating an inordinate number of queries. By profiling queries, they can quickly find those that are causing a problem on a certain database. Once discovered, they can either get rid of these queries entirely, or fix those that may have been hung up by using a bad index or no index at all. This continual finding, fixing, and culling process helps purge their application of performance issues.

  3.  Working from packet captures. During a recent period of high volume, the team deployed new Redis capacity. They were seeing spikes in latency and needed to confirm that they had provisioned enough resources to match the heavy traffic. Because VividCortex allowed them to monitor packet captures— and not just queries—they could see the latency increases introduced by the new connections caused by the Redis deploy. In the past, it was difficult to gain visibility into this dimension, as traditional database monitoring solutions don't enable engineering teams to see packet traffic. However, when given the right monitoring tool, they were able to see the exact metrics they knew they should look for and monitor, throughout the traffic surge.

  4.  Developer testing of queries. Developers across different teams in the organization are responsible for not only writing new queries, but performance testing them as well. They use performance monitoring to evaluate poor-performing queries and confirm that they have either been fixed or removed. By giving developers access to monitoring tools, Shopify eliminates the need for complicated back-and-forths between devs and DBAs, to ensure that code pushes won't have a detrimental effect in production. Administrative functions such as role-based and team-based access control make it possible for even non-DBA engineers to navigate sophisticated monitoring platforms without risk. Shopify has over 500 users on their VividCortex install, benefitting from SAML integration and Okta single sign-on. That means users of many different specialties and approaches can have an input on Shopify's monitoring.

Eliminating the Guesswork

With hundreds of instances to monitor and dozens of developers, DBAs, and engineers on the team, seamless coordination on issue resolution is a real challenge. Prior to their experience with VividCortex, when a server started to slow down, Shopify's DBAs and developers were forced to jump into their internal chat room, to discuss the cause of the problem. Without hard hard performance data, issues could evade detection and stay buried.

Today, when an issue arises, the team still jumps into the chat room, but they're now armed with hard evidence — screenshots of performance charts they can share with their colleagues, and Shopify thrives with this data-driven analysis. Working together, the team members move fast and ship often, now able to deliver the company's established high-performance standards to even the deepest tiers of the system.

↧

Roland Bouman's blog: Need a Mondrian .WAR? Check out XMondrian.

February 11, 2017, 2:39 am
$
0
0
Roland Bouman's blog: Need a Mondrian .WAR? Check out XMondrian.: Error: The Mondrian XML: Mondrian Error:Internal error: No suitable connection found (00HSBE02)

Hi Prabu,



this:



 Error: The Mondrian XML: Mondrian Error:Internal error: No suitable connection found (00HSBE02)



Usually means the jar containing the JDBC driver is not in your classpath. Put the mysql-connector .jar file inside the lib folder of tomcat, or inside the lib folder of the mondrian web application.
↧

MySQL in the Cloud - Pros and Cons of Amazon RDS

February 13, 2017, 6:55 am
$
0
0
Related resources
 Infrastructure Automation - Deploying ClusterControl and MySQL-based systems on AWS using Ansible
 Press Release: Severalnines and WooServers bring ClusterControl to web hosting

Moving your data into a public cloud service is a big decision. All the major cloud vendors offer cloud database services, with Amazon RDS for MySQL being probably the most popular.

In this blog, we’ll have a close look at what it is, how it works, and compare its pros and cons.

RDS (Relational Database Service) is an Amazon Web Services offering. In short, it is a Database as a Service, where Amazon deploys and operates your database. It takes care of tasks like backup and patching the database software, as well as high availability. A few databases are supported by RDS, we are here mainly interested in MySQL though - Amazon supports MySQL and MariaDB. There is also Aurora, which is Amazon’s clone of MySQL, improved, especially in area of replication and high availability.

Deploying MySQL via RDS

Let’s take a look at the deployment of MySQL via RDS. We picked MySQL and then we are presented with couple of deployment patterns to pick from.

Main choice is - do we want to have high availability or not? Aurora is also promoted.

Next dialog box gives us some options to customize. You can pick one of many MySQL versions - several 5.5, 5.6 and 5.7 versions are available. Database instance - you can choose from typical instance sizes available in a given region.

Next option is a pretty important choice - do you want to use multi-AZ deployment or not? This is all about high availability. If you don’t want to use multi-AZ deployment, a single instance will be installed. In case of failure, a new one will be spun up and its data volume will be remounted to it. This process takes some time, during which your database will not be available. Of course, you can minimize this impact by using slaves and promoting one of them, but it’s not an automated process. If you want to have automated high availability, you should use multi-AZ deployment. What will happen is that two database instances will be created. One is visible to you. A second instance, in a separate availability zone, is not visible to the user. It will act as a shadow copy, ready to take over the traffic once the active node fails. It is still not a perfect solution as traffic has to be switched from the failed instance to the shadow one. In our tests, it took ~45s to perform a failover but, obviously, it may depend on instance size, I/O performance etc. But it’s much better than non-automated failover where only slaves are involved.

Finally, we have storage settings - type, size, PIOPS (where applicable) and database settings - identifier, user and password.

In the next step, a few more options are waiting for user input.

We can choose where the instance should be created: VPC, subnet, should it be publicly available or not (as in - should a public IP be assigned to the RDS instance), availability zone and VPC Security Group. Then, we have database options: first schema to be created, port, parameter and option groups, whether metadata tags should be included in snapshots or not, encryption settings.

Next, backup options - how long do you want to keep your backups? When would you like them taken? Similar setup is related to maintenances - sometimes Amazon administrators have to perform maintenance on your RDS instance - it will happen within a predefined window which you can set here. Please note, there is no option not to pick at least 30 minutes for the maintenance window, that’s why having multi-AZ instance on production is really important. Maintenance may result in node restart or lack of availability for some time. Without multi-AZ, you need to accept that downtime. With multi-AZ deployment, failover happens.

Finally, we have settings related to additional monitoring - do we want to have it enabled or not?

Managing RDS

In this chapter we will take a closer look at how to manage MySQL RDS. We will not go through every option available out there, but we’d like to highlight some of the features Amazon made available.

Snapshots

MySQL RDS uses EBS volumes as storage, so it can use EBS snapshots for different purposes. Backups, slaves - all based on snapshots. You can create snapshots manually or they can be taken automatically, when such need arises. It is important to keep in mind that EBS snapshots, in general (not only on RDS instances), adds some overhead to I/O operations. If you want to take a snapshot, expect your I/O performance to drop. Unless you use multi-AZ deployment, that is. In such case, the “shadow” instance will be used as a source of snapshots and no impact will be visible on the production instance.

Backups

Backups are based on snapshots. As mentioned above, you can define your backup schedule and retention when you create a new instance. Of course, you can edit those settings afterwards, through the “modify instance” option.

At any time you can restore a snapshot - you need to go to the snapshot section, pick the snapshot you want to restore, and you will be presented with a dialog similar to the one you’ve seen when you created a new instance. This is not a surprise as you can only restore a snapshot into a new instance - there is no way to restore it on one of the existing RDS instances. It may come as a surprise, but even in cloud environment, it may make sense to reuse hardware (and instances you already have). In a shared environment, performance of a single virtual instance may differ - you may prefer to stick to the performance profile that you are already familiar with. Unfortunately, it’s not possible in RDS.

Another option in RDS is point-in-time recovery - very important feature, a requirement for anyone who need to take good care of her data. Here things are more complex and less bright. For starters, it’s important to keep in mind that MySQL RDS hides binary logs from the user. You can change a couple of settings and list created binlogs, but you don’t have direct access to them - to make any operation, including using them for recovery, you can only use the UI or CLI. This limits your options to what Amazon allows you to do, and it allows you to restore your backup up to the latest “restorable time” which happens to be calculated in 5 minutes interval. So, if your data has been removed at 9:33a, you can restore it only up to the state at 9:30a. Point-in-time recovery works the same way as restoring snapshots - a new instance is created.

Scale-out, replication

MySQL RDS allows scale-out through adding new slaves. When a slave is created, a snapshot of the master is taken and it is used to create a new host. This part works pretty well. Unfortunately, you cannot create any more complex replication topology like one involving intermediate masters. You are not able to create a master - master setup, which leaves any HA in the hands of Amazon (and multi-AZ deployments). From what we can tell, there is no way to enable GTID (not that you could benefit from it as you don’t have any control over the replication, no CHANGE MASTER in RDS), only regular, old-fashioned binlog positions.

Lack of GTID makes it not feasible to use multithreaded replication - while it is possible to set a number of workers using RDS parameter groups, without GTID this is unusable. Main issue is that there is no way to locate a single binary log position in case of a crash - some workers could have been behind, some could be more advanced. If you use the latest applied event, you’ll lose data that is not yet applied by those “lagging” workers. If you will use the oldest event, you’ll most likely end up with “duplicate key” errors caused by events applied by those workers which are more advanced. Of course, there is a way to solve this problem but it is not trivial and it is time-consuming - definitely not something you could easily automate.

Users created on MySQL RDS don’t have SUPER privilege so operations, which are simple in stand-alone MySQL, are not trivial in RDS. Amazon decided to use stored procedures to empower the user to do some of those operations. From what we can tell, a number of potential issues are covered although it hasn’t always been the case - we remember when you couldn’t rotate to the next binary log on the master. A master crash + binlog corruption could render all slaves broken - now there is a procedure for that: rds_next_master_log.

A slave can be manually promoted to a master. This would allow you to create some sort of HA on top of multi-AZ mechanism (or bypassing it) but it has been made pointless by the fact that you cannot reslave any of existing slaves to the new master. Remember, you don’t have any control over the replication. This makes the whole exercise futile - unless your master can accommodate all of your traffic. After promoting a new master, you are not able to failover to it because it does not have any slaves to handle your load. Spinning up new slaves will take time as EBS snapshots have to be created first and this may take hours. Then, you need to warm up the infrastructure before you can put load on it.

Lack of SUPER privilege

As we stated earlier, RDS does not grant users SUPER privilege and this becomes annoying for someone who is used to having it on MySQL. Take it for granted that, in the first weeks, you will learn how often it is required to do things that you do rather frequently - such as killing queries or operating the performance schema. In RDS, you will have to stick to predefined list of stored procedures and use them instead of doing things directly. You can list all of them using the following query:

SELECT specific_name FROM information_schema.routines;

As with replication, a number of tasks are covered but if you ended up in a situation which is not yet covered, then you’re out of luck.

Interoperability and Hybrid Cloud Setups

This is another area where RDS is lacking flexibility. Let’s say you want to build a mixed cloud/on-premises setup - you have a RDS infrastructure and you’d like to create a couple of slaves on premises. The main problem you’ll be facing is that there is no way to move data out of RDS except to take a logical dump. You can take snapshots of RDS data but you don’t have access to them and you cannot move them away from AWS. You also don’t have physical access to the instance to use xtrabackup, rsync or even cp. The only option for you is to use mysqldump, mydumper or similar tools. This adds complexity (character set and collation settings have a potential to cause problems) and is time-consuming (it takes long time to dump and load data using logical backup tools).

It is possible to setup replication between RDS and an external instance (in both ways, so migrating data into RDS is also possible), but it can be a very time-consuming process.

On the other hand, if you want to stay within an RDS environment and span your infrastructure across the atlantic or from east to west coast US, RDS allows you to do that - you can easily pick a region when you create a new slave.

Unfortunately, if you’d like to move your master from one region to the other, this is virtually not possible without downtime - unless your single node can handle all of your traffic.

Security

While MySQL RDS is a managed service, not every aspect related to security is taken care of by Amazon’s engineers. Amazon calls it “Shared Responsibility Model”. In short, Amazon takes care of the security of the network and storage layer (so that data is transferred in a secure way), operating system (patches, security fixes). On the other hand, user has to take care of the rest of the security model. Make sure traffic to and from RDS instance is limited within VPC, ensure that database level authentication is done right (no password-less MySQL user accounts), verify that API security is ensured (AMI’s are set correctly and with minimal required privileges). User should also take care of firewall settings (security groups) to minimize exposure of RDS and the VPC it’s in to external networks. It’s also the user’s responsibility to implement data at rest encryption - either on the application level or on the database level, by creating an encrypted RDS instance in the first place.

Database level encryption can be enabled only on the instance creation, you cannot encrypt an existing, already running database.

RDS limitations

If you plan to use RDS or if you are already using it, you need to be aware of limitations that come with MySQL RDS.

Lack of SUPER privilege can be, as we mentioned, very annoying. While stored procedures take care of a number of operations, it is a learning curve as you need to learn to do things in a different way. Lack of SUPER privilege can also create problems in using external monitoring and trending tools - there are still some tools which may require this priviledge for some part of its functionality.

Lack of direct access to MySQL data directory and logs makes it harder to perform actions which involves them. It happens every now and then that a DBA needs to parse binary logs or tail error, slow query or general log. While it is possible to access those logs on RDS, it is more cumbersome than doing whatever you need by logging into shell on the MySQL host. Downloading them locally also takes some time and adds additional latency to whatever you do.

Lack of control over replication topology, high availability only in multi-AZ deployments. Given that you don’t have a control over the replication, you cannot implement any kind of high availability mechanism into your database layer. It doesn’t matter that you have several slaves, you cannot use some of them as master candidates because even if you promote a slave to a master, there is no way to reslave the remaining slaves off this new master. This forces users to use multi-AZ deployments and increase costs (the “shadow” instance doesn’t come free, user has to pay for it).

Reduced availability through planned downtime. When deploying an RDS instance, you are forced to pick a weekly time window of 30 minutes during which maintenance operations may be executed on your RDS instance. On the one hand, this is understandable as RDS is a Database as a Service so hardware and software upgrades of your RDS instances are managed by AWS engineers. On the other hand, this reduce your availability because you cannot prevent your master database from going down for the duration of the maintenance period. Again, in this case using multi-AZ setup increases availability as changes happen first on the shadow instance and then failover is executed. Failover itself, though, is not transparent so, one way or the other, you lose the uptime. This forces you to design your app with unexpected MySQL master failures in mind. Not that it’s a bad design pattern - databases can crash at any time and your application should be built in a way it can withstand even the most dire scenario. It’s just that with RDS, you have limited options for high availability.

Reduced options for high availability implementation. Given the lack of flexibility in the replication topology management, the only feasible high availability method is multi-AZ deployment. This method is good but there are tools for MySQL replication which would minimize the downtime even further. For example, MHA or ClusterControl when used in connection with ProxySQL can deliver (under some conditions like lack of long running transactions) transparent failover process for the application. While on RDS, you won’t be able to use this method.

Reduced insight into performance of your database. While you can get metrics from MySQL itself, sometimes it’s just not enough to get a full 10k feet view of the situation. At some point, the majority of users will have to deal with really weird issues caused by faulty hardware or faulty infrastructure - lost network packets, abruptly terminated connections or unexpectedly high CPU utilization. When you have an access to your MySQL host, you can leverage lots of tools that help you to diagnose state of a Linux server. When using RDS, you are limited to what metrics are available in Cloudwatch, Amazon’s monitoring and trending tool. Any more detailed diagnosis require contacting support and asking them to check and fix the problem. This may be quick but it also can be a very long process with a lot of back and forth email communication.

Vendor lock-in caused by complex and time-consuming process of getting data out of the MySQL RDS. RDS doesn’t grant access to MySQL data directory so there is no way to utilize industry standard tools like xtrabackup to move data in a binary way. On the other hand, the RDS under the hood is a MySQL maintained by Amazon, it is hard to tell if it is 100% compatible with upstream or not. RDS is only available on AWS, so you would not be able to do a hybrid setup.

Summary

MySQL RDS has both strengths and weaknesses. This is a very good tool for those who’d like to focus on the application without having to worry about operating the database. You deploy a database and start issuing queries. No need for building backup scripts or setting up monitoring solution because it’s already done by AWS engineers - all you need to do is to use it.

There is also a dark side of the MySQL RDS. Lack of options to build more complex setups and scaling outside of just adding more slaves. Lack of support for better high availability than what’s proposed under multi-AZ deployments. Cumbersome access to MySQL logs. Lack of direct access to MySQL data directory and lack of support for physical backups, which makes it hard to move the data out of the RDS instance.

To sum it up, RDS may work fine for you if you value ease of use over detailed control of the database. You need to keep in mind that, at some point in the future, you may outgrow MySQL RDS. We are not necessarily talking here about performance only. It’s more about your organization’s needs for more complex replication topology or a need to have better insight into database operations to deal quickly with different issues that arise from time to time. In that case, if your dataset already has grown in size, you may find it tricky to move out of the RDS. Before making any decision to move your data into RDS, information managers must consider their organization's requirements and constraints in specific areas.

In next couple of blog posts we will show you how to take your data out of the RDS into a separate location. We will discuss both migration to EC2 and to on-premises infrastructure.

Tags: 
MySQL
MariaDB
amazon
AWS
cloud database
rds
dbaas
↧
Search

JSON Support in PostgreSQL, MySQL, MongoDB, and SQL Server

February 9, 2017, 5:42 am
$
0
0

Updated 2/10/2017

If you've been watching the evolution of database technologies over the past few years, you've seen how quickly JSON has quickly cemented its position in major database servers. Due to its use in the web front-end, JSON has overtaken XML in APIs, and it’s spread through all the layers in the stack one step at a time.

Most major databases supported XML in some fashion for a while, too, but developer uptake wasn’t universal. JSON adoption among developers is nearly universal today, however. (The king is dead, long live the king!) But how good is JSON support in the databases we know and love? We’ll do a comparison in this blog post.

crown-1.jpg
Image Credit - Jason Train

What is JSON Support?

First – what does it even mean for a database to support JSON?

You could easily argue that JSON support has been in databases for a long time. After all, all you have to do is store a blob of text that is correctly formatted in the database, and applications can store, retrieve, and manipulated as usual. But the level of JSON support that we are seeing in databases today exceeds that. JSON is becoming natively supported as a datatype, with functions and operators and all of the decoration that goes along with it.

For the purposes of this article, we don’t insist upon JSON being a natively supported data type that is distinct from other datatypes. It is enough for JSON to be stored as a blob, and manipulated with functions. However, as you will see, most databases go well beyond that.

JSON Support in MySQL

MySQL was late to the party. For a long time, developers, including VividCortex, stored JSON in blobs in MySQL, and either manipulated it in the application or used see compiled functions or stored procedures to manipulate it. That was always been suboptimal. MySQL, and particularly InnoDB, are not great at storing and manipulating blob data. It can be stored inefficiently, with a lot of overhead.

Luckily, with MySQL 5.7 that all changed. As of MySQL 5.7, JSON has been a natively supported data type. 


Read the free ebook "Configuring MySQL for Performance."

In addition, MySQL 5.7 introduced a set of JSON functions for manipulating JSON columns directly. This is an even more recent update -- as a previous version of this blog post noted, when MySQL 5.7 first introduced JSON, its function prefixes were designated with JSN_ , in case JSON function ever became part of the SQL standard. As current MySQL documentation notes, "For functions that take a JSON argument, an error occurs if the argument is not a valid JSON value," and, "Unless otherwise indicated, the JSON functions were added in MySQL 5.7.8."

Lastly in MySQL 5.7, it became possible to index values buried inside JSON documents by using virtual columns.

In the newest MySQL release, MySQL 8.0, several updates were added for document database support, such as JSON_ARRAYAGG() and JSON_OBJECTAGG(), along with the JSON operator ->>. MySQL 8.0's release notes describe the new operator as "similar to the -> operator, but [it] performs JSON unquoting of the value as well. For a JSON column mycol and JSON path expression mypath, the following three expressions are equivalent."

In older versions of MySQL, a set of UDF functions had been available for quite a while for the community to install into their databases and manipulate JSON. There are also community UDFs in the MySQL UDF repository.

JSON Support in PostgreSQL

JSON has a longer history of support in PostgreSQL. I have not used it personally, but friends of mine have been talking to me about it at least since the 9.2 release. In this release, JSON was a natively supported datatype, and there were a couple of functions available for querying and manipulating it.

In the 9.3 release, support was greatly extended. In addition to the functions, there were a half-dozen operators, and these are still available in the latest PostgreSQL release, 9.6. Personally, I find these operators hard to read, and I don’t think they add much to SQL. I think they would be better off as functions. But that’s just my personal preference.

In version 9.4, the JSON datatype was supplemented by a JSONB datatype, which is still supported in 9.6. JSONB includes another half dozen operators, making the SQL look even more like Perl. There are also a couple of dozen JSONB functions, one for each corresponding JSON function, and PostgreSQL 9.6 added JSONB_Insert().

Perhaps one of the biggest advantages of JSONB over plaintext JSON is that it supports indexing. You can create functional indexes in Postgres, of course, so you can still index plaintext JSON; but it’s better and more efficient to index JSONB than JSON.

JSON Support in MongoDB

It may seem a little silly to include this section in this blog post, especially if you are familiar with MongoDB. However, even if it’s a little obvious, we don’t want to leave it unstated. And there are non-obvious things, too.

It’s very simple: in MongoDB, JSON is the native datatype, and JavaScript is used to access it natively. In reality, it is stored as JSONB, not plaintext JSON.


Start a free trial of VividCortex and see all the activity of your various database types in a unified view!

However, until versions of MongoDB from a few years ago, the data was stored with no compression. The recommended way of mitigating this problem was to use short field names, which wasn’t a very good alternative, frankly. As of MongoDB version 3.0, the storage engine began to include compression by default, much improving the situation. You can also use Percona’s TokuMX, which includes a storage engine with high-performance, transactions, and native compression.

Indexing is natively supported as well, naturally. This has been included since the first MongoDB releases. However, indexing has improved a lot over time, including the addition of sparse indexes (since not all documents will contain all fields you are indexing).

JSON Support in SQL Server

JSON support in Microsoft SQL server arrived in the 2016 release of the product. However, in contrast to the other databases we have discussed, SQL Server doesn't include a native JSON datatype. Instead, the functionality is quite similar to the native XML functionality that has existed in SQL Server for a long time. If you are familiar with SQL Server, you will immediately recognize the OPENJSON() function. It operates on NVARCHAR values. You can learn more in Microsoft's documentation on SQL Server 2016, which has been updated recently.

Indexing is also limited. No native JSON indexes; just fulltext indexing.

You can read more details on the MSDN blog.

Conclusion

One of the big reasons why people are interested in JSON support in databases is that they want to use fewer types of databases. The modern technology stack is beginning to introduce significant sprawl as people use different databases in particular areas, taking advantage of their strengths to gain efficiency. However, this polyglot persistence increases the technology surface area enough that it can become quite difficult to monitor, manage, develop, and operate such a diverse set of databases.

One potential answer to this problem is to continue using work horses such as MySQL and Postgres, replacing specialized JSON native databases with the new JSON functionality. For the record, MongoDB is not the only JSON native database. There are many databases, including RethinkDB, that deal with JSON natively.

A big difference between the JSON support in MySQL, Postgres, and MongoDB is that in MongoDB, this is the native transport across the wire as well. JSON is native end to end in this database, whereas in other databases it is typically shoehorned into row and column storage in the client and on the wire, as well as the programming API.

Still, keeping technology diversity down can be a big enough reason to continue using the reliable and trusted databases of yore.

↧
↧

Percona Blog Poll: What Database Engine Are You Using to Store Time Series Data?

February 10, 2017, 7:19 am
$
0
0
TIme Series Data

TIme Series DataTake Percona’s blog poll on what database engine you are using to store time series data.

Time series data is some of the most actionable data available when it comes to analyzing trends and making predictions. Simply put, time series data is data that is indexed not just by value, but by time as well – allowing you to view value changes over time as they occur. Obvious uses include the stock market, web traffic, user behavior, etc.

With the increasing number of smart devices in the Internet of Things (IoT), being able to track data over time is more and more important. With time series data, you can measure and make predictions on things like energy consumption, pH values, water consumption, data from environment-aware machines like smart cars, etc. The sensors used in IoT devices and systems generate huge amounts of time-series data.

How is all of this data collected, segmented and stored? We’d like to hear from you: what database engine are you using to store time series data? Please take a few seconds and answer the following poll. Which are you using? Help the community learn what database engines help solve critical database issues. Please select from one to three database engines as they apply to your environment. Feel free to add comments below if your engine isn’t listed.

Note: There is a poll embedded within this post, please visit the site to participate in this post's poll.
↧

Shinguz: MySQL and MariaDB authentication against pam_unix

February 13, 2017, 9:02 am
$
0
0

The PAM authentication plug-in is an extension included in MySQL Enterprise Edition (since 5.5) and in MariaDB (since 5.2).

MySQL authentication against pam_unix


Check if plug-in is available:

# ll lib/plugin/auth*so
-rwxr-xr-x 1 mysql mysql 42937 Sep 18  2015 lib/plugin/authentication_pam.so
-rwxr-xr-x 1 mysql mysql 25643 Sep 18  2015 lib/plugin/auth.so
-rwxr-xr-x 1 mysql mysql 12388 Sep 18  2015 lib/plugin/auth_socket.so
-rwxr-xr-x 1 mysql mysql 25112 Sep 18  2015 lib/plugin/auth_test_plugin.so

Install PAM plug-in:

mysql> INSTALL PLUGIN authentication_pam SONAME 'authentication_pam.so';

Check plug-in information:

mysql> SELECT * FROM information_schema.plugins WHERE plugin_name = 'authentication_pam'\G
*************************** 1. row ***************************
           PLUGIN_NAME: authentication_pam
        PLUGIN_VERSION: 1.1
         PLUGIN_STATUS: ACTIVE
           PLUGIN_TYPE: AUTHENTICATION
   PLUGIN_TYPE_VERSION: 1.1
        PLUGIN_LIBRARY: authentication_pam.so
PLUGIN_LIBRARY_VERSION: 1.7
         PLUGIN_AUTHOR: Georgi Kodinov
    PLUGIN_DESCRIPTION: PAM authentication plugin
        PLUGIN_LICENSE: PROPRIETARY
           LOAD_OPTION: ON

It seems like this set-up is persisted and survives a database restart because of the mysql schema table:

mysql> SELECT * FROM mysql.plugin;
+--------------------+-----------------------+
| name               | dl                    |
+--------------------+-----------------------+
| authentication_pam | authentication_pam.so |
+--------------------+-----------------------+

Configuring PAM on Ubuntu/Debian:

#%PAM-1.0
#
# /etc/pam.d/mysql
#
@include common-auth
@include common-account
@include common-session-noninteractive

Create the database user matching to the O/S user:

mysql> CREATE USER 'oli'@'localhost'
IDENTIFIED WITH authentication_pam AS 'mysql'
;
mysql> GRANT ALL PRIVILEGES ON test.* TO 'oli'@'localhost';

Verifying user in the database:

mysql> SELECT user, host, authentication_string FROM mysql.user WHERE user = 'oli';
+-----------+-----------+-------------------------------------------+
| user      | host      | authentication_string                     |
+-----------+-----------+-------------------------------------------+
| oli       | localhost | mysql                                     |
+-----------+-----------+-------------------------------------------+

mysql> SHOW CREATE USER 'oli'@'localhost';
+-----------------------------------------------------------------------------------------------------------------------------------+
| CREATE USER for oli@localhost                                                                                                     |
+-----------------------------------------------------------------------------------------------------------------------------------+
| CREATE USER 'oli'@'localhost' IDENTIFIED WITH 'authentication_pam' AS 'mysql' REQUIRE NONE PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK |
+-----------------------------------------------------------------------------------------------------------------------------------+

Connection tests:

# mysql --user=oli --host=localhost
ERROR 2059 (HY000): Authentication plugin 'mysql_clear_password' cannot be loaded: plugin not enabled

# mysql --user=oli --host=localhost --enable-cleartext-plugin --password=wrong
ERROR 1045 (28000): Access denied for user 'oli'@'localhost' (using password: YES)

# tail /var/log/auth.log
Feb 13 15:15:14 chef unix_chkpwd[31600]: check pass; user unknown
Feb 13 15:15:14 chef unix_chkpwd[31600]: password check failed for user (oli)

# mysql --user=oli --host=localhost --enable-cleartext-plugin --password=rigth
ERROR 1045 (28000): Access denied for user 'oli'@'localhost' (using password: YES)

# tail /var/log/auth.log
Feb 13 15:15:40 chef unix_chkpwd[31968]: check pass; user unknown
Feb 13 15:15:40 chef unix_chkpwd[31968]: password check failed for user (oli)

Some research led to the following result: The non privileged mysql user is not allowed to access the file /etc/shadow thus it should be added to the group shadow to make it work:

# ll /sbin/unix_chkpwd
-rwxr-sr-x 1 root shadow 35536 Mar 16  2016 /sbin/unix_chkpwd

# usermod -a -G shadow mysql


Connection tests:

# mysql --user=oli --host=localhost --enable-cleartext-plugin --password=rigth

mysql> SELECT USER(), CURRENT_USER(), @@proxy_user;
+---------------+----------------+--------------+
| USER()        | CURRENT_USER() | @@proxy_user |
+---------------+----------------+--------------+
| oli@localhost | oli@localhost  | NULL         |
+---------------+----------------+--------------+

MariaDB authentication against pam_unix


Check if plug-in is available:

# ll lib/plugin/auth*so
-rwxr-xr-x 1 mysql mysql 12462 Nov  4 14:37 lib/plugin/auth_0x0100.so
-rwxr-xr-x 1 mysql mysql 33039 Nov  4 14:37 lib/plugin/auth_gssapi_client.so
-rwxr-xr-x 1 mysql mysql 80814 Nov  4 14:37 lib/plugin/auth_gssapi.so
-rwxr-xr-x 1 mysql mysql 19015 Nov  4 14:37 lib/plugin/auth_pam.so
-rwxr-xr-x 1 mysql mysql 13028 Nov  4 14:37 lib/plugin/auth_socket.so
-rwxr-xr-x 1 mysql mysql 23521 Nov  4 14:37 lib/plugin/auth_test_plugin.so

Install PAM plug-in:

mysql> INSTALL SONAME 'auth_pam';

Check plug-in information:

mysql> SELECT * FROM information_schema.plugins WHERE plugin_name = 'pam'\G
*************************** 1. row ***************************
           PLUGIN_NAME: pam
        PLUGIN_VERSION: 1.0
         PLUGIN_STATUS: ACTIVE
           PLUGIN_TYPE: AUTHENTICATION
   PLUGIN_TYPE_VERSION: 2.0
        PLUGIN_LIBRARY: auth_pam.so
PLUGIN_LIBRARY_VERSION: 1.11
         PLUGIN_AUTHOR: Sergei Golubchik
    PLUGIN_DESCRIPTION: PAM based authentication
        PLUGIN_LICENSE: GPL
           LOAD_OPTION: ON
       PLUGIN_MATURITY: Stable
   PLUGIN_AUTH_VERSION: 1.0

Configuring PAM on Ubuntu/Debian:

#%PAM-1.0
#
# /etc/pam.d/mysql
#
@include common-auth
@include common-account
@include common-session-noninteractive

Create the database user matching to the O/S user:

mysql> CREATE USER 'oli'@'localhost'
IDENTIFIED VIA pam USING 'mariadb'
;
mysql> GRANT ALL PRIVILEGES ON test.* TO 'oli'@'localhost';

Verifying user in the database:

mysql> SELECT user, host, authentication_string FROM mysql.user WHERE user = 'oli';
+------+-----------+-----------------------+
| user | host      | authentication_string |
+------+-----------+-----------------------+
| oli  | localhost | mariadb               |
+------+-----------+-----------------------+

Connection tests:

# mysql --user=oli --host=localhost --password=wrong
ERROR 2059 (HY000): Authentication plugin 'dialog' cannot be loaded: /usr/local/mysql/lib/plugin/dialog.so: cannot open shared object file: No such file or directory

# tail /var/log/auth.log
Feb 13 17:11:16 chef mysqld: pam_unix(mariadb:auth): unexpected response from failed conversation function
Feb 13 17:11:16 chef mysqld: pam_unix(mariadb:auth): conversation failed
Feb 13 17:11:16 chef mysqld: pam_unix(mariadb:auth): auth could not identify password for [oli]
Feb 13 17:11:16 chef mysqld: pam_winbind(mariadb:auth): getting password (0x00000388)
Feb 13 17:11:16 chef mysqld: pam_winbind(mariadb:auth): Could not retrieve user's password

# mysql --user=oli --host=localhost --password=wrong --plugin-dir=$PWD/lib/plugin
ERROR 1045 (28000): Access denied for user 'oli'@'localhost' (using password: NO)
Feb 13 17:11:30 chef mysqld: pam_unix(mariadb:auth): authentication failure; logname= uid=1001 euid=1001 tty= ruser= rhost=  user=oli
Feb 13 17:11:30 chef mysqld: pam_winbind(mariadb:auth): getting password (0x00000388)
Feb 13 17:11:30 chef mysqld: pam_winbind(mariadb:auth): pam_get_item returned a password
Feb 13 17:11:30 chef mysqld: pam_winbind(mariadb:auth): request wbcLogonUser failed: WBC_ERR_AUTH_ERROR, PAM error: PAM_USER_UNKNOWN (10), NTSTATUS: NT_STATUS_NO_SUCH_USER, Error message was: No such user

Add mysql user to the shadow group:

# ll /sbin/unix_chkpwd
-rwxr-sr-x 1 root shadow 35536 Mar 16  2016 /sbin/unix_chkpwd

# usermod -a -G shadow mysql

Connection tests:

# mysql --user=oli --host=localhost --password=right --plugin-dir=$PWD/lib/plugin

mysql> SELECT USER(), CURRENT_USER(), @@proxy_user;
+---------------+----------------+--------------+
| USER()        | CURRENT_USER() | @@proxy_user |
+---------------+----------------+--------------+
| oli@localhost | oli@localhost  | NULL         |
+---------------+----------------+--------------+

Taxonomy upgrade extras: 
authentication
pam
security
plugin
plug-in
↧

MySQL may return results in non-deterministic order with ‘order by’

February 13, 2017, 11:15 am
$
0
0

Whenever we want a query’s results sorted, you may think of using the clause “order by.” You are correct: most of the time, MySQL will return the results in expected order with “order by.”

Be careful, though. Sometimes MySQL may return results in the non-deterministic order even with “order by.”

Indeed, if a query is ordered by a non-unique column, it may return results in an unexpected order. I tested the below example on MySQL 5.1.73, 5.5.54 and 5.6.19 and got the same result. However, when I applied the same example on MySQL 5.7.17, it returned the results in an unexpected order differently.

Follow me step-by-step and see how MySQL returns results in a non-deterministic order. Step 1-4 is for MySQL 5.1.73, 5.5.54 and 5.6.19, Step 5 is for MySQL 5.7.17. After the example, I will explain the reason behind this output.

Step 1. Create the table as below:

CREATE TABLE `user_rankings` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`sort_order` int(11) NOT NULL,
`equals_role` varchar(40) NOT NULL DEFAULT '',
`equals_user_id` int(11) NOT NULL,
`metric_posts` int(11) NOT NULL DEFAULT '0',
`metric_pageviews` int(11) NOT NULL DEFAULT '0',
`metric_logins` int(11) NOT NULL DEFAULT '0',
`metric_minutes_online` int(11) NOT NULL DEFAULT '0',
`average_message_rating` int(11) NOT NULL DEFAULT '0',
`registration_age` int(11) NOT NULL DEFAULT '0',
`metric_tags` int(11) NOT NULL DEFAULT '0',
`rank_formula_enabled` tinyint(1) DEFAULT NULL,
`rank_formula` varchar(5000) DEFAULT NULL,
`rank_name` varchar(80) NOT NULL,
`rank_bold` tinyint(1) NOT NULL DEFAULT '0',
`rank_color` varchar(6) NOT NULL DEFAULT '',
`rank_icon` varchar(255) NOT NULL DEFAULT '',
`rank_icon_right` varchar(255) NOT NULL DEFAULT '',
`rank_icon_thread` varchar(255) NOT NULL DEFAULT '',
`grant_roles` varchar(4100) NOT NULL DEFAULT '',
`remove_roles` varchar(4100) NOT NULL DEFAULT '',
`kudos_weight` int(11) DEFAULT NULL,
`deleted` tinyint(1) NOT NULL DEFAULT '0',
`row_version` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `IDX_USER_RANKINGS_ROW_VERSION` (`row_version`)
) ENGINE=InnoDB AUTO_INCREMENT=32 DEFAULT CHARSET=utf8;

Step 2. Insert into the table 31 rows of data as below:

INSERT INTO `user_rankings` VALUES  (1,0,'Administrator,Ipsos Admin',-1,0,0,0,0,0,0,0,1,'hasRole(\"Administrator\") || hasRole(\"Ipsos Admin\") || hasRole(\"Acat_Moderator\") || hasRole(\"Bcat_Moderator\") || hasRole(\"Ccat_Moderator\") || hasRole(\"Dcat_Moderator\") || hasRole(\"Ecat_Moderator\") || hasRole(\"Fcat_Moderator\") || hasRole(\"Gcat_Moderator\") || hasRole(\"Hcat_Moderator\") || hasRole(\"Icat_Moderator\") || hasRole(\"Jcat_Moderator\") || hasRole(\"Kcat_Admin\") || hasRole(\"Lcat_Admin\") || hasRole(\"Mcat_Admin\") || hasRole(\"Ncat_Admin\") || hasRole(\"Ocat_Admin\") || hasRole(\"Pcat_Admin\") || hasRole(\"Qcat_Admin\") || hasRole(\"Rcat_Admin\") || hasRole(\"Scat_Admin\") || hasRole(\"Tcat_Admin\") || hasRole(\"Ucat_Admin\") || hasRole(\"Vcat_Admin\") || hasRole(\"Wcat_Admin\") || hasRole(\"Xcat_Admin\") || hasRole(\"Ycat_Admin\") || hasRole(\"Zcat_Admin\") || hasRole(\"Dcat_Moderator\") || hasRole(\"Ccat_Moderator\")  || hasRole(\"Bcat_Moderator\")  || hasRole(\"Ecat_Moderator\")','Community Manager',1,'009966','/i/rank_icons/admin.gif','','','','',7,0,'2016-10-25 00:16:34'), (2,1,'Moderator',-1,0,0,0,0,0,0,0,0,'','Moderator',1,'009966','/i/rank_icons/moderator.gif','','','','',7,0,'2016-07-07 01:48:03'), (3,2,'Member',-1,0,0,0,0,0,0,0,0,'hasRole(\"Member\") || hasRole(\"Acat_Member\") || hasRole(\"Bcat_Member\") || hasRole(\"Ccat_Member\") || hasRole(\"Dcat_Member\") || hasRole(\"Ecat_Member\") || hasRole(\"Fcat_Member\") || hasRole(\"Gcat_Member\") || hasRole(\"Hcat_Member\") || hasRole(\"Icat_Member\") || hasRole(\"Jcat_Member\") || hasRole(\"Kcat_Member\") || hasRole(\"Lcat_Member\") || hasRole(\"Mcat_Member\") || hasRole(\"Ncat_Member\") || hasRole(\"Ocat_Member\") || hasRole(\"Pcat_Member\") || hasRole(\"Qcat_Member\") || hasRole(\"Rcat_Member\") || hasRole(\"Scat_Member\") || hasRole(\"Tcat_Member\") || hasRole(\"Ucat_Member\") || hasRole(\"Vcat_Member\") || hasRole(\"Wcat_Member\") || hasRole(\"Xcat_Member\") || hasRole(\"Ycat_Member\") || hasRole(\"Zcat_Member\")','Member',0,'666666','','','','','',-1,0,'2016-09-27 07:37:42'),(4,3,'Client',-1,0,0,0,0,0,0,0,1,'hasRole(\"Client\") || hasRole(\"Acat_Client\") || hasRole(\"Bcat_Client\") || hasRole(\"Ccat_Client\") || hasRole(\"Dcat_Client\") || hasRole(\"Ecat_Client\") || hasRole(\"Fcat_Client\") || hasRole(\"Gcat_Client\") || hasRole(\"Hcat_Client\") || hasRole(\"Icat_Client\") || hasRole(\"Jcat_Client\") || hasRole(\"Kcat_Client\") || hasRole(\"Lcat_Client\") || hasRole(\"Mcat_Client\") || hasRole(\"Ncat_Client\") || hasRole(\"Ocat_Client\") || hasRole(\"Pcat_Client\") || hasRole(\"Qcat_Client\") || hasRole(\"Rcat_Client\") || hasRole(\"Scat_Client\") || hasRole(\"Tcat_Client\") || hasRole(\"Ucat_Client\") || hasRole(\"Vcat_Client\") || hasRole(\"Wcat_Client\") || hasRole(\"Xcat_Client\") || hasRole(\"Ycat_Client\") || hasRole(\"Zcat_Client\")','VIP',1,'000000','/html/rank_icons/VIP.png','','','','',3,0,'2016-09-27 07:33:14'),(5,4,'Administrator',-1,0,0,0,0,0,0,0,1,'(overall_posts >= 805) && (minutes_online >= 6125) && (net_kudos_weight_received >= 1225) && (net_kudos_weight_given >= 350) && hasRole(\"Administrator\")','Super Star',1,'fd4403','/html/rank_icons/rank_superstar.png','','','','',3,0,'2016-07-07 01:48:03'),(6,5,'Administrator',-1,0,0,0,0,0,0,0,1,'(overall_posts >= 630) && (minutes_online >= 4775) && (net_kudos_weight_received >= 1000) && (net_kudos_weight_given >= 240) && hasRole(\"Administrator\")','Rising Star',1,'fb8717','/html/rank_icons/rank_risingstar.png','','','','',3,0,'2016-07-07 01:48:03'),(7,6,'',-1,0,0,0,0,0,0,0,1,'(overall_posts >= 480) && (minutes_online >= 3650) && (net_kudos_weight_received >= 800) && (net_kudos_weight_given >= 160)','Star',1,'fcb223','/html/rank_icons/rank_star.png','','','','',3,0,'2016-07-07 01:48:03'), (8,7,'',-1,0,0,0,0,0,0,0,1,'(overall_posts >= 355) && (minutes_online >= 2750) && (net_kudos_weight_received >= 625) && (net_kudos_weight_given >= 100)','Honored Member',1,'663399','/html/rank_icons/rank_honoredmember.gif','','','','',2,1,'2016-07-25 19:20:02'), (9,8,'',-1,0,0,0,0,0,0,0,1,'(overall_posts >= 280) && (minutes_online >= 2050) && (net_kudos_weight_received >= 475) && (net_kudos_weight_given >= 60)','Esteemed Member',1,'9933CC','/html/rank_icons/rank_esteemedmember.gif','','','','',2,1,'2016-07-25 19:19:43'),(10,9,'',-1,0,0,0,0,0,0,0,1,'(overall_posts >= 215) && (minutes_online >= 1375) && (net_kudos_weight_received >= 350) && (net_kudos_weight_given >= 30)','Respected Member',1,'CC00CC','/html/rank_icons/rank_respectedmember.gif','','','','',2,1,'2016-07-25 19:19:27'),(11,10,'',-1,0,0,0,0,0,0,0,1,'(overall_posts >= 150) && (minutes_online >= 775) && (net_kudos_weight_received >= 250) && (net_kudos_weight_given >= 5)','Trusted Member',1,'FF66CC','/html/rank_icons/rank_trustedmember.gif','','','','',2,1,'2016-07-25 19:19:12'),(12,11,'',-1,0,0,0,0,0,0,0,1,'(overall_posts >= 90) && (logins >= 10) && (minutes_online >= 225)','Super Contributor',0,'0f68d6','/html/rank_icons/rank_supercontributor.png','','','','',1,1,'2016-07-25 19:18:52'),(13,12,'',-1,0,0,0,0,0,0,0,1,'(overall_posts >= 45) && (logins >= 7) && (minutes_online >= 135)','Regular Contributor',0,'02b1cb','/html/rank_icons/rank_Regularcontributor.png','','','','',1,1,'2016-07-25 19:18:43'),(14,13,'',-1,0,0,0,0,0,0,0,1,'(overall_posts >= 10) && (logins >= 5) && (minutes_online >= 75) && (net_kudos_weight_received >= 15)','Liked Contributor',0,'02cb28','/html/rank_icons/rank_kudoedcontributor.png','','','','',1,1,'2016-07-25 19:18:07'),(15,14,'',-1,0,0,0,0,0,0,0,1,'(overall_posts >= 10) && (logins >= 5) && (minutes_online >= 75)','Frequent Contributor',0,'66CC00','','','','','',1,1,'2016-07-25 19:17:44'),(16,15,'',-1,0,0,0,0,0,0,0,1,'(overall_posts >= 4) && (logins >= 2) && (minutes_online >= 20)','Occasional Contributor',0,'000099','','','','','',1,1,'2016-07-25 19:17:29'), (17,16,'',-1,0,0,0,0,0,0,0,1,'(overall_posts >= 1)','New Contributor',0,'3399FF','','','','','',1,1,'2016-07-25 19:17:18'),(18,17,'',-1,0,0,0,0,0,0,0,1,'(logins >= 8)','Regular Visitor',0,'000000','','','','','',-1,1,'2016-07-25 19:17:05'), (19,18,'',-1,0,0,0,0,0,0,0,1,'(logins >= 6)','Frequent Visitor',0,'000000','','','','','',-1,1,'2016-07-25 19:16:54'),(20,7,'',-1,0,0,0,0,0,0,0,1,'(logins >= 4)','Visitor',0,'666666','','','','','',-1,0,'2016-07-25 19:20:02'), (21,8,'',-1,0,0,0,0,0,0,0,0,'','New Member',0,'666666','','','','','',-1,0,'2016-07-25 19:20:02'),(22,21,'Staff',-1,0,0,0,0,0,0,0,0,'','Staff',0,'0000CC','','','','','',-1,1,'2016-07-25 19:16:33'),(23,22,'',-1,8000,0,0,0,0,300000,0,1,'(((posts - deleted_posts) + net_idea_threads) >= 8000) && (registrationAge >= 300000) && ((net_kudos_weight_received +(net_accepted_solutions*10)) >= 1500)','Honored Contributor',1,'660099','','','','','',-1,1,'2016-07-25 19:16:17'),(24,23,'',-1,5000,0,0,0,0,250000,0,1,'(((posts - deleted_posts) + net_idea_threads) >= 5000) && (registrationAge >= 250000) && ((net_kudos_weight_received + (net_accepted_solutions*10)) >= 800)','Esteemed Contributor',1,'660099','','','','','',-1,1,'2016-07-25 19:16:06'),(25,24,'',-1,2000,0,0,0,0,200000,0,1,'(((posts - deleted_posts) + net_idea_threads) >= 2000) && (registrationAge >= 200000) && ((net_kudos_weight_received + (net_accepted_solutions*10)) >= 400)','Respected Contributor',1,'660099','','','','','',-1,1,'2016-07-25 19:15:55'),(26,25,'',-1,1000,0,0,0,0,165000,0,1,'(((posts - deleted_posts) + net_idea_threads) >= 1000) && (registrationAge >= 165000) && ((net_kudos_weight_received + (net_accepted_solutions*10)) >= 200)','Trusted Contributor',1,'660099','','','','','',-1,1,'2016-07-25 19:15:35'),(27,26,'',-1,500,0,0,0,0,125000,0,1,'(((posts - deleted_posts) + net_idea_threads) >= 500) && (registrationAge >= 125000) && ((net_kudos_weight_received + (net_accepted_solutions*10)) >= 75)','Valued Contributor',1,'660099','','','','','',-1,1,'2016-07-25 19:15:21'),(28,27,'',-1,20,0,0,0,0,10080,0,0,'','Contributor',0,'3300CC','','','','','',-1,1,'2016-07-25 19:15:09'),(29,28,'',-1,0,0,18,0,0,129600,0,1,'(logins >=18) && (registrationAge >= 43200) && (message_views >= 70)','Senior Member',0,'000000','','','','','',-1,1,'2016-07-25 19:14:55'),(30,29,'',-1,0,0,16,0,0,86400,0,1,'(logins >=16) && (registrationAge >= 43200) && (message_views >= 50)','Established Member',0,'000000','','','','','',-1,1,'2016-07-25 19:14:46'),(31,30,'',-1,0,0,2,0,0,2880,0,0,'','Occasional Visitor',0,'000000','','','','','',-1,1,'2016-07-25 19:14:28');

Step 3. Execute the queries order by the column sort_order which is not unique

mysql# SELECT id, rank_name ,sort_order FROM user_rankings where sort_order in (7,8) ORDER BY sort_order;
+----+-----------------+------------+
| id | rank_name       | sort_order |
+----+-----------------+------------+
|  8 | Honored Member  |          7 |
| 20 | Visitor         |          7 |
|  9 | Esteemed Member |          8 |
| 21 | New Member      |          8 |
+----+-----------------+------------+
4 rows in set (0.00 sec)

mysql# select id,rank_name,sort_order from user_rankings order by sort_order;
+----+------------------------+------------+
| id | rank_name              | sort_order |
+----+------------------------+------------+
|  1 | Community Manager      |          0 |
|  2 | Moderator              |          1 |
|  3 | Member                 |          2 |
|  4 | VIP                    |          3 |
|  5 | Super Star             |          4 |
|  6 | Rising Star            |          5 |
|  7 | Star                   |          6 |
| 20 | Visitor                |          7 |
|  8 | Honored Member         |          7 |
| 21 | New Member             |          8 |
|  9 | Esteemed Member        |          8 |
| 10 | Respected Member       |          9 |
| 11 | Trusted Member         |         10 |
| 12 | Super Contributor      |         11 |
| 13 | Regular Contributor    |         12 |
| 14 | Liked Contributor      |         13 |
| 15 | Frequent Contributor   |         14 |
| 16 | Occasional Contributor |         15 |
| 17 | New Contributor        |         16 |
| 18 | Regular Visitor        |         17 |
| 19 | Frequent Visitor       |         18 |
| 22 | Staff                  |         21 |
| 23 | Honored Contributor    |         22 |
| 24 | Esteemed Contributor   |         23 |
| 25 | Respected Contributor  |         24 |
| 26 | Trusted Contributor    |         25 |
| 27 | Valued Contributor     |         26 |
| 28 | Contributor            |         27 |
| 29 | Senior Member          |         28 |
| 30 | Established Member     |         29 |
| 31 | Occasional Visitor     |         30 |
+----+------------------------+------------+
31 rows in set (0.00 sec)
mysql# select *  from user_rankings order by sort_order;
+----+------------+---------------------------+----------------+--------------+------------------+---------------+-----------------------+------------------------+------------------+-------------+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------+-----------+------------+----------------------------------------------+-----------------+------------------+-------------+--------------+--------------+---------+---------------------+
| id | sort_order | equals_role               | equals_user_id | metric_posts | metric_pageviews | metric_logins | metric_minutes_online | average_message_rating | registration_age | metric_tags | rank_formula_enabled | rank_formula                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | rank_name              | rank_bold | rank_color | rank_icon                                    | rank_icon_right | rank_icon_thread | grant_roles | remove_roles | kudos_weight | deleted | row_version         |
+----+------------+---------------------------+----------------+--------------+------------------+---------------+-----------------------+------------------------+------------------+-------------+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------+-----------+------------+----------------------------------------------+-----------------+------------------+-------------+--------------+--------------+---------+---------------------+
|  1 |          0 | Administrator,Ipsos Admin |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | hasRole("Administrator") || hasRole("Ipsos Admin") || hasRole("Acat_Moderator") || hasRole("Bcat_Moderator") || hasRole("Ccat_Moderator") || hasRole("Dcat_Moderator") || hasRole("Ecat_Moderator") || hasRole("Fcat_Moderator") || hasRole("Gcat_Moderator") || hasRole("Hcat_Moderator") || hasRole("Icat_Moderator") || hasRole("Jcat_Moderator") || hasRole("Kcat_Admin") || hasRole("Lcat_Admin") || hasRole("Mcat_Admin") || hasRole("Ncat_Admin") || hasRole("Ocat_Admin") || hasRole("Pcat_Admin") || hasRole("Qcat_Admin") || hasRole("Rcat_Admin") || hasRole("Scat_Admin") || hasRole("Tcat_Admin") || hasRole("Ucat_Admin") || hasRole("Vcat_Admin") || hasRole("Wcat_Admin") || hasRole("Xcat_Admin") || hasRole("Ycat_Admin") || hasRole("Zcat_Admin") || hasRole("Dcat_Moderator") || hasRole("Ccat_Moderator")  || hasRole("Bcat_Moderator")  || hasRole("Ecat_Moderator") | Community Manager      |         1 | 009966     | /i/rank_icons/admin.gif                      |                 |                  |             |              |            7 |       0 | 2016-10-24 17:16:34 |
|  2 |          1 | Moderator                 |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    0 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | Moderator              |         1 | 009966     | /i/rank_icons/moderator.gif                  |                 |                  |             |              |            7 |       0 | 2016-07-06 18:48:03 |
|  3 |          2 | Member                    |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    0 | hasRole("Member") || hasRole("Acat_Member") || hasRole("Bcat_Member") || hasRole("Ccat_Member") || hasRole("Dcat_Member") || hasRole("Ecat_Member") || hasRole("Fcat_Member") || hasRole("Gcat_Member") || hasRole("Hcat_Member") || hasRole("Icat_Member") || hasRole("Jcat_Member") || hasRole("Kcat_Member") || hasRole("Lcat_Member") || hasRole("Mcat_Member") || hasRole("Ncat_Member") || hasRole("Ocat_Member") || hasRole("Pcat_Member") || hasRole("Qcat_Member") || hasRole("Rcat_Member") || hasRole("Scat_Member") || hasRole("Tcat_Member") || hasRole("Ucat_Member") || hasRole("Vcat_Member") || hasRole("Wcat_Member") || hasRole("Xcat_Member") || hasRole("Ycat_Member") || hasRole("Zcat_Member")                                                                                                                                                                      | Member                 |         0 | 666666     |                                              |                 |                  |             |              |           -1 |       0 | 2016-09-27 00:37:42 |
|  4 |          3 | Client                    |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | hasRole("Client") || hasRole("Acat_Client") || hasRole("Bcat_Client") || hasRole("Ccat_Client") || hasRole("Dcat_Client") || hasRole("Ecat_Client") || hasRole("Fcat_Client") || hasRole("Gcat_Client") || hasRole("Hcat_Client") || hasRole("Icat_Client") || hasRole("Jcat_Client") || hasRole("Kcat_Client") || hasRole("Lcat_Client") || hasRole("Mcat_Client") || hasRole("Ncat_Client") || hasRole("Ocat_Client") || hasRole("Pcat_Client") || hasRole("Qcat_Client") || hasRole("Rcat_Client") || hasRole("Scat_Client") || hasRole("Tcat_Client") || hasRole("Ucat_Client") || hasRole("Vcat_Client") || hasRole("Wcat_Client") || hasRole("Xcat_Client") || hasRole("Ycat_Client") || hasRole("Zcat_Client")                                                                                                                                                                      | VIP                    |         1 | 000000     | /html/rank_icons/VIP.png                     |                 |                  |             |              |            3 |       0 | 2016-09-27 00:33:14 |
|  5 |          4 | Administrator             |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 805) && (minutes_online >= 6125) && (net_kudos_weight_received >= 1225) && (net_kudos_weight_given >= 350) && hasRole("Administrator")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | Super Star             |         1 | fd4403     | /html/rank_icons/rank_superstar.png          |                 |                  |             |              |            3 |       0 | 2016-07-06 18:48:03 |
|  6 |          5 | Administrator             |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 630) && (minutes_online >= 4775) && (net_kudos_weight_received >= 1000) && (net_kudos_weight_given >= 240) && hasRole("Administrator")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | Rising Star            |         1 | fb8717     | /html/rank_icons/rank_risingstar.png         |                 |                  |             |              |            3 |       0 | 2016-07-06 18:48:03 |
|  7 |          6 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 480) && (minutes_online >= 3650) && (net_kudos_weight_received >= 800) && (net_kudos_weight_given >= 160)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Star                   |         1 | fcb223     | /html/rank_icons/rank_star.png               |                 |                  |             |              |            3 |       0 | 2016-07-06 18:48:03 |
|  8 |          7 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 355) && (minutes_online >= 2750) && (net_kudos_weight_received >= 625) && (net_kudos_weight_given >= 100)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Honored Member         |         1 | 663399     | /html/rank_icons/rank_honoredmember.gif      |                 |                  |             |              |            2 |       1 | 2016-07-25 12:20:02 |
| 20 |          7 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (logins >= 4)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | Visitor                |         0 | 666666     |                                              |                 |                  |             |              |           -1 |       0 | 2016-07-25 12:20:02 |
|  9 |          8 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 280) && (minutes_online >= 2050) && (net_kudos_weight_received >= 475) && (net_kudos_weight_given >= 60)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | Esteemed Member        |         1 | 9933CC     | /html/rank_icons/rank_esteemedmember.gif     |                 |                  |             |              |            2 |       1 | 2016-07-25 12:19:43 |
| 21 |          8 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    0 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | New Member             |         0 | 666666     |                                              |                 |                  |             |              |           -1 |       0 | 2016-07-25 12:20:02 |
| 10 |          9 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 215) && (minutes_online >= 1375) && (net_kudos_weight_received >= 350) && (net_kudos_weight_given >= 30)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | Respected Member       |         1 | CC00CC     | /html/rank_icons/rank_respectedmember.gif    |                 |                  |             |              |            2 |       1 | 2016-07-25 12:19:27 |
| 11 |         10 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 150) && (minutes_online >= 775) && (net_kudos_weight_received >= 250) && (net_kudos_weight_given >= 5)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | Trusted Member         |         1 | FF66CC     | /html/rank_icons/rank_trustedmember.gif      |                 |                  |             |              |            2 |       1 | 2016-07-25 12:19:12 |
| 12 |         11 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 90) && (logins >= 10) && (minutes_online >= 225)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | Super Contributor      |         0 | 0f68d6     | /html/rank_icons/rank_supercontributor.png   |                 |                  |             |              |            1 |       1 | 2016-07-25 12:18:52 |
| 13 |         12 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 45) && (logins >= 7) && (minutes_online >= 135)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | Regular Contributor    |         0 | 02b1cb     | /html/rank_icons/rank_Regularcontributor.png |                 |                  |             |              |            1 |       1 | 2016-07-25 12:18:43 |
| 14 |         13 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 10) && (logins >= 5) && (minutes_online >= 75) && (net_kudos_weight_received >= 15)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | Liked Contributor      |         0 | 02cb28     | /html/rank_icons/rank_kudoedcontributor.png  |                 |                  |             |              |            1 |       1 | 2016-07-25 12:18:07 |
| 15 |         14 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 10) && (logins >= 5) && (minutes_online >= 75)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Frequent Contributor   |         0 | 66CC00     |                                              |                 |                  |             |              |            1 |       1 | 2016-07-25 12:17:44 |
| 16 |         15 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 4) && (logins >= 2) && (minutes_online >= 20)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | Occasional Contributor |         0 | 000099     |                                              |                 |                  |             |              |            1 |       1 | 2016-07-25 12:17:29 |
| 17 |         16 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 1)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | New Contributor        |         0 | 3399FF     |                                              |                 |                  |             |              |            1 |       1 | 2016-07-25 12:17:18 |
| 18 |         17 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (logins >= 8)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | Regular Visitor        |         0 | 000000     |                                              |                 |                  |             |              |           -1 |       1 | 2016-07-25 12:17:05 |
| 19 |         18 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (logins >= 6)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | Frequent Visitor       |         0 | 000000     |                                              |                 |                  |             |              |           -1 |       1 | 2016-07-25 12:16:54 |
| 22 |         21 | Staff                     |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    0 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | Staff                  |         0 | 0000CC     |                                              |                 |                  |             |              |           -1 |       1 | 2016-07-25 12:16:33 |
| 23 |         22 |                           |             -1 |         8000 |                0 |             0 |                     0 |                      0 |           300000 |           0 |                    1 | (((posts - deleted_posts) + net_idea_threads) >= 8000) && (registrationAge >= 300000) && ((net_kudos_weight_received +(net_accepted_solutions*10)) >= 1500)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Honored Contributor    |         1 | 660099     |                                              |                 |                  |             |              |           -1 |       1 | 2016-07-25 12:16:17 |
| 24 |         23 |                           |             -1 |         5000 |                0 |             0 |                     0 |                      0 |           250000 |           0 |                    1 | (((posts - deleted_posts) + net_idea_threads) >= 5000) && (registrationAge >= 250000) && ((net_kudos_weight_received + (net_accepted_solutions*10)) >= 800)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Esteemed Contributor   |         1 | 660099     |                                              |                 |                  |             |              |           -1 |       1 | 2016-07-25 12:16:06 |
| 25 |         24 |                           |             -1 |         2000 |                0 |             0 |                     0 |                      0 |           200000 |           0 |                    1 | (((posts - deleted_posts) + net_idea_threads) >= 2000) && (registrationAge >= 200000) && ((net_kudos_weight_received + (net_accepted_solutions*10)) >= 400)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Respected Contributor  |         1 | 660099     |                                              |                 |                  |             |              |           -1 |       1 | 2016-07-25 12:15:55 |
| 26 |         25 |                           |             -1 |         1000 |                0 |             0 |                     0 |                      0 |           165000 |           0 |                    1 | (((posts - deleted_posts) + net_idea_threads) >= 1000) && (registrationAge >= 165000) && ((net_kudos_weight_received + (net_accepted_solutions*10)) >= 200)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Trusted Contributor    |         1 | 660099     |                                              |                 |                  |             |              |           -1 |       1 | 2016-07-25 12:15:35 |
| 27 |         26 |                           |             -1 |          500 |                0 |             0 |                     0 |                      0 |           125000 |           0 |                    1 | (((posts - deleted_posts) + net_idea_threads) >= 500) && (registrationAge >= 125000) && ((net_kudos_weight_received + (net_accepted_solutions*10)) >= 75)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | Valued Contributor     |         1 | 660099     |                                              |                 |                  |             |              |           -1 |       1 | 2016-07-25 12:15:21 |
| 28 |         27 |                           |             -1 |           20 |                0 |             0 |                     0 |                      0 |            10080 |           0 |                    0 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | Contributor            |         0 | 3300CC     |                                              |                 |                  |             |              |           -1 |       1 | 2016-07-25 12:15:09 |
| 29 |         28 |                           |             -1 |            0 |                0 |            18 |                     0 |                      0 |           129600 |           0 |                    1 | (logins >=18) && (registrationAge >= 43200) && (message_views >= 70)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Senior Member          |         0 | 000000     |                                              |                 |                  |             |              |           -1 |       1 | 2016-07-25 12:14:55 |
| 30 |         29 |                           |             -1 |            0 |                0 |            16 |                     0 |                      0 |            86400 |           0 |                    1 | (logins >=16) && (registrationAge >= 43200) && (message_views >= 50)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Established Member     |         0 | 000000     |                                              |                 |                  |             |              |           -1 |       1 | 2016-07-25 12:14:46 |
| 31 |         30 |                           |             -1 |            0 |                0 |             2 |                     0 |                      0 |             2880 |           0 |                    0 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | Occasional Visitor     |         0 | 000000     |                                              |                 |                  |             |              |           -1 |       1 | 2016-07-25 12:14:28 |
+----+------------+---------------------------+----------------+--------------+------------------+---------------+-----------------------+------------------------+------------------+-------------+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------+-----------+------------+----------------------------------------------+-----------------+------------------+-------------+--------------+--------------+---------+---------------------+
31 rows in set (0.00 sec)

Step 4. After deleting a few rows of the table, executed the queries again

mysql# delete from user_rankings where id>21;
Query OK, 10 rows affected (0.01 sec)

mysql# select id,rank_name,sort_order from user_rankings order by sort_order;
+----+------------------------+------------+
| id | rank_name              | sort_order |
+----+------------------------+------------+
|  1 | Community Manager      |          0 |
|  2 | Moderator              |          1 |
|  3 | Member                 |          2 |
|  4 | VIP                    |          3 |
|  5 | Super Star             |          4 |
|  6 | Rising Star            |          5 |
|  7 | Star                   |          6 |
|  8 | Honored Member         |          7 |
| 20 | Visitor                |          7 |
| 21 | New Member             |          8 |
|  9 | Esteemed Member        |          8 |
| 10 | Respected Member       |          9 |
| 11 | Trusted Member         |         10 |
| 12 | Super Contributor      |         11 |
| 13 | Regular Contributor    |         12 |
| 14 | Liked Contributor      |         13 |
| 15 | Frequent Contributor   |         14 |
| 16 | Occasional Contributor |         15 |
| 17 | New Contributor        |         16 |
| 18 | Regular Visitor        |         17 |
| 19 | Frequent Visitor       |         18 |
+----+------------------------+------------+
21 rows in set (0.00 sec)

mysql# delete from user_rankings where sort_order>9;
Query OK, 9 rows affected (0.04 sec)

mysql# select id,rank_name,sort_order from user_rankings order by sort_order;
+----+-------------------+------------+
| id | rank_name         | sort_order |
+----+-------------------+------------+
|  1 | Community Manager |          0 |
|  2 | Moderator         |          1 |
|  3 | Member            |          2 |
|  4 | VIP               |          3 |
|  5 | Super Star        |          4 |
|  6 | Rising Star       |          5 |
|  7 | Star              |          6 |
|  8 | Honored Member    |          7 |
| 20 | Visitor           |          7 |
|  9 | Esteemed Member   |          8 |
| 21 | New Member        |          8 |
| 10 | Respected Member  |          9 |
+----+-------------------+------------+
12 rows in set (0.00 sec)

Step 5. On MySQL 5.7.17, returned the results in unexpected order differently 

mysql# SELECT id,rank_name,sort_order FROM user_rankings where sort_order in('7','8') ORDER BY sort_order;
+----+-----------------+------------+
| id | rank_name       | sort_order |
+----+-----------------+------------+
|  8 | Honored Member  |          7 |
| 20 | Visitor         |          7 |
|  9 | Esteemed Member |          8 |
| 21 | New Member      |          8 |
+----+-----------------+------------+
4 rows in set (0.00 sec)

mysql# select * from user_rankings ORDER BY sort_order;
...
|  8 |          7 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 355) && (minutes_online >= 2750) && (net_kudos_weight_received >= 625) && (net_kudos_weight_given >= 100)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Honored Member         |         1 | 663399     | /html/rank_icons/rank_honoredmember.gif      |                 |                  |             |              |            2 |       1 | 2016-07-25 12:20:02 |
| 20 |          7 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (logins >= 4)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | Visitor                |         0 | 666666     |                                              |                 |                  |             |              |           -1 |       0 | 2016-07-25 12:20:02 |
|  9 |          8 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    1 | (overall_posts >= 280) && (minutes_online >= 2050) && (net_kudos_weight_received >= 475) && (net_kudos_weight_given >= 60)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | Esteemed Member        |         1 | 9933CC     | /html/rank_icons/rank_esteemedmember.gif     |                 |                  |             |              |            2 |       1 | 2016-07-25 12:19:43 |
| 21 |          8 |                           |             -1 |            0 |                0 |             0 |                     0 |                      0 |                0 |           0 |                    0 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | New Member             |         0 | 666666     |                                              |                 |                  |             |              |           -1 |       0 | 2016-07-25 12:20:02 |
...
31 rows in set (0.03 sec)

mysql# SELECT id,rank_name,sort_order FROM user_rankings ORDER BY sort_order;
...
| 20 | Visitor                |          7 |
|  8 | Honored Member         |          7 |
|  9 | Esteemed Member        |          8 |
| 21 | New Member             |          8 |
...
31 rows in set (0.03 sec)

mysql# delete from user_rankings where id>21;
Query OK, 10 rows affected (0.18 sec)

mysql# SELECT id,rank_name,sort_order FROM user_rankings ORDER BY sort_order;
...
|  8 | Honored Member         |          7 |
| 20 | Visitor                |          7 |
| 21 | New Member             |          8 |
|  9 | Esteemed Member        |          8 |
...
21 rows in set (0.00 sec)

mysql# delete from user_rankings where sort_order>9;
Query OK, 9 rows affected (0.05 sec)

mysql# SELECT id,rank_name,sort_order FROM user_rankings ORDER BY sort_order;
...
|  8 | Honored Member    |          7 |
| 20 | Visitor           |          7 |
|  9 | Esteemed Member   |          8 |
| 21 | New Member        |          8 |
...
12 rows in set (0.00 sec)

Some relevant variables:

mysql# show variables like '%sort%';
+--------------------------------+---------------------+
| Variable_name                  | Value               |
+--------------------------------+---------------------+
| innodb_disable_sort_file_cache | OFF                 |
| innodb_ft_sort_pll_degree      | 2                   |
| innodb_sort_buffer_size        | 1048576             |
| max_length_for_sort_data       | 1024                |
| max_sort_length                | 1024                |
| myisam_max_sort_file_size      | 9223372036853727232 |
| myisam_sort_buffer_size        | 8388608             |
| sort_buffer_size               | 262144              |
+--------------------------------+---------------------+
8 rows in set (0.00 sec)

Please note, there are 4 tuples out of 31 rows with a duplicated sort_order value:

  • two having sort_order=7
  • the other two having sort_order=8

In step 3, the query with WHERE sort_order in (7,8) ORDER BY sort_order returned tuples in the ‘correct’ order:

  • the row with id=8 in front of the row with id=20
  • the row with id=9 in front of the row with id=21

When the query selected all of the columns and all the tuples with ORDER BY sort_order, it returned in the ‘correct’ order again:

  • the row with id=8 in front of the row with id=20
  • the row with id=9 in front of the row with id=21

But when the query selected part of the columns and all the tuples with ORDER BY sort_order, it returned in the ‘wrong’ order:

  • the row with id=20 in front of the row with id=8
  • the row with id=21 in front of the row with id=9

Additionally, in step 4 after deleting the last 10 rows with id>21, we can see half of the 4 tuples returned in ‘correct’ order:

  • the row with id=8 in front of the row with id=20

The other half still returned in ‘wrong’ order:

  • the row with id=21 in front of the row with id=9

After deleting 9 more rows with sort_order>9, now we can see all the 4 tuples returned in the ‘correct’ order:

  • the row with id=8 in front of the row with id=20
  • the row with id=9 in front of the row with id=21

When I applied the same table with the same data on MySQL 5.7.17, it behaved differently as below:
When the query selected part of the columns and all the tuples with ORDER BY sort_order, the tuples with sort_order=7 returned in ‘wrong’ order:

  • the row with id=20 in front of the row with id=8

The tuples with sort_order=8 returned in ‘correct’ order:

  • the row with id=9 in front of the row with id=21

After deleting the last 10 rows with id>21, we can see the situation reversed:
The tuples with sort_order=7 returned in ‘correct’ order:

  • the row with id=8 in front of the row with id=20

The tuples with sort_order=8 returned in ‘wrong’ order:

  • the row with id=21 in front of the row with id=9

After deleting 9 more rows with sort_order>9, now we can see all the 4 tuples returned in ‘correct’ order:

  • the row with id=8 in front of the row with id=20
  • the row with id=9 in front of the row with id=21

How did this happen?

Without diving into the code deeply, I will explain by highlighting the documentation for MySQL 5.7 ORDER BY optimization.

Citing the key sections below:

“MySQL has two filesort algorithms for sorting and retrieving results. The original method uses only the ORDER BY columns. The modified method uses not just the ORDER BY columns, but all the columns referenced by the query.”

“… the optimizer uses the modified algorithm only if the total size of the extra columns in the sort tuple does not exceed the value of the max_length_for_sort_data system variable.”

“The modified filesort algorithm incorporates an optimization to avoid reading the rows twice: It records the sort key value, but instead of the row ID, it records the columns referenced by the query” “…When the sort buffer becomes full, sort the tuples by sort key value in memory and write it to a temporary file.”

Here is the explanation
Let’s take a look at the filesort_summary for each query that the optimizer follows:
When the query select all the columns, because the tuple in the example table is over the max_length_for_sort_data which is set to 1024, it selected the original filesort method and so the results were returned in ‘correct’ order based on rowId:

"filesort_summary": {
  "rows": 31,
  "examined_rows": 31,
  "number_of_tmp_files": 0,
  "sort_buffer_size": 6096,
  "sort_mode": "<sort_key, rowid>"
}

When the query selected part of the columns, because the total size of the extra columns is lower than the max_length_for_sort_data (which is set to 1024 by default), MySQL used the modified file sort algorithm. Since it sorted the tuples by sort key value (the order by column sort_order), which has duplicated values, the algorithm may return the rows in random order. In some circumstances, MySQL may return the results in the ‘wrong’ order, while in others, MySQL may return the results in the ‘correct’ order.

"filesort_summary": {
  "rows": 31,
  "examined_rows": 31,
  "number_of_tmp_files": 0,
  "sort_buffer_size": 99822,
  "sort_mode": "<sort_key, additional_fields>"
}

When our example query has a WHERE clause, so that there are only a few rows being sorted, in the case that most likely the results were returned in ‘correct’ order.

#
"filesort_summary": {
  "rows": 4,
  "examined_rows": 31,
  "number_of_tmp_files": 0,
  "sort_buffer_size": 99822,
  "sort_mode": "<sort_key, additional_fields>"
}

After deleting most of the rows, there are only a few rows being sorted, in the case that most likely the results were returned in ‘correct’ order.

"filesort_summary": {
  "rows": 12,
  "examined_rows": 12,
  "number_of_tmp_files": 0,
  "sort_buffer_size": 99822,
  "sort_mode": "<sort_key, additional_fields>"
}

It is true that since multiple rows have identical values in the ORDER BY column, the server is free to return those rows in any order. In other words, the sort order of those rows is non-deterministic due to the non-unique order by column.

Bearing this in mind, if you use LIMIT N, M following with ORDER BY non-unique column, you may get unexpected results based on different circumstances such as sort buffer related variables, the data size of each tuple, number of rows returned, data size returned etc. All of these factors might affect the choice of the file sort algorithm or even differences inside the algorithm itself. See this bug for more information: https://bugs.mysql.com/bug.php?id=72076

But why did MySQL 5.7 return the results in the order differently? Again, the relevant documentation hints at this:

“As of MySQL 5.7.3, the modified filesort algorithm includes an additional optimization designed to enable more tuples to fit into the sort buffer…”

And here is the optimizer filesort_summary on the query returning the results in the order differently:

# "filesort_summary": {
  "rows": 31,
  "examined_rows": 31,
  "number_of_tmp_files": 0,
  "sort_buffer_size": 100208,
  "sort_mode": "<sort_key, packed_additional_fields>"
}

Conclusion

For MySQL, results may be returned in a non-deterministic order with ‘ORDER BY‘ on a non-unique column. The only reliable way to get the expected order of your results is to explicitly define your query’s ORDER BY clause to handle potential duplicate tuples. Then test it.

In the end, ordering by unique columns will always return results in deterministic order.

Pythian offers end-to-end MySQL Services from planning and Implementation to ongoing operational support. Contact us to learn more.

↧

MySQL & MariaDB load balancing with ProxySQL & ClusterControl: introduction webinar

February 14, 2017, 2:15 am
$
0
0

Proxies are building blocks of high availability setups for MySQL and MariaDB. They can detect failed nodes and route queries to hosts which are still available. If your master failed and you had to promote one of your slaves, proxies will detect such topology changes and route your traffic accordingly. More advanced proxies can do much more: route traffic based on precise query rules, cache queries or mirror them. They can even be used to implement different types of sharding.

Introducing ProxySQL!

Join us for this live joint webinar with ProxySQL’s creator, René Cannaò, who will tell us more about this new proxy and its features. We will also show you how you can deploy ProxySQL using ClusterControl. And we will give you an early walk-through of some of the  exciting ClusterControl features for ProxySQL that we have planned.

Date, Time & Registration

Europe/MEA/APAC

Tuesday, February 28th at 09:00 GMT (UK) / 10:00 CET (Germany, France, Sweden)

Register Now

North America/LatAm

Tuesday, February 28th at 9:00 Pacific Time (US) / 12:00 Eastern Time (US)

Register Now

Agenda

  1. Introduction
  2. ProxySQL concepts (René Cannaò)
    • Hostgroups
    • Query rules
    • Connection multiplexing
    • Configuration management
  3. Demo of ProxySQL setup in ClusterControl (Krzysztof Książek)
  4. Upcoming ClusterControl features for ProxySQL

Speakers

René Cannaò, Creator & Founder, ProxySQL. René has 10 years of working experience as a System, Network and Database Administrator mainly on Linux/Unix platform. In the last 4-5 years his experience was focused mainly on MySQL, working as Senior MySQL Support Engineer at Sun/Oracle and then as Senior Operational DBA at Blackbird, (formerly PalominoDB). In this period he built an analytic and problem solving mindset and he is always eager to take on new challenges, especially if they are related to high performance. And then he created ProxySQL …

Krzysztof Książek, Senior Support Engineer at Severalnines, is a MySQL DBA with experience managing complex database environments for companies like Zendesk, Chegg, Pinterest and Flipboard.

We look forward to “seeing” you there and to insightful discussions!

If you have any questions or would like a personalised live demo, please do contact us.

Tags: 
MySQL
MariaDB
load balancing
proxysql
clustercontrol
↧
↧

How does a relational database work

February 14, 2017, 12:38 am
$
0
0
Introduction While doing my High-Performance Java Persistence training, I came to realize that it’s worth explaining how a relational database works, as otherwise, it is very difficult to grasp many transaction-related concepts like atomicity, durability, and checkpoints. In this post, I’m going to give a high-level explanation of how a relational database works internally while … Continue reading How does a relational database work →
↧

PHP-SQL-Parser updated with pull requests

February 14, 2017, 8:21 am
$
0
0
I took some time this morning to check out and merge some existing pull requests for PHP-SQL-Parser, the most popular SQL parser for MySQL and PHP.

I'm thinking about adding a Validate($conn,$sql) method to the parser to validate the syntax of the SQL against the given MySQL database.
↧
Search

Releasing BSL 1.1

February 14, 2017, 2:29 pm
$
0
0
Releasing BSL 1.1 kajarno Tue, 02/14/2017 - 17:29

When we originally released BSL 1.0 together with MariaDB MaxScale 2.0 in August 2016, we were acutely aware that using a license not approved by the OSI equates swearing in the church (or your other favourite place of worship). That’s why we didn’t beat the BSL drum, just the MaxScale drum. We tested the waters, and after a while approached the authorities on Open Source. Specifically, based on earlier relationships from the late 1990s, we asked Bruce Perens to review the BSL. Bruce Perens is the founder of the Open Source Initiative, together with Eric Raymond (The Cathedral and the Bazaar).

Bruce had a number of comments, that have flown into what is now BSL 1.1. From the perspective of MariaDB MaxScale users, nothing really changes. The changes aim at enabling the BSL to be a viable license for other vendors than ourselves, by setting consistent expectations for users and customers, about what the BSL is.

Take the Creative Commons. Great thing, but there’s many varieties of it, so you need to spend time evaluating the fine print. Given that BSL also allows for a parameter section, Bruce saw a need to constrain that parameter section. We concur. Out of three parameters, we managed to put clear constraints on two.

First, the duration of the time window, as defined by the Change Date. “This software will convert to Open Source in ninety years” is a possible BSL 1.0 parameter, but it makes little sense. Now there is an automatic cap at four years. That said, we encourage a shorter window (and for MariaDB MaxScale, the window is between two and three years).

Second, the Open Source license when the time window closes, as defined by the Change License. Even amongst OSI approved Open Source licenses, there are some odd ones. Now we require the Change License to be GPL compatible (you may pick between GPLv2 and GPLv3, or any other license that can be combined properly with GPL software). And you can add additional change licenses as you desire.

Third, the Use Limitation. There we failed at identifying a reasonable least common denominator, as software comes in so many different shapes. Even “non-commercial use” is hard to define. As an example, we want to enable App developers to distribute their free versions under the BSL, where a payment trigger would be getting rid of ads. And what exactly is commercial use? Internal use, where you save a lot of cost, in producing value for your own customers? If the users earn money – directly or indirectly – the developer should be fed.

The outcome: We updated the BSL to accommodate these changes. We straightened the language, with the help of Heather Meeker of O’Melveny & Myers LLP [*]. We updated our MaxScale specific and vendor oriented BSL FAQs. And while our next MariaDB MaxScale releases [**] will use BSL 1.1, the more important part is that other vendors are in a better shape using a license endorsed by Bruce Perens.

[*] Heather is author of Open (Source) For Business: A Practical Guide to Open Source Licensing, and has been a pro-bono counsel to the Mozilla, GNOME, and Python foundations, so who better to ask than her.

[**] To be specific:

  • MariaDB MaxScale 1.4.5 (released 8 Feb 2017) is GPL and the following maintenance release MariaDB MaxScale 1.4.6 (whenever that may be) will remain GPL.
  • MariaDB MaxScale 2.0.4 (also released 8 Feb 2017) is BSL 1.0 but the following maintenance release MariaDB MaxScale 2.0.5 (whenever that may be) will move to BSL 1.1.
  • MariaDB MaxScale 2.1.0 will be released under BSL 1.1 from the start.  

So, our suggestion to our fellow software vendors: Take the BSL. Build your business model around it, over time. During the time window, you’ll pay your developer salaries, and when the window is closed, you’ve contributed to the world of Open Source. It’s authentic, it’s honest, it’s fair for your users, your customers, your developers and your shareholders alike. Go BSL!

When we originally released BSL 1.0 together with MariaDB MaxScale 2.0 in August 2016, we were acutely aware that using a license not approved by the OSI equates swearing in the church (or your other favourite place of worship). That’s why we didn’t beat the BSL drum, just the MaxScale drum. We tested the waters, and after a while approached the authorities on Open Source. Specifically, based on earlier relationships from the late 1990s, we asked Bruce Perens to review the BSL. Bruce Perens is the founder of the Open Source Initiative, together with Eric Raymond (The Cathedral and the Bazaar).

Login or Register to post comments

↧

Log Buffer #504: A Carnival of the Vanities for DBAs

February 14, 2017, 5:15 pm
$
0
0

This edition of Log Buffer covers Oracle, Microsoft SQL Server, and MySQL.

Oracle:

Data Pump or Data Pain Part02 – tablespace/user

Monday Spotlight: It’s About Security – Oracle Exadata SL6

0001_specify_date_format_mask.md copy

OGG Custom Adapters: How to include a unique identifier for every record in custom adapter?

One of the optimizer enhancements that appeared in 12.2 for SQL is the “band join” that makes certain types of merge join much more efficient.

Microsoft SQL Server:

How to import data to Azure SQL Data Warehouse using SSIS

Using Index-Organized Tables in Oracle

Query Performance Tuning – A Methodical Approach

SQL Server Temporal Tables: How-To Recipes

Fix Commvault Backup Failures After an Upgrade

MySQL:

MySQL may return results in non-deterministic order with ‘order by’

Shinguz: MySQL and MariaDB authentication against pam_unix

MySQL in the Cloud – Pros and Cons of Amazon RDS

With 500+ VividCortex Users, Shopify Eliminates High Latency Queries From Redis and MySQL

An Introduction to MariaDB’s Data at Rest Encryption (DARE)

↧
↧

MySQL on Docker: Composing the Stack

February 15, 2017, 12:43 am
$
0
0

Docker 1.13 introduces a long-awaited feature called compose-file support, which allow us to define our containers with a nice simple config file instead of a single long command. If you have a look at our previous “MySQL on Docker” blog posts, we used multiple long command lines to run containers and services. By using compose-file, containers are easily specified for deployment. This reduces the risk for human error as you do not have to remember long commands with multiple parameters.

In this blog post, we’ll show you how to use compose-file by using simple examples around MySQL deployments. We assume you have Docker Engine 1.13 installed on 3 physical hosts and Swarm mode is configured on all hosts.

Introduction to Compose-file

In the Compose-file, you specify everything in YAML format as opposed to trying to remember all the arguments we have to pass to Docker commands. You can define services, networks and volumes here. The definition will be picked up by Docker and it is very much like passing command-line parameters to “docker run|network|volume” command.

As an introduction, we are going to deploy a simple standalone MySQL container. Before you start writing a Compose file, you first need to know the run command. Taken from our first MySQL on Docker blog series, let’s compose the following “docker run” command:

$ docker run --detach \
--name=test-mysql \
--publish 6603:3306 \
--env="MYSQL_ROOT_PASSWORD=mypassword" \
-v /storage/docker/mysql-datadir:/var/lib/mysql \
mysql

The docker-compose command will look for a default file called “docker-compose.yml” in the current directory. So, let’s first create the required directories beforehand:

$ mkdir -p ~/compose-files/mysql/single
$ mkdir -p /storage/docker/mysql-datadir
$ cd ~/compose-files/mysql/single

In YAML, here is what should be written:

version: '2'

services:
  mysql:
    image: mysql
    container_name: test-mysql
    ports:
      - 6603:3306
    environment:
      MYSQL_ROOT_PASSWORD: "mypassword"
    volumes:
      - /storage/docker/mysql-datadir:/var/lib/mysql

Save the above content into “~/compose-files/mysql/single/docker-compose.yml”. Ensure you are in the current directory ~/compose-files/mysql/single, then fire it up by running the following command:

$ docker-compose up -d
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating test-mysql

Verify if the container is running in detached mode:

[root@docker1 single]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
379d5c15ef44        mysql               "docker-entrypoint..."   8 minutes ago       Up 8 minutes        0.0.0.0:6603->3306/tcp   test-mysql

Congratulations! We have now got a MySQL container running with just a single command.

Deploying a Stack

Compose-file simplifies things, it provides us with a clearer view on how the infrastructure should look like. Let’s create a container stack that consists of a website running on Drupal, using a MySQL instance under a dedicated network and link them together.

Similar to above, let’s take a look at the command line version in the correct order to build this stack:

$ docker volume create mysql_data
$ docker network create drupal_mysql_net --driver=bridge
$ docker run -d --name=mysql-drupal --restart=always -v mysql_data:/var/lib/mysql --net=drupal_mysql_net -e MYSQL_ROOT_PASSWORD="mypassword" -e MYSQL_DATABASE="drupal" mysql
$ docker run -d --name=drupal -p 8080:80 --restart=always -v /var/www/html/modules -v /var/www/html/profiles -v /var/www/html/themes -v /var/www/html/sites --link mysql:mysql --net=drupal_mysql_net drupal

To start composing, let’s first create a directory for our new stack:

$ mkdir -p ~/compose-files/drupal-mysql
$ cd ~/compose-files/drupal-mysql

Then, create write content of docker-compose.yml as per below:

version: '2'

services:
  mysql:
    image: mysql
    container_name: mysql-drupal
    environment:
      MYSQL_ROOT_PASSWORD: "mypassword"
      MYSQL_DATABASE: "drupal"
    volumes:
      - mysql_data:/var/lib/mysql
    restart: always
    networks:
      - drupal_mysql_net

  drupal:
    depends_on:
      - mysql
    image: drupal
    container_name: drupal
    ports:
      - 8080:80
    volumes:
      - /var/www/html/modules
      - /var/www/html/profiles
      - /var/www/html/themes
      - /var/www/html/sites
    links:
      - mysql:mysql
    restart: always
    networks:
      - drupal_mysql_net

volumes:
  mysql_data:

networks:
  drupal_mysql_net:
    driver: bridge

Fire them up:

$ docker-compose up -d
..
Creating network "drupalmysql_drupal_mysql_net" with driver "bridge"
Creating volume "drupalmysql_mysql_data" with default driver
Pulling drupal (drupal:latest)...
..
Creating mysql-drupal
Creating drupal

Docker will perform the deployment as follows:

  1. Create network
  2. Create volume
  3. Pull images
  4. Create mysql-drupal (since container “drupal” is dependent on it)
  5. Create the drupal container

At this point, our architecture can be illustrated as follows:

We can then specify ‘mysql’ as the MySQL host in the installation wizard page since both containers are linked together. That’s it. To tear them down, simply run the following command under the same directory:

$ docker-compose down

The corresponding containers will be terminated and removed accordingly. Take note that the docker-compose command is bound to the individual physical host running Docker. In order to run on multiple physical hosts across Swarm, it needs to be treated differently by utilizing “docker stack” command. We’ll explain this in the next section.

Composing a Stack Across Swarm

Firstly, make sure the Docker engine is running on v1.13 and Swarm mode is enabled and in ready state:

$ docker node ls
ID                           HOSTNAME       STATUS  AVAILABILITY  MANAGER STATUS
8n8t3r4fvm8u01yhli9522xi9 *  docker1.local  Ready   Active        Reachable
o1dfbbnmhn1qayjry32bpl2by    docker2.local  Ready   Active        Reachable
tng5r9ax0ve855pih1110amv8    docker3.local  Ready   Active        Leader

In order to use the stack feature for Docker Swarm mode, we have to use the Docker Compose version 3 format. We are going to deploy a setup similar to the above, apart from a 3-node Galera setup as the MySQL backend. We already explained in details in this blog post.

Firstly, create a directory for our new stack:

$ mkdir -p ~/compose-files/drupal-galera
$ cd ~/compose-files/drupal-galera

Then add the following lines into “docker-compose.yml”:

version: '3'

services:

  galera:
    deploy:
      replicas: 3
      restart_policy:
        condition: on-failure
        delay: 30s
        max_attempts: 3
        window: 60s
      update_config:
        parallelism: 1
        delay: 10s
        max_failure_ratio: 0.3
    image: severalnines/pxc56
    environment:
      MYSQL_ROOT_PASSWORD: "mypassword"
      CLUSTER_NAME: "my_galera"
      XTRABACKUP_PASSWORD: "mypassword"
      DISCOVERY_SERVICE: '192.168.55.111:2379,192.168.55.112:2379,192.168.55.207:2379'
      MYSQL_DATABASE: 'drupal'
    networks:
      - galera_net

  drupal:
    depends_on:
      - galera
    deploy:
      replicas: 1
    image: drupal
    ports:
      - 8080:80
    volumes:
      - drupal_modules:/var/www/html/modules
      - drupal_profile:/var/www/html/profiles
      - drupal_theme:/var/www/html/themes
      - drupal_sites:/var/www/html/sites
    networks:
      - galera_net

volumes:
  drupal_modules:
  drupal_profile:
  drupal_theme:
  drupal_sites:

networks:
  galera_net:
    driver: overlay

Note that the Galera image that we used (severalnines/pxc56) requires a running etcd cluster installed on each of the Docker physical host. Please refer to this blog post on the prerequisite steps.

One of the important parts in our compose-file is the max_attempts parameter under restart_policy section. We have to specify a hard limit on the number of restarts in case of failure. This will make the deployment process safer because, by default, the Swarm scheduler will never give up in attempting to restart containers. If this happens, the process loop will fill up the physical host’s disk space with unusable containers when the scheduler cannot bring the containers up to the desired state. This is a common approach when handling stateful services like MySQL. It’s better to bring them down altogether rather than make them run in an inconsistent state.

To start them all, just execute the following command in the same directory where docker-compose.yml resides:

$ docker stack deploy --compose-file=docker-compose.yml my_drupal

Verify the stack is created with 2 services (drupal and galera):

$ docker stack ls
NAME       SERVICES
my_drupal  2

We can also list the current tasks in the created stack. The result is a combined version of “docker service ps my_drupal_galera” and “docker service ps my_drupal_drupal” commands:

$ docker stack ps my_drupal
ID            NAME                IMAGE                      NODE           DESIRED STATE  CURRENT STATE           ERROR  PORTS
609jj9ji6rxt  my_drupal_galera.1  severalnines/pxc56:latest  docker3.local  Running        Running 7 minutes ago
z8mcqzf29lbq  my_drupal_drupal.1  drupal:latest              docker1.local  Running        Running 24 minutes ago
skblp9mfbbzi  my_drupal_galera.2  severalnines/pxc56:latest  docker1.local  Running        Running 10 minutes ago
cidn9kb0d62u  my_drupal_galera.3  severalnines/pxc56:latest  docker2.local  Running        Running 7 minutes ago

Once we get the CURRENT STATE as RUNNING, we can start the Drupal installation by connecting to any of the Docker host IP address or hostname on port 8080, as in this case we used docker3 (albeit the drupal container is deployed on docker1), http://192.168.55.113:8080/. Proceed with the installation and specify ‘galera’ as the MySQL host and ‘drupal’ as the database name (as defined in the compose-file under MYSQL_DATABASE environment variable):

That’s it. The stack deployment was simplified by using Compose-file. At this point, our architecture is looking something like this:

Lastly, to remove the stack, just run the following command:

$ docker stack rm my_drupal
Removing service my_drupal_galera
Removing service my_drupal_drupal
Removing network my_drupal_galera_net
Related resources
 MySQL on Docker: Deploy a Homogeneous Galera Cluster with etcd
 MySQL on Docker: Introduction to Docker Swarm Mode and Multi-Host Networking
 MySQL Docker Containers: Understanding the basics

Conclusion

Using compose-file can save you time and reduce the risk for human error, as compared to when working with long command lines. This is a perfect tool for you to master before working with multi-container Docker applications, dealing with multiple deployment environments (e.g dev, test, staging, pre-prod, prod) and handling much more complex services, just like MySQL Galera Cluster. Happy containerizing!

Tags: 
compose
docker compose
MySQL
galera
docker swarm
kubernetes
↧

Docker Images for Percona Server for MySQL Group Replication

February 15, 2017, 8:27 am
$
0
0
Group Replication

Group ReplicationIn this blog post, we’ll point to a new Docker image for Percona Server for MySQL Group Replication.

Our most recent release of Percona Server for MySQ (Percona Server for MySQL 5.7.17) comes with Group Replication plugins. Unfortunately, since this technology is very new, it requires some fairly complicated steps to setup and get running. To help with that process, I’ve prepare Docker images that simplify its setup procedures.

You can find the image here: https://hub.docker.com/r/perconalab/pgr-57/.

To start the first node (bootstrap the group):

docker run -d -p 3306 --net=clusternet -e MYSQL_ROOT_PASSWORD=passw0rd -e CLUSTER_NAME=cluster1 perconalab/pgr-57

To add nodes into the group after:

docker run -d -p 3306 --net=clusternet -e MYSQL_ROOT_PASSWORD=passw0rd -e CLUSTER_NAME=cluster1 -e CLUSTER_JOIN=CONTAINER_ID_FROM_THE_FIRST_STEP perconalab/pgr-57

You can also get a full script that starts “N” number of nodes, here: https://github.com/Percona-Lab/percona-docker/blob/master/pgr-57/start_node.sh

 

↧

Kerberos for SQLyog by MariaDB Connector/C

February 15, 2017, 12:06 pm
$
0
0
Kerberos for SQLyog by MariaDB Connector/C julienfritsch Wed, 02/15/2017 - 15:06

MariaDB is an open source enterprise database with one of the most active and fastest-growing communities in the world. MariaDB Enterprise delivers the security, high availability and scalability required for mission-critical applications, and the management and monitoring tools today’s enterprises rely on.

SQLyog is included in MariaDB Enterprise and it helps DBAs, developers and database architects save time writing queries visualized with syntax checking, designing visually complex queries, and many other powerful features for visualization, synchronization and management. This 12.4 release introduces ‘read-only’ connections as well as support for the MariaDB auth_gssapi (Kerberos) plugin.

Kerberos is an authentication protocol that works on the basis of 'tickets' to allow nodes communicating over a non-secure network to prove their identity to one another in a secure manner. Typically Kerberos is used within corporate/internal environments for two purposes: security and authentication – replacing the computer local passwords with tickets on a distributed network system, thereby eliminating the risk for transmitted passwords over the network to be intercepted.

The MariaDB auth_gssapi (Kerberos) plugin was available with MariaDB Connector/C 3.0 (currently beta), but based on the high demand for this, we backported it to the MariaDB Connector/C 2.3 which is used in SQLyog.

More information on Connector/C here.

Download SQLyog 12.4 from our Customer Portal here.

Want to learn more, please contact us here.

This blog post was inspired by this page.

SQLyog is included in MariaDB Enterprise and it helps DBAs, developers and database architects save time writing queries visualized with syntax checking, designing visually complex queries, and many other powerful features for visualization, synchronization and management. This 12.4 release introduces ‘read-only’ connections as well as support for the MariaDB auth_gssapi (Kerberos) plugin.

Login or Register to post comments

↧
Viewing all 18842 articles
Browse latest View live
Search
<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>