Quantcast
Channel: Planet MySQL
Viewing all 18838 articles
Browse latest View live

New PERFORMANCE_SCHEMA defaults in MySQL 5.7.7

$
0
0

I thought it was worth a moment to reiterate on the new Performance Schema related defaults that MySQL 5.7.7 brings to the table, for various reasons.

For one, most of you might have noticed that profiling was marked as deprecated in MySQL 5.6.7. So it is expected that you invest into learning more about Performance Schema (and Mark’s sys schema!).

Second, there are lots of virtual environments and appliances out there running Community Edition MySQL where Performance Schema can be a useful tool for analyzing performance. Thus, expect to see more articles about using PERFORMANCE_SCHEMA and SYS_SCHEMA from us!

Third, we have more and more junior readers who might benefit from light reads such as this. :)

The new defaults that I wanted to highlight are mentioned in the MySQL 5.7.7 release notes:
– The MySQL sys schema is now installed by default during data directory installation.
– The events_statements_history and events_transactions_history consumers now are enabled by default.

Note that if you are upgrading from an earlier version of MySQL to 5.7.7 to get these goodies you will need to run mysql_upgrade and restart the database for the above changes to take effect.

So what do these mean?

If you haven’t had a chance to dig into PERFORMANCE_SCHEMA, check out the quick start guide here. PERFORMANCE_SCHEMA is a nify tool (implemented as a union of a storage engine and a schema in MySQL) to monitor MySQL server execution at a low level, with a focus on performance metrics. It monitors for events that have been “instrumented”, such as function calls, OS wait times, synchronization calls, etc. With performance nomenclature “instruments” are essentially “probes”. The events that the instruments generate can be processed by consumers. Note that not all instruments or consumers are enabled by default.

Some would say that the structure of PERFORMANCE_SCHEMA may be complex and may not be very DBA-friendly. This is what led to the birth of SYS_SCHEMA. For those who are not familiar with Mark Leith’s SYS_SCHEMA and prefer TL;DR – it provides human friendly views, functions and procedures that can help you analyze database usage using PERFORMANCE_SCHEMA. If you haven’t had a chance to check it out yet you might want to read Miguel’s article on using the sys schema or Alexander Rubin’s article about using it in multitenant environments and give it a spin!

I welcome the fact that events_statements_history and events_transactions_history consumers are enabled by default in MySQL 5.7.7 as it means that we get some handy performance details available to us out of the box in vanilla MySQL. Note that these are per-thread tables and by default the history length (the length of the number of entries present; more on those variables here) is automatically sized, thus you may need to increase them.

What details do you get off the bat with them?

Consider the following example:

mysql> select * from performance_schema.events_statements_history where event_id=353G
*************************** 1. row ***************************
              THREAD_ID: 20
               EVENT_ID: 353
           END_EVENT_ID: 456
             EVENT_NAME: statement/sql/select
                 SOURCE: mysqld.cc:963
            TIMER_START: 1818042501405000
              TIMER_END: 1818043715449000
             TIMER_WAIT: 1214044000
              LOCK_TIME: 67000000
               SQL_TEXT: select * from imdb.title limit 100
                 DIGEST: ec93c38ab021107c2160259ddee31faa
            DIGEST_TEXT: SELECT * FROM `imdb` . `title` LIMIT ?
         CURRENT_SCHEMA: performance_schema
            OBJECT_TYPE: NULL
          OBJECT_SCHEMA: NULL
            OBJECT_NAME: NULL
  OBJECT_INSTANCE_BEGIN: NULL
            MYSQL_ERRNO: 0
      RETURNED_SQLSTATE: NULL
           MESSAGE_TEXT: NULL
                 ERRORS: 0
               WARNINGS: 0
          ROWS_AFFECTED: 0
              ROWS_SENT: 100
          ROWS_EXAMINED: 100
CREATED_TMP_DISK_TABLES: 0
     CREATED_TMP_TABLES: 0
       SELECT_FULL_JOIN: 0
 SELECT_FULL_RANGE_JOIN: 0
           SELECT_RANGE: 0
     SELECT_RANGE_CHECK: 0
            SELECT_SCAN: 1
      SORT_MERGE_PASSES: 0
             SORT_RANGE: 0
              SORT_ROWS: 0
              SORT_SCAN: 0
          NO_INDEX_USED: 1
     NO_GOOD_INDEX_USED: 0
       NESTING_EVENT_ID: NULL
     NESTING_EVENT_TYPE: NULL
1 row in set (0.00 sec)

As you can see from above you get similar data that you are used to seeing from EXPLAINs and slow query logs, such as query run time, locking time, rows sent/examined, etc. For instance, in above output my query obtained about a 100 rows (lines 26-27), avoided creating temp tables (lines 28-29) and didn’t have to sort (lines 36-38) and no index was used (line 39) and it ran for about 121 ms (TIMER_END-TIMER_START). The list of details provided is not as abundant as it could be but I imagine that with newer releases the list may grow.

If you want to read on and are curious about how to use Performance Schema for profiling check out Jervin’s great article here!

The post New PERFORMANCE_SCHEMA defaults in MySQL 5.7.7 appeared first on MySQL Performance Blog.


PlanetMySQL Voting: Vote UP / Vote DOWN

OurSQL Episode 208: Looking Back

$
0
0

In this episode we discuss our timeless podcasts. For those who are new and listening for the first time - welcome! This podcast is a great jumping off point to learn about specific topics we've covered. If you have been with us for some or all of the time, thank you for being with us, and we hope this trip down memory lane brings you nostalgia - it did for us.


PlanetMySQL Voting: Vote UP / Vote DOWN

WITH RECURSIVE and MySQL

$
0
0
If you have been using certain DBMSs, or reading recent versions of the SQL standard, you are probably aware of the so-called "WITH clause" of SQL.
Some call it Subquery Factoring. Others call it Common Table Expression.
In its simplest form, this feature is a kind of "boosted derived table".

Assume that a table T1 has three columns:

CREATE TABLE T1(
YEAR INT, # 2000, 2001, 2002 ...
MONTH INT, # January, February, ...
SALES INT # how much we sold on that month of that year
);
Now I want to know the sales trend (increase/decrease), year after year:

SELECT D1.YEAR, (CASE WHEN D1.S>D2.S THEN 'INCREASE' ELSE 'DECREASE' END) AS TREND
FROM
(SELECT YEAR, SUM(SALES) AS S FROM T1 GROUP BY YEAR) AS D1,
(SELECT YEAR, SUM(SALES) AS S FROM T1 GROUP BY YEAR) AS D2
WHERE D1.YEAR = D2.YEAR-1;
Both derived tables are based on the same subquery text, but usually a DBMS is not smart enough to recognize it. Thus, it will evaluate "SELECT YEAR, SUM(SALES)... GROUP BY YEAR" twice! A first time to fill D1, a second time to fill D2. This limitation is sometimes stated as "it's not possible to refer to a derived table twice in the same query".
Such double evaluation can lead to a serious performance problem. Using WITH, this limitation does not exist, and the following statement evaluates the subquery only once:

WITH D AS (SELECT YEAR, SUM(SALES) AS S FROM T1 GROUP BY YEAR)
SELECT D1.YEAR, (CASE WHEN D1.S>D2.S THEN 'INCREASE' ELSE 'DECREASE' END) AS TREND
FROM
D AS D1,
D AS D2
WHERE D1.YEAR = D2.YEAR-1;
This already demonstrates one benefit of WITH.
In MySQL, WITH is not yet supported. But it can be emulated with a view:

CREATE VIEW D AS (SELECT YEAR, SUM(SALES) AS S FROM T1 GROUP BY YEAR);
SELECT D1.YEAR, (CASE WHEN D1.S>D2.S THEN 'INCREASE' ELSE 'DECREASE' END) AS TREND
FROM
D AS D1,
D AS D2
WHERE D1.YEAR = D2.YEAR-1;
DROP VIEW D;
Instead of a view, I could as well create D as a normal table. But not as a temporary table, because in MySQL a temporary table cannot be referred twice in the same query, as mentioned in the manual.

After this short introduction, showing the simplest form of WITH, I would like to turn to the more complex form of WITH: the RECURSIVE form.
According to the SQL standard, to use the recursive form, you should write WITH RECURSIVE. However, looking at some other DBMSs, they seem to not require the RECURSIVE word.
WITH RECURSIVE is a powerful construct. For example, it can do the same job as Oracle's CONNECT BY clause (you can check out some example conversions between both constructs).
Let's walk through an example, to understand what WITH RECURSIVE does.

Assume you have a table of employees (this is a very classical example of WITH RECURSIVE):

CREATE TABLE EMPLOYEES (
ID INT PRIMARY KEY,
NAME VARCHAR(100),
MANAGER_ID INT,
INDEX (MANAGER_ID),
FOREIGN KEY (MANAGER_ID) REFERENCES EMPLOYEES(ID)
);
INSERT INTO EMPLOYEES VALUES
(333, "Yasmina", NULL),
(198, "John", 333),
(29, "Pedro", 198),
(4610, "Sarah", 29),
(72, "Pierre", 29),
(692, "Tarek", 333);
In other words, Yasmina is CEO, John and Tarek report to her. Pedro reports to John, Sarah and Pierre report to Pedro.
In a big company, they would be thousands of rows in this table.

Now, let's say that you would like to know, for each employee: "how many people are, directly and indirectly, reporting to him/her"? Here is how I would do it. First, I would make a list of people who are not managers: with a subquery I get the list of all managers, and using NOT IN (subquery) I get the list of all non-managers:

SELECT ID, NAME, MANAGER_ID, 0 AS REPORTS
FROM EMPLOYEES
WHERE ID NOT IN (SELECT MANAGER_ID FROM EMPLOYEES WHERE MANAGER_ID IS NOT NULL);
Then I would insert the results into a new table named EMPLOYEES_EXTENDED; EXTENDED stands for "extended with more information", the new information being the fourth column named REPORTS: it is a count of people who are reporting directly or indirectly to the employee. Because  we have listed people who are not managers, they have a value of 0 in the REPORTS column.
Then, we can produce the rows for "first level" managers (the direct managers of non-managers):

SELECT M.ID, M.NAME, M.MANAGER_ID, SUM(1+E.REPORTS) AS REPORTS
FROM EMPLOYEES M JOIN EMPLOYEES_EXTENDED E ON M.ID=E.MANAGER_ID
GROUP BY M.ID, M.NAME, M.MANAGER_ID;
Explanation: for a row of M (that is, for an employee), the JOIN will produce zero or more rows, one per non-manager directly reporting to the employee.
Each such non-manager contributes to the value of REPORTS for his manager, through two numbers: 1 (the non-manager himself), and the number of direct/indirect reports of the non-manager (i.e. the value of REPORTS for the non-manager).
Then I would empty EMPLOYEES_EXTENDED, and fill it with the rows produced just above, which describe the first level managers.
Then the same query should be run again, and it would produce information about the "second level" managers. And so on.
Finally, at one point Yasmina will be the only row of EMPLOYEES_EXTENDED, and when we run the above SELECT again, the JOIN will produce no rows, because E.MANAGER_ID will be NULL (she's the CEO). We are done.

It's time for a recap: EMPLOYEES_EXTENDED has been a kind of "temporary buffer", which has successively held non-managers, first level managers, second level managers, etc. We have used recursion. The answer to the original problem is: the union of all the successive content of EMPLOYEES_EXTENDED.
Non-managers have been the start of the recursion, which is usually called "the anchor member" or "the seed". The SELECT query which moves from one step of  recursion to the next one, is the "recursive member". The complete statement looks like this:

WITH RECURSIVE
# The temporary buffer, also used as UNION result:
EMPLOYEES_EXTENDED
AS
(
# The seed:
SELECT ID, NAME, MANAGER_ID, 0 AS REPORTS
FROM EMPLOYEES
WHERE ID NOT IN (SELECT MANAGER_ID FROM EMPLOYEES WHERE MANAGER_ID IS NOT NULL)
UNION ALL
# The recursive member:
SELECT M.ID, M.NAME, M.MANAGER_ID, SUM(1+E.REPORTS) AS REPORTS
FROM EMPLOYEES M JOIN EMPLOYEES_EXTENDED E ON M.ID=E.MANAGER_ID
GROUP BY M.ID, M.NAME, M.MANAGER_ID
)
# what we want to do with the complete result (the UNION):
SELECT * FROM EMPLOYEES_EXTENDED;
MySQL does not yet support WITH RECURSIVE, but it is possible to code a generic stored procedure which can easily emulate it. Here is how you would call it:

CALL WITH_EMULATOR(
"EMPLOYEES_EXTENDED",
"
SELECT ID, NAME, MANAGER_ID, 0 AS REPORTS
FROM EMPLOYEES
WHERE ID NOT IN (SELECT MANAGER_ID FROM EMPLOYEES WHERE MANAGER_ID IS NOT NULL)
",
"
SELECT M.ID, M.NAME, M.MANAGER_ID, SUM(1+E.REPORTS) AS REPORTS
FROM EMPLOYEES M JOIN EMPLOYEES_EXTENDED E ON M.ID=E.MANAGER_ID
GROUP BY M.ID, M.NAME, M.MANAGER_ID
",
"SELECT * FROM EMPLOYEES_EXTENDED",
0,
""
);
You can recognize, as arguments of the stored procedure, every member of the WITH standard syntax: name of the temporary buffer, query for the seed, query for the recursive member, and what to do with the complete result. The last two arguments - 0 and the empty string - are details which you can ignore for now.

Here is the result returned by this stored procedure:

+------+---------+------------+---------+
| ID | NAME | MANAGER_ID | REPORTS |
+------+---------+------------+---------+
| 72 | Pierre | 29 | 0 |
| 692 | Tarek | 333 | 0 |
| 4610 | Sarah | 29 | 0 |
| 29 | Pedro | 198 | 2 |
| 333 | Yasmina | NULL | 1 |
| 198 | John | 333 | 3 |
| 333 | Yasmina | NULL | 4 |
+------+---------+------------+---------+
7 rows in set
Notice how Pierre, Tarek and Sarah have zero reports, Pedro has two, which looks correct... However, Yasmina appears in two rows! Odd? Yes and no. Our algorithm starts from non-managers, the "leaves" of the tree (Yasmina being the root of the tree). Then our algorithm looks at first level managers, the direct parents of leaves. Then at second level managers. But Yasmina is both a first level manager (of the nonmanager Tarek) and a third level manager (of the nonmanagers Pierre, Tarek and Sarah). That's why she appears twice in the final result: once for the "tree branch" which ends at leaf Tarek, once for the tree branch which ends at leaves Pierre, Tarek and Sarah. The first tree branch contributes 1 direct/indirect report. The second tree branch contributes 4. The right number, which we want, is the sum of the two: 5. Thus we just need to change the final query, in the CALL:

CALL WITH_EMULATOR(
"EMPLOYEES_EXTENDED",
"
SELECT ID, NAME, MANAGER_ID, 0 AS REPORTS
FROM EMPLOYEES
WHERE ID NOT IN (SELECT MANAGER_ID FROM EMPLOYEES WHERE MANAGER_ID IS NOT NULL)
",
"
SELECT M.ID, M.NAME, M.MANAGER_ID, SUM(1+E.REPORTS) AS REPORTS
FROM EMPLOYEES M JOIN EMPLOYEES_EXTENDED E ON M.ID=E.MANAGER_ID
GROUP BY M.ID, M.NAME, M.MANAGER_ID
",
"
SELECT ID, NAME, MANAGER_ID, SUM(REPORTS)
FROM EMPLOYEES_EXTENDED
GROUP BY ID
, NAME, MANAGER_ID
",
0,
""
);
And here is finally the proper result:

+------+---------+------------+--------------+
| ID | NAME | MANAGER_ID | SUM(REPORTS) |
+------+---------+------------+--------------+
| 29 | Pedro | 198 | 2 |
| 72 | Pierre | 29 | 0 |
| 198 | John | 333 | 3 |
| 333 | Yasmina | NULL | 5 |
| 692 | Tarek | 333 | 0 |
| 4610 | Sarah | 29 | 0 |
+------+---------+------------+--------------+
6 rows in set
Let's finish by showing the body of the stored procedure. You will notice that it does heavy use of dynamic SQL, thanks to prepared statements. Its body does not depend on the particular problem to solve, it's reusable as-is for other WITH RECURSIVE use cases. I have added comments inside the body, so it should be self-explanatory. If it's not, feel free to drop a comment on this post, and I will explain further.

# Usage: the standard syntax:
# WITH RECURSIVE recursive_table AS
# (initial_SELECT
# UNION ALL
# recursive_SELECT)
# final_SELECT;
# should be translated by you to
# CALL WITH_EMULATOR(recursive_table, initial_SELECT, recursive_SELECT,
# final_SELECT, 0, "").

# ALGORITHM:
# 1) we have an initial table T0 (actual name is an argument
# "recursive_table"), we fill it with result of initial_SELECT.
# 2) We have a union table U, initially empty.
# 3) Loop:
# add rows of T0 to U,
# run recursive_SELECT based on T0 and put result into table T1,
# if T1 is empty
# then leave loop,
# else swap T0 and T1 (renaming) and empty T1
# 4) Drop T0, T1
# 5) Rename U to T0
# 6) run final select, send relult to client

# This is for *one* recursive table.
# It would be possible to write a SP creating multiple recursive tables.

delimiter |

CREATE PROCEDURE WITH_EMULATOR(
recursive_table varchar(100), # name of recursive table
initial_SELECT varchar(65530), # seed a.k.a. anchor
recursive_SELECT varchar(65530), # recursive member
final_SELECT varchar(65530), # final SELECT on UNION result
max_recursion int unsigned, # safety against infinite loop, use 0 for default
create_table_options varchar(65530) # you can add CREATE-TABLE-time options
# to your recursive_table, to speed up initial/recursive/final SELECTs; example:
# "(KEY(some_column)) ENGINE=MEMORY"
)

BEGIN
declare new_rows int unsigned;
declare show_progress int default 0; # set to 1 to trace/debug execution
declare recursive_table_next varchar(120);
declare recursive_table_union varchar(120);
declare recursive_table_tmp varchar(120);
set recursive_table_next = concat(recursive_table, "_next");
set recursive_table_union = concat(recursive_table, "_union");
set recursive_table_tmp = concat(recursive_table, "_tmp");
# If you need to reference recursive_table more than
# once in recursive_SELECT, remove the TEMPORARY word.
SET @str = # create and fill T0
CONCAT("CREATE TEMPORARY TABLE ", recursive_table, " ",
create_table_options, " AS ", initial_SELECT);
PREPARE stmt FROM @str;
EXECUTE stmt;
SET @str = # create U
CONCAT("CREATE TEMPORARY TABLE ", recursive_table_union, " LIKE ", recursive_table);
PREPARE stmt FROM @str;
EXECUTE stmt;
SET @str = # create T1
CONCAT("CREATE TEMPORARY TABLE ", recursive_table_next, " LIKE ", recursive_table);
PREPARE stmt FROM @str;
EXECUTE stmt;
if max_recursion = 0 then
set max_recursion = 100; # a default to protect the innocent
end if;
recursion: repeat
# add T0 to U (this is always UNION ALL)
SET @str =
CONCAT("INSERT INTO ", recursive_table_union, " SELECT * FROM ", recursive_table);
PREPARE stmt FROM @str;
EXECUTE stmt;
# we are done if max depth reached
set max_recursion = max_recursion - 1;
if not max_recursion then
if show_progress then
select concat("max recursion exceeded");
end if;
leave recursion;
end if;
# fill T1 by applying the recursive SELECT on T0
SET @str =
CONCAT("INSERT INTO ", recursive_table_next, " ", recursive_SELECT);
PREPARE stmt FROM @str;
EXECUTE stmt;
# we are done if no rows in T1
select row_count() into new_rows;
if show_progress then
select concat(new_rows, " new rows found");
end if;
if not new_rows then
leave recursion;
end if;
# Prepare next iteration:
# T1 becomes T0, to be the source of next run of recursive_SELECT,
# T0 is recycled to be T1.
SET @str =
CONCAT("ALTER TABLE ", recursive_table, " RENAME ", recursive_table_tmp);
PREPARE stmt FROM @str;
EXECUTE stmt;
# we use ALTER TABLE RENAME because RENAME TABLE does not support temp tables
SET @str =
CONCAT("ALTER TABLE ", recursive_table_next, " RENAME ", recursive_table);
PREPARE stmt FROM @str;
EXECUTE stmt;
SET @str =
CONCAT("ALTER TABLE ", recursive_table_tmp, " RENAME ", recursive_table_next);
PREPARE stmt FROM @str;
EXECUTE stmt;
# empty T1
SET @str =
CONCAT("TRUNCATE TABLE ", recursive_table_next);
PREPARE stmt FROM @str;
EXECUTE stmt;
until 0 end repeat;
# eliminate T0 and T1
SET @str =
CONCAT("DROP TEMPORARY TABLE ", recursive_table_next, ", ", recursive_table);
PREPARE stmt FROM @str;
EXECUTE stmt;
# Final (output) SELECT uses recursive_table name
SET @str =
CONCAT("ALTER TABLE ", recursive_table_union, " RENAME ", recursive_table);
PREPARE stmt FROM @str;
EXECUTE stmt;
# Run final SELECT on UNION
SET @str = final_SELECT;
PREPARE stmt FROM @str;
EXECUTE stmt;
# No temporary tables may survive:
SET @str =
CONCAT("DROP TEMPORARY TABLE ", recursive_table);
PREPARE stmt FROM @str;
EXECUTE stmt;
# We are done :-)
END|

delimiter ;
In the SQL Standard, WITH RECURSIVE allows some nice additional tweaks (depth-first or breadth-first ordering, cycle detection). In future posts I will show how to emulate them too.

PlanetMySQL Voting: Vote UP / Vote DOWN

MariaDB 10.1.4 Overview and Highlights

$
0
0

MariaDB 10.1.4 was recently released, and is available for download here:

https://downloads.mariadb.org/mariadb/10.1.4/

This is the 2nd beta, and 5th overall, release of MariaDB 10.1. Now that it is beta, there were not as many major changes in this release (compared to 10.1.3), but there were a few notable items as well as many overall bugs fixed (I counted 367).

Since it’s beta, I’ll only cover the major changes and additions, and omit covering general bug fixes (feel free to browse them all here).

To me, these are the highlights:

Of course it goes without saying that do not use this for production systems as it is only the 2nd beta release of 10.1. However, I definitely recommend installing it on a test server and testing it out. And if you happen to be running a previous version of 10.1, then you should definitely upgrade to this latest release.

You can read more about the 10.1.4 release here:

https://mariadb.com/kb/en/mariadb-1014-release-notes/

And if interested, you can review the full list of changes in 10.1.4 (changelogs) here:

https://mariadb.com/kb/en/mariadb-1014-changelog/

Hope this helps.


PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL 5.5.44 Overview and Highlights

$
0
0

MySQL 5.5.44 was recently released (it is the latest MySQL 5.5, is GA), and is available for download here:

http://dev.mysql.com/downloads/mysql/5.5.html

This release, similar to the last 5.5 release, is mostly uneventful.

There were 0 “Functionality Added or Changed” items this time, and just 15 overall bugs fixed.

Out of the 15 bugs, there were 5 InnoDB bugs (1 of which also spans partitioning), 1 security-related bug, 1 performance-related, and 3 additional potential crashing bugs. Here are the ones worth noting:

  • InnoDB: An assertion was raised on shutdown due to XA PREPARE transactions holding explicit locks.
  • InnoDB: Removal of a foreign key object from the data dictionary cache during error handling caused the server to exit.
  • InnoDB: SHOW ENGINE INNODB STATUS output showed negative reservation and signal count values due to a counter overflow error.
  • InnoDB: Estimates that were too low for the size of merge chunks in the result sorting algorithm caused a server exit.
  • InnoDB; Partitioning: The CREATE_TIME column of the INFORMATION_SCHEMA.TABLES table now shows the correct table creation time for partitioned InnoDB tables. The CREATE_TIME column of the INFORMATION_SCHEMA.PARTITIONS table now shows the correct partition creation time for a partition of partitioned InnoDB tables. The UPDATE_TIME column of the INFORMATION_SCHEMA.TABLES table now shows when a partitioned InnoDB table was last updated by an INSERT, DELETE, or UPDATE. The UPDATE_TIME column of the INFORMATION_SCHEMA.PARTITIONS table now shows when a partition of a partitioned InnoDB table was last updated. (Bug #69990)
  • Security-related: A user with a name of event_scheduler could view the Event Scheduler process list without the PROCESS privilege.
  • Performance-related: Certain queries for the INFORMATION_SCHEMA TABLES and COLUMNS tables could lead to excessive memory use when there were large numbers of empty InnoDB tables. (Bug #72322)
  • Crashing Bug: SHOW VARIABLES mutexes were being locked twice, resulting in a server exit.
  • Crashing Bug: Under certain conditions, the libedit command-line library could write outside an array boundary and cause a client program crash.
  • Crashing Bug: For a prepared statement with an ORDER BY that refers by column number to a GROUP_CONCAT() expression that has an outer reference, repeated statement execution could cause a server exit.

I don’t think I’d call any of these urgent for all, but if running 5.5, especially if not a very recent 5.5, you should consider upgrading.

For reference, the full 5.5.44 changelog can be viewed here:

http://dev.mysql.com/doc/relnotes/mysql/5.5/en/news-5-5-44.html

Hope this helps.


PlanetMySQL Voting: Vote UP / Vote DOWN

Reporting Across Shards

$
0
0
If you have chosen to split your data across boxes, and architected your app to not query across boxes there is still a case where you will need to. Data mining, reports and data health checks require hitting all servers at some point. The case I am going over is sessions and figuring out the Session Length without taking averages of averages which is wrong.


Let's assume you have a session table of the following

mysql> describe sessions;
+----------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+---------------------+------+-----+---------+-------+
| user_id | bigint(20) unsigned | NO | PRI | 0 | |
| added_ms | bigint(20) unsigned | NO | PRI | 0 | |
| appVer | varchar(8) | YES | | NULL | |
| device | bigint(20) unsigned | YES | MUL | NULL | |
| start | int(10) unsigned | NO | MUL | NULL | |
| stop | int(10) unsigned | NO | | NULL | |
+----------+---------------------+------+-----+---------+-------+




The data is federated (distributed) by user_id. This table exists across 1000s of servers. How do you get the average session length for the month of May?






  • The question already scopes the process to hit every single server

  • Second we can't just take AVG((stop-start)) and then sum and divide that by the number of shards

  • We can't pull all the data in memory

  • We don't want to have to pull the data and upload it to BigQuery or Amazon RedShift

  • We want a daily report at some point








SELECT SUM((stop-start)) as sess_diff, count(*) as sess_sample FROM sessions WHERE start BETWEEN $start AND $stop AND stop>start





The above SQL statement says for the connection to a single server give me the sum of the session delta and count the corresponding rows in the set. In this case the SUM of SUMs (sum of session_delta) is the numerator and the sum of sess_sample is the denominator.





Now do this across all servers and finally write some client code to take a few rows < 1000 to report the number.









$total = 0;
$sessions_diff = 0;

foreach ($rows as $shard_id => $result) {

$sessions_diff = \bcadd($sessions_diff, $result[0]['sess_diff']);
$total = \bcadd($total, $result[0]['sess_sample']);
}


Now the session_avg = sessions_diff/total

Tada a query that can take hours if done on a traditional mining server is done in ms.



PlanetMySQL Voting: Vote UP / Vote DOWN

Setup and configure MySQL backup using Holland and Xtrabackup

$
0
0
Setting up a database backup is a primary task for database administrators and we see perl and shell scripts wrapped around few of the backup-tools in practice. With right tools things can look easy and today we shall look into one of that! MySQL Database backup can be done with mysqldump, mysqlhotbackup, xtrabackup, lvm / […]
PlanetMySQL Voting: Vote UP / Vote DOWN

Is 80% of RAM how you should tune your innodb_buffer_pool_size?

$
0
0

It seems these days if anyone knows anything about tuning InnoDB, it’s that you MUST tune your innodb_buffer_pool_size to 80% of your physical memory. This is such prolific tuning advice, it seems engrained in many a DBA’s mind.  The MySQL manual to this day refers to this rule, so who can blame the DBA?  The question is: does it makes sense?

What uses the memory on your server?

Before we question such advice, let’s consider what can take up RAM in a typical MySQL server in their broad categories.  This list isn’t necessarily complete, but I think it outlines the large areas a MySQL server could consume memory.

  • OS Usage: Kernel, running processes, filesystem cache, etc.
  • MySQL fixed usage: query cache, InnoDB buffer pool size, mysqld rss, etc.
  • MySQL workload based usage: connections, per-query buffers (join buffer, sort buffer, etc.)
  • MySQL replication usage:  binary log cache, replication connections, Galera gcache and cert index, etc.
  • Any other services on the same server: Web server, caching server, cronjobs, etc.

There’s no question that for tuning InnoDB, the innodb_buffer_pool_size is the most important variable.  It’s expected to occupy most of the RAM on a dedicated MySQL/Innodb server, but of course other local services may affect how it is tuned.  If it (and other memory consumption on the server) is too large, swapping can kick in and degrade your performance rapidly.

Further, the workload of the MySQL server itself may cause a lot of variation.  Does the server have a lot of open connections and active query workload consuming memory?  The memory consumption caused by this can be dramatically different server to server.

Finally, replication mechanisms like Galera have their own memory usage pattern and can require some adjustments to your buffer pool.

We can see clearly that the 80% rule isn’t as nuanced as reality.

A rule of thumb

However, for the sake of argument, let’s say the 80% rule is a starting point.  A rule of thumb to help us get a quick tuning number to get the server running.  Assuming we don’t know anything really about the workload on the system yet, but we know that the system is dedicated to InnoDB, how might our 80% rule play out?

Total Server RAMBuffer pool with 80% ruleRemaining RAM
1G800MB200MB
16G13G3G
32G26G6G
64G51G13G
128G102G26G
256G205G51G
512G409G103G
1024G819G205G

At lower numbers, our 80% rule looks pretty reasonable.  However, as we get into large servers, it starts to seem less sane.  For the rule to hold true, it must mean that workload memory consumption increases in proportion to needed size of the buffer pool, but that usually isn’t the case.  Our server that has 1TB of RAM likely doesn’t need 205G of that to handle things like connections and queries (likely MySQL couldn’t handle that many active connections and queries anyway).

So, if you really just spent all that money on a beefy server do you really want to pay a 20% tax on that resource because of this rule of thumb?

The origins of the rule

At one of my first MySQL conferences, probably around 2006-2007 when I worked at Yahoo, I attended an InnoDB tuning talk hosted by Heikki Tuuri (the original author of InnoDB) and Peter Zaitsev.  I distinctly remember asking about the 80% rule because at the time Yahoo had some beefy 64G servers and the rule wasn’t sitting right with me.

Heikki’s answer stuck with me.  He said something to the effect of (not a direct quote): “Well, the server I was testing on had 1GB of RAM and 80% seemed about right”.  He then, if memory serves, clarified it and said it would not apply similarly to larger servers.

How should you tune?

80% is maybe a great start and rule of thumb.  You do want to be sure the server has plenty of free RAM for the OS and the usually unknown workload.  However, as we can see above, the larger the server, the more likely the rule will wind up wasting RAM.   I think for most people it starts and ends at the rule of thumb, mostly because changing the InnoDB buffer pool requires a restart in current releases.

So what’s a better rule of thumb?  My rule is that you tune the innodb_buffer_pool_size as large as possible without using swap when the system is running the production workload.  This sounds good in principle, but again, it requires a bunch of restarts and may be easier said than done.

Fortunately MySQL 5.7 and it’s online buffer pool resize feature should make this an easier principle to follow.  Seeing lots of free RAM (and/or filesystem cache usage)?  Turn the buffer pool up dynamically.  Seeing some swap activity?  Just turn it down with no restart required.   In practice, I suspect there will be some performance related hiccups of using this feature, but it is at least a big step in the right direction.

The post Is 80% of RAM how you should tune your innodb_buffer_pool_size? appeared first on MySQL Performance Blog.


PlanetMySQL Voting: Vote UP / Vote DOWN

Using Perl and MySQL to Automatically Respond to Retweets on Twitter

$
0
0

In my previous post, I showed you a way to store tweets in MySQL, and then use Perl to automatically publish them on Twitter.

In this post, we will look at automatically sending a “thank you” to people who retweet your tweets — and we will be using Perl and MySQL again.

Just like in the first post, you will need to register your application with Twitter via apps.twitter.com, and obtain the following:

consumer_key
consumer_secret
access_token
access_token_secret

One caveat: Twitter has a rate limit on how often you may connect with your application — depending upon what you are trying to do. See the Rate Limiting and Rate Limits API docs for more information. What this means is that if you are going to put this into a cron job (or use some other automated scheduler), I wouldn’t run it more than once every 15 minutes or so.

We will also be using the same tables we created in the first posttweets and history — as well as a new table named retweets. The retweets table will contain all of the user names and tweet ID’s for those retweets we have discovered and for which we’ve already sent a “thank you” tweet response.

The Perl script will connect to your tweet history table and retrieve a set of your tweet ID’s, with the most recent tweet first. The script will then connect to Twitter and check to see if there are any retweets for each given ID. If an existing retweet is found, then the script will check your retweets table to see if you have already thanked the user for their retweet. If this is a new retweet, then the script will connect to Twitter, send the “thank you” message to that user, and finally insert the user name and tweet ID combination into the retweets table. This will ensure that you do not send repeat “thank you” responses.

Here is a flow chart to help explain what the script does:

We will be using the retweets(id) API call to see if a tweet ID was retweeted, and then we will send the “thank you” tweet via the update API call. You can find more information about the Perl Twitter module at Net::Twitter::Lite::WithAPIv1_1.

First we will need to create the retweets table, where we will store the information about our tweets that were retweeted. Here is the CREATE TABLE statement for the retweets table:

CREATE TABLE `retweets` (
  `id` int(8) NOT NULL AUTO_INCREMENT,
  `tweet_id` bigint(24) DEFAULT NULL,
  `user_name` varchar(24) DEFAULT NULL,
  `retweet_update` varchar(36) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1

Then you will need to edit the below script and insert the consumer_key, consumer_secret, access_token, and access_token_secret values for your application (which you get when registering your app with Twitter), and edit the accessTweets file (more on that shortly) used by the ConnectToMySql subroutine. (You may also want to comment-out the debug style “print” calls).

#!/usr/bin/perl
 
use Net::Twitter::Lite::WithAPIv1_1;
use DBI;
use DBD::mysql;

my $Database = "tweets";

# Credentials for your twitter application
# you will need to subsitute your own application information for these four variables
my $nt = Net::Twitter::Lite::WithAPIv1_1->new(
      traits              => [qw/API::RESTv1_1/],
      consumer_key        => "$consumer_key",
      consumer_secret     => "$consumer_secret",
      access_token        => "$access_token",
      access_token_secret => "$access_token_secret",
      ssl                 => 1
);

# Grab the last X number of tweets to check for retweets
# - determined by the number after "limit"

$dbh = ConnectToMySql($Database);
$query = "select tweet_id, tweet_update FROM history order by tweet_update desc, id limit 10";	
$sth = $dbh->prepare($query);
$sth->execute();

# loop through our results - one tweet at a time
while (@data = $sth->fetchrow_array()) {

	$tweet_id = $data[0];
	$tweet_update = $data[1];

	print "----------------------------------------------------------------------\n";
	print "Checking:  $tweet_id $tweet_update\n";
	print "----------------------------------------------------------------------\n";

		# Connect to twitter and see if anyone retweeted this tweet
		my $results = eval { $nt->retweets($tweet_id)};

		for my $status ( @$results ) {
       
			$user_name = "$status->{user}{screen_name}";
			$retweet_update = "$status->{created_at}";

			# see if this person has retweeted this before, and we already
			# have a record of the retweet in our database
					
			$dbh2 = ConnectToMySql($Database);
			$query2 = "select tweet_id, user_name FROM retweets where tweet_id = '$tweet_id' and user_name = '$user_name' limit 1";	
			$sth2 = $dbh2->prepare($query2);
			$sth2->execute();
    
			@data2 = $sth2->fetchrow_array();
    
			# Uncomment if you want to see it in action
			# print "Query: $query\n";
		
			# Check to see if we had any results, and if not, then insert
			# tweet into database and send them a "thank you" tweet
			if (length($data2[0]) prepare($query3);
				$sth3->execute();

				# Uncomment if you want to see it in action
				# print "Query2: $query2\n";


				# ----------------------------------------------------------------------------
				# send tweet
				# ----------------------------------------------------------------------------

				# This pause is just to slow down the action - you can remove this line if you want
				sleep 5;
				
				my $nt = Net::Twitter::Lite::WithAPIv1_1->new(
					traits              => [qw/API::RESTv1_1/],
					consumer_key        => "$consumer_key",
					consumer_secret     => "$consumer_secret",
					access_token        => "$access_token",
					access_token_secret => "$access_token_secret",
					ssl                 => 1
					);
					
					# Here is the message you want to send - 
					# the thank you to the user who sent the retweet
					$tweet = "\@$user_name thanks for the retweet!";

					# send thank you tweet
					my $results = eval { $nt->update("$tweet") };

						undef @data2;
						undef @data3;
					}
		
					else
				
					{
  						# we have already thanked this user - as their name and this tweet-id was found in the database
    					print "----- Found tweet: $tweet_id\n";
    
						while (@data2) {

							print "----------------------------------------------------------------------\n";
							print "Checking retweet by $user_name for $tweet_id\n";
							print "Found retweet:  $tweet_id $user_name $retweet_update \n";

							$tweet_id = $data2[0];
							$user_name = $data2[1];
					
							print "***** Retweet by $user_name already in database \n";
							print "----------------------------------------------------------------------\n";
					
							#exit;
							undef @data2;
							undef @data3;

							# This pause is just to slow down the action - you can remove this line if you want
							sleep 5;

							# end while
							}
						
					# end else
					}

		# end for my $status ( @$results ) {
		}

# end while
}

exit;

#----------------------------------------------------------------------
sub ConnectToMySql {
#----------------------------------------------------------------------

   my ($db) = @_;

   open(PW, "connect($connectionInfo,$userid,$passwd);
   return $l_dbh;

}

In the ConnectToMySql subroutine, I store the MySQL login credentials in a text file one directory below where my Perl script is located. This accessTweets file should contain the following information:




I tested this on two Twitter accounts and everything worked well for me, but please let me know if you have any problems. Please keep in mind, however, that I am not the best Perl programmer, nor am I an expert on the Twitter API, so there are likely better/easier way to do this.

That’s it for now. I hope that this was interesting and may prove helpful. THANK YOU for using MySQL!

 


Tony Darnell is a Principal Sales Consultant for MySQL, a division of Oracle, Inc. MySQL is the world’s most popular open-source database program. Tony may be reached at info [at] ScriptingMySQL.com and on LinkedIn.

PlanetMySQL Voting: Vote UP / Vote DOWN

VividCortex Announces Database Monitoring for MongoDB

$
0
0

VividCortex, the monitoring solution for the modern data system, now supports MongoDB in addition to MySQL, PostgreSQL and Redis.

Charlottesville, Virginia (PRWEB) June 01, 2015 – VividCortex, the monitoring solution for the modern data system, announces immediate availability of database monitoring for MongoDB. Earlier this year, VividCortex expanded its cloud-based tool from MySQL to PostgreSQL and Redis. This is another step toward providing the most robust monitoring solution for today’s diverse, distributed systems as data grows exponentially.

MongoDB is a rapidly growing NoSQL database that is developer-friendly and designed to scale. VividCortex applies those same attributes to database monitoring. Since launching in April of 2014, VividCortex has taken significant strides to give users unprecedented, comprehensive insight into system performance, enabling IT teams across verticals to run at increased efficiency. The results are improved time to market, decreased downtime, and reduced infrastructure costs. Josh Prunier, DBA at NetProspex, puts it simply: “VividCortex is a database performance monitoring solution for enterprise systems. It makes a DBA’s and developers’ job easier.”

The unique technical approach of VividCortex includes:

  • Query insight at microsecond detail, including CPU, I/O disk and network consumption
  • The ability to view and compare database and OS performance and key events over time
  • 1-second, high-resolution data measures vital metrics that directly affect system performance
  • Actionable intelligence through a visually streamlined interface that is easy-to-use
  • Less than one percent overhead by providing lightweight agent software and performing the computationally-intensive analysis and correlation in the VividCortex cloud

VividCortex was recently recognized for its contribution to the MySQL community and plans to have a similar impact within the MongoDB ecosystem. Even during the beta period, VividCortex’s MongoDB customers were impressed, as Chris Clark, DevOps engineer at Parse.ly states: “VividCortex’s highly granular data, including detailed information about each query made against our MongoDB clusters, is invaluable. Installation was a breeze, and the system immediately spotted two problematic queries that had evaded all our other monitoring systems.”

Pricing and Availability

A free trial of VividCortex for MongoDB is available through a simple signup. Installation takes minutes and requires no server restarts or configuration changes. Users gain access to an intuitive interface and responsive support staff within 90 seconds.

About VividCortex

Database management software is at the core of IT systems, but often operates as a black box. VividCortex has created the first Database Performance Monitoring software, a comprehensive tool designed specifically to provide actionable insight and a high definition window into the inner workings of databases with unprecedented detail, accuracy, and ease-of-use. Read our blog at VividCortex.com/blog/.

PRWeb Press Release


PlanetMySQL Voting: Vote UP / Vote DOWN

JSON Support in PostgreSQL, MySQL, MongoDB, and SQL Server

$
0
0

Unless you’ve been hiding under a rock for a few years, you probably know that JSON is quickly gaining support in major database servers. Due to its use in the web front-end, JSON has overtaken XML in APIs, and it’s spreading through all the layers in the stack one step at a time.

Most major databases have supported XML in some fashion for a while, too, but developer uptake wasn’t universal. JSON adoption amongst developers is nearly universal today, however. (The king is dead, long live the king!) But how good is JSON support in the databases we know and love? We’ll do a comparison in this blog post.

What is JSON Support?

First – what does it even mean for a database to support JSON?

You could easily argue that JSON support has been in databases for a long time. After all, all you have to do is store a blob of text that is correctly formatted in the database, and applications can store, retrieve, and manipulated as usual. But the level of JSON support that we are seeing in databases today exceeds that. JSON is becoming natively supported as a datatype, with functions and operators and all of the decoration that goes along with it.

For the purposes of this article, we don’t insist upon JSON being a natively supported data type that is distinct from other datatypes. It is enough for JSON to be stored as a blob, and manipulated with functions. However, as you will see, most databases go well beyond that.

JSON Support in MySQL

MySQL is late to the party. For a while now, developers, including VividCortex, have been storing JSON in blobs in MySQL, and either manipulating it in the application, or using see compiled functions or stored procedures to manipulate it.

This has always been sub optimal. MySQL, and particularly InnoDB, are not great at storing and manipulating blob data. It can be stored inefficiently, with a lot of overhead. Native compression can be tricky to get right, so JSON blobs can take up a lot of space. And a lot of the querying and string manipulation functions in MySQL don’t do very well with UTF-8 text in some cases.

In the upcoming MySQL 5.7 version, however, this all changes. In MySQL 5.7, JSON is a natively supported data type. Internally, JSON will actually be stored as BSON. This is a compact binary format that can be translated to and from the JSON text format efficiently and easily.

In addition, MySQL 5.7 will have a set of JSON functions for manipulating JSON columns directly. One of the weirdnesses of this is that the functions are named with a JSN_ prefix. This seems a little odd, but as Morgan Tocker, community relations manager for MySQL, explained to me in an email, it is to avoid any namespace conflicts if JSON functions become an official part of the SQL standard.

Finally, MySQL 5.7 will support indexing values buried inside JSON documents by using virtual columns, another new feature included in the 5.7 release candidate.

In older versions of MySQL, a set of UDF functions has been available for quite a while for the community to install into their databases and manipulate JSON. There are also community UDFs in the MySQL UDF repository.

JSON Support in PostgreSQL

JSON has been supported for a while longer in Postgres. I have not used it personally, but friends of mine have been talking to me about it at least since the 9.2 release. In this release, JSON was a natively supported datatype, and there were a couple of functions available for querying and manipulating it.

In the 9.3 release, support was greatly extended. In addition to the functions, there are a half a dozen operators. Personally, I find these operators hard to read, and I don’t think they add much to SQL. I think they would be better off as functions. But that’s just my personal preference.

In addition to the operators, there are 10 or so new functions too.

In version 9.4, the JSON datatype is supplemented by a JSONB datatype. Another half a dozen operators come along with this, further making the SQL look like Perl. There are also a couple of dozen JSONB functions, one for each corresponding JSON function.

Perhaps one of the biggest advantages of JSONB over plaintext JSON is that it supports indexing. You can create functional indexes in Postgres, of course, so you can still index plaintext JSON; but it’s better and more efficient to index JSONB than JSON.

JSON Support in MongoDB

It may seem a little silly to include this section in this blog post, especially if you are familiar with MongoDB. However, even if it’s a little obvious, we don’t want to leave it unstated. And there are non-obvious things, too.

It’s very simple: in MongoDB, JSON is the native datatype, and JavaScript is used to access it natively. In reality, it is stored as JSONB, not plaintext JSON.

However, until recent versions of MongoDB, the data was stored with no compression. The recommended way of mitigating this problem was to use short field names, which wasn’t a very good alternative, frankly. In MongoDB version 3.0, the new storage engine includes compression by default, making the situation much improved. You can also use Percona’s TokuMX, which includes a storage engine with high-performance, transactions, and native compression.

Indexing is natively supported as well, naturally. This has been included since the first MongoDB releases. However, indexing has improved a lot over time, including the addition of sparse indexes (since not all documents will contain all fields you are indexing).

JSON Support in SQL Server

JSON support in Microsoft SQL server will be coming in the 2016 release of the product. However, in contrast to the other databases we have discussed, there will be no native datatype. Instead, the functionality will be quite similar to the native XML functionality that has existed in SQL Server for a long time. If you are familiar with SQL Server, you will immediately recognize the OPENJSON() function. It operates on NVARCHAR values.

Indexing will also be rather limited. No native JSON indexes; just fulltext indexing.

You can read more details on the MSDN blog.

Conclusion

One of the big reasons that people are interested in JSON support in databases is that they want to use fewer types of databases. The modern technology stack is beginning to introduce significant sprawl as people use different databases in particular areas, taking advantage of their strengths to gain efficiency. However, this polyglot persistence increases the technology surface area enough that it can become quite difficult to monitor, manage, develop, and operate such a diverse set of databases.

One potential answer to this problem is to continue using work horses such as MySQL and Postgres, replacing specialized JSON native databases with the new JSON functionality. For the record, MongoDB is not the only JSON native database. There are many databases, including RethinkDB, that deal with JSON natively.

A big difference between the JSON support in MySQL, Postgres, and MongoDB is that in MongoDB, this is the native transport across the wire as well. JSON is native end to end in this database, whereas in other databases it is typically shoehorned into row and column storage in the client and on the wire, as well as the programming API.

Still, keeping technology diversity down can be a big enough reason to continue using the reliable and trusted databases of yore.


PlanetMySQL Voting: Vote UP / Vote DOWN

Protect Your Data: Row-level Security in MariaDB 10.0

$
0
0
Tue, 2015-06-02 19:07
geoff_montee_g

Most MariaDB users are probably aware of the privilege system available in MariaDB and MySQL. Privileges control what databases, tables, columns, procedures, and functions a particular user account can access. For example, if an application stored credit card data in the database, this data should probably be protected from most users. To make that happen, the DBA might disallow access to the table or column storing this sensitive data.

However, sometimes the privilege system isn't sufficient to secure data. Sometimes data needs to be secured beyond tables and columns. In those cases, row-level security (sometimes abbreviated RLS) may be necessary. Possible use cases for row-level security are:

  • A government agency might only allow a user to see a row based on classification (CONFIDENTIAL, SECRET, TOP SECRET) and other factors.
  • An e-commerce site storing credit card information might only allow users to see the credit cards tied to their account.
  • A hospital or clinic might only allow staff to see records for patients that they are authorized to see.
  • A regional store manager might only be able to see employment records for employees and inventory records for stores in their region.

MariaDB's privilege system does not support row-level privileges, so developers and DBAs need to find another way to implement row-level security.

Sometimes, the row-level security logic is taken care of by the application. Other times, it can be more effective or better design to put the row-level security logic into the database. For example, if multiple applications use the same database, it might be better for the database to handle security. That way, the security functionality only has to be designed once, and it works the same for every application.

In this blog post, I will show a very simple way to implement row-level security in MariaDB 10.0 using the following features:

Of course, this is just a simple example. This is not the only way to implement row-level security in MariaDB.

Security Labels and Policies

To implement row-level security, you need two things:

  • Some way to label the data. This might be the name of the owner of the data, or a classification level (CONFIDENTIAL, SECRET, TOP SECRET), or it might be something else entirely.
  • Some rules or policies that outline which users can see data labelled with each security label.

Real world security labels and policies can be very complicated. There might be a hierarchical system of labels, or there might be several groups of labels that contribute different authorization information to the policy.

In this example, we will use a very simple labelling system. Data will be labelled using colors. For a user to access data labelled with the red security label, the user needs to be granted access to the red security label. For the user to access data labelled blue, the user needs to be granted access to the blue security label. The labels of each color work exactly the same way.

Now, let's start creating the database objects.

First, let's create a database to store access information.

CREATE DATABASE accesses;

Second, let's store the possible security labels. Bitstrings can be a good way to efficiently store a lot of security labels. Each label is assigned a bit field, and then bitwise operations can be used to get/set individual labels from the bitstring.

We will use bitstrings to store the labels that a user can access, so let's also store the bit field of the label in a BIT column.

CREATE TABLE accesses.security_labels (
	id INT AUTO_INCREMENT PRIMARY KEY,
	security_label VARCHAR(50),
	label_value BIT(5)
);

INSERT INTO accesses.security_labels (security_label, label_value) VALUES
	('red', b'00001'),
	('blue', b'00010'),
	('green', b'00100'),
	('yellow', b'01000'),
	('purple', b'10000');

Third, let's store the actual access levels for the user accounts.

CREATE TABLE accesses.user_accesses (
	id INT AUTO_INCREMENT PRIMARY KEY,
	user VARCHAR(50),
	access_label_values BIT(5)
);

INSERT INTO accesses.user_accesses (user, access_label_values) VALUES
	('root@localhost', b'11111'),
	('alice@localhost', b'00011'),
	('bob@localhost', b'11100'),
	('trudy@localhost', b'00000');

Fourth, let's create a stored function to represent our row-level security policy.

The function takes user name X and security label Y, and it returns true if the user is allowed to access the label. Notice that the function uses the bitwise AND (&) operator to get the individual label's bit field from the bitstring column.

DELIMITER //

CREATE FUNCTION accesses.access_check (v_user VARCHAR(50), v_security_label VARCHAR(50)) 
RETURNS BOOLEAN
NOT DETERMINISTIC
READS SQL DATA
SQL SECURITY INVOKER
BEGIN
	SELECT label_value INTO @v_label_value
	FROM accesses.security_labels
	WHERE security_label = v_security_label;
	
	SELECT @v_label_value & access_label_values INTO @v_label_check
	FROM accesses.user_accesses 
	WHERE user = v_user;

	IF @v_label_check = @v_label_value THEN
		RETURN true;
	ELSE
		RETURN false;
	END IF;
END
//

DELIMITER ;

Now, let's test out the function with a few user and label combinations.

MariaDB [(none)]> SELECT accesses.access_check('alice@localhost', 'red');
+-------------------------------------------------+
| accesses.access_check('alice@localhost', 'red') |
+-------------------------------------------------+
|                                               1 |
+-------------------------------------------------+
1 row in set (0.00 sec)

MariaDB [(none)]> SELECT accesses.access_check('alice@localhost', 'blue');
+--------------------------------------------------+
| accesses.access_check('alice@localhost', 'blue') |
+--------------------------------------------------+
|                                                1 |
+--------------------------------------------------+
1 row in set (0.00 sec)

MariaDB [(none)]> SELECT accesses.access_check('alice@localhost', 'green');
+---------------------------------------------------+
| accesses.access_check('alice@localhost', 'green') |
+---------------------------------------------------+
|                                                 0 |
+---------------------------------------------------+
1 row in set (0.00 sec)

MariaDB [(none)]> SELECT accesses.access_check('bob@localhost', 'red');
+-----------------------------------------------+
| accesses.access_check('bob@localhost', 'red') |
+-----------------------------------------------+
|                                             0 |
+-----------------------------------------------+
1 row in set (0.00 sec)

MariaDB [(none)]> SELECT accesses.access_check('bob@localhost', 'blue');
+------------------------------------------------+
| accesses.access_check('bob@localhost', 'blue') |
+------------------------------------------------+
|                                              0 |
+------------------------------------------------+
1 row in set (0.00 sec)

MariaDB [(none)]> SELECT accesses.access_check('bob@localhost', 'green');
+-------------------------------------------------+
| accesses.access_check('bob@localhost', 'green') |
+-------------------------------------------------+
|                                               1 |
+-------------------------------------------------+
1 row in set (0.00 sec)

Protecting the Data

Now that the user accounts' accesses are set up, let's set up some data to protect.

First, let's create a regular table with some labeled data.

CREATE DATABASE unprotected;

CREATE TABLE unprotected.important_data (
	id INT AUTO_INCREMENT PRIMARY KEY,
	data VARCHAR(50),
	security_label VARCHAR(50)
);

INSERT INTO unprotected.important_data (data, security_label) VALUES
	('correct', 'red'),
	('horse', 'blue'),
	('battery', 'green'),
	('stapler', 'yellow'),
	('correcter', 'purple');

Second, let's create a view that queries the unprotected table in a secure manner.

CREATE DATABASE protected;

CREATE 
SQL SECURITY DEFINER
VIEW protected.important_data
AS
	SELECT *
	FROM unprotected.important_data uid
	WHERE accesses.access_check(SESSION_USER(), uid.security_label)
WITH CHECK OPTION;

Some things to note here:

  • The protected.important_data view queries the unprotected.important_data table.
  • The view adds a WHERE clause that filters the results based on the accesses of SESSION_USER().
  • SESSION_USER() has to be used, rather than CURRENT_USER(), since the view is defined with SQL SECURITY DEFINER.
  • SQL SECURITY DEFINER has to be used, since the view's invoker (i.e. a normal user) usually won't have privileges to directly access the unprotected.important_data table or the accesses.access_check function. (Giving regular users direct access to these objects may allow ways to bypass the security mechanisms.)
  • The WITH CHECK OPTION makes it so that users can only insert and update data that they are authorized to see. Depending on the type of data, if a user is inserting data that they aren't authorized to see, it could mean that a security incident of some kind (potentially outside the database) has already occurred, which allowed the user to receive that data.

Testing the Interface

Now that everything is set up, let's create some user accounts and test it out.

First, create an anonymous account and grant it access to the protected database.

CREATE USER ''@'localhost';
GRANT SELECT, INSERT, UPDATE, DELETE ON protected.* TO ''@'localhost';

Now we can log in as any user to this database.

[gmontee@localhost ~]$ mysql -u alice --execute="SELECT SESSION_USER(), CURRENT_USER();"
+-----------------+----------------+
| SESSION_USER()  | CURRENT_USER() |
+-----------------+----------------+
| alice@localhost | @localhost     |
+-----------------+----------------+
[gmontee@localhost ~]$ mysql -u bob --execute="SELECT SESSION_USER(), CURRENT_USER();"
+----------------+----------------+
| SESSION_USER() | CURRENT_USER() |
+----------------+----------------+
| bob@localhost  | @localhost     |
+----------------+----------------+

Now let's test out some queries using different user accounts.

[gmontee@localhost ~]$ mysql -u root --execute="SELECT * FROM protected.important_data"
+----+-----------+----------------+
| id | data      | security_label |
+----+-----------+----------------+
|  1 | correct   | red            |
|  2 | horse     | blue           |
|  3 | battery   | green          |
|  4 | stapler   | yellow         |
|  5 | correcter | purple         |
+----+-----------+----------------+
[gmontee@localhost ~]$ mysql -u alice --execute="SELECT * FROM protected.important_data"
+----+---------+----------------+
| id | data    | security_label |
+----+---------+----------------+
|  1 | correct | red            |
|  2 | horse   | blue           |
+----+---------+----------------+
[gmontee@localhost ~]$ mysql -u bob --execute="SELECT * FROM protected.important_data"
+----+-----------+----------------+
| id | data      | security_label |
+----+-----------+----------------+
|  3 | battery   | green          |
|  4 | stapler   | yellow         |
|  5 | correcter | purple         |
+----+-----------+----------------+
[gmontee@localhost ~]$ mysql -u trudy --execute="SELECT * FROM protected.important_data"
[gmontee@localhost ~]$ mysql -u alice --execute="SELECT * FROM protected.important_data WHERE security_label='purple'"
[gmontee@localhost ~]$ mysql -u alice --execute="SELECT * FROM protected.important_data WHERE security_label='red'"
+----+---------+----------------+
| id | data    | security_label |
+----+---------+----------------+
|  1 | correct | red            |
+----+---------+----------------+

The row-level security mechanism built into the view appears to work great. But what happens if these users try to query the actual table, rather than the view?

[gmontee@localhost ~]$ mysql -u root --execute="SELECT * FROM unprotected.important_data"
+----+-----------+----------------+
| id | data      | security_label |
+----+-----------+----------------+
|  1 | correct   | red            |
|  2 | horse     | blue           |
|  3 | battery   | green          |
|  4 | stapler   | yellow         |
|  5 | correcter | purple         |
+----+-----------+----------------+
[gmontee@localhost ~]$ mysql -u alice --execute="SELECT * FROM unprotected.important_data"
ERROR 1142 (42000) at line 1: SELECT command denied to user ''@'localhost' for table 'important_data'
[gmontee@localhost ~]$ mysql -u bob --execute="SELECT * FROM unprotected.important_data"
ERROR 1142 (42000) at line 1: SELECT command denied to user ''@'localhost' for table 'important_data'
[gmontee@localhost ~]$ mysql -u trudy --execute="SELECT * FROM unprotected.important_data"
ERROR 1142 (42000) at line 1: SELECT command denied to user ''@'localhost' for table 'important_data'

The root account can query the original table, but our other accounts don't have sufficient privileges.

Conclusion

Although MariaDB 10.0 doesn't have a built-in row-level security mechanism, it is still fairly easy to implement row-level security with built-in features.

Has anyone been using row-level security implementations in MariaDB? Do you have any suggestions on how to improve MariaDB to make this better?

About the Author

geoff_montee_g's picture

Geoff Montee is a Support Engineer with MariaDB. He has previous experience as a Database Administrator/Software Engineer with the U.S. Government, and as a System Administrator and Software Developer at Florida State University.


PlanetMySQL Voting: Vote UP / Vote DOWN

Replication in real-time from Oracle and MySQL into data warehouses and analytics

$
0
0
Analyzing transactional data is becoming increasingly common, especially as the data sizes and complexity increase and transactional stores are no longer to keep pace with the ever-increasing storage. Although there are many techniques available for loading data, getting effective data in real-time into your data warehouse store is a more difficult problem. VMware Continuent provides
PlanetMySQL Voting: Vote UP / Vote DOWN

Adding a unique constraint with more than 16 columns in MariaDB

$
0
0

When I started writing this post I planned to follow up on my series of posts by creating a unique constraint with more than 16 columns using a MariaDB virtual column the same way I used a MySQL generated column in my most recent post. During my testing I abandoned that plan when I discovered two things:

  1. MariaDB virtual columns impose a 252 character limit on the expression that defines the column. This works for concatenating lots of columns with very short names like I did in my last post, but in the real world it's easy to find an example where column names are long enough that a concatenate expression involving more than 16 columns is longer than 252 characters.
  2. MariaDB doesn't have the same 16 column limit on indexes; instead it imposes a limit of 32 columns. Thus I can add a unique constraint on 17-32 columns in MariaDB without having to do anything special.

So I can't use MariaDB virtual columns as a workaround to add a unique constraint on more than 16 columns, but it probably doesn't matter because I actually don't need that workaround as long as my unique constraint includes no more than 32 columns.

Read on for code examples.

My original use case of creating a 20 column unique constraint works as-is in MariaDB, so in order to get error 1070 I have to go beyond 32 columns. Since the resulting concatenated column will be larger than 767 bytes, I need to make sure I'm using innodb_file_format=BARRACUDA, innodb_large_prefix=ON, and ROW_FORMAT=DYNAMIC as I've written about in the past.

With those settings in place, I create a table with 40 columns:

``` MariaDB [test]> CREATE TABLE IF NOT EXISTS even_more_columns (

->   c1 int not null,
->   c2 int not null,
->   c3 int not null,
->   c4 int not null,
->   c5 int not null,
->   c6 int not null,
->   c7 int not null,
->   c8 int not null,
->   c9 int not null,
->   c10 int not null,
->   c11 char(8) not null,
->   c12 char(8) not null,
->   c13 char(8) not null,
->   c14 char(8) not null,
->   c15 char(8) not null,
->   c16 char(8) not null,
->   c17 char(8) not null,
->   c18 char(8) not null,
->   c19 char(8) not null,
->   c20 char(8) not null,
->   c21 int not null,
->   c22 int not null,
->   c23 int not null,
->   c24 int not null,
->   c25 int not null,
->   c26 int not null,
->   c27 int not null,
->   c28 int not null,
->   c29 int not null,
->   c30 int not null,
->   c31 char(8) not null,
->   c32 char(8) not null,
->   c33 char(8) not null,
->   c34 char(8) not null,
->   c35 char(8) not null,
->   c36 char(8) not null,
->   c37 char(8) not null,
->   c38 char(8) not null,
->   c39 char(8) not null,
->   c40 char(8) not null
-> ) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC;

Query OK, 0 rows affected (0.02 sec) ```

To prove that I don't need the workaround, I create a unique constraint on 32 columns:

``` MariaDB [test]> alter table even_more_columns

->   add unique index unique_constraint_32col (c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,
->     c11,c12,c13,c14,c15,c16,c17,c18,c19,c20,
->   c21,c22,c23,c24,c25,c26,c27,c28,c29,c30,c31,c32);

Query OK, 0 rows affected (0.05 sec) Records: 0 Duplicates: 0 Warnings: 0 ```

But if I try to add a unique constraint on 33 columns I get an error:

``` MariaDB [test]> alter table even_more_columns

->   add unique index unique_constraint_33col (c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,
->     c11,c12,c13,c14,c15,c16,c17,c18,c19,c20,
->   c21,c22,c23,c24,c25,c26,c27,c28,c29,c30,c31,c32,c33);

ERROR 1070 (42000): Too many key parts specified; max 32 parts allowed ```

If I try to create a unique constraint on all 40 columns using a virtual column I get the error about the 252 character expression limit:

``` MariaDB [test]> alter table even_more_columns

->   add column all_columns varchar(512) as (
->   concat(c1,'-',c2,'-',c3,'-',c4,'-',c5,'-',c6,'-',c7,'-',c8,'-',c9,'-',c10,'-',
->     c11,'-',c12,'-',c13,'-',c14,'-',c15,'-',c16,'-',c17,'-',c18,'-',c19,'-',c20,'-',
->     c21,'-',c22,'-',c23,'-',c24,'-',c25,'-',c26,'-',c27,'-',c28,'-',c29,'-',c30,'-',
->     c31,'-',c32,'-',c33,'-',c34,'-',c35,'-',c36,'-',c37,'-',c38,'-',c39,'-',c40)
->   ) persistent,
->   add unique index unique_constraint (all_columns);

ERROR 1470 (HY000): String ' concat(c1,'-',c2,'-',c3,'-',c4,'-',c5,'-',c6,'-',c7,'-',c8,'-',c9,'' is too long for VIRTUAL COLUMN EXPRESSION (should be no longer than 252) ```

In the end, the good news is that MariaDB supports my original use case of creating a unique constraint on 20 columns.

As for virtual columns, MariaDB had a big head start on MySQL since it added virtual columns in version 5.2 which went GA more than 4 years ago. MySQL generated columns are still not GA yet, but they appear to have some advantages over MariaDB virtual columns: As far as I know MySQL generated columns do not have a 252 character limit on the generated column expression, and based on the MySQL 5.7.7 labs release it appears that generated columns will allow you to create an index on a virtual (not stored) generated column, which MariaDB does not support.


PlanetMySQL Voting: Vote UP / Vote DOWN

Optimizing Percona XtraDB Cluster for write hotspots

$
0
0

Some applications have a heavy write workload on a few records – for instance when incrementing a global counter: this is called a write hotspot. Because you cannot update the same row simultaneously from multiple threads, this can lead to performance degradation. When using Percona XtraDB Cluster (PXC), some users try to solve this specific issue by writing on multiple nodes at the same time. Good idea or bad idea? Read on!

Simultaneous writes on a standalone InnoDB server

Say you have these 3 transactions being run simultaneously (id is the primary key of the table):

# T1
UPDATE t SET ... WHERE id = 100
# T2
UPDATE t SET ... WHERE id = 100
# T3
UPDATE t SET ... WHERE id = 101

All transactions will require a row lock on the record they want to modify. So T3 can commit at the same time than T1 and/or T2, because it will not lock the same record as T1 and T2.

But T1 and T2 cannot execute simultaneously because they need to set a lock on the same record. Let’s assume T2 is executed by InnoDB first, how long does T1 need to wait? It is essentially the time needed for T2 to execute.

Simultaneous writes on multiple nodes (PXC)

Now is it any different if you have a 3-node PXC cluster and if you want to run T1 on node1 on T2 on node2? Let’s review step by step how the cluster will execute these queries:

1. T1 starts executing on node1: a row lock is set on the record where id=100. T2 also starts executing on node2 and also sets a row lock on the record where id=100. How is it possible that 2 transactions set the same lock on the same record? Remember that locking in Galera is optimistic, meaning that a transaction can only set locks on the node where it is executing, but never on the remote nodes: here, T1 sets a lock on node1 and T2 sets a lock on node2.

step1

2. Let’s assume T2 reaches commit before T1. T2 is then synchronously replicated on all nodes and it gets a global sequence number. At this point, a certification test is run on node2 to determine whether there is any write conflicts between T2 and other “in-flight” transactions in the cluster. Go to the next section if you want some clarification about “in-flight” transactions and the certification test.

Back to our transactions: T2 is the first transaction to try to commit, so no other transaction is “in-flight”: the certification test will succeed and InnoDB will be able to apply the transaction on node1. On node2, the same certification test will be run and T2 will be put in the apply queue (and it will be applied at some point in the near future).

step2

Ok, wait a minute! No other transaction is “in-flight”, really? What about T1 on node1? Actually T1 is simply a local transaction on node1 and it is not known by the cluster yet. Therefore it is not what I called an “in-flight” transaction and it does not play any role in the certification test.

3. Now T1 reaches commit on node1. It is then synchronously replicated to all nodes and a certification test will run on node1. If T1 and T2 did commit simultaneously, there is a good chance that when T1 starts committing, T2 is still in the apply queue of node1. In this case, the certification test for T1 will fail. Why? To make sure that T2 will commit on all nodes no matter what, any other transaction that wants to set a lock on the record where id=100 will be rejected.

Then if the certification test for T1 fails, T1 is rolled back. The only option to commit T1 is to retry executing it.

step3

Let’s assume that this second try is successful, how long did T1 have to wait before being executing? Essentially we had to execute T1 twice so we had to replicate it twice, each replication taking 1 network RTT, we had to roll T1 back on node1 (rollback is expensive in InnoDB) and the application had to decide that T1 had to be executed a second time. That is a lot more work and wait compared to the scenario on a single server.

So where is the fundamental problem when we tried to write on several nodes? Galera uses optimistic locking, so we had to go very far in the execution of T1 before realizing that the query will not succeed. Multi-node writing is therefore not a good solution at all when the system sees write hotspots.

“In-flight” transactions and certification test

“In-flight” transactions are transactions that have already been applied on at least one node of the cluster but not on all nodes. Remember that even if replicating transactions is synchronous, applying committed transactions on remote nodes is not. So a transaction Tx can be committed and applied on node1 but not on node2: on node2, Tx will simply sit in an apply queue, waiting to be executed. Tx is then an “in-flight” transaction.

The goal of the certification test is to make sure that no transaction can prevent Tx to execute on node2: as Tx is already on node1 and as we want data consistency, we must make sure that Tx will execute successfully no matter what can happen. And the magic of Galera is that the test is deterministic so group communication is not necessary when a node runs a certification test.

Conclusion

So what are your options with PXC when the workload has write hotspots? The most obvious one is to write on a single node: then you will have the same locking model as with a standalone InnoDB server. Performance will not be as good as with a standalone server as you will have to pay for synchronous replication, but you will avoid the very expensive write conflicts.

Another option would be to rearchitect your application to write less often. For instance, for a global counter, you could maintain it with Redis and only periodically flush the value to PXC.

And if you want to understand more about the benefits and drawback of writing on multiple nodes of a PXC cluster, you can read these two posts.

The post Optimizing Percona XtraDB Cluster for write hotspots appeared first on MySQL Performance Blog.


PlanetMySQL Voting: Vote UP / Vote DOWN

New Features Webinar: ClusterControl 1.2.10 - Fully Programmable DevOps Platform - Live Demo

$
0
0

Following the release of ClusterControl 1.2.10 a week ago, we are excited to demonstrate this latest version of the product on Tuesday next week, June 9th.

Join our CTO, Johan Andersson, who will be discussing and demonstrating the new ClusterControl DSL, Integrated Developer Studio and Database Advisors, which are some of the cool new features we’ve introduced with ClusterControl 1.2.10.

New Features Webinar: ClusterControl 1.2.10

DATE & TIME

Europe/MEA/APAC
Tuesday, June 9th at 09:00 (UK) / 10:00 CEST (Germany, France, Sweden)
Register Now

North America/LatAm
Tuesday, June 9th at 10:00 Pacific Time (US) / 13:00 Eastern Time (US)
Register Now

 

SPEAKER


Johan Andersson, CTO, Severalnines

Highlights of ClusterControl 1.2.10 include:

  • ClusterControl DSL (Domain Specific Language)
  • Integrated Developer Studio (Developer IDE)
  • Database Advisors/JS bundle
  • On-premise Deployment of MySQL / MariaDB Galera Cluster (New implementation)
  • Detection of long running and deadlocked transactions (Galera)
  • Detection of most advanced (last committed) node in case of cluster failure (Galera)
  • Registration of manually added nodes with ClusterControl
  • Failover and Slave Promotion in MySQL 5.6 Replication setups
  • General front-end optimizations

For additional details about the release:

Join us for this live webinar, where we’ll be discussing and demonstrating the latest features of ClusterControl!

We look forward to “seeing” you there and to insightful discussions!

If you have any questions or would like a personalised live demo, please do contact us.

 

ABOUT CLUSTERCONTROL

Setting up, maintaining and operating a database cluster can be tricky. ClusterControl gives you the power to deploy, manage, monitor and scale entire clusters efficiently and reliably. ClusterControl supports a variety of MySQL-based clusters (Galera, NDB, 5.6 Replication), MariaDB as well as MongoDB/TokuMX-based clusters and Postgres. With over 7,000 users to date, ClusterControl is the leading, platform independent automation and management solution for the MySQL, MongoDB and Postgres databases. 

Blog category:


PlanetMySQL Voting: Vote UP / Vote DOWN

SQL Load Balancing Benchmark - Comparing Performance of ProxySQL vs MaxScale

$
0
0
In the MySQL ecosystem there are few load balancers there are also open-source, and ProxySQL is one of the few proxies that works at the application layer and therefore is SQL aware.
In this blog post we will benchmark ProxySQL against MaxScale, another popular proxy for MySQL.
The idea to compare ProxySQL vs MaxScale came after reading an interesting blog post of Krzysztof Książek on SQL Load Balancing Benchmark, comparing performance of MaxScale vs HAProxy.

Disclaimer: ProxySQL is not GA yet, therefore please do not use it in production.



Sysbench setup

I wanted to setup a similar sysbench setup to what Krzysztof used in his benchmark, but it is slightly different:
a) instead of using a MySQL cluster with Galera, I setup a cluster with 1 master and 3 slaves. Since the workload was meant to be completely read-only and in-memory, the 2 setups are functionally identical;
b) instead of using AWS instances I used 4 physical servers: server A was running as a master and servers B, C and D were running as slaves. Since the master was idle (remember, this is a read-only workload that use only the slaves), I used the same box to also run sysbench and all the various proxies.

Benchmark were executed running the follow:
./sysbench \
--test=./tests/db/oltp.lua \
--num-threads=$THREADS \
--max-requests=0 \
--max-time=600 \
--mysql-user=rcannao \
--mysql-password=rcannao \
--mysql-db=test \
--db-driver=mysql \
--oltp-tables-count=128 \
--oltp-read-only=on \
--oltp-skip-trx=on  \
--report-interval=1 \
--oltp-point-selects=100 \
--oltp-table-size=400000 \
--mysql-host=127.0.0.1 \
--mysql-port=$PORT \
run

The versions used are:
Percona Server 5.6.22
sysbench 0.5
ProxySQL at commit a47136e with debugging disabled
MaxScale 1.0.5 GA
HAProxy 1.4.15

ProxySQL and MaxScale: few design differences


In the benchmark executed by Krzysztof, MaxScale was configured to listen on port 4006 where the service "RW Split Router" was running, and on port 4008 where the service "Read Connection Router" was running.
To my understand:
a) RW Split Router performs read/write split, parsing the queries and tracking the state of the transaction;
b) Read Connection Router performs a simple network forwarding, connecting clients to backends;
c) the two services, to operate, need to listen on different ports.

ProxySQL is, by design, different.

ProxySQL and RW split


ProxySQL performs a very simple query analysis to determine where the queries need to be send.
ProxySQL decides where a query needs to be forwarded based on a user configurable chain of rules, where a DBA can specify various matching criteria like username, schemaname, if there is an active transaction (feature not completely implemented), and a regular expression to match the query.
Matching against a regular expression provides better speed than building a syntax tree, and having a chain of rules that match with either regex or other attributes allows a great degree of flexibility compared to hardcoded routing policies.
Therefore, to implemented a basic read/write split, ProxySQL was configured in a way that:
a) all the queries matching '^SELECT.*FOR UPDATE$' were sent to master ;
b) all the queries not matching the previous rules but matching '^SELECT.*' were sent to slaves.
c) by default, all traffic not matching any of the previous rules was sent to master;

Considering the 3 rules listed above, all traffic generated by sysbench was always sent to slaves.

Additionally, while ProxySQL doesn't perform any syntax parsing to determine the target of a query, no matter what routing rules are in place, it also performs a very simple query analysis to determine what type of statement is being executed and generate statistics based on these. That is, ProxySQL is counting the type of statements that is executing, and these information are accessible through ProxySQL itself.

As already pointed in previous articles, one of the main idea behind ProxySQL is that the DBA is now the one controlling and defining query routing rules, making routing completely transparent to the developers, eliminates the politics behind DBAs depending on developers for such tweaking
of the setup, and therefore increasing interaction speed.

ProxySQL and Fast Forwarding

I think that the way MaxScale implements different modules listening on different port is a very interesting approach, yet it forces the developers to enforce some sort of read/write split in the application: connect to port 4006 if you want R/W split, or port 4008 if you want RO load balancing.
My aim in ProxySQL is that the application should have a single connection point, ProxySQL, and the proxy should determine what to do with the incoming requests. In other words, the application should just connect to ProxySQL and this should take care of the rests, according to its configuration.
To do so, ProxySQL should always authenticate the client before applying any rule. Therefore I thought that a quick feature to implement is Fast Forwarding based on username: when a specific user connects, all its requests are forwarded to the backends without any query processing or connection pool.
In other words, ProxySQL's Fast Forwarding is a concept similar to MaxScale's Read Connection, but uses the same port as the R/W split module and the matching criteria is the client's username instead of listener port.
Note that ProxySQL already support multiple listeners, but the same rules apply to all ports; in future versions, ProxySQL will support matching criteria also based on listener's port behaving in a similar way of MaxScale, but will also add additional matching criteria like the source of the connection.



Performance benchmarks


As said previously, on the same host where sysbench was running I also configured ProxySQL, MaxScale and HAProxy.
In the blog post published by Severalnines, one of the comment states that MaxScale was very slow with few connections, on physical hardware.
Therefore, the first benchmark I wanted to run was exactly at low number of connections, and progressively increase the number of connections.
ProxySQL and MaxScale were both configured with just 1 worker thread, and HAProxy was configured with only 1 process.

Please note that in the follows benchmark worker threads and connections are two completely different entities:
1) a connection is defined as a client connection;
2) a worker thread is a thread inside the proxy, either ProxySQL, MaxScale or HAProxy (even if HAProxy uses processes and not threads).
What could cause confusion is the fact that in sysbench a thread is a connection: from a proxy prospective, it is just a connection.

Benchmark with 1 worker thread




Tagline:
maxscale rw = MaxScale with RW Split Router
maxscale rr = MaxScale with Read Connection Router
proxysql rw = ProxySQL with query routing enabled
proxysql ff = ProxySQL with fast forwarding enabled

Average throughput in QPS:

ConnectionsHAProxyMaxScale RWMaxScale RRProxySQL RWProxySQL FF
13703.36709.99722.273534.923676.04
414506.452815.72926.4413125.6714275.66
826628.445690.225833.7723000.9824514.94
3254570.2614722.9722969.7341072.5151998.35
25653715.7913902.9242227.4645348.5958210.93

In the above graphs we can easily spot that:
a) indeed, MaxScale performance are very low when running with just few connections (more details below);
b) for any proxy, performance become quite unstable when the number of connections increases;
c) proxysql-ff is very close to the performance of haproxy;
d) with only 1 or 4 client connections, ProxySQL provides 5 times more throughput than MaxScale in both modules; with only 8 client connections ProxySQL provides 4 times more throughput than MaxScale in R/W split, and 4.3 times more in fast forward mode;
e) at 32 client connections, proxysql-rw provides 2.8x more throughput than maxscale-rw, and proxysql-ff provides 2.3x more than maxscale-rr ;
f) 4 proxies configurations (haproxy, maxscale-rw, proxysql-rw, proxysql-ff) behave similarly at 32 or 256 client's connections, while maxscale-rr almost double its throughput at 256 connections vs 32 connections: in other words, when the number of connections is high some bottleneck is taken away.

Below are also the graphs of average throughput, average and 95% response time at low number of connections.





Fortunately, I have access to physical hardware (not AWS instances) and I was able to reproduce the issue reported in that comment: MaxScale seems to be very slow when running with just few connections.
Although, for comparison, I tried a simple benchmark on AWS and I found that MaxScale doesn't behave as bad as on physical server.
After these interesting results, I tried running the same benchmark connecting to MaxScale and ProxySQL not through TCP but through Unix Domain Socket, with further interesting results.
Unfortunately, I didn't have a version of HAProxy that accepted connections via UDS, so I ran benchmark against HAProxy using TCP connections.















Average throughput in QPS:
ConnectionsHAProxyMaxScale RWMaxScale RRProxySQL RWProxySQL FF
13703.363276.853771.153716.193825.81
414506.4511780.2714807.4513333.0314729.59
826628.4415203.9327068.8124504.4225538.57
3254570.2616370.6944711.2546846.0458016.03
25653715.7914689.7345108.5454229.2971981.32

In the above graphs we can easily spot that:
a) MaxScale is no longer slow when running with just few connections: the performance bottleneck at low number of connections is not present when using UDS instead of TCP;
b) again, for any proxy, performance become quite unstable when the number of connections increase;
d) maxscale-rw is the slowest configuration at any number of connections;d) with an increased number of client connections, performance of MaxScale reaches its limits with an average QPS of 16.4k reads/s peaking at 32 connections for maxscale-rw , and an average QPS of 45.1k reads/s peaking at 256 connections for maxscale-rr;
e) with an increased number of client connections, performance of ProxySQL reaches its limits with an average QPS of 54.2k reads/s peaking at 256 connections for proxysql-rw , and an average QPS of 72.0k reads/s peaking at 256 connections for proxysql-ff .

As pointed already, with an increased number of connections the performance become quite unstable, although it is easy to spot that:
1) in R/W split mode, ProxySQL can reached a throughput over 3 times higher than MaxScale;
2) ProxySQL in Fast Forward mode can reach a throughput of 33% more than MaxScale in Read Connection Router mode;
3) ProxySQL in R/W split mode is faster than MaxScale in simple Read Connection Router mode.

The above points that while MaxScale has a readconnroute module with a low latency, none of the two MaxScale's module scale very well. The bottleneck seems to be that MaxScale uses a lot of CPU, as already pointed by Krzysztof in his blog post, therefore it quickly saturates its CPU resources without being able to scale.

Of course, it is possible to scale adding more threads: more results below!


MaxScale and TCP

At this stage I knew that, on physical hardware:
- ProxySQL was running well when clients were connecting via TCP or UDS at any number of connections;
- MaxScale was running well when clients were connecting via UDS at any number of connections;
- MaxScale was running well when clients were connecting via TCP with a high number of connections;
- MaxScale was not running well when clients were connecting via TCP with a low number of connections.

My experience with networking programming quickly drove me to where the bottleneck could be.
This search returns no results:
https://github.com/mariadb-corporation/MaxScale/search?utf8=%E2%9C%93&q=TCP_NODELAY

In other words, MaxScale never disabled the Nagle's algorithm, adding latency to any communication with the client. The problem is noticeable only at low number of connections because at high number of connections the latency introduced by Nagle's algorithm become smaller compared to the overall latency caused by processing multiple clients. For reference:
http://en.wikipedia.org/wiki/Nagle%27s_algorithm

I will also soon open a bug report against MaxScale.

What I can't understand, and I would appreciate if someone's else can comment on this, is why Nagle's algorithm doesn't seem to have any effect on AWS or other virtualization environments.
In any case, this is a very interesting example of how software behave differently on physical hardware and virtualization environments.

Because MaxScale performs on average, 5x more slowly at low number of connections via TCP, the follow graphs only use UDS for ProxySQL and MaxScale: the performance of MaxScale on TCP were too low to be considered.






Benchmark with 2 worker threads

Because MaxScale performs really bad at low number of connections via TCP due the Nagle's algorithm on physical hardware, I decided to run all the next benchmark connecting to MaxScale and ProxySQL only through UDS. HAProxy will still be used for comparison, even if connections are through TCP sockets.
I know it is not fair to compare performance of connections via TCP (HAProxy) against connections via UDS (for ProxySQL and MaxScale), but HAProxy is used only for reference.














Average throughput in QPS:
ConnectionsHAProxyMaxScale RWMaxScale RRProxySQL RWProxySQL FF
414549.6111627.1614185.8813697.0314795.74
827492.4121865.3927863.9425540.6127747.1
3281301.3229602.8463553.7762350.8977449.45
256109867.6628329.873751.2481663.75125717.18
512105999.8426696.669488.7181734.18128512.32
1024103654.9727340.4763446.6174747.25118992.24


Notes with 2 worker threads (for MaxScale and ProxySQL) or 2 worker processes (HAProxy):
a) once again, for any proxy, performance become quite unstable when the number of connections increase. Perhaps this is not a bug in the proxies, but it is a result of how the kernel schedules processes;
b) up to 32 client connections, performance of 2 workers is very similar to performance of 1 worker no matter the proxy. Each proxy configuration has its different performance, but it performs the same with either 1 or 2 workers;
c) maxscale-rw reaches its average peak at 32 connections, reaching 29.6k reads/s;
d) maxscale-rr reaches its average peak at 256 connections, reaching 73.8k reads/s;
e) proxysql-rw reaches its average peak at 512 connections, reaching 81.7k reads/s;
f) proxysql-ff reaches its average peak at 512 connections, reaching 128.5k reads/s;

As pointed already, with an increased number of connections the performance become quite unstable, but as in the workload with just one worker thread it is easy to spot that:
1) in R/W split mode, ProxySQL can reach a throughput of nearly 3 times higher than MaxScale;
2) ProxySQL in Fast Forward mode can reach a throughput of 74% more than MaxScale in Read Connection Router mode;
3) ProxySQL in R/W split mode is faster than MaxScale in simple Read Connection Router mode.

The above points confirms what said previously: ProxySQL uses less CPU resources, therefore it is able to scale a lot better than MaxScale with an increased number of client connections.




Benchmark with 4 worker threads

I ran more benchmark using 4 worker threads for ProxySQL and MaxScale, and 4 worker processes  for HAProxy.










Average throughput in QPS:
ConnectionsHAProxyMaxScale RWMaxScale RRProxySQL RWProxySQL FF
1650258.2141939.850621.7446265.6551280.99
3289501.3350339.8187192.5870321.1785846.94
256174666.0952294.7117709.3115056.5183602.6
512176398.3346777.17114743.73112982.78188264.03
2048157304.080107052.01102456.38187906.29


What happens with 4 worker threads/processes?
a) as with 1 or 2 workers, for any proxy, performance become quite unstable when the number of connections increase, but this time the fluctuations seems more smooth. Yet, ProxySQL seems the most stable proxy at high number of connections;
b) at 32 connections, ProxySQL and HAProxy gives similar throughput at either 2 or 4 workers;
c) at 32 connections, MaxScale provides more throughput with 4 workers than at 2 workers, showing that MaxScale needs more CPU power to provide better throughput;
d) at 32 connections, HAProxy, ProxySQL and MaxScale provide similar reads/s if they do not analyze traffic (89.5k , 85.8k and 87.2k);
e) using R/W functionality, at 16 connections ProxySQL provides 10% more reads/s than MaxScale (46.3k vs 41.9k), and at 32 connections ProxySQL provides 40% more reads/s than MaxScale (70.3k vs 50.3k);
f) MaxScale in R/W mode wasn't able to run 2048 client's connections;
g) maxscale-rw reaches its average peak at 256 connections, with 52.3k reads/s;
h) maxscale-rr reaches its average peak at 256 connections, with 117.7k reads/s;
i) proxysql-rw reaches its average peak at 256 connections, with 115.1k reads/s;
j) proxysql-ff reaches its average peak at 512 connections, with 188.3k reads/s;
 
Few more notes on scalability with 4 threads:
1) in R/W split mode, ProxySQL can reached a throughput over 2 times higher than MaxScale;
2) ProxySQL in Fast Forward mode can reach a throughput of 60% more than MaxScale in Read Connection Router mode;
3) ProxySQL in R/W split mode is, for the first time, slightly slower than MaxScale in simple Read Connection Router mode (115.1k vs 117.7k).


Note on transport layer load balancing

I consider important only the benchmark related to R/W split because only this provides SQL load balancing; HAProxy, ProxySQL with fast forward and MaxScale with readconnroute module do not provide SQL load balancing, but are present in the benchmark above to provide some reference of the overhead caused by processing SQL traffic.
Furthermore, the performance of MaxScale's readconnroute cannot be compared with the performance of HAProxy or ProxySQL. From a user's prospective, I would prefer to use HAProxy because it can provide way better performance.


Conclusions

One of the main focus while developing ProxySQL is that it must be a very fast proxy, to introduce almost no latency. This goal seems to be very well achieved, and ProxySQL is able to process MySQL traffic with very little overhead, and it is able to scale very well.
In all the benchmark listed above ProxySQL is able to scale easily.
In fact, in R/W split mode (highly configurable in ProxySQL, but hardcoded in MaxScale), ProxySQL is able to provide up to 5 times more throughput than MaxScale, depending from workload.

Since ProxySQL in query processing mode (R/W split) provides more throughput than MaxScale's readconnroute in the majority of the cases, I would always use ProxySQL's query processing that implements important features like query routing, query rewrite, query caching, statistics, connection poll, etc.
At today, the only reason why I wouldn't use ProxySQL in production is that ProxySQL is not GA ... yet!






PlanetMySQL Voting: Vote UP / Vote DOWN

MMUG13: Practical MySQL Optimisation y Galera Cluster presentations

$
0
0

English: Madrid MySQL Users Group will be holding their next meeting on 17th June at 18:00h at EIE Spain in Madrid. Dimitri Vanoverbeke and Stéphane Combaudon from Percona will be offering two presentations for us:

  • Practical MySQL optimisations
  • Galera Cluster – introduction and where it fits in the MySQL eco-system

I think this is an excellent moment to learn new things and meet new people. If you’re in Madrid and are interested please come along.  More information can be found here at the Madrid MySQL Users Group page.

Español: El día 17 de junio a las 18:00 Madrid MySQL Users Group tendrá su próxima reunión en las oficinas de EIE Spain.  Dimitri Vanoverbeke y Stéphane Combaudon de Percona nos ofrecerá dos presentaciones (en inglés):

  • Practical MySQL optimisations
  • Galera Cluster – introduction and where it fits in the MySQL eco-system

Creo que será una oportunidad excelente para aprender algo nuevo y para conocer gente nueva. Si estás en Madrid y interesado nos gustaría verte.  Se puede encontrar más información aquí en la página de Madrid MySQL Users Group.

 


PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL as an Oracle DBA

$
0
0

A quick question, which I’ve asked once before –

if you were an Oracle DBA, then learned MySQL, what do you wish you knew before you got started?

Also, what was helpful to you as you learned?  (Websites, #mysql on irc, documents, etc.)  What do you wish you had (or want to have now) if you are a DBA for both Oracle databases and MySQL databases?

Ie, what would be good to give an Oracle DBA who wants to start learning or supporting MySQL as well?  Please respond with comments here, or directly email me at ben-dot-krug-at-oracle-dot-com.

Thanks!



PlanetMySQL Voting: Vote UP / Vote DOWN

Percona XtraDB Cluster 5.6.24-25.11 is now available

$
0
0

Percona XtraDB Cluster 5.6.24-25.11Percona is glad to announce the new release of Percona XtraDB Cluster 5.6 on June 3rd 2015. Binaries are available from downloads area or from our software repositories.

Based on Percona Server 5.6.24-72.2 including all the bug fixes in it, Galera Replicator 3.11, and on Codership wsrep API 25.11, Percona XtraDB Cluster 5.6.24-25.11 is now the current General Availability release. All of Percona’s software is open-source and free, and all the details of the release can be found in the 5.6.24-25.11 milestone at Launchpad.

New Features:

  • Percona XtraDB Cluster now allows reads in non-primary state by introducing a new session variable wsrep_dirty_reads. This variable is boolean and is OFF by default. When set to ON, a Percona XtraDB Cluster node accepts queries that only read, but not modify data even if the node is in the non-PRIM state (#1407770).
  • Percona XtraDB Cluster now allows queries against INFORMATION_SCHEMA and PERFORMANCE_SCHEMA even with variables wsrep_ready and wsrep_dirty_reads set to OFF. This allows monitoring applications to monitor the node when it is even in non-PRIM state (#1409618).
  • wsrep_replicate_myisam variable is now both global and session variable (#1280280).
  • Percona XtraDB Cluster now uses getifaddrs for node address detection (#1252700).
  • Percona XtraDB Cluster has implemented two new status variables: wsrep_cert_bucket_count and wsrep_gcache_pool_size for better instrumentation of galera memory usage. Variable wsrep_cert_bucket_count shows the number of cells in the certification index hash-table and variable wsrep_gcache_pool_size shows the size of the page pool and/or dynamic memory allocated for gcache (in bytes).

Bugs Fixed:

  • Using concurrent REPLACE, LOAD DATA REPLACE or INSERT ON DUPLICATE KEY UPDATE statements in the READ COMMITTED isolation level or with the innodb_locks_unsafe_for_binlog option enabled could lead to a unique-key constraint violation. Bug fixed #1308016.
  • Using the Rolling Schema Upgrade as a schema upgrade method due to conflict with wsrep_desync would allows only one ALTER TABLE to run concurrently. Bugs fixed #1330944 and #1330941.
  • SST would resume even when the donor was already detected as being in SYNCED state. This was caused when wsrep_desync was manually set to OFF which caused the conflict and resumed the donor sooner. Bug fixed #1288528.
  • DDL would fail on a node when running a TOI DDL, if one of the nodes has the table locked. Bug fixed #1376747.
  • xinet.d mysqlchk file was missing type = UNLISTED to work out of the box. Bug fixed #1418614.
  • Conflict between enforce_storage_engine and wsrep_replicate_myisam for CREATE TABLE has been fixed. Bug fixed #1435482.
  • A specific trigger execution on the master server could cause a slave assertion error under row-based replication. The trigger would satisfy the following conditions: 1) it sets a savepoint; 2) it declares a condition handler which releases this savepoint; 3) the trigger execution passes through the condition handler. Bug fixed #1438990.
  • Percona XtraDB Cluster Debian init script was testing connection with wrong credentials. Bug fixed #1439673.
  • Race condition between IST and SST in xtrabackup-v2 SST has been fixed. Bugs fixed #1441762, #1443881, and #1451524.
  • SST will now fail when move-back fails instead of continuing and failing at the next step. Bug fixed #1451670.
  • Percona XtraDB Cluster .deb binaries were built without fast mutexes. Bug fixed #1457118.
  • The error message text returned to the client in the non-primary mode is now more descriptive ("WSREP has not yet prepared node for application use"), instead of "Unknown command" returned previously. Bug fixed #1426378.
  • Out-of-bount memory access issue in seqno_reset() function has been fixed.
  • wsrep_local_cached_downto would underflow when the node on which it is queried has no writesets in gcache.

Other bugs fixed: #1290526.

Help us improve our software quality by reporting any bugs you encounter using our bug tracking system. As always, thanks for your continued support of Percona!

The post Percona XtraDB Cluster 5.6.24-25.11 is now available appeared first on MySQL Performance Blog.


PlanetMySQL Voting: Vote UP / Vote DOWN
Viewing all 18838 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>