-----     / \     -----
| A | -> / X \ -> | B |
-----    -----    -----

• binary logging is enabled everywhere: log-bin=binlog,
• log-slave-updates is disabled everywhere,
• the replication user 'repl'@'%' exists on all nodes with its password being slavepass,
• the user 'repl'@'%' has the right GRANTs for accepting a MaxScale connection (search "user list" in Install and Configure HOWTO for more details).

-----
| A |
-----
  |
  +-------------+
  |             |
 / \           / \
/ X \         / Y \
-----         -----
  |             |
  +------+      |
  |      |      |
-----  -----  -----
| B |  | C |  | D |
-----  -----  -----

# On A in a MySQL client session.
CREATE DATABASE test_mbls;
CREATE TABLE test_mbls.t1
  (user BIGINT PRIMARY KEY, pass BIGINT DEFAULT NULL);
INSERT INTO test_mbls.t1 VALUES (1, 0);
INSERT INTO test_mbls.t1 VALUES (2, 0);
INSERT INTO test_mbls.t1 VALUES (3, 0);

# On X in bash.
sudo service maxscale stop; 

# On A in a MySQL client session.
INSERT INTO test_mbls.t1 VALUES (4, 0);
INSERT INTO test_mbls.t1 VALUES (5, 0);

# On A in bash.
sudo killall -9 mysqld_safe;
sudo killall -9 mysqld;

-\-/-
| A |
-/-\-

 / \           / \
/ X \         / Y \
-----         -----
  |             |
  +------+      |
  |      |      |
-----  -----  -----
| B |  | C |  | D |
-----  -----  -----

-\-/-
| A |
-/-\-

 / \           / \
/ X \ <------ / Y \
-----         -----
  |             |
  +------+      |
  |      |      |
-----  -----  -----
| B |  | C |  | D |
-----  -----  -----

1. run "SHOW MASTER LOGS;" to get the binary log filename,
2. then run enough "FLUSH BINARY LOGS;" to get one file further than the last binary log available on X and Y,
3. then run "STOP SLAVE; RESET SLAVE ALL;" to drop all replication configuration from C,
4. and then run "PURGE BINARY LOGS BEFORE NOW();" to forget all binary logs on C that would exist/conflict on X and Y.

# On C in a MySQL client session.
INSERT INTO test_mbls.t1 VALUES (6, 0);
DELETE FROM test_mbls.t1 WHERE user = 3;

• sudo service maxscale stop
• Edit the MaxScale configuration file to replicate from C but DO NOT start MaxScale yet.
• Create, in the binary log directory, a new 4 bytes binary log file with the name of the first binary log available on C (the next file in the sequence of binary logs):
  xxd -r <<< "0000000: fe62 696e" > $right_binlog_file
• Make sure the new binary log file has the right ownership and permissions (the same as the other binary log files).
• sudo service maxscale start

-\-/-  -----
| A |  | C |
-/-\-  -----
         |
  +------+------+
  |             |
 / \           / \
/ X \         / Y \
-----         -----
  |             |
-----         -----
| B |         | D |
-----         -----

• never serve binary log event downstream before having received the complete transaction,
• be able to truncate its local binary log at the last completed transaction before creating the next binary log.

• the 2 nodes running MySQL 5.6 (I did tests with MySQL 5.6.24 using Amazon VMs),
• binary logging enabled on both master (A) and slave (B): log-bin=binlog,
• log-slave-updates DISABLED on both nodes,
• the replication user 'repl'@'%' existing on both nodes with its password being slavepass.

-----    -----
| A | -> | B |
-----    -----

• Follow the instructions in https://mariadb.com/my_portal/download/maxscale
  (You will need a MariaDB account do access this page.)
  (You might need to accept importing the Maxscale-GPG-KEY when running yum.)
  (If all fails, try to access the URL reported by yum and fix it in /etc/yum.repos.d/maxscale.repo.)

cd /usr/local/mariadb-maxscale/etc/;
sudo cp MaxScale_BinlogServer_template.cnf MaxScale.cnf;
sudo chmod og-rwx MaxScale.cnf;
ls -l MaxScale.cnf;

• (No need to change the user and password, we are using the same as in the template),
• Change version_string=5.6.15-log to version_string=5.6.24-log
  (MaxScale will advertise itself to the master as 5.6.24-log),
• Change server-id=1000000000 to server-id=<a uniq server-id>
  (this is the server-id used by MaxScale to connect to the master; MaxScale will present itself to slaves using the server-id of the master),
• Change filestem=mysql-bin to filestem=binlog
  (this is the basename of the binary log file to download from the master; it must be the same as what is configured on the master; and a prerequisite/simplification above is to configure all nodes with  log-bin=binlog),
• In the [master] section, change address=master.example.com
  to address=<hostname or IP address of your master>
  ([master] is the servers option referenced in the [Binlog_Service] section which indicates the Binlog Router which master to download binary logs from),
• In the [Binlog Listener] section, change port=5306 to port=3306
  (this section makes the link between the module/router section and network/listener layer and we want the MaxScale Binlog Router to listen on the same port as a standard MySQL).

• MaxScale is running as root by default after yum install: bug number to come soon...
• The Binlog Router is not crash-safe: bug number to come soon... (see the Operations HOWTO for a work-around),
• If MySQL is stopped on the master when MaxScale is started, MaxScale will not download GRANTs after successfully connecting to the master: bug number to come soon... (to work around that bug, restart MaxScale after starting the master),
• Configuring the Binlog Router to download a non-existing binlog file from the master lead to a connection storm to the master and to a log storm in MaxScale: bug number to come soon...
• A CHANGE MASTER TO MaxScale without file and position triggers a log storm in MaxScale: bug number to come soon...

Data inconsistencies in replication environments are a pretty common. There are lots of posts that explain how to fix those using pt-table-checksum and pt-table-sync. Usually we only care about the data but from time to time we receive this question in support:

How can I check the table definition consistency between servers?

Replication also allow us to have different table definition between master and slaves. For example, there are some cases that you need some indexes on slaves for querying purposes but are not really needed on the master. There are some other cases where those differences are just a mistake that needs to be fixed.

mysqldiff, included in Oracle’s MySQL Utilities, can help us to find those differences and get the information we need to fix those them. In this post I’m going to show you how to use it with an example.

Find table definition inconsistencies

mysqldiff allows us to find those inconsistencies checking the differences between the tables on the same server (different databases) or on different servers (also possible on different databases). In this example I’m going to search for differences in table definitions between two different servers, server1 and server2.

The command line is pretty simple. This is used to compare the tables on “test” database:

mysqldiff --server1=user@host1 --server2=user@host2 test:test

If the database name is different:

mysqldiff --server1=user@host1 --server2=user@host2 testdb:anotherdb

If the table name is different:

mysqldiff --server1=user@host1 --server2=user@host2 testdb.table1:anotherdb.anothertable

Now I want to check the table definition consistency between two servers. The database’s name is “employees”:

# mysqldiff --force --server1=root:msandbox@127.0.0.1:21489 --server2=root:msandbox@127.0.0.1:21490 employees:employees
# WARNING: Using a password on the command line interface can be insecure.
# server1 on 127.0.0.1: ... connected.
# server2 on 127.0.0.1: ... connected.
# Comparing `employees` to `employees`                             [PASS]
# Comparing `employees`.`departments` to `employees`.`departments`   [FAIL]
# Object definitions differ. (--changes-for=server1)
#
--- `employees`.`departments`
+++ `employees`.`departments`
@@ -1,6 +1,6 @@
 CREATE TABLE `departments` (
   `dept_no` char(4) NOT NULL,
-  `dept_name` varchar(40) NOT NULL,
+  `dept_name` varchar(256) DEFAULT NULL,
   PRIMARY KEY (`dept_no`),
   UNIQUE KEY `dept_name` (`dept_name`)
 ) ENGINE=InnoDB DEFAULT CHARSET=latin1
# Comparing `employees`.`dept_emp` to `employees`.`dept_emp`       [PASS]
# Comparing `employees`.`dept_manager` to `employees`.`dept_manager`   [PASS]
# Comparing `employees`.`employees` to `employees`.`employees`     [FAIL]
# Object definitions differ. (--changes-for=server1)
#
--- `employees`.`employees`
+++ `employees`.`employees`
@@ -5,5 +5,6 @@
   `last_name` varchar(16) NOT NULL,
   `gender` enum('M','F') NOT NULL,
   `hire_date` date NOT NULL,
-  PRIMARY KEY (`emp_no`)
+  PRIMARY KEY (`emp_no`),
+  KEY `last_name` (`last_name`,`first_name`)
 ) ENGINE=InnoDB DEFAULT CHARSET=latin1
# Comparing `employees`.`salaries` to `employees`.`salaries`       [PASS]
# Comparing `employees`.`titles` to `employees`.`titles`           [PASS]
Compare failed. One or more differences found.

There are at least two differences. One in departments table and another one in employees table. The output is similar to diff. By default the tool stops after finding the first difference. That’s why we use –force, to tell the tool to continue checking all the tables.

It shows us that on departments the dept_name is varchar(40) on server1 and varchar(256) on server2. For “employees” table, it has a KEY (last_name, first_name) on the server2 that is not present on server1. Why is it taking server2 as a reference? Because of this line:

# Object definitions differ. (--changes-for=server1)

So, the changes shown on the diff are for server1. If you want server2 to be the one to be changed and server1 used as reference, then –changes-for=server2 would be needed.

In some cases the diff output is not really useful. We actually need a SQL query to do the changes on the server. We just need to add –difftype=sql to the command line:

# mysqldiff --force --difftype=sql --server1=root:msandbox@127.0.0.1:21489 --server2=root:msandbox@127.0.0.1:21490 employees:employees
[...]
# Comparing `employees`.`departments` to `employees`.`departments`   [FAIL]
# Transformation for --changes-for=server1:
ALTER TABLE `employees`.`departments`
  DROP INDEX dept_name,
  ADD UNIQUE INDEX dept_name (dept_name),
  CHANGE COLUMN dept_name dept_name varchar(256) NULL;
[...]
# Comparing `employees`.`employees` to `employees`.`employees`     [FAIL]
# Transformation for --changes-for=server1:
#
ALTER TABLE `employees`.`employees`
  DROP PRIMARY KEY,
  ADD PRIMARY KEY(`emp_no`),
  ADD INDEX last_name (last_name,first_name);

As we can see, the tool is not perfect. There are two problems here:

1- On “departments table” it drops a UNIQUE key that is present in both servers only to add it again. Waste of time and resources.

2- On “employees” table it drops and recreate the PRIMARY KEY, again something that is not needed a all.

I have created a bug report but this also teaches us a good lesson. Don’t just copy and paste commands without first double checking it.

What mysqldiff runs under the hood?

Mostly queries on INFORMATION_SCHEMA. These are the ones used to check inconsistencies on departments:

SHOW CREATE TABLE `departments`;
SELECT TABLE_SCHEMA, TABLE_NAME, ENGINE, AUTO_INCREMENT, AVG_ROW_LENGTH, CHECKSUM, TABLE_COLLATION, TABLE_COMMENT, ROW_FORMAT, CREATE_OPTIONS
  FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = 'employees' AND TABLE_NAME = 'departments';
SELECT ORDINAL_POSITION, COLUMN_NAME, COLUMN_TYPE, IS_NULLABLE,
         COLUMN_DEFAULT, EXTRA, COLUMN_COMMENT, COLUMN_KEY
  FROM INFORMATION_SCHEMA.COLUMNS
  WHERE TABLE_SCHEMA = 'employees' AND TABLE_NAME = 'departments';
SELECT PARTITION_NAME, SUBPARTITION_NAME, PARTITION_ORDINAL_POSITION,
         SUBPARTITION_ORDINAL_POSITION, PARTITION_METHOD, SUBPARTITION_METHOD,
         PARTITION_EXPRESSION, SUBPARTITION_EXPRESSION, PARTITION_DESCRIPTION
  FROM INFORMATION_SCHEMA.PARTITIONS
  WHERE TABLE_SCHEMA = 'employees' AND TABLE_NAME = 'departments';
SELECT CONSTRAINT_NAME, COLUMN_NAME, REFERENCED_TABLE_SCHEMA,
         REFERENCED_TABLE_NAME, REFERENCED_COLUMN_NAME
  FROM INFORMATION_SCHEMA.KEY_COLUMN_USAGE
  WHERE TABLE_SCHEMA = 'employees' AND TABLE_NAME = 'departments' AND
        REFERENCED_TABLE_SCHEMA IS NOT NULL;
SELECT TABLE_SCHEMA, TABLE_NAME, ENGINE, AUTO_INCREMENT, AVG_ROW_LENGTH, CHECKSUM, TABLE_COLLATION, TABLE_COMMENT, ROW_FORMAT, CREATE_OPTIONS
  FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = 'employees' AND TABLE_NAME = 'departments';
SELECT ORDINAL_POSITION, COLUMN_NAME, COLUMN_TYPE, IS_NULLABLE,
         COLUMN_DEFAULT, EXTRA, COLUMN_COMMENT, COLUMN_KEY
  FROM INFORMATION_SCHEMA.COLUMNS
  WHERE TABLE_SCHEMA = 'employees' AND TABLE_NAME = 'departments';
SELECT PARTITION_NAME, SUBPARTITION_NAME, PARTITION_ORDINAL_POSITION,
         SUBPARTITION_ORDINAL_POSITION, PARTITION_METHOD, SUBPARTITION_METHOD,
         PARTITION_EXPRESSION, SUBPARTITION_EXPRESSION, PARTITION_DESCRIPTION
  FROM INFORMATION_SCHEMA.PARTITIONS
  WHERE TABLE_SCHEMA = 'employees' AND TABLE_NAME = 'departments';

As a summary, it checks partitions, row format, collation, constraints and so on.

Conclusion

There are different tools for different purposes. We can check the data consistency with pt-table-checkum/pt-table-sync but also the table definitions with mysqldiff.

The post Checking table definition consistency with mysqldiff appeared first on MySQL Performance Blog.

Pivot tables and the "Show Parents" functionality

SELECT  Measures.Quantity     ON COLUMNS
,       Time.Quarters.Members ON ROWS
FROM    SteelWheelsSales

Implementing Show/Hide Parents

A pure MDX expression

The Ancestors() function

1. A member for which to find ancestor members (members at a higher level that contain the argument member)
2. An argument that specifies how many levels to traverse up.

• As a level: the second argument specifies a level and all ancestors up to that level will be retrieved
• As a integer representing a distance: the second argument specifies the number of levels that will be traversed upwards

Converting the set of Ancestor members to a scalar value

WITH
MEMBER  Measures.[Time Ancestors]
AS      SetToStr(
          Ancestors(
            Time.CurrentMember, 
            CInt(
              Time.CurrentMember.Properties("LEVEL_NUMBER")
            )
          )
        )
SELECT  Measures.[Time Ancestors] ON COLUMNS
,       Time.Quarters.Members ON ROWS
FROM    SteelWheelsSales

• The string representation returned by SetToStr() looks very much like how one would write the set down as a MDX set literal (is that a thing? It should be :-). While entirely correct, it does not look very friendly and it is certainly quite a bit different from what our GUI tools present to end-users
• The order of the members. It looks like Ancestors() returns the members in order of upward traversal, that is to say, from lower levels (=higher level numbers) to higher levels (=lower level numbers). The fancy way of saying that is that our result suggests that Ancestors() returns its members in post-natural order. We'd like the members to be in natural order, that is to say, in descending order of level (from high to low). Note that the specification of Ancestors() does not specify or require any particular order. So in the general case we should not rely on the results to be in any particular order.

• Order() is general purpose function that can be used to order the members of a set by an arbitrary numberic expression.
• Hierarchize() is designed to order members into hierarchical order, that is to say, the members are ordered by their level number and by the level number of any of its ancestors.

WITH
MEMBER  Measures.[Time Ancestors]
AS      SetToStr(
          Hierarchize(
            Ancestors(
              Time.CurrentMember, 
              CInt(
                Time.CurrentMember.Properties("LEVEL_NUMBER")
              )
            )
          )
        )
SELECT  Measures.[Time Ancestors] ON COLUMNS
,       Time.Quarters.Members ON ROWS
FROM    SteelWheelsSales

1. A set. This is where we'll feed the Ancestors() expression in
2. A string expression. This expression will be evaluated for each member in the set passed as first argument. We'll use this to retrieve the caption of the current member of the hiearchy for which we're generating the ancestors list.
3. A separator. Generate() concatenates the result values returned by the string expression passed as second argument, and this string will be used to separate those values. Since we want to obtain a comma-separated list, we'll use the literal string ", " for this argument.

WITH
MEMBER  Measures.[Time Ancestors]
AS      Generate(
          Hierarchize(
            Ancestors(
              Time.CurrentMember, 
              CInt(
                Time.CurrentMember.Properties("LEVEL_NUMBER")
              )
            )
          )
        , Time.CurrentMember.Properties("MEMBER_CAPTION")
        , ","
        )
SELECT  Measures.[Time Ancestors] ON COLUMNS
,       Time.Quarters.Members ON ROWS
FROM    SteelWheelsSales

• Tens of PB - maybe that doesn't fit in memory after all
• High availability - automated fast failover in less than 30 seconds courtesy of lossless semisync and a lot of hard work from the MySQL PE team. See Fast Master Failover without Data Loss.
• 170M QPS at peak - sorry, no tweet to share yet

The MySQL 5.6.4 release introduced support for fractional values within the temporal datatypes: TIME, DATETIME, and TIMESTAMP. Hence the storage requirement and encoding differs for them in comparison to older (5.5 and earlier) temporal datatypes. The storage format for the temporal datatypes in the old format are not space efficient either, and recreating tables having both the new and old formats can be a long and tedious process. For these reasons, we wanted to make it easier for users to identify precisely which tables, if any, need to be upgraded.

In my previous blog post, where we looked at the process of upgrading old MySQL-5.5 format temporals to the MySQL-5.6 format, there was the question about how one would go about identifying whether a table actually contained temporal columns in the old format or not (thus needing to be upgraded). Based on the feedback we received from one of our customers, and also for the benefit of all our MySQL users who plan to upgrade tables having such columns to the new format, we have introduced a new server option in 5.6.24 called show_old_temporals. When this variable is enabled, the SHOW CREATE TABLE behavior for that session is changed so that we use comments to clearly mark the temporal columns that are using the old binary format. For example:

mysql> SET SESSION show_old_temporals=ON;
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> SHOW WARNINGS;
+---------+------+-------------------------------------------------------------------------------+
| Level   | Code | Message |
+---------+------+-------------------------------------------------------------------------------+
| Warning | 1287 | '@@show_old_temporals' is deprecated and will be removed in a future release. |
+---------+------+-------------------------------------------------------------------------------+
1 row in set (0.00 sec) 

mysql> SHOW CREATE TABLE ts;
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ts    | CREATE TABLE `ts` (
  `f_time` time /* 5.5 binary format */ DEFAULT NULL,
  `f_timestamp` timestamp /* 5.5 binary format */ NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `f_datetime` datetime /* 5.5 binary format */ DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Also a similar comment is added to the ‘COLUMN_TYPE’ field in the Information_Schema.COLUMNS table:

mysql> SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE table_name='ts';
+---------------+--------------+------------+-------------+------------------+-------------------+-------------+-----------+--------------------------+------------------------+-------------------+---------------+--------------------+--------------------+----------------+-----------------------------------+------------+-----------------------------+---------------------------------+----------------+
| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | COLUMN_NAME | ORDINAL_POSITION | COLUMN_DEFAULT    | IS_NULLABLE | DATA_TYPE | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_SCALE | DATETIME_PRECISION | CHARACTER_SET_NAME | COLLATION_NAME | COLUMN_TYPE                       | COLUMN_KEY | EXTRA                       | PRIVILEGES                      | COLUMN_COMMENT |
+---------------+--------------+------------+-------------+------------------+-------------------+-------------+-----------+--------------------------+------------------------+-------------------+---------------+--------------------+--------------------+----------------+-----------------------------------+------------+-----------------------------+---------------------------------+----------------+
| def           | test         | ts         | f_time      |                1 | NULL              | YES         | time      |                     NULL |                   NULL |              NULL |          NULL |                  0 | NULL               | NULL           | time /* 5.5 binary format */      |            |                             | select,insert,update,references |                |
| def           | test         | ts         | f_timestamp |                2 | CURRENT_TIMESTAMP | NO          | timestamp |                     NULL |                   NULL |              NULL |          NULL |                  0 | NULL               | NULL           | timestamp /* 5.5 binary format */ |            | on update CURRENT_TIMESTAMP | select,insert,update,references |                |
| def           | test         | ts         | f_datetime  |                3 | NULL              | YES         | datetime  |                     NULL |                   NULL |              NULL |          NULL |                  0 | NULL               | NULL           | datetime /* 5.5 binary format */  |            |                             | select,insert,update,references |                |
+---------------+--------------+------------+-------------+------------------+-------------------+-------------+-----------+--------------------------+------------------------+-------------------+---------------+--------------------+--------------------+----------------+-----------------------------------+------------+-----------------------------+---------------------------------+----------------+
3 rows in set (0.00 sec)

When show_old_temporals is OFF (the default), then both SHOW CREATE TABLE and Information_Schema.COLUMNS will provide the standard behavior and output.

As listed in my previous blog post, there are disadvantages of having tables with temporal columns in the old format, and hence we will remove support for them entirely in a future release. For this reason, the show_old_temporals option is already deprecated and will also be removed in a future release. Its value is only temporary, and it will be removed at the same time that we remove support for the old temporal formats.

We really hope that this new feature makes users’ lives easier when upgrading to MySQL 5.6 and later! We also look forward to your feedback! You can leave a comment here on the blog post or in a support ticket. If you feel that you encountered any related bugs, please do let us know via a bug report.

As always, THANK YOU for using MySQL!

I recently hosted a webinar introducing MySQL Cluster and then looking into what’s new in the latest version (MySQL Cluster 7.4) in some more detail. The replay of the MySQL Cluster 7.4 webinar is now available here. Alternatively if just want to skim through the charts then scroll down.

Abstract

MySQL Cluster powers the subscriber databases of major communication services providers as well as next generation web, cloud, social and mobile applications. It is designed to deliver:

• Real-time, in-memory performance for both OLTP and analytics workloads
• Linear scale-out for both reads and writes
• 99.999% High Availability
• Transparent, cross-shard transactions and joins
• Update-Anywhere Geographic replication
• SQL or native NoSQL APIs
• All that while still providing full ACID transactions.

Understand some of the highlights of MySQL Cluster 7.4:

• 200 Million queries per minute
• Active-Active geographic replication with conflict detection and resolution
• 5x faster on-line maintenance activities
• Enhanced reporting for memory and database operations

Charts

When optimizing queries and investigating performance issues, MySQL comes with built in support for profiling queries aka

SET profiling = 1;

Because profiling will be removed soon (already deprecated on MySQL 5.6 ad 5.7); the built-in profiling capability can only be enabled per session. This means that you cannot capture profiling information for queries running from other connections. If you are using Percona Server, the profiling option for log_slow_verbosity is a nice alternative, unfortunately, not everyone is using Percona Server.

Now, for a quick demo: I execute a simple query and profile it below. Note that all of these commands are executed from a single session to my test instance.

mysql> SHOW PROFILES;
+----------+------------+----------------------------------------+
| Query_ID | Duration   | Query                                  |
+----------+------------+----------------------------------------+
|        1 | 0.00011150 | SELECT * FROM sysbench.sbtest1 LIMIT 1 |
+----------+------------+----------------------------------------+
1 row in set, 1 warning (0.00 sec)
mysql> SHOW PROFILE SOURCE FOR QUERY 1;
+----------------------+----------+-----------------------+------------------+-------------+
| Status               | Duration | Source_function       | Source_file      | Source_line |
+----------------------+----------+-----------------------+------------------+-------------+
| starting             | 0.000017 | NULL                  | NULL             |        NULL |
| checking permissions | 0.000003 | check_access          | sql_parse.cc     |        5797 |
| Opening tables       | 0.000021 | open_tables           | sql_base.cc      |        5156 |
| init                 | 0.000009 | mysql_prepare_select  | sql_select.cc    |        1050 |
| System lock          | 0.000005 | mysql_lock_tables     | lock.cc          |         306 |
| optimizing           | 0.000002 | optimize              | sql_optimizer.cc |         138 |
| statistics           | 0.000006 | optimize              | sql_optimizer.cc |         381 |
| preparing            | 0.000005 | optimize              | sql_optimizer.cc |         504 |
| executing            | 0.000001 | exec                  | sql_executor.cc  |         110 |
| Sending data         | 0.000025 | exec                  | sql_executor.cc  |         190 |
| end                  | 0.000002 | mysql_execute_select  | sql_select.cc    |        1105 |
| query end            | 0.000003 | mysql_execute_command | sql_parse.cc     |        5465 |
| closing tables       | 0.000004 | mysql_execute_command | sql_parse.cc     |        5544 |
| freeing items        | 0.000005 | mysql_parse           | sql_parse.cc     |        6969 |
| cleaning up          | 0.000006 | dispatch_command      | sql_parse.cc     |        1874 |
+----------------------+----------+-----------------------+------------------+-------------+
15 rows in set, 1 warning (0.00 sec)

To demonstrate how we can achieve the same with Performance Schema, we first identify our current connection id. In the real world, you might want to get the connection/processlist id of the thread you want to watch i.e. from

SHOW PROCESSLIST

mysql> SELECT THREAD_ID INTO @my_thread_id
    -> FROM threads WHERE PROCESSLIST_ID = CONNECTION_ID();
Query OK, 1 row affected (0.00 sec)

Next, we identify the bounding EVENT_IDs for the statement stages. We will look for the statement we wanted to profile using the query below from the

events_statements_history_long

mysql> SELECT THREAD_ID, EVENT_ID, END_EVENT_ID, SQL_TEXT, NESTING_EVENT_ID
    -> FROM events_statements_history_long
    -> WHERE THREAD_ID = @my_thread_id
    ->   AND EVENT_NAME = 'statement/sql/select'
    -> ORDER BY EVENT_ID DESC LIMIT 3 G
*************************** 1. row ***************************
       THREAD_ID: 13848
        EVENT_ID: 419
    END_EVENT_ID: 434
        SQL_TEXT: SELECT THREAD_ID INTO @my_thread_id
FROM threads WHERE PROCESSLIST_ID = CONNECTION_ID()
NESTING_EVENT_ID: NULL
*************************** 2. row ***************************
       THREAD_ID: 13848
        EVENT_ID: 374
    END_EVENT_ID: 392
        SQL_TEXT: SELECT * FROM sysbench.sbtest1 LIMIT 1
NESTING_EVENT_ID: NULL
*************************** 3. row ***************************
       THREAD_ID: 13848
        EVENT_ID: 353
    END_EVENT_ID: 364
        SQL_TEXT: select @@version_comment limit 1
NESTING_EVENT_ID: NULL
3 rows in set (0.02 sec)

From the results above, we are mostly interested with the EVENT_ID and END_EVENT_ID values from the second row, this will give us the stage events of this particular query from the

events_stages_history_long

mysql> SELECT EVENT_NAME, SOURCE, (TIMER_END-TIMER_START)/1000000000 as 'DURATION (ms)'
    -> FROM events_stages_history_long
    -> WHERE THREAD_ID = @my_thread_id AND EVENT_ID BETWEEN 374 AND 392;
+--------------------------------+----------------------+---------------+
| EVENT_NAME                     | SOURCE               | DURATION (ms) |
+--------------------------------+----------------------+---------------+
| stage/sql/init                 | mysqld.cc:998        |        0.0214 |
| stage/sql/checking permissions | sql_parse.cc:5797    |        0.0023 |
| stage/sql/Opening tables       | sql_base.cc:5156     |        0.0205 |
| stage/sql/init                 | sql_select.cc:1050   |        0.0089 |
| stage/sql/System lock          | lock.cc:306          |        0.0047 |
| stage/sql/optimizing           | sql_optimizer.cc:138 |        0.0016 |
| stage/sql/statistics           | sql_optimizer.cc:381 |        0.0058 |
| stage/sql/preparing            | sql_optimizer.cc:504 |        0.0044 |
| stage/sql/executing            | sql_executor.cc:110  |        0.0008 |
| stage/sql/Sending data         | sql_executor.cc:190  |        0.0251 |
| stage/sql/end                  | sql_select.cc:1105   |        0.0017 |
| stage/sql/query end            | sql_parse.cc:5465    |        0.0031 |
| stage/sql/closing tables       | sql_parse.cc:5544    |        0.0037 |
| stage/sql/freeing items        | sql_parse.cc:6969    |        0.0056 |
| stage/sql/cleaning up          | sql_parse.cc:1874    |        0.0006 |
+--------------------------------+----------------------+---------------+
15 rows in set (0.01 sec)

As you can see the results are pretty close, not exactly the same but close. SHOW PROFILE shows Duration in seconds, while the results above is in milliseconds.

Some limitations to this method though:

• As we’ve seen it takes a few hoops to dish out the information we need. Because we have to identify the statement we have to profile manually, this procedure may not be easy to port into tools like the sys schema or pstop.
• Only possible if Performance Schema is enabled (by default its enabled since MySQL 5.6.6, yay!)
• Does not cover all metrics compared to the native profiling i.e. CONTEXT SWITCHES, BLOCK IO, SWAPS
• Depending on how busy the server you are running the tests, the sizes of the history tables may be too small, as such you either have to increase or loose the history to early i.e.

  performance_schema_events_stages_history_long_size

  variable. Using ps_history might help in this case though with a little modification to the queries.
• The resulting Duration per event may vary, I would think this may be due to the additional as described on performance_timers table. In any case we hope to get this cleared up as result when this bug is fixed.

The post Profiling MySQL queries from Performance Schema appeared first on MySQL Performance Blog.

Performance Schema is used extensively both internally and within the MySQL community, and I expect even more usage with the new SYS Schema and the Performance Schema enhancements in 5.7. Performance Schema is the single best tool available for monitoring MySQL Server internals and execution details at a lower level. Having said that, we are also no stranger to the fact that any monitoring tool comes with an additional cost to performance. Hence It has always been an important question to find out just how much it costs us when Performance Schema is turned ON and to see what we can do to make it perform as fast as possible.

I have been using MySQL Performance Schema for the past year. This is my first blog on Performance Schema and here I am primarily concerned with the performance impact and characteristics around the default values.

Performance Schema Default ON vs OFF

The question I will try and answer here is this: How well does MySQL scale with performance_schema=ON (default parameters and configuration) compared to performance_schema=OFF?

Test Details

I used Sysbench to get TPS benchmarks. Just to ensure that I have some variation in my tests I ran Sysbench in both the CPU bound and the Disk bound environments to gather the stats. All tests were performed on the same server and with the same MySQL Server build. I used the latest development release (at the time), 5.7.6-m16, for my tests. With MySQL 5.7.6, 311 instruments are enabled by default.

Test Results

I was very happy to see the answers after the tests ran! They clearly show that the numbers are quite reasonable (within 1% – 3%) if we consider the huge benefits offered by Performance Schema with the default settings. More Details below:

CPU Bound

Sysbench Read Only Mode

Sysbench Read Write Mode

Disk Bound

Sysbench Read Only Mode

Sysbench Read Write Mode

performance_schema=ON shows a dip of 0.45%

Conclusion

In my tests we see average overhead results of:

• Just over 2% in the CPU Bound OLTP RW (2.23%) and OLTP_RO (2.40%) runs
• Less than 1% in the Disk Bound OLTP RW (0.41%) and OLTP_RO (0.92%) runs

Overall, as we see I think you can expect a performance impact of approximately 1-3% with performance_schema=ON vs OFF (again, using the default configuration). The most important takeaway is that while there are still some dips in raw performance numbers—which is expected—the overall impact is relatively small. We will also continue to do all that we can to lower the overhead and performance impact of Performance Schema even further in the future! I hope that these benchmarks may help anyone looking forward to using Performance Schema extensively, but are concerned about what performance impact may come with it.

As I’ve demonstrated here—due to all of the helpful input from the MySQL Community, and the work that we’ve done to lower the related impact and overhead—the Performance Schema can truly be said to offer great power, and do so without great cost!

That’s all for now. Thank you for using MySQL!

Appendix: Test Details.

Table1: 311 Instruments

Table2: MySQL Details

Table3: Test Machine Details

Table4: Sysbench Configuration Details

Table5: MySQLD Configuration Details

Want to introduce our MySQL-AutoXtraBackup command line tool, using Percona Xtrabackup in core. Looking for contributor

Project Structure:

XtraBackup is powerfull and open-source hot online backup tool for MySQL from Percona.
This script is using XtraBackup for full and incremental backups, also for preparing and recovering taken backups.
Here is project path tree (default location is /home/MySQL-AutoXtraBackup):

* backup_dir — The main folder for storing backups.
* master_backup_script — Full and Incremental backup taker script.
* backup_prepare — Backup prepare and restore script.
* partial_recovery — Partial table recovery script.
* general_conf — All-in-one config file’s and config reader class folder.
* setup.py — Setup file.
* autoxtrabackup.py — Commandline Tool provider script.

Here you can watch Demo Usage Video.

The post MySQL-AutoXtrabackup command line tool for using Percona Xtrabackup appeared first on Azerbaijan MySQL UG.

This Log Buffer Edition covers Oracle, MySQL, SQL Server blog posts from around the world.

Oracle:

• Why the Internet of Things should matter to you
• Modifying Sales Behavior Using Oracle SPM – Written by Tyrice Johnson
• SQLcl: Run a Query Over and Over, Refresh the Screen
• Data Integration Tips: ODI 12.1.3 – Convert to Flow
• JRE 1.8.0_45 Certified with Oracle E-Business Suite

SQL Server:

• What’s this, a conditional WHERE clause that doesn’t use dynamic SQL?
• The job of a DBA requires a fusion of skill and knowledge. To acquire this requires a craftsman mindset. Craftsmen find that the better they get at the work, the more enjoyable the work gets, and the more successful they become.
• Using SQL to perform cluster analysis to gain insight into data with unknown groups
• There are times when you don’t what to return a complete set of records. When you have this kind of requirement to only select the TOP X number of items Transact SQL (TSQL) has the TOP clause to meet your needs.
• Spatial Data in SQL Server has special indexing because it has to perform specialised functions.

MySQL:

Profiling MySQL queries from Performance Schema

How to Easily Identify Tables With Temporal Types in Old Format!

The Perfect Server – CentOS 7.1 with Apache2, Postfix, Dovecot, Pure-FTPD, BIND and ISPConfig 3

Database Security – How to fully SSL-encrypt MySQL Galera Cluster and ClusterControl

MDX: retrieving the entire hierarchy path with Ancestors()

It turns out that configuring Perl wasn’t the last step for my student instance. It appears that I neglected to configure my student instance to support Java connectivity to MySQL. This post reviews the configuration of Java to run programs against MySQL. It also covers the new syntax on how you register a DriverManager, and avoid Java compilation errors with the older syntax.

In prior posts, I’ve shown how to use Perl , PHP, Python, and Ruby languages to query a MySQL database on Linux.

You need to install the Open JDK libraries with the yum utility command:

yum install -y java-1.7.0-openjdk*

It should generate the following log output:

Loaded plugins: langpacks, refresh-packagekit
Package 1:java-1.7.0-openjdk-1.7.0.75-2.5.4.2.fc20.x86_64 already installed and latest version
Package 1:java-1.7.0-openjdk-headless-1.7.0.75-2.5.4.2.fc20.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package java-1.7.0-openjdk-accessibility.x86_64 1:1.7.0.75-2.5.4.2.fc20 will be installed
--> Processing Dependency: java-atk-wrapper for package: 1:java-1.7.0-openjdk-accessibility-1.7.0.75-2.5.4.2.fc20.x86_64
---> Package java-1.7.0-openjdk-demo.x86_64 1:1.7.0.75-2.5.4.2.fc20 will be installed
---> Package java-1.7.0-openjdk-devel.x86_64 1:1.7.0.75-2.5.4.2.fc20 will be installed
---> Package java-1.7.0-openjdk-javadoc.noarch 1:1.7.0.75-2.5.4.2.fc20 will be installed
---> Package java-1.7.0-openjdk-src.x86_64 1:1.7.0.75-2.5.4.2.fc20 will be installed
--> Running transaction check
---> Package java-atk-wrapper.x86_64 0:0.30.4-4.fc20 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
================================================================================
 Package                          Arch   Version                  Repository
                                                                           Size
================================================================================
Installing:
 java-1.7.0-openjdk-accessibility x86_64 1:1.7.0.75-2.5.4.2.fc20  updates  32 k
 java-1.7.0-openjdk-demo          x86_64 1:1.7.0.75-2.5.4.2.fc20  updates 1.9 M
 java-1.7.0-openjdk-devel         x86_64 1:1.7.0.75-2.5.4.2.fc20  updates 9.2 M
 java-1.7.0-openjdk-javadoc       noarch 1:1.7.0.75-2.5.4.2.fc20  updates  14 M
 java-1.7.0-openjdk-src           x86_64 1:1.7.0.75-2.5.4.2.fc20  updates  39 M
Installing for dependencies:
 java-atk-wrapper                 x86_64 0.30.4-4.fc20            fedora   71 k
 
Transaction Summary
================================================================================
Install  12 Packages (+1 Dependent package)
 
Total download size: 163 M
Installed size: 765 M
Downloading packages:
(1/6): java-1.7.0-openjdk-accessibility-1.7.0.75-2.5.4.2.f |  32 kB  00:00     
(2/6): java-1.7.0-openjdk-demo-1.7.0.75-2.5.4.2.fc20.x86_6 | 1.9 MB  00:02     
(3/6): java-1.7.0-openjdk-devel-1.7.0.75-2.5.4.2.fc20.x86_ | 9.2 MB  00:05     
(4/6): java-1.7.0-openjdk-javadoc-1.7.0.75-2.5.4.2.fc20.no |  14 MB  00:04     
(5/6): java-atk-wrapper-0.30.4-4.fc20.x86_64.rpm           |  71 kB  00:00     
(6/6): java-1.7.0-openjdk-src-1.7.0.75-2.5.4.2.fc20.x86_6  |  39 MB  00:23     
--------------------------------------------------------------------------------
Total                                             4.5 MB/s | 163 MB  00:36     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction (shutdown inhibited)
  Installing : java-atk-wrapper-0.30.4-4.fc20.x86_64                       3/13 
  Installing : 1:java-1.7.0-openjdk-accessibility-1.7.0.75-2.5.4.2.fc20    4/13 
  Installing : 1:java-1.7.0-openjdk-devel-1.7.0.75-2.5.4.2.fc20.x86_64     9/13 
  Installing : 1:java-1.7.0-openjdk-src-1.7.0.75-2.5.4.2.fc20.x86_64      10/13 
  Installing : 1:java-1.7.0-openjdk-javadoc-1.7.0.75-2.5.4.2.fc20.noarc   12/13 
  Installing : 1:java-1.7.0-openjdk-demo-1.7.0.75-2.5.4.2.fc20.x86_64     13/13 
  Verifying  : 1:java-1.7.0-openjdk-demo-1.7.0.75-2.5.4.2.fc20.x86_64      2/13 
  Verifying  : 1:java-1.7.0-openjdk-javadoc-1.7.0.75-2.5.4.2.fc20.noarc    3/13 
  Verifying  : java-atk-wrapper-0.30.4-4.fc20.x86_64                       5/13 
  Verifying  : 1:java-1.7.0-openjdk-accessibility-1.7.0.75-2.5.4.2.fc20    6/13 
  Verifying  : 1:java-1.7.0-openjdk-devel-1.7.0.75-2.5.4.2.fc20.x86_64     8/13 
  Verifying  : 1:java-1.7.0-openjdk-src-1.7.0.75-2.5.4.2.fc20.x86_64      12/13 
 
Installed:
  java-1.7.0-openjdk-accessibility.x86_64 1:1.7.0.75-2.5.4.2.fc20               
  java-1.7.0-openjdk-demo.x86_64 1:1.7.0.75-2.5.4.2.fc20                        
  java-1.7.0-openjdk-devel.x86_64 1:1.7.0.75-2.5.4.2.fc20                       
  java-1.7.0-openjdk-javadoc.noarch 1:1.7.0.75-2.5.4.2.fc20                     
  java-1.7.0-openjdk-src.x86_64 1:1.7.0.75-2.5.4.2.fc20                         
 
Dependency Installed:
  java-atk-wrapper.x86_64 0:0.30.4-4.fc20                                       
 
Complete!

You can find the Java compiler’s version with the following command:

javac -version

It should show you the following Java version:

javac 1.7.0_75

Next, you need to install the mysql-connector-java library with yum like this:

yum install -y mysql-connector-java

It should generate the following installation output:

Loaded plugins: langpacks, refresh-packagekit
mysql-connectors-community                                  | 2.5 kB  00:00     
mysql-tools-community                                       | 2.5 kB  00:00     
mysql56-community                                           | 2.5 kB  00:00     
pgdg93                                                      | 3.6 kB  00:00     
updates/20/x86_64/metalink                                  |  15 kB  00:00     
Resolving Dependencies
--> Running transaction check
---> Package mysql-connector-java.noarch 1:5.1.28-1.fc20 will be installed
--> Processing Dependency: jta >= 1.0 for package: 1:mysql-connector-java-5.1.28-1.fc20.noarch
--> Processing Dependency: slf4j for package: 1:mysql-connector-java-5.1.28-1.fc20.noarch
--> Running transaction check
---> Package geronimo-jta.noarch 0:1.1.1-15.fc20 will be installed
---> Package slf4j.noarch 0:1.7.5-3.fc20 will be installed
--> Processing Dependency: mvn(log4j:log4j) for package: slf4j-1.7.5-3.fc20.noarch
--> Processing Dependency: mvn(javassist:javassist) for package: slf4j-1.7.5-3.fc20.noarch
--> Processing Dependency: mvn(commons-logging:commons-logging) for package: slf4j-1.7.5-3.fc20.noarch
--> Processing Dependency: mvn(commons-lang:commons-lang) for package: slf4j-1.7.5-3.fc20.noarch
--> Processing Dependency: mvn(ch.qos.cal10n:cal10n-api) for package: slf4j-1.7.5-3.fc20.noarch
--> Running transaction check
---> Package apache-commons-lang.noarch 0:2.6-13.fc20 will be installed
---> Package apache-commons-logging.noarch 0:1.1.3-8.fc20 will be installed
--> Processing Dependency: mvn(logkit:logkit) for package: apache-commons-logging-1.1.3-8.fc20.noarch
--> Processing Dependency: mvn(avalon-framework:avalon-framework-api) for package: apache-commons-logging-1.1.3-8.fc20.noarch
---> Package cal10n.noarch 0:0.7.7-3.fc20 will be installed
---> Package javassist.noarch 0:3.16.1-6.fc20 will be installed
---> Package log4j.noarch 0:1.2.17-14.fc20 will be installed
--> Processing Dependency: mvn(org.apache.geronimo.specs:geronimo-jms_1.1_spec) for package: log4j-1.2.17-14.fc20.noarch
--> Processing Dependency: mvn(javax.mail:mail) for package: log4j-1.2.17-14.fc20.noarch
--> Running transaction check
---> Package avalon-framework.noarch 0:4.3-9.fc20 will be installed
--> Processing Dependency: xalan-j2 for package: avalon-framework-4.3-9.fc20.noarch
---> Package avalon-logkit.noarch 0:2.1-13.fc20 will be installed
--> Processing Dependency: tomcat-servlet-3.0-api for package: avalon-logkit-2.1-13.fc20.noarch
---> Package geronimo-jms.noarch 0:1.1.1-17.fc20 will be installed
---> Package javamail.noarch 0:1.5.0-6.fc20 will be installed
--> Running transaction check
---> Package tomcat-servlet-3.0-api.noarch 0:7.0.52-2.fc20 will be installed
---> Package xalan-j2.noarch 0:2.7.1-22.fc20 will be installed
--> Processing Dependency: xerces-j2 for package: xalan-j2-2.7.1-22.fc20.noarch
--> Processing Dependency: osgi(org.apache.xerces) for package: xalan-j2-2.7.1-22.fc20.noarch
--> Running transaction check
---> Package xerces-j2.noarch 0:2.11.0-17.fc20 will be installed
--> Processing Dependency: xml-commons-resolver >= 1.2 for package: xerces-j2-2.11.0-17.fc20.noarch
--> Processing Dependency: xml-commons-apis >= 1.4.01 for package: xerces-j2-2.11.0-17.fc20.noarch
--> Processing Dependency: osgi(org.apache.xml.resolver) for package: xerces-j2-2.11.0-17.fc20.noarch
--> Processing Dependency: osgi(javax.xml) for package: xerces-j2-2.11.0-17.fc20.noarch
--> Running transaction check
---> Package xml-commons-apis.noarch 0:1.4.01-14.fc20 will be installed
---> Package xml-commons-resolver.noarch 0:1.2-14.fc20 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
================================================================================
 Package                    Arch       Version                Repository   Size
================================================================================
Installing:
 mysql-connector-java       noarch     1:5.1.28-1.fc20        updates     1.3 M
Installing for dependencies:
 apache-commons-lang        noarch     2.6-13.fc20            fedora      281 k
 apache-commons-logging     noarch     1.1.3-8.fc20           updates      78 k
 avalon-framework           noarch     4.3-9.fc20             fedora       87 k
 avalon-logkit              noarch     2.1-13.fc20            fedora       87 k
 cal10n                     noarch     0.7.7-3.fc20           fedora       37 k
 geronimo-jms               noarch     1.1.1-17.fc20          fedora       32 k
 geronimo-jta               noarch     1.1.1-15.fc20          fedora       21 k
 javamail                   noarch     1.5.0-6.fc20           fedora      606 k
 javassist                  noarch     3.16.1-6.fc20          fedora      626 k
 log4j                      noarch     1.2.17-14.fc20         fedora      449 k
 slf4j                      noarch     1.7.5-3.fc20           fedora      173 k
 tomcat-servlet-3.0-api     noarch     7.0.52-2.fc20          updates     207 k
 xalan-j2                   noarch     2.7.1-22.fc20          updates     1.9 M
 xerces-j2                  noarch     2.11.0-17.fc20         updates     1.1 M
 xml-commons-apis           noarch     1.4.01-14.fc20         fedora      227 k
 xml-commons-resolver       noarch     1.2-14.fc20            fedora      108 k
 
Transaction Summary
================================================================================
Install  1 Package (+16 Dependent packages)
 
Total download size: 7.3 M
Installed size: 10 M
Downloading packages:
(1/17): apache-commons-logging-1.1.3-8.fc20.noarch.rpm      |  78 kB  00:00     
(2/17): apache-commons-lang-2.6-13.fc20.noarch.rpm          | 281 kB  00:00     
(3/17): avalon-framework-4.3-9.fc20.noarch.rpm              |  87 kB  00:00     
(4/17): avalon-logkit-2.1-13.fc20.noarch.rpm                |  87 kB  00:00     
(5/17): cal10n-0.7.7-3.fc20.noarch.rpm                      |  37 kB  00:00     
(6/17): geronimo-jms-1.1.1-17.fc20.noarch.rpm               |  32 kB  00:00     
(7/17): geronimo-jta-1.1.1-15.fc20.noarch.rpm               |  21 kB  00:00     
(8/17): javamail-1.5.0-6.fc20.noarch.rpm                    | 606 kB  00:00     
(9/17): javassist-3.16.1-6.fc20.noarch.rpm                  | 626 kB  00:00     
(10/17): log4j-1.2.17-14.fc20.noarch.rpm                    | 449 kB  00:00     
(11/17): slf4j-1.7.5-3.fc20.noarch.rpm                      | 173 kB  00:00     
(12/17): mysql-connector-java-5.1.28-1.fc20.noarch.rpm      | 1.3 MB  00:01     
(13/17): tomcat-servlet-3.0-api-7.0.52-2.fc20.noarch.rpm    | 207 kB  00:00     
(14/17): xalan-j2-2.7.1-22.fc20.noarch.rpm                  | 1.9 MB  00:00     
(15/17): xerces-j2-2.11.0-17.fc20.noarch.rpm                | 1.1 MB  00:00     
(16/17): xml-commons-apis-1.4.01-14.fc20.noarch.rpm         | 227 kB  00:00     
(17/17): xml-commons-resolver-1.2-14.fc20.noarch.rpm        | 108 kB  00:00     
--------------------------------------------------------------------------------
Total                                              1.3 MB/s | 7.3 MB  00:05     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction (shutdown inhibited)
  Installing : xml-commons-apis-1.4.01-14.fc20.noarch                      1/17 
  Installing : geronimo-jms-1.1.1-17.fc20.noarch                           2/17 
  Installing : xml-commons-resolver-1.2-14.fc20.noarch                     3/17 
  Installing : xerces-j2-2.11.0-17.fc20.noarch                             4/17 
  Installing : xalan-j2-2.7.1-22.fc20.noarch                               5/17 
  Installing : javamail-1.5.0-6.fc20.noarch                                6/17 
  Installing : log4j-1.2.17-14.fc20.noarch                                 7/17 
  Installing : tomcat-servlet-3.0-api-7.0.52-2.fc20.noarch                 8/17 
  Installing : avalon-framework-4.3-9.fc20.noarch                          9/17 
  Installing : avalon-logkit-2.1-13.fc20.noarch                           10/17 
  Installing : apache-commons-logging-1.1.3-8.fc20.noarch                 11/17 
  Installing : javassist-3.16.1-6.fc20.noarch                             12/17 
  Installing : cal10n-0.7.7-3.fc20.noarch                                 13/17 
  Installing : apache-commons-lang-2.6-13.fc20.noarch                     14/17 
  Installing : slf4j-1.7.5-3.fc20.noarch                                  15/17 
  Installing : geronimo-jta-1.1.1-15.fc20.noarch                          16/17 
  Installing : 1:mysql-connector-java-5.1.28-1.fc20.noarch                17/17 
  Verifying  : geronimo-jta-1.1.1-15.fc20.noarch                           1/17 
  Verifying  : geronimo-jms-1.1.1-17.fc20.noarch                           2/17 
  Verifying  : xalan-j2-2.7.1-22.fc20.noarch                               3/17 
  Verifying  : apache-commons-lang-2.6-13.fc20.noarch                      4/17 
  Verifying  : slf4j-1.7.5-3.fc20.noarch                                   5/17 
  Verifying  : log4j-1.2.17-14.fc20.noarch                                 6/17 
  Verifying  : avalon-framework-4.3-9.fc20.noarch                          7/17 
  Verifying  : xerces-j2-2.11.0-17.fc20.noarch                             8/17 
  Verifying  : cal10n-0.7.7-3.fc20.noarch                                  9/17 
  Verifying  : avalon-logkit-2.1-13.fc20.noarch                           10/17 
  Verifying  : 1:mysql-connector-java-5.1.28-1.fc20.noarch                11/17 
  Verifying  : xml-commons-resolver-1.2-14.fc20.noarch                    12/17 
  Verifying  : xml-commons-apis-1.4.01-14.fc20.noarch                     13/17 
  Verifying  : javassist-3.16.1-6.fc20.noarch                             14/17 
  Verifying  : tomcat-servlet-3.0-api-7.0.52-2.fc20.noarch                15/17 
  Verifying  : javamail-1.5.0-6.fc20.noarch                               16/17 
  Verifying  : apache-commons-logging-1.1.3-8.fc20.noarch                 17/17 
 
Installed:
  mysql-connector-java.noarch 1:5.1.28-1.fc20                                   
 
Dependency Installed:
  apache-commons-lang.noarch 0:2.6-13.fc20                                      
  apache-commons-logging.noarch 0:1.1.3-8.fc20                                  
  avalon-framework.noarch 0:4.3-9.fc20                                          
  avalon-logkit.noarch 0:2.1-13.fc20                                            
  cal10n.noarch 0:0.7.7-3.fc20                                                  
  geronimo-jms.noarch 0:1.1.1-17.fc20                                           
  geronimo-jta.noarch 0:1.1.1-15.fc20                                           
  javamail.noarch 0:1.5.0-6.fc20                                                
  javassist.noarch 0:3.16.1-6.fc20                                              
  log4j.noarch 0:1.2.17-14.fc20                                                 
  slf4j.noarch 0:1.7.5-3.fc20                                                   
  tomcat-servlet-3.0-api.noarch 0:7.0.52-2.fc20                                 
  xalan-j2.noarch 0:2.7.1-22.fc20                                               
  xerces-j2.noarch 0:2.11.0-17.fc20                                             
  xml-commons-apis.noarch 0:1.4.01-14.fc20                                      
  xml-commons-resolver.noarch 0:1.2-14.fc20                                     
 
Complete!

I must write too much Java code for the Windows platform because I didn’t notice the change in how the DriverManager should be instantiated. Initially, I wrote the program using the following declaration for the DriverManager class:

30

      DriverManager.registerDriver(new com.mysql.jdbc.Driver());

While it worked on Windows, the same syntax in the MySQL.java program raised two errors on the Linux server. One for the declaration of the com.mysql.jdbc.Driver class and another trying to declare an instance of Driver class.

These are the two errors:

MySQL.java:5: error: package com.mysql.jdbc does not exist
import com.mysql.jdbc.Driver;
                     ^
MySQL.java:31: error: package com.mysql.jdbc does not exist
      DriverManager.registerDriver(new com.mysql.jdbc.Driver());
                                                     ^

I rewrote the MySQL.java program as follows, and it works on both implementations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

// Import classes.
import java.sql.*;
 
/* You can't include the following on Linux without raising an exception. */
// import com.mysql.jdbc.Driver;
 
public class MySQL {
  public MySQL() {
    /* Declare variables that require explicit assignments because
       they're addressed in the finally block. */
    Connection conn = null;
    Statement stmt = null;
    ResultSet rset = null;
 
    /* Declare other variables. */
    String url;
    String username = "student";
    String password = "student";
    String database = "studentdb";
    String hostname = "localhost";
    String port = "3306";
    String sql;
 
    /* Attempt a connection. */
    try {
      // Set URL.
      url = "jdbc:mysql://" + hostname + ":" + port + "/" + database;
 
      // Create instance of MySQL.
      Class.forName ("com.mysql.jdbc.Driver").newInstance();
      conn = DriverManager.getConnection (url, username, password);
 
      // Query the version of the database.
      sql = "SELECT version()";
      stmt = conn.createStatement();
      rset = stmt.executeQuery(sql);
 
      System.out.println ("Database connection established");
 
      // Read row returns for one column.
      while (rset.next()) {
        System.out.println("MySQL Version [" + rset.getString(1) + "]"); }
 
    }
    catch (SQLException e) {
      System.err.println ("Cannot connect to database server:");
      System.out.println(e.getMessage());
    }
    catch (ClassNotFoundException e) {
      System.err.println ("Cannot connect to database server:");
      System.out.println(e.getMessage());
    }
    catch (InstantiationException e) {
      System.err.println ("Cannot connect to database server:");
      System.out.println(e.getMessage());
    }
    catch (IllegalAccessException e) {
      System.err.println ("Cannot connect to database server:");
      System.out.println(e.getMessage());
    }
    finally {
      if (conn != null) {
        try {
          rset.close();
          stmt.close();
          conn.close();
          System.out.println ("Database connection terminated");
        }
        catch (Exception e) { /* ignore close errors */ }
      }
    }
  }
  /* Unit test. */
  public static void main(String args[]) {
    new MySQL();
  }
}

The old approach to the DriverManager and Driver classes disallows the use of the ClassNotFoundException, InstantiationException, and IllegalAccessException classes. The new syntax works on Linux, Mac OS X, and Windows. If you’re running on Mac OS X, you need to import the following additional library in the MySQL.java program:

import com.apple.eawt.*;

Before you compile the MySQL.java program, you need to put the mysql-connector-java.jar and your present working directory into your environment’s $CLASSPATH variable. You can set the $CLASSPATH variable at the command-line or embed the following in your .bashrc file:

export CLASSPATH=/usr/share/java/mysql-connector-java.jar:.

If you embedded it in the .bashrc file, you need to source that file or restart your terminal session, which resources the .bashrc for you. You can source your .bashrc file from an active Terminal session in your home directory with this syntax:

. ./.bashrc

If you’re new to Java and the MySQL Connector/J, you compile the MySQL.java program with the following syntax. At least, it works when you have the MySQL.java source file in the present working directory and want to create the class file in the same directory. You can find more about the javac command-line at the www.tutorialpoint.com web site>

javac -verbose -cp . MySQL.java

Then, you can run it with the class file with this syntax:

java MySQL

It should return the following:

Database connection established
MySQL Version [5.6.24]
Database connection terminated

If you’d prefer to return data, you can replace line 34 in the MySQL.java program with a query against a table, like:

34

      sql = "SELECT item_title, item_rating FROM item";

Then, change line 42 in the MySQL.java program with syntax to manage the output, like:

42

        System.out.println(rset.getString(1) + ", " + rset.getString(2)); }

Recompile it, and rerun the MySQL class file with this syntax:

java MySQL

It should return the following:

Database connection established
The Hunt for Red October, PG
Star Wars I, PG
Star Wars II, PG
Star Wars II, PG
Star Wars III, PG13
The Chronicles of Narnia, PG
RoboCop, Mature
Pirates of the Caribbean, Teen
The Chronicles of Narnia, Everyone
MarioKart, Everyone
Splinter Cell, Teen
Need for Speed, Everyone
The DaVinci Code, Teen
Cars, Everyone
Beau Geste, PG
I Remember Mama, NR
Tora! Tora! Tora!, G
A Man for All Seasons, G
Hook, PG
Around the World in 80 Days, G
Harry Potter and the Chamber of Secrets, PG
Camelot, G
Database connection terminated

As always, I hope this helps those looking for a solution.

As you know there is a great toolset named “MySQL Utilities”, which you can use for solving various administrative tasks.
mysqldiskusage utility is for calculating MySQL Server’s disk usage and generating informative reports.
Of course this project is open source and everybody could review the source code.
A few words about how mysqldiskusage calculates database disk usage will be crucial for understanding algorithm.
The source tree is: mysql-utilities-1.5.4/scripts/mysqldiskusage.py
If you open this Python file you will see (line 169-175) :

 # We do database disk usage by default.
    try:
        diskusage.show_database_usage(servers[0], datadir, args, options)
    except UtilError:
        _, e, _ = sys.exc_info()
        print("ERROR: %s" % e.errmsg)
        sys.exit(1)

By default it shows database disk usage and calling another function named show_database_usage from mysql-utilities-1.5.4/mysql/utilities/command/diskusage.py file.
Now if we open up this diskusage.py file and search for show_database_usage function, you should see that in turn this function uses another function named _build_db_list. From _build_db_list it gets back all necessary information as in code stated clearly(line 550-562):

# Get list of databases with sizes and formatted when necessary
    columns, rows, db_total = _build_db_list(server, res, dblist, datadir,
                                             fmt == "grid",
                                             have_read, verbosity,
                                             include_empty or do_all,
                                             is_remote)

    if not quiet:
        print "# Database totals:"
    print_list(sys.stdout, fmt, columns, rows, no_headers)
    if not quiet:
        _print_size("\nTotal database disk usage = ", db_total)
        print

Now we know that all calculations are happened in _build_db_list function. If you search and find this function(begins from line 360) you can see that, in fact mysqldiskusage is calculating database disk usage as follows:

1. It finds (data_length + index_length) from information_schema.tables per database manner
2. Then it sum ups (data_length + index_length) with misc_files variable data which is in fact returned by _get_db_dir_size function.

But what is this misc_files? Logically misc_files must be .opt and .frm files. So misc_files must not be an “.MYD” , “.MYI”, “.IBD”, “general_log”, “slow_log”.
So in fact mysqldiskusage calculates database disk usage as -> (data_length + index_length)[size in bytes] + (.opt+.frm)[size in bytes].

First of all we must insist on not using information_schema for accurate disk usage calculation because of simple rule: “InnoDB preallocates pages(16Kib) for further table usage, but data_length column will not show these pages”
As proof of concept let’s create sample empty table:

mysql> create database test;
Query OK, 1 row affected (0,01 sec)

mysql> use test;
Database changed

mysql> select data_length, index_length from information_schema.tables where table_schema='test' and table_name='t1';
+-------------+--------------+
| data_length | index_length |
+-------------+--------------+
|       16384 |            0 |
+-------------+--------------+
1 row in set (0,00 sec)

If we consider that our exact table size is 16384 bytes, we are in wrong direction.
In fact if we use OS commands we could see that the exact size of table is 98304 bytes:

[root@node1 ~]# ls -lt /var/lib/mysql/test/
total 208
-rw-rw----. 1 mysql mysql 98304 Apr 18 11:44 t1.ibd

So when we create an InnoDB table it is preallocates 6 pages(16Kib*6 = 98304) but only 1 page shown up from data_length column.

Now let’s come back to our misc_files or exactly _get_db_dir_size() function.
From source code we can see that there is no a check for “.IBD” files:

...
for item in os.listdir(folder):
        name, ext = os.path.splitext(item)
        if ext.upper() not in (".MYD", ".MYI") and \
           name.upper() not in ('SLOW_LOG', 'GENERAL_LOG'):
            itemfolder = os.path.join(folder, item)
...

Because of this calculation is wrong for databases as we see from output:

[root@node1 ~]# mysqldiskusage --server=root_pass -vvv
# Source on localhost: ... connected.
# Database totals:
+---------------------+--------------+--------------+--------------+--------------+
| db_name             | db_dir_size  |   data_size  |  misc_files  |       total  |
+---------------------+--------------+--------------+--------------+--------------+
| employees           | 242.523.049  | 205.979.648  | 242.523.049  | 448.502.697  |

As you see it sum ups data_size with misc_files and gets back total as 448.502.697 bytes. But in fact our employees database is exactly 242.523.049 bytes. And of course the:
Total database disk usage = 450.940.391 bytes or 430,05 MB is wrong as well.
For further exploration and how to patch source code see related BUG REPORT #76703.

The post mysqldiskusage – source code examination appeared first on Azerbaijan MySQL UG.

Last week I posted how to configure and test Ruby and MySQL. Somebody asked me how to handle a dynamic list of columns. So, here’s a quick little program to show you how to read the dynamic list of column:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

require 'rubygems'
require 'mysql'
 
# Begin block.
begin
  # Create a new connection resource.
  db = Mysql.new('localhost','student','student','studentdb')
 
  # Create a result set.
  rs = db.query('SELECT item_title, item_rating FROM item')
  # Read through the result set hash.
  rs.each do | row |
    out = ""
    i = 0
    while i < db.field_count
      # Check if not last column.
      if i < db.field_count - 1
        out += "#{row[i]}, "
      else
        out += "#{row[i]}"
      end
      i += 1
    end
    puts "#{out}"
  end
  # Release the result set resources.
  rs.free
rescue Mysql::Error => e
  # Print the error.
  puts "ERROR #{e.errno} (#{e.sqlstate}): #{e.error}"
  puts "Can't connect to MySQL database specified."
  # Signal an error.
  exit 1
ensure
  # Close the connection when it is open.
  db.close if db
end

The new logic on lines 13 through 22 reads the list of columns into a comma delimited list of values. The if-block checks to make sure it doesn’t append a comma to the last column in the list. It prints output like:

The Hunt for Red October, PG
Star Wars I, PG
Star Wars II, PG
Star Wars II, PG
Star Wars III, PG13
The Chronicles of Narnia, PG
RoboCop, Mature
Pirates of the Caribbean, Teen
The Chronicles of Narnia, Everyone
MarioKart, Everyone
Splinter Cell, Teen
Need for Speed, Everyone
The DaVinci Code, Teen
Cars, Everyone
Beau Geste, PG
I Remember Mama, NR
Tora! Tora! Tora!, G
A Man for All Seasons, G
Hook, PG
Around the World in 80 Days, G
Harry Potter and the Sorcerer's Stone, PG
Camelot, G

As always, I hope this helps those looking for a solution.