This presentation was made at LSPE event in Bangalore (India) held at Walmart labs on 10-03-2018. This presentation focuses how we have harnessed the power of Ansible at Mydbops.

With MySQL 8.0, the error logging subsystem has been redesigned to use the new component architecture.

Thanks to this new redesign, now the log events can be filtered, the output can be sent to multiple destinations (different formats like JSON). All that is controlled by system variables.

This work gives the possibility for a log event to become the raw material for log processing by more modern and automated systems like filebeat for beats, kibana, etc…

Let’s check the default configuration:

mysql> select * from global_variables where VARIABLE_NAME like 'log_error_%';
+---------------------+----------------------------------------+
| VARIABLE_NAME       | VARIABLE_VALUE                         |
+---------------------+----------------------------------------+
| log_error_services  | log_filter_internal; log_sink_internal |
| log_error_verbosity | 2                                      |
+---------------------+----------------------------------------+

This means that log events will follow the following flow: first pass through log_filter_internal (built-in filter component), then through log_sink_internal (buit-in log writer component).

To enable a log component you need to use INSTALL COMPONENT command and set the log_error_services global variable as wished. To disable it use  UNINSTALL COMPONENT.

Currently the available log components are in lib/plugins:

• component_log_filter_dragnet.so
• component_log_sink_json.so
• component_log_sink_syseventlog.so
• component_log_sink_test.so

LOG OUTPUT

To specify a new output format, you need to use a log writer component (sink). So let’s try to use one.

To load a component , you need its URN. This is ‘file://‘ + the filename without the .so extension. Example, to load the writer to json component, you enable it like this:

mysql> INSTALL COMPONENT 'file://component_log_sink_json';
Query OK, 0 rows affected (0.14 sec)
mysql> SET GLOBAL log_error_services = 'log_filter_internal; log_sink_internal; log_sink_json';
Query OK, 0 rows affected (0.01 sec)
mysql> select * from global_variables where VARIABLE_NAME like 'log_error_%';
+---------------------+-------------------------------------------------------+
| VARIABLE_NAME       | VARIABLE_VALUE                                        |
+---------------------+-------------------------------------------------------+
| log_error_services  | log_filter_internal; log_sink_internal; log_sink_json |
| log_error_verbosity | 2                                                     |
+---------------------+-------------------------------------------------------+
2 rows in set (0.00 sec)

Now if I generate an entry, I will have the error in the standard error log file and also in a new json file (having the name of the error log, specified in log_error variable) with a number and the .json extension. More info here in the manual.

Let’s have a look at the new entry generated by loading the group_replication plugin in this sandbox instance.

in the traditional error log:

2018-03-13T09:13:45.846708Z 24 [ERROR] [MY-011596] [Repl] Plugin group_replication reported: 'binlog_checksum should be NONE for Group Replication'
2018-03-13T09:13:45.853491Z 24 [ERROR] [MY-011660] [Repl] Plugin group_replication reported: 'Unable to start Group Replication on boot'

and in the json error log:

{
  "prio": 1,
  "err_code": 11596,
  "subsystem": "Repl",
  "component": "plugin:group_replication",
  "SQL_state": "HY000",
  "source_file": "plugin.cc",
  "function": "check_if_server_properly_configured",
  "msg": "Plugin group_replication reported: 'binlog_checksum should be NONE for Group Replication'",
  "time": "2018-03-13T09:13:45.846708Z",
  "thread": 24,
  "err_symbol": "ER_GRP_RPL_BINLOG_CHECKSUM_SET",
  "label": "Error"
}
{
  "prio": 1,
  "err_code": 11660,
  "subsystem": "Repl",
  "component": "plugin:group_replication",
  "SQL_state": "HY000",
  "source_file": "plugin.cc",
  "function": "plugin_group_replication_init",
  "msg": "Plugin group_replication reported: 'Unable to start Group Replication on boot'",
  "time": "2018-03-13T09:13:45.853491Z",
  "thread": 24,
  "err_symbol": "ER_GRP_RPL_FAILED_TO_START_ON_BOOT",
  "label": "Error"
}

FILTERS

The new error log service gives you the possibility to use components to filter the events.

The default built-in, log_filter_internal, filters the events only based on their priority, you can specify it using the global variable log_error_verbosity (default is 2).

But there is another component available that allows you to filter on rules that you define: log_filter_dragnet

Let’s try to setup this last one:

mysql> INSTALL COMPONENT 'file://component_log_filter_dragnet'
mysql> SET GLOBAL log_error_services = 'log_filter_dragnet; log_sink_internal; log_sink_json';

mysql> SELECT * from global_variables where VARIABLE_NAME like 'log_error_ser%';
+--------------------+------------------------------------------------------+
| VARIABLE_NAME      | VARIABLE_VALUE                                       |
+--------------------+------------------------------------------------------+
| log_error_services | log_filter_dragnet; log_sink_internal; log_sink_json |
+--------------------+------------------------------------------------------+

and we can already check the available dragnet rule:

mysql> select * from global_variables where VARIABLE_NAME like 'dragnet%'\G
*************************** 1. row ***************************
 VARIABLE_NAME: dragnet.log_error_filter_rules
VARIABLE_VALUE: IF prio>=INFORMATION THEN drop. IF EXISTS source_line THEN unset source_line.
1 row in set (0.30 sec)

You can find much more information about the dragnet rule language in the manual.

CONCLUSION

As you can see the new Error Logging Service is much more powerful than prior of MySQL 8.0. It’s again another example of a new feature in 8.0.

And for those that want even more or something very specific, don’t forget that now you have also the possibility to modify directly the error logging as you have the possibility to create your own components.

Please join Percona’s Principal Support Engineer, Sveta Smirnova, as she presents Basic External MySQL Troubleshooting Tools on March 15, 2018 at 10:00 am PDT (UTC-7) / 1:00 pm EDT (UTC-4).

In my troubleshooting webinar series, I normally like to discuss built-in instruments available via the SQL interface. While they are effective and help to understand what is going on, external tools are also designed to make life of a database administrator easier.

In this webinar, I will discuss the external tools, toolkits and graphical instruments most valued by Support teams and customers. I will show the main advantages of these tools, and provide examples on how to effectively use them.

I will cover Percona Toolkit, MySQL Utilities, MySQL Sandbox, Percona Monitoring and Management (PMM) and a few other instruments.

Register for the webinar now.

Sveta Smirnova, Principal Technical Services Engineer

Sveta joined Percona in 2015. Her main professional interests are problem-solving, working with tricky issues, bugs, finding patterns that can quickly solve typical issues, and teaching others how to deal with MySQL issues, bugs and gotchas effectively. Before joining Percona Sveta worked as Support Engineer in the MySQL Bugs Analysis Support Group in MySQL AB-Sun-Oracle. She is the author of the book “MySQL Troubleshooting” and JSON UDF functions for MySQL.

In this blog post, we’ll look at how to navigate some of the complexities of multi-source GTID replication.

GTID replication is often a real challenge for DBAs, especially if this has to do with multi-source GTID replication. A while back, I came across a really interesting customer environment with shards where multi-master, multi-source, multi-threaded MySQL 5.6 MIXED replication was active. This is a highly complex environment that has both pros and cons, introducing risks as a trade-off for specific customer requirements.

This is the set up of part of this environment:

I started looking into this setup when a statement broke replication between db1 and db10. Replication broke due to a statement executed on a schema that was not present on db10. This also resulted in changes originating from db1 to not being pushed down to db100 as db10, as we stopped the replication thread (for db1 channel).

On the other hand, replication was not stopped on db2 because the schema in question was present on db2. Replication between db2 and db20 was broken as well because the schema was not present in db20.

In order to fix db1->db10 replication, four GTID sets were injected in db10.

Here are some interesting blog posts regarding how to handle/fix GTID replication issues:

• https://www.percona.com/blog/2013/03/26/repair-mysql-5-6-gtid-replication-by-injecting-empty-transactions/
• https://www.percona.com/blog/2016/11/10/database-daily-ops-series-gtid-replication/
• https://www.percona.com/blog/2016/12/01/database-daily-ops-series-gtid-replication-binary-logs-purge/

After injecting the GTID sets, we started replication again and everything ran fine.

After that, we had to check the db2->db20 replication, which, as I’ve already said, was broken as well. In this case, injecting only the first GTID trx into db20 instead of all of those causing issues on db10 was enough!

You may wonder how this is possible. Right? The answer is that the rest of them were replicated from db10 to db20, although the channel was not the same.

Another strange thing is the fact that although the replication thread for the db2->db20 channel was stopped (broken), checking the slave status on db20 showed that Executed_Gtid_Set was moving for all channels even though Retrieved_Gtid_Set for the broken one was stopped! So what was happening there?

This raised my curiosity, so I decided to do some further investigation and created scenarios regarding other strange things that could happen. An interesting one was about the replication filters. In our case, I thought “What would happen in the following scenario … ?”

Let’s say we write a row from db1 to db123.table789. This row is replicated to db10 (let’s say using channel 1) and to db2 (let’s say using channel2). On channel 1, we filter out the db123.% tables, on channel2 we don’t. db1 writes the row and the entry to the binary log. db2 writes the row after reading the entry from the binary log and subsequently writes the entry to its own binary log and replicates this change to db20. This change is also replicated to db10. So now, on db10 (depending on which channel finds the GTID first) it either gets filtered on channel1 and written to its own bin log at just start…commit with any actual DDL/DML removed, or if it is read first on channel2 (db1->db2 and then db20->db10) then it is NOT filtered out and executed instead. Is this correct? It definitely ISN’T!

Points of interest

You can find answers to the above questions in the points of interest listed below. Although it’s not really clear through the official documentation, this is what happens with GTID replication and multi-source GTID replication:

• As we know GTID sets are unique across all nodes in a given cluster. In multi-source replication, Executed_Gtid_Set is common for all channels. This means that regardless the originating channel, when a GTID transaction is executed it is recorded in all channels’ Executed_Gtid_Set. Although it’s logical (each database is unique, so if a trx is going to affect a database it shouldn’t be tightened to a single channel regardless of the channel it uses), the documentation doesn’t provide much info around this.
• When we have multi-source, multi-level replication, there are cases where the GTID sets originating from one master can end up on one slave via different replication paths. It’s not clear if it applies any special algorithm (although it doesn’t seem that there could be one), but the preferred method seems to be FIFO. The fastest wins! This means that GTID sets can travel to the slave via different channels, and it’s related to how fast the upper-level slaves can commit changes. In fact, the path doesn’t really matter as it only executes each GTID trx once.
• Replication filters are global regardless the channel. This means they apply each filter to all channels. This is normal as we can’t define a replication filter per channel. In order to be able to debug such cases, adding a small replication delay per channel seems a good idea.

Today, yet another blog post about improvements in MySQL 8.0 related to Performance_Schema. Before MySQL 8.0 it was not always easy to get an example of the queries you could find in Performance_Schema when looking for statements summaries. You had to link several tables (even from sys) to achieve this goal as I explained it in this post.

Now in MySQL 8.0, we have changed the table events_statements_summary_by_digest. This table now contains 6 extra columns:

• QUANTILE_95 : stores the 95th percentile of the statement latency, in picoseconds.
• QUANTILE_99 : stores the 99th percentile of the statement latency, in picoseconds.
• QUANTILE_999 : stores the 99.9th percentile of the statement latency, in picoseconds.
• QUERY_SAMPLE_TEXT : captures a query sample that can be used with EXPLAIN to get a query plan.
• QUERY_SAMPLE_SEEN : stores the timestamp of the query.
• QUERY_SAMPLE_TIMER_WAIT : stores the query sample execution time.

FIRST_SEEN and LAST_SEEN have also been modified to use fractional seconds. The previous definition was:

  Field: LAST_SEEN
   Type: timestamp
   Null: NO
    Key: 
Default: 0000-00-00 00:00:00
  Extra: 

Now it’s

  Field: LAST_SEEN
   Type: timestamp(6)
   Null: NO
    Key: 
Default: 0000-00-00 00:00:00.000000
  Extra: 

The main goal is to capture a full example query like it was made in production with some key information about this query example and to make it easily accessible.

Proudly announcing the release of the latest stable release of ProxySQL 1.4.7 as of the 14th March 2018.

ProxySQL is a high performance, high availability, protocol aware proxy for MySQL. It can be downloaded here, and freely usable and accessible according to GPL license.

ProxySQL 1.4.7 includes a number of important improvements and bug fixes including:

New features:

• Added MySQL Query Rules Fast Routing which extends MySQL Query Rules to provide immediate routing based on user & schema (especially important when supporting a large number of schemas). Details of this feature will be described in a future blog post.
• Added new metrics for hostgroup manager and frontend queries, specifically:
  • MyHGM_myconnpoll_destroy
  • MyHGM_myconnpoll_get
  • MyHGM_myconnpoll_get_ok
  • MyHGM_myconnpoll_push
  • MyHGM_myconnpoll_reset
  • Queries_frontends_bytes_recv
  • Queries_frontends_bytes_sent
• Implemented querying of stats_mysql_global status variables using SELECT @@ syntax #1375
• Added the monitor_read_only_max_timeout_count variable to allow multiple attempts on read only check #1206

Bug fixes:

• Add rw lock to ensure metadata aren't updated while running STMT_PREPARE_RESPONSE
• MyComQueryCmd not initialized could cause crash #1370
• Incorrect parsing of SET NAMES ... COLLATE #1357
• Embed main-bundle.min.css in web UI #1354
• Handle additional whitespace parse error case on SET commands #1373
• Reset connection creation time on CHANGE USER #1393
• Added fix for utf8_unicode_ci in MariaDB Client Connector C #1396
• Fixed a bug where a server disappears when RO=1 becomes RO=0 (similar to #1039)

A special thanks to all the people that reports bugs: this makes each version of ProxySQL better than the previous one.

Please report any bugs or feature requests on github issue tracker

The MariaDB project is pleased to announce the immediate availability of MariaDB Connector/J 2.2.3 and MariaDB Connector/J 1.7.3. See the release notes and changelogs for details and visit mariadb.com/downloads/connector to download.

Download MariaDB Connector/J 2.2.3

Release Notes Changelog About MariaDB Connector/J

Download MariaDB Connector/J 1.7.3

Release Notes Changelog About MariaDB Connector/J

• Community
• MariaDB Releases

The MariaDB project is pleased to announce the immediate availability of MariaDB Connector/J 2.2.3 and MariaDB Connector/J 1.7.3. See the release notes and changelogs for details.

If you’ve been delaying nominating someone or something for a MySQL Community Award, now is the time to submit it. You can submit it quickly via a google form, or go through the trouble of tweeting or emailing it.

After March 15, the Committee will being voting.

As a reminder: there are categories for Community Contributor (a person), Application, and Corporate Contributor. This is a fantastic way to honor new and old community members for the work they do – often in their spare time, and to give thanks to the corporations that help the community.

The original post with more details is here: http://mysqlawards.org/mysql-community-awards-2018/

In this blog, I will provide answers to the Q & A for the Basic Internal Troubleshooting Tools for MySQL Server webinar.

First, I want to thank everybody for attending my February 15, 2018, webinar on troubleshooting tools for MySQL. The recording and slides for the webinar are available here. Below is the list of your questions that I was unable to answer fully during the webinar.

Q: How do we prevent the schema prefix from appearing in the show create view. This is causing issue with restore on another server with a different DB. See the issue here and reproducible test case: https://gist.github.com/anonymous/30cb138b598fec46be762789397796b6

A: I shortened the example in order to fit it in this blog:

mysql> create table t1(f1 int);
Query OK, 0 rows affected (3.47 sec)
mysql> create view v1 as select * from t1;
Query OK, 0 rows affected (0.21 sec)
mysql> show create view v1G
*************************** 1. row ***************************
                View: v1
         Create View: CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select `t1`.`f1` AS `f1` from `t1`
character_set_client: utf8
collation_connection: utf8_general_ci
1 row in set (0.00 sec)
mysql> select * from information_schema.views where table_schema='test'G
*************************** 1. row ***************************
       TABLE_CATALOG: def
        TABLE_SCHEMA: test
          TABLE_NAME: v1
     VIEW_DEFINITION: select `test`.`t1`.`f1` AS `f1` from `test`.`t1`
        CHECK_OPTION: NONE
        IS_UPDATABLE: YES
             DEFINER: root@localhost
       SECURITY_TYPE: DEFINER
CHARACTER_SET_CLIENT: utf8
COLLATION_CONNECTION: utf8_general_ci
1 row in set (0.00 sec)

The issue you experienced happened because even if you created a view as

SELECT foo FROM table1;

SELECT foo FROM your_schema.table1;

*.frm

sveta@Thinkie:~/build/ps-5.7/mysql-test$ cat var/mysqld.1/data/test/v1.frm
TYPE=VIEW
query=select `test`.`t1`.`f1` AS `f1` from `test`.`t1`
md5=5840f59d1287385629fcb9b948e53d96
updatable=1
algorithm=0
definer_user=root
definer_host=localhost
suid=2
with_check_option=0
timestamp=2018-02-24 10:27:45
create-version=1
source=select * from t1
client_cs_name=utf8
connection_cl_name=utf8_general_ci
view_body_utf8=select `test`.`t1`.`f1` AS `f1` from `test`.`t1`

You cannot prevent the schema prefix from being stored. If you restore the view on a different server with a different database name, you should edit the view definition manually. If you already restored the view that points to a non-existent schema, just recreate it.

VIEW

Q: What is thread/sql/compress_gtid_table in performance_schema.threads?

A:

thread/sql/compress_gtid_table

• thread/

   is a group of instruments. In this case, it is the instruments that are visible in the

  THREADS

   table.

• thread/sql/

   is the group of instruments that are part of the server kernel code. If you are not familiar with MySQL source tree, download the source code tarball and check its content. The main components are:

  • sql

      – server kernel

  • storage

     – where storage engines code located (

    storage/innobase

      is InnoDB code,

    storage/myisam

      is MyISAM code and so on)

  • vio

     – input-output functions

  • mysys

     – code, shared between all parts of the server

  • client

     – client library and utilities

  • strings

     – functions to work with strings

This is not full list. For more information consult MySQL Internals Manual

• thread/sql/compress_gtid_table

   is the name of the particular instrument.

Unfortunately, there is no link to source code for instrumented threads in the table

THREADS

sql

compress_gtid_table

sql/rpl_gtid_persist.cc

/**
The main function of the compression thread.
- compress the gtid_executed table when get a compression signal.
@param p_thd Thread requesting to compress the table
@return
@retval 0 OK. always, the compression thread will swallow any error
for going to wait for next compression signal until
it is terminated.
*/
extern "C" {
static void *compress_gtid_table(void *p_thd)
{

You can also find the description of mysql.gtid_executed compression in the User Reference Manual.

You can follow the same actions to find out what other MySQL threads are doing.

Q: How does a novice on MySQL learn the core basics about MySQL. The documentation can be very vast which surpasses my understanding right now. Are there any good intro books you can recommend for a System Admin?

A: I learned MySQL a long time ago, and a book that I can recommend written for version 5.0. This is “MySQL 5.0 Certification Study Guide” by Paul DuBois,‎ Stefan Hinz and Carsten Pedersen. The book is in two parts: one is devoted to SQL developers and explains how to run and tune queries. The second part is for DBAs and describes how to tune MySQL server. I asked my colleagues to suggest more modern books for you, and this one is still on the list for many. This is in all cases an awesome book for beginners, just note that MySQL has changed a lot since 5.0 and you need to deepen your knowledge after you finish reading this book.

Another book that was recommended is “MySQL” by Paul DuBois. It is written for beginners and has plenty of content. Paul DuBois has been working on (and continues to work on) the official MySQL documentation for many years, and knows MySQL in great detail.

Another book is “Murach’s MySQL” by Joel Murach, which is used as a course book in many colleges for “Introduction into Databases” type classes.

For System Administrators, you can read “Systems Performance: Enterprise and the Cloud” by Brendan Gregg. This book talks about how to tune operating systems for performance. This is one of the consistent tasks we have to do when administering MySQL. I also recommend that you study Brendan Gregg’s website, which is a great source of information for everyone who is interested in operating system performance tuning.

After you finish the books for novices, you can check out “High Performance MySQL, 3rd Edition” by Peter Zaitsev, Vadim Tkachenko, Baron Schwartz and “MySQL Troubleshooting” by Sveta Smirnova (yours truly =) ). These two books require at least basic MySQL knowledge, however.

Q: Does the database migration goes on same way? Do these tools work for migration as well?

A: The tools I discussed in this webinar are available for any version of MySQL/Percona/MariaDB server. You may use them for migration. For example, it is always useful to compare configuration (

SHOW GLOBAL VARIABLES

Q: How can we take backup of a single schema from a MySQL AWS instance without affecting the performance of applications. An AWS RDS instance to be more clear. mysqldump we cannot use in RDS instance in the current scenario.

A: You can connect to your RDS instance with mysqldump from your local machine, exactly like your MySQL clients connect to it. Then you can collect a dump of a single database, table or even specify the option –where to limit the resulting set to only a portion of the table. Note, by default

mysqldump

mysqldump

--single-transaction

Alternatively, you can use AWS Database Migration Service, which allows you to replicate your databases. Then you can take a backup of a single schema using whatever method you like.

Q: Why do some sites suggest to turn off information and performance schema? Is it important to keep it on or turn it off?

A: You cannot turn off Information Schema. It is always available.

Performance Schema in earlier versions (before 5.6.14) was resource-consuming, even if it was idle while enabled. These limitations were fixed a long time ago, and you don’t need to keep it off. At least unless you hit some new bug.

Q: How do we handle storage level threshold if a data file size grows and reaches max threshold when unnoticed? Can you please help on this question?

A: Do you mean what will happen if the data file grows until filesystem has no space? In this case, clients receive the error

"OS error code 28: No space left on device"

Q: What are the performance bottlenecks when enabling performance_schema. Is there any benchmark we can have?

A: Just enabling Performance Schema in version 5.6 and up does not cause any performance issue. With version 5.7, it can also start with almost zero allocated memory, so it won’t affect your other buffers. The Performance Schema causes impact when you enable particular instruments. Most of them are instruments that start with the name

events_waits_*

Q: Suggest us some tips about creating a real-time dashboards for the same as we have some replication environment? it would be great if you can help us here for building business level dashboards

A: This is topic for yet another webinar or, better still, a tutorial. For starters, I recommend you to check out the “MySQL Replication” dashboard in PMM and extend it using the metrics that you need.

Thanks for attending the webinar on internal troubleshooting tools for MySQL.

In this blog post, we’ll look at how to create PMM custom graphs and dashboards to track what you need to see in your database.

Percona Monitoring and Management (PMM)‘s default set of graphs is pretty complete: it covers most of the stuff a DBA requires to fully visualize database servers. However, sometimes custom information is needed in graphical form. Otherwise, you just feel your PMM deployment is a missing a graph.

Recently, a customer request came in asking for a better understanding of a specific metric: table growth, or more specifically the daily table growth (in bytes) for the last 30 days.

The graph we came up with looks like this:
. . .which graphs the information that comes from this query:

increase(mysql_info_schema_table_size{instance="$host",component="data_length"}[1d])

But what does that query mean, and how do I create one myself? I’m glad you asked! Let’s go deep into the technical details!

Before creating any graph, we must ensure that we have the data that will represent graphically. So, the first step is to ensure data collection.

Data collection

This data is already collected by the Percona mysqld_exporter, as defined in the “Collector Flags” table from the GitHub repo: https://github.com/percona/mysqld_exporter/#collector-flags

Cool! Now we need a Prometheus query in order to get the relevant data. Luckily, the Prometheus documentation is very helpful and we came up with a query in no time.

Prometheus query

What do we need for the query? In this case, it is a metric, a label and a time range. Every PMM deployment has access to the Prometheus console by adding “/prometheus” to the URL. The console is incredibly helpful when playing with queries. The console looks like this:

The metric

The time series values collected by the exporter are stored in the metrics inside of Prometheus. For our case, the metric name is called mysql_info_schema_table_size, which I figured out by using the Prometheus console “Expression” text input and its autocomplete feature. This shows you the options available as you’re writing. All the metrics collected by mysqld_export start with “mysql”.

The label

Labels are different per metric, but they are intuitively named. We need the instance and component labels. Instance is the hostname and component is equivalent to the column name of a MySQL table. The component we need is “data_length”.

The time frame

This is easy: since is a daily value, the time frame is 1d. 

The time frame is not mandatory, but it is a parameter asked for by the function we’re going to use to calculate the increase, which is called increase().

That’s how we ended up with the query that feeds the metrics, which end up in here:

You will notice it’s using a variable: $host. We define that variable in the dashboard creation, explained below.

PMM dashboard

PMM best practice is to take a copy of the existing dashboard using Setting > Save as…, since edits to Percona-provided dashboards are not preserved during upgrades. In this example, we will start with an empty dashboard.

Adding a new dashboard is as easy as clicking the “New” button from the Grafana dropdown menu:

After that, you choose the type of element that you want on a new row, which is a Graph in this case:

We like to use variables for our graphs – changing which server we analyze, for example. To add variables to the dashboard, we need to head up to the Templating option and add the variables:

Make sure you put a meaningful name for your dashboard, and you’re all set! A good practice will be to export the JSON definition of your dashboard as a backup for future recovery, or to just share it with others.

The final dashboard is called “MySQL Table Size” and holds another graph showing the table size during the timeframe for the top ten biggest tables. It looks like this:

The top right of the screen has some drop down links, the ones that look like this:

You can add links on the “Link” tab of the dashboard settings:

In case you are wondering, the query for the “Table size” graph is:

topk(10,sort_desc(sum(mysql_info_schema_table_size{instance="$host",component=~".*length"}) by (schema, table)))

So next time you want to enhance PMM and you know that there is data already inside Prometheus, but PMM lacks the visualization you want, just add it! Create a new graph and put it to your own custom dashboard!

This is our second blog in the ProxySQL Series ( Blog I MySQL Replication Read-write Split up ). Will cover how to integrate ProxySQL with MHA to handle failover of Database servers.

We already have Master – Slave replication setup behind ProxySQL from previous blog [ProxySQL On MySQL Replication]

For this setup we have added one more node for MHA Manager , Which will keep eye on Master and Slave status.

• node5 (172.17.0.5) , MHA Manager

ProxySQL can be greatly configured with MHA for Highly available setup with zero downtime.

MHA role in failover :

MHA tool is used for failover.During failover, MHA promotes most updated slave (slave with most recent transactions) as new master and apply CHANGE MASTER command on new slave and change read_only flag on new master and slave.

ProxySQL role in failover :

When failover happened (due to crash or manual for any maintenance activity) ProxySQL will detect the change (checking read_only flag) and promotes new master server’s IP into writers hostgroup and start sending traffic on new master.

Each row in mysql_replication_hostgroups table in proxysql represent a pair of writer_hostgroup and reader_hostgroup .
ProxySQL will monitor the value of read_only from mysql_server_read_only_log for all the servers.
If read_only=1 the host is copied/moved to the reader_hostgroup, while if read_only=0 the host is copied/moved to the writer_hostgroup .

Installing MHA

If replication is classic binlog/pos format based then install MHA node on all hosts involving (manager,master,slaves), for GTID based replication it has to be installed only on the manager node.

Install MHA node :

Install MHA on all DB nodes and MHA manager server. More information

apt-get -y install libdbd-mysql-perl
dpkg -i mha4mysql-node_0.56-0_all.deb

Install MHA manager :

Only install on MHA manager server.

#dependencies
apt-get install -y  libdbi-perl libdbd-mysql-perl libconfig-tiny-perl liblog-dispatch-perl libparallel-forkmanager-perl libnet-amazon-ec2-perl

dpkg -i mha4mysql-manager_0.56-0_all.deb

Configuration changes :

Changes only on node5 (172.17.0.5) , MHA Manager :

Create directories :

mkdir -p /etc/mha/ /var/log/mha/

Config file :

cat /etc/mha/cluster1.conf

[server default]
# mysql user and password
user=root
password=xxx
# replication user password
repl_user=repl
repl_password=xxx

remote_workdir=/var/tmp
# working directory on the manager
manager_workdir=/var/log/mha/
# manager log file
manager_log=/var/log/mha/mha.log
ping_interval=15

*/As we don't have to deal with VIP's here, disable master_ip_failover_script */
#master_ip_failover_script=/usr/local/bin/master_ip_failover 
master_ip_online_change_script=/usr/local/bin/master_ip_online_change 
master_binlog_dir=/data/log/
secondary_check_script=/etc/mha/mha_prod/failover_triggered.sh
report_script=/etc/mha/mha_prod/failover_report.sh
master_pid_file=/var/run/mysqld/mysqld.pid 
ssh_user=root 
log_level=debug 
#set this to 0 if YOU ARE SURE THIS CAN"T BREAK YOUR REPLICATION 
check_repl_filter=1 
[server1] 
hostname=172.17.0.1 
port=3306 
[server2] 
hostname=172.17.0.2 
port=3306 
[server3] 
hostname=172.17.0.3 
port=3306 
no_master=1 

master_ip_failover : Script used to switch virtual IP address.
master_ip_online_change : Script used in switchover when master is online or dead.

NOTE: Don’t forget to comment out the “FIX ME” lines in the above scripts.

Custom scripts : Below scripts are optional

secondary_check_script It is always good to double check the availability of master.  More info

report_script  :  With this script we can configure alerts or email when failover completes. In Detail

Now run test against the cluster using below two scripts :
– masterha_check_repl
– masterha_check_ssh

Please note If this check fails then MHA will refuse to run any kind of failover.

root@MHA-Node# /etc/mha # masterha_check_ssh --conf=/etc/mha/cluster1.cnf
-- truncated long output
[info] All SSH connection tests passed successfully.

root@MHA-Node#  masterha_check_repl --conf=/etc/mha/cluster1.cnf
172.17.0.1(172.17.0.1:3306) (current master)
 +--172.17.0.2(172.17.0.2:3306)
 +--172.17.0.3(172.17.0.3:3306)

-- truncated long output
MySQL Replication Health is OK.

To run an Manual failover :

• To Execute manual failover , Make sure your MHA manager is stooped.
• new_master_host – Its optional parameter if you want to select new master. If we don’s specify any value then most updated slave considered as new Master.

masterha_master_switch --master_state=alive --conf=/etc/mha/cluster1.conf
--orig_master_is_new_slave [--new_master_host=] 

Automatic failover :

We need to run masterha_manager is background to monitor cluster status :

nohup masterha_manager --conf=/etc/mha/cluster1.cnf  < /dev/null > /var/log/mha/mha.log 2>&1 &

When Auto failover happen , In case of Master Crash , Logs look like

tail -f /var/log/mha/mha.log
 ----- Failover Report -----

Master 172.17.0.1(172.17.0.1:3306) is down!
Started automated(non-interactive) failover.
172.17.0.1(172.17.0.1:3306)
Selected 172.17.0.2(172.17.0.2:3306) as a new master.
172.17.0.2(172.17.0.2:3306): OK: Applying all logs succeeded.
172.17.0.2(172.17.0.2:3306): OK: Activated master IP address.
172.17.0.3(172.17.0.3:3306): OK: Slave started, replicating from 172.17.0.2(172.17.0.2:3306)
    172.17.0.2(172.17.0.2:3306): Resetting slave info succeeded.
    Master failover to 172.17.0.2(172.17.0.2:3306) completed successfully.

we can also check the status of masterha_manager:

RUNNING :
root@mysql-monitoring /etc/mha # masterha_check_status --conf=/etc/mha/cluster1.cnf
cluster1 (pid:15810) is running(0:PING_OK), master:172.17.0.1

NOT RUNNING :
root@mysql-monitoring /etc/mha # masterha_check_status --conf=/etc/mha/cluster1.cnf
cluster1 is stopped(2:NOT_RUNNING).

Remember masterha_manager script stops working in two situation :

1. After automatic fail-over , for that we need to manually start above script to continue monitoring clusters nodes
2. If any replication is not running on any slaves defined in cluster or any slave server is down.

Check backend status at ProxySQL :

Below table of ProxySQL shows , what is the current master and its slaves after failover with their ONLINE status.

Admin > select hostgroup,srv_host,status,Queries,Bytes_data_sent,Latency_us from stats_mysql_connection_pool where hostgroup in (0,1);
+-----------+------------+----------+---------+-----------------+------------+
| hostgroup | srv_host   | status   | Queries | Bytes_data_sent | Latency_us |
+-----------+------------+----------+---------+-----------------+------------+
| 0         | 172.17.0.1 | ONLINE   | 12349   | 76543232        | 144        |
| 1         | 172.17.0.2 | ONLINE   | 22135   | 87654356        | 190        |
| 1         | 172.17.0.3 | ONLINE   | 22969   | 85344235        | 110        |
| 1         | 172.17.0.1 | ONLINE   | 1672    | 4534332         | 144        |
+-----------+------------+----------+---------+-----------------+------------+

Preserve relay logs and purge regularly :

As we have two slaves in this setup , MHA is keeping relay logs for recovering other slaves. [To ensure that it disables  relay_log_purge ]

We need to periodically purges old relay logs like binary logs. MHA Node has a command line tool purge_relay_logs to do that

purge_relay_logs removes relay logs without blocking SQL threads. Relay logs need to be purged regularly (i.e. once per day, once per 6 hours, etc), so purge_relay_logs should be regularly invoked on each slave server at different time. It can be scheduled as a cron too.

[root@mysql-slave1]$

cat /etc/cron.d/purge_relay_logs
#purge relay logs after every 5 hours 
0 */5 * * * /usr/bin/purge_relay_logs --user=mha --password=PASSWORD --workdir=/data/archive_relay --disable_relay_log_purge >> /var/log/mha/purge_relay_logs.log 2>&1

Above script will purge relay logs and set relay_log_purge = 0 [OFF] to avoid automatic relay purge .

More Details : https://github.com/yoshinorim/mha4mysql-manager/wiki/Requirements

We can also have MySQL Utilities to perform the failover in ProxySQL too.The main advantage of using MHA-ProxySQL integration is, it avoids need for VIP or re-defining DNS after MHA failover , they are taken care by ProxySQL.

Thank you to everyone who joined us at our second annual MariaDB user conference, M|18, in New York City on February 26 and 27. DBAs, open source enthusiasts, engineers, executives and more from all over the world came together to explore and learn.

Couldn’t make the event or want to relive your favorite session?

Watch 40+ M|18 session recordings on demand.

Session highlights

• The welcome keynote from MariaDB’s CEO, Michael Howard, announced several new initiatives, including MariaDB Labs, a research division focused on solving extreme challenges within the database industry on the topics of machine learning; distributed computing; and next-generation chips, storage and memory.

• “Corporate Banking and Future-Ready Technology,” with Ng Peng Khim and Joan Tay from DBS (Development Bank of Singapore), provided an update on the bank’s journey from Oracle Enterprise to MariaDB TX.
• “Massive Scale with MariaDB,” with ServiceNow’s Tim Yim, revealed that the company has a mind-blowing 85,000 MariaDB databases that ServiceNow manages with a custom “multi-instance deployment” implementation.
• “How We Made the Move to MariaDB at FNI,” with William Woods from Financial Network, Inc., shared the considerations, including security, that led FNI to migrate from Oracle to MariaDB.
• “MariaDB AX Panel Discussion on Analytical Use Cases,” featuring Aziz Vahora from Pinger, Jack Sprague from Center of Information Management, and Patrice Linel from Genus Plc., provided an opportunity to hear directly from MariaDB AX users from diverse industries and learn why they chose MariaDB.
• “How Facebook Migrated to MyRocks,” with Facebook’s Yoshinori Matsunobu, covered how Facebook created the MyRocks storage engine (which is in RC for MariaDB Server 10.3) to replace InnoDB in the Facebook users database, and highlighted MyRocks features being developed.
• “Panel Discussion: Open Source in the Enterprise,” featured Bill Montgomery of Red Hat, Dheeraj Golla of Copart, and Phil Mazza of Financial Network, Inc., sharing why they chose to adopt open source, the business and technical challenges they faced and how other enterprises can succeed with open source software.
• “How Copart Switched to MariaDB and Reduced Costs During Growth,” featured Dheeraj Golla and Pravin Malali of Copart explaining why Copart chose to standardize on MariaDB TX, how they convinced operational and development teams to adopt it, and how they’re now evaluating MariaDB AX as a replacement for their current analytics database.

Networking with games

Learning opportunities abounded at M|18, all while having fun. At the opening-night party and closing reception, attendees enjoyed food, drink and conversation – plus a little good-natured competition.

#MARIADBM18

Thanks to the attendees and speakers, M|18 was trending on Twitter. Here are a few of our favorite conference tweets.

Had great time at conference. Helped to reinforce architectural changes needed in our company and how mariadb suite helps. #MARIADBM18

— Adam A. Lang (@AdamALang) 

Have had a blast at #MARIADBM18 - lots learnt and even more to think about. Thanks #Mariadb

— Tudor Davies (@DerBroader71) 

Thank you MariaDB for all of your hard work creating a great M18 conference! #MARIADBM18 pic.twitter.com/FqYxU8kNZO

— Lou Zircher (@ndp3188) 

An extra thanks …

… to our sponsors for their generous support.

M|19 will be announced soon!

In the next few weeks, we’ll release the dates for our next MariaDB user conference, M|19. Be on the lookout for the announcement!

MariaDB's second annual MariaDB user conference, M|18, took place in New York City on February 26 and 27. DBAs, open source enthusiasts, engineers, executives and more from all over the world came together to explore and learn. Couldn’t make the event or want to relive your favorite session? Watch 45+ M|18 session recordings on demand.

In this blog post, we’ll look at how you can verify query performance using ProxySQL.

In the previous blog post, I showed you how many information can you get from the “stats.stats_mysql_query_digest” table in ProxySQL. I also mentioned you could even collect and graph these metrics. I will show you this is not just theory, it is possible.

These graphs could be very useful to understand the impact of the changes what you made on the query count or execution time.

I used our all-time favorite benchmark tool called Sysbench. I was running the following query:

UPDATE sbtest1 SET c=? WHERE k=?

There was no index on “k” when I started the test. During the test, I added an index. We expect to see some changes in the graphs.

I selected the “stats.stats_mysql_query_digest” into a file in every second, then I used Percona Monitoring and Management (PMM) to create graphs from the metrics. (I am going write another blog post on how can you use PMM to create graphs from any kind of metrics.)

Without the index, the update was running only 2-3 times per second. By adding the index, it went up to 400-500 hundred. We can see the results immediately on the graph.

Let’s see the average execution time:

Without the index, it took 600000-700000 microseconds, which is around 0.7s. By adding an index, it dropped to 0.01s. This is a big win, but most importantly we can see the effects on the query response time and query count if we are making some changes to the schema, query or configuration as well.

Conclusion

If you already have a ProxySQL server collecting and graphing these metrics, they could be quite useful when you are optimizing your queries. They can help make sure you are moving in the right direction with your tunings/modifications.

The survey results from StackOverflow’s developers survey are already here, and we can now declare the most popular databases for 2018.

Without further ado, let’s look into the results:

So what can we learn from these results?

1. The most popular database is MySQL, and not by far comes SQL Server. This wasn’t very surprising, as they were the top two databases last year as well. This year, more than half of the respondents (58.7%) are using MySQL. It seems RDBMS databases and specifically MySQL are here to stay for at least few years ahead.
2. SQL Server has a large market share as well (41.2%) This is also an increase from last year (30.8%). Microsoft is pushing SQL Server with their Windows Servers, and that way gaining significant market share. In addition, recently Microsoft introduced some AI capabilities in SQL Server over Azure, so we believe it will gain them some more new customers wishing to optimize their database using some artificial intelligence.
3. PostgreSQL is gaining lots of traction in the last few years. Developers working with Postgres are very pleased with the product, both in terms of capabilities and performance. We believe this trend will continue and we’ll see larger usage rates next year.
4. RDBMS databases are still significantly more common than NoSQL databases such as MongoDB.
5. Relatively new technologies are starting to gain market share in the databases world – Redis (first release at 2009) and Cassandra (first release at 2008).
6. Almost 1/4 of all programmers (21.6%) are using SQLite, which is a lite SQL database which is based on a single file. This database has large popularity in small desktop applications and applications where an embedded database is required (for example, mobile phone apps).
7. One might argue that many beginner developers are using MySQL and that’s why the numbers are so high. Well, if you’ll look at the same chart but only for Professional Developers, the numbers look very similar.

To summarize, RDBMS and MySQL are still very popular among tech companies and programmers. NoSQL databases are probably not here to replace those needs, but to be used to solve different requirements.

Disclaimer: charts and data copyrights are of StackOverflow. EverSQL is in no way associated with StackOverflow.