Replication is one of the well-known features that allows us to build an identical copy of a database. It is supported in almost every RDBMS. The advantages of replication may be huge, especially HA (High Availability) and load balancing. But what if we need to build replication between 2 heterogeneous databases like MySQL and PostgreSQL? Can we continuously replicate changes from a MySQL database to a PostgreSQL database? The answer to this question is pg_chameleon.

For replicating continuous changes, pg_chameleon uses the mysql-replication library to pull the row images from MySQL, which are transformed into a jsonb object. A pl/pgsql function in postgres decodes the jsonb and replays the changes into the postgres database. In order to setup this type of replication, your mysql binlog_format must be “ROW”.

A few points you should know before setting up this tool :

1. Tables that need to be replicated must have a primary key.
2. Works for PostgreSQL versions > 9.5 and MySQL > 5.5
3. binlog_format must be ROW in order to setup this replication.
4. Python version must be > 3.3

When you initialize the replication, pg_chameleon pulls the data from MySQL using the CSV format in slices, to prevent memory overload. This data is flushed to postgres using the COPY command. If COPY fails, it tries INSERT, which may be slow. If INSERT fails, then the row is discarded.

To replicate changes from mysql, pg_chameleon mimics the behavior of a mysql slave. It creates the schema in postgres, performs the initial data load, connects to MySQL replication protocol, stores the row images into a table in postgres. Now, the respective functions in postgres decode those rows and apply the changes. This is similar to storing relay logs in postgres tables and applying them to a postgres schema. You do not have to create a postgres schema using any DDLs. This tool automatically does that for the tables configured for replication. If you need to specifically convert any types, you can specify this in the configuration file.

The following is just an exercise that you can experiment with and implement if it completely satisfies your requirement. We performed these tests on CentOS Linux release 7.4.

Prepare the environment

Set up Percona Server for MySQL

InstallMySQL 5.7 and add appropriate parameters for replication.

In this exercise, I have installed Percona Server for MySQL 5.7 using YUM repo.

yum install http://www.percona.com/downloads/percona-release/redhat/0.1-6/percona-release-0.1-6.noarch.rpm
yum install Percona-Server-server-57
echo "mysql ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
usermod -s /bin/bash mysql
sudo su - mysql

pg_chameleon requires the following the parameters to be set in your my.cnf file (parameter file of your MySQL server). You may add the following parameters to /etc/my.cnf

binlog_format= ROW
binlog_row_image=FULL
log-bin = mysql-bin
server-id = 1

Now start your MySQL server after adding the above parameters to your my.cnf file.

$ service mysql start

Fetch the temporary root password from mysqld.log, and reset the root password using mysqladmin

$ grep "temporary" /var/log/mysqld.log
$ mysqladmin -u root -p password 'Secret123!'

Now, connect to your MySQL instance and create sample schema/tables. I have also created an emp table for validation.

$ wget http://downloads.mysql.com/docs/sakila-db.tar.gz
$ tar -xzf sakila-db.tar.gz
$ mysql -uroot -pSecret123! < sakila-db/sakila-schema.sql
$ mysql -uroot -pSecret123! < sakila-db/sakila-data.sql
$ mysql -uroot -pSecret123! sakila -e "create table emp (id int PRIMARY KEY, first_name varchar(20), last_name varchar(20))"

Create a user for configuring replication using pg_chameleon and give appropriate privileges to the user using the following steps.

$ mysql -uroot -p
create user 'usr_replica'@'%' identified by 'Secret123!';
GRANT ALL ON sakila.* TO 'usr_replica'@'%';
GRANT RELOAD, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'usr_replica'@'%';
FLUSH PRIVILEGES;

While creating the user in your mysql server (‘usr_replica’@’%’), you may wish to replace % with the appropriate IP or hostname of the server on which pg_chameleon is running.

Set up PostgreSQL

Install PostgreSQL and start the database instance.

You may use the following steps to install PostgreSQL 10.x

yum install https://yum.postgresql.org/10/redhat/rhel-7.4-x86_64/pgdg-centos10-10-2.noarch.rpm
yum install postgresql10*
su - postgres
$/usr/pgsql-10/bin/initdb
$ /usr/pgsql-10/bin/pg_ctl -D /var/lib/pgsql/10/data start

As seen in the following logs, create a user in PostgreSQL using which pg_chameleon can write changed data to PostgreSQL. Also create the target database.

postgres=# CREATE USER usr_replica WITH ENCRYPTED PASSWORD 'secret';
CREATE ROLE
postgres=# CREATE DATABASE db_replica WITH OWNER usr_replica;
CREATE DATABASE

Steps to install and setup replication using pg_chameleon

Step 1: In this exercise, I installed Python 3.6 and pg_chameleon 2.0.8 using the following steps. You may skip the python install steps if you already have the desired python release. We can create a virtual environment if the OS does not include Python 3.x by default.

yum install gcc openssl-devel bzip2-devel wget
cd /usr/src
wget https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tgz
tar xzf Python-3.6.6.tgz
cd Python-3.6.6
./configure --enable-optimizations
make altinstall
python3.6 -m venv venv
source venv/bin/activate
pip install pip --upgrade
pip install pg_chameleon

Step 2: This tool requires a configuration file to store the source/target server details, and a directory to store the logs. Use the following command to let pg_chameleon create the configuration file template and the respective directories for you.

$ chameleon set_configuration_files

The above command would produce the following output, which shows that it created some directories and a file in the location where you ran the command.

creating directory /var/lib/pgsql/.pg_chameleon
creating directory /var/lib/pgsql/.pg_chameleon/configuration/
creating directory /var/lib/pgsql/.pg_chameleon/logs/
creating directory /var/lib/pgsql/.pg_chameleon/pid/
copying configuration example in /var/lib/pgsql/.pg_chameleon/configuration//config-example.yml

Copy the sample configuration file to another file, lets say, default.yml

$ cd .pg_chameleon/configuration/
$ cp config-example.yml default.yml

Here is how my default.yml file looks after adding all the required parameters. In this file, we can optionally specify the data type conversions, tables to skipped from replication and the DML events those need to skipped for selected list of tables.

---
#global settings
pid_dir: '~/.pg_chameleon/pid/'
log_dir: '~/.pg_chameleon/logs/'
log_dest: file
log_level: info
log_days_keep: 10
rollbar_key: ''
rollbar_env: ''
# type_override allows the user to override the default type conversion into a different one.
type_override:
  "tinyint(1)":
    override_to: boolean
    override_tables:
      - "*"
#postgres  destination connection
pg_conn:
  host: "localhost"
  port: "5432"
  user: "usr_replica"
  password: "secret"
  database: "db_replica"
  charset: "utf8"
sources:
  mysql:
    db_conn:
      host: "localhost"
      port: "3306"
      user: "usr_replica"
      password: "Secret123!"
      charset: 'utf8'
      connect_timeout: 10
    schema_mappings:
      sakila: sch_sakila
    limit_tables:
#      - delphis_mediterranea.foo
    skip_tables:
#      - delphis_mediterranea.bar
    grant_select_to:
      - usr_readonly
    lock_timeout: "120s"
    my_server_id: 100
    replica_batch_size: 10000
    replay_max_rows: 10000
    batch_retention: '1 day'
    copy_max_memory: "300M"
    copy_mode: 'file'
    out_dir: /tmp
    sleep_loop: 1
    on_error_replay: continue
    on_error_read: continue
    auto_maintenance: "disabled"
    gtid_enable: No
    type: mysql
    skip_events:
      insert:
#        - delphis_mediterranea.foo #skips inserts on the table delphis_mediterranea.foo
      delete:
#        - delphis_mediterranea #skips deletes on schema delphis_mediterranea
      update:

Step 3: Initialize the replica using this command:

$ chameleon create_replica_schema --debug

The above command creates a schema and nine tables in the PostgreSQL database that you specified in the .pg_chameleon/configuration/default.yml file. These tables are needed to manage replication from source to destination. The same can be observed in the following log.

db_replica=# \dn
List of schemas
Name | Owner
---------------+-------------
public | postgres
sch_chameleon | target_user
(2 rows)
db_replica=# \dt sch_chameleon.t_*
List of relations
Schema | Name | Type | Owner
---------------+------------------+-------+-------------
sch_chameleon | t_batch_events | table | target_user
sch_chameleon | t_discarded_rows | table | target_user
sch_chameleon | t_error_log | table | target_user
sch_chameleon | t_last_received | table | target_user
sch_chameleon | t_last_replayed | table | target_user
sch_chameleon | t_log_replica | table | target_user
sch_chameleon | t_replica_batch | table | target_user
sch_chameleon | t_replica_tables | table | target_user
sch_chameleon | t_sources | table | target_user
(9 rows)

Step 4: Add the source details to pg_chameleon using the following command. Provide the name of the source as specified in the configuration file. In this example, the source name is mysql and the target is postgres database defined under pg_conn.

$ chameleon add_source --config default --source mysql --debug

Once you run the above command, you should see that the source details are added to the t_sources table.

db_replica=# select * from sch_chameleon.t_sources;
-[ RECORD 1 ]-------+----------------------------------------------
i_id_source | 1
t_source | mysql
jsb_schema_mappings | {"sakila": "sch_sakila"}
enm_status | ready
t_binlog_name |
i_binlog_position |
b_consistent | t
b_paused | f
b_maintenance | f
ts_last_maintenance |
enm_source_type | mysql
v_log_table | {t_log_replica_mysql_1,t_log_replica_mysql_2}
$ chameleon show_status --config default
Source id Source name Type Status Consistent Read lag Last read Replay lag Last replay
----------- ------------- ------ -------- ------------ ---------- ----------- ------------ -------------
1 mysql mysql ready Yes N/A N/A

Step 5: Initialize the replica/slave using the following command. Specify the source from which you are replicating the changes to the PostgreSQL database.

$ chameleon init_replica --config default --source mysql --debug

Initialization involves the following tasks on the MySQL server (source).

1. Flush the tables with read lock
2. Get the master’s coordinates
3. Copy the data
4. Release the locks

The above command creates the target schema in your postgres database automatically.
In the default.yml file, we mentioned the following schema_mappings.

schema_mappings:
sakila: sch_sakila

So, now it created the new schema scott in the target database db_replica.

db_replica=# \dn
List of schemas
Name | Owner
---------------+-------------
public | postgres
sch_chameleon | usr_replica
sch_sakila | usr_replica
(3 rows)

Step 6: Now, start replication using the following command.

$ chameleon start_replica --config default --source mysql

Step 7: Check replication status and any errors using the following commands.

$ chameleon show_status --config default
$ chameleon show_errors

This is how the status looks:

$ chameleon show_status --source mysql
Source id Source name Type Status Consistent Read lag Last read Replay lag Last replay
----------- ------------- ------ -------- ------------ ---------- ----------- ------------ -------------
1 mysql mysql running No N/A N/A
== Schema mappings ==
Origin schema Destination schema
--------------- --------------------
sakila sch_sakila
== Replica status ==
--------------------- ---
Tables not replicated 0
Tables replicated 17
All tables 17
Last maintenance N/A
Next maintenance N/A
Replayed rows
Replayed DDL
Skipped rows

Now, you should see that the changes are continuously getting replicated from MySQL to PostgreSQL.

Step 8:  To validate, you may insert a record into the table in MySQL that we created for the purpose of validation and check that it is replicated to postgres.

$ mysql -u root -pSecret123! -e "INSERT INTO sakila.emp VALUES (1,'avinash','vallarapu')"
mysql: [Warning] Using a password on the command line interface can be insecure.
$ psql -d db_replica -c "select * from sch_sakila.emp"
 id | first_name | last_name
----+------------+-----------
  1 | avinash    | vallarapu
(1 row)

In the above log, we see that the record that was inserted to the MySQL table was replicated to the PostgreSQL table.

You may also add multiple sources for replication to PostgreSQL (target).

Reference : http://www.pgchameleon.org/documents/

Please refer to the above documentation to find out about the many more options that are available with pg_chameleon

The post Replication from Percona Server for MySQL to PostgreSQL using pg_chameleon appeared first on Percona Database Performance Blog.

Please join Percona’s CEO, Peter Zaitsev as he presents MySQL vs MongoDB – Choosing the Right Technology for Your Application on Thursday, August 23, 2018, at 10:30 AM PDT (UTC-7) / 1:30 PM EDT (UTC-4).

Are you considering to adopt the most popular open source relational database or the most popular open source NoSQL database? Which one is right for your particular application?

In this presentation, we will look into advantages and disadvantages of both and examine the applications where MySQL or MongoDB are the most appropriate choice.

Register Now

The post Webinar Thurs 8/23: MySQL vs MongoDB – Choosing the Right Technology for Your Application appeared first on Percona Database Performance Blog.

There are many organizations where front/web-facing applications use MySQL and back end processing uses PostgreSQL®. Any system integration between these applications generally involves the replication—or duplication—of data from system to system. We recently blogged about pg_chameleon which can be used replicate data from MySQL® to PostgreSQL. mysql_fdw can play a key role in eliminating the problem of replicating/duplicating data. In order to eliminate maintaining the same data physically in both postgres and MySQL, we can use mysql_fdw. This allows PostgreSQL to access MySQL tables and to use them as if they are local tables in PostgreSQL. mysql_fdw can be used, too, with Percona Server for MySQL, our drop-in replacement for MySQL.

This post is to showcase how easy it is to set that up and get them working together. We will address a few points that we skipped while discussing about FDWs in general in our previous post

Preparing MySQL for fdw connectivity

On the MySQL server side, we need to set up a user to allow for access to MySQL from the PostgreSQL server side. We recommend Percona Server for MySQL if you are setting it up for the first time.

mysql> create user 'fdw_user'@'%' identified by 'Secret!123';

This user needs to have privileges on the tables which are to be presented as foreign tables in PostgreSQL.

mysql> grant select,insert,update,delete on EMP to fdw_user@'%';
Query OK, 0 rows affected (0.00 sec)
mysql> grant select,insert,update,delete on DEPT to fdw_user@'%';
Query OK, 0 rows affected (0.00 sec)

Installing mysql_fdw on PostgreSQL server

Under the hood, MySQL FDW (mysql_fdw) facilitates the use of PostgreSQL server as a client for MySQL Server, which means it can then fetch data from the MySQL database as a client. Obviously, mysql_fdw uses MySQL client libraries. Nowadays, many Linux distributions are packaged with MariaDB® libraries. This works well enough for mysql_fdw to function. If we install mysql_fdw from the PGDG repo, then mariadb-devel.x86_64 packages will be installed alongside other development packages. To switch to Percona packages as client libraries, you need to have the Percona development packages too.

sudo yum install Percona-Server-devel-57-5.7.22-22.1.el7.x86_64.rpm

Now we should be able to install the mysql_fdw from PGDG repository:

sudo yum install mysql_fdw_10.x86_64

Connect to the PostgreSQL server where we are going to create the foreign table, and using the command line tool, create mysql_fdw extension:

postgres=# create extension mysql_fdw;
CREATE EXTENSION

Create a server definition to point to the MySQL server running on a host machine by specifying the hostname and port:

postgres=# CREATE SERVER mysql_svr  FOREIGN DATA WRAPPER mysql_fdw OPTIONS (host 'hr',port '3306');
CREATE SERVER

Now we can create a user mapping. This maps the database user in PostgreSQL to the user on the remote server (MySQL). While creating the user mapping, we need to specify the user credentials for the MySQL server as shown below. For this demonstration, we are using PUBLIC user in PostgreSQL. However, we could use a specific user as an alternative.

postgres=# CREATE USER MAPPING FOR PUBLIC SERVER mysql_svr OPTIONS (username 'fdw_user',password 'Secret!123');
CREATE USER MAPPING

Import schema objects

Once we complete the user mapping, we can import the foreign schema.

postgres=# IMPORT FOREIGN SCHEMA hrdb FROM SERVER mysql_svr INTO public;

Or we have the option to import only selected tables from the foreign schema.

postgres=# IMPORT FOREIGN SCHEMA hrdb limit to ("EMP","DEPT") FROM SERVER mysql_svr INTO public;

This statement says that the tables “EMP” and “DEPT” from the foreign schema named “hrdb” in mysql_server need to be imported into the  public schema of the PostgreSQL database.

FDWs in PostgreSQL allow us to import the tables to any schema in postgres.

Let’s create a schema in postgres:

postgres=# create schema hrdb;
postgres=# IMPORT FOREIGN SCHEMA hrdb limit to ("EMP","DEPT") FROM SERVER mysql_svr INTO hrdb;

Suppose we need the foreign table to be part of multiple schemas of PostgreSQL. Yes, it is possible.

postgres=# create schema payroll;
CREATE SCHEMA
postgres=# create schema finance;
CREATE SCHEMA
postgres=# create schema sales;
CREATE SCHEMA
postgres=# IMPORT FOREIGN SCHEMA  hrdb limit to ("EMP","DEPT") FROM SERVER mysql_svr INTO payroll;
IMPORT FOREIGN SCHEMA
postgres=# IMPORT FOREIGN SCHEMA  hrdb limit to ("EMP","DEPT") FROM SERVER mysql_svr INTO finance;
IMPORT FOREIGN SCHEMA
postgres=# IMPORT FOREIGN SCHEMA  hrdb limit to ("EMP","DEPT") FROM SERVER mysql_svr INTO sales;
IMPORT FOREIGN SCHEMA

You might be wondering if there’s a benefit to doing this. Yes, since in a multi-tenant environment, it allows us to centralize many of the master/lookup tables. These can even sit in a remote server, and that can be MySQL as well!.

IMPORTANT: PostgreSQL extensions are database specific. So if you have more than one database inside a PostgreSQL instance/cluster, you have to create a separate fdw extension, foreign server definition and user mapping.

Foreign tables with a subset of columns

Another important property of foreign tables is that you can have a subset of columns if you are not planning to issue DMLs on the remote table. For example MySQL’s famous sample database Sakila contains a table “film” with the following definition

CREATE TABLE `film` (
`film_id` smallint(5) unsigned NOT NULL AUTO_INCREMENT,
`title` varchar(255) NOT NULL,
`description` text,
`release_year` year(4) DEFAULT NULL,
`language_id` tinyint(3) unsigned NOT NULL,
`original_language_id` tinyint(3) unsigned DEFAULT NULL,
`rental_duration` tinyint(3) unsigned NOT NULL DEFAULT '3',
`rental_rate` decimal(4,2) NOT NULL DEFAULT '4.99',
`length` smallint(5) unsigned DEFAULT NULL,
`replacement_cost` decimal(5,2) NOT NULL DEFAULT '19.99',
`rating` enum('G','PG','PG-13','R','NC-17') DEFAULT 'G',
`special_features` set('Trailers','Commentaries','Deleted Scenes','Behind the Scenes') DEFAULT NULL,
`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`film_id`),
KEY `idx_title` (`title`),
KEY `idx_fk_language_id` (`language_id`),
KEY `idx_fk_original_language_id` (`original_language_id`),
CONSTRAINT `fk_film_language` FOREIGN KEY (`language_id`) REFERENCES `language` (`language_id`) ON UPDATE CASCADE,
CONSTRAINT `fk_film_language_original` FOREIGN KEY (`original_language_id`) REFERENCES `language` (`language_id`) ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=1001 DEFAULT CHARSET=utf8

Imagine that we don’t need all of these fields to be available to the PostgreSQL database and its application. In such cases, we can create a foreign table with only the necessary columns in the PostgreSQL side. For example:

CREATE FOREIGN TABLE film (
film_id smallint NOT NULL,
title varchar(255) NOT NULL,
) SERVER mysql_svr OPTIONS (dbname 'sakila', table_name 'film');

The challenges of incompatible syntax and datatypes

There are many syntactical differences between MySQL and PostgreSQL. Consequently, you may need to manually intervene to create foreign tables. For example, MySQL tables accepts definition of enumerations in place, whereas PostgreSQL expects enumeration types to be defined before creating the table like this:

CREATE TYPE rating_t AS enum('G','PG','PG-13','R','NC-17');

Many such things are not handled perfectly. So it is better to specify them as a text datatype. The same applies to the set datatype.

CREATE FOREIGN TABLE film (
film_id smallint NOT NULL,
title varchar(255) NOT NULL,
rating text,
special_features text
) SERVER mysql_svr OPTIONS (dbname 'sakila', table_name 'film');

I’m used to receiving scepticism from people about treating enum and set as text. Well, please don’t forget that we are not storing them in PostgreSQL, the text datatype is just a method for handling input and output from the table. The data is pulled and pushed from the foreign server, which is MySQL, and this converts these texts into the corresponding enumeration before storing them.

IMPORTANT : mysql_fdw has the capability to do data type conversion (casting) automatically behind the scenes when a user fires DML against foreign tables.

Generally, DML against a remote MySQL database from the PostgreSQL side can be quite challenging because of the architecture differences. These impose restrictions, such as the first column of the foreign table must be unique. We will cover these in more depth in a future post.

Handling views on the MySQL side

While foreign tables are not limited tables on the MySQL side, a view can also be mapped as a foreign table. Let’s create a view in the MySQL database.

mysql> create view v_film as select film_id,title,description,release_year from film;

PostgreSQL can treat this view as a foreign table:

postgres=# CREATE FOREIGN TABLE v_film (
film_id smallint,
title varchar(255) NOT NULL,
description text,
release_year smallint ) SERVER mysql_svr OPTIONS (dbname 'sakila', table_name 'v_film');
CREATE FOREIGN TABLE

Views on the top of foreign table on PostgreSQL

PostgreSQL allows us to create views on the top of foreign tables. This might even be pointing to a view on the remote MySQL server. Let’s try creating a view using the newly created foreign table v_film.

postgres=# create view v2_film as select film_id,title from v_film;
postgres=# explain verbose select * from v2_film;
QUERY PLAN
--------------------------------------------------------------------------
Foreign Scan on public.v_film  (cost=10.00..1010.00 rows=1000 width=518)
Output: v_film.film_id, v_film.title
Local server startup cost: 10
Remote query: SELECT `film_id`, `title` FROM `sakila`.`v_film`
(4 rows)

Materializing the foreign tables (Materialized Views)

One of the key features mysql_fdw implements is the ability to support persistent connections. After query execution, the connection to the remote MySQL database is not dropped. Instead it retains the connection for the next query from the same session. Nevertheless, in some situations, there will be concerns about continuously streaming data from the source database (MySQL) to the destination (PostgreSQL). If you have a frequent need to access data from foreign tables, you could consider the option of materializing the data locally. It is possible to create a materialized view on the top of the foreign table.

postgres=# CREATE MATERIALIZED VIEW mv_film as select * from film;
SELECT 1000

Whenever required, we can just refresh the materialized view.

postgres=# REFRESH MATERIALIZED VIEW mv_film;
REFRESH MATERIALIZED VIEW

Automated Cleanup

One of the features I love about the FDW framework is its ability to clean up foreign tables in a single shot. This is very useful when we setup foreign table for a temporary purpose, like data migration. At the very top level, we can drop the extension, PostgreSQL will walk through the dependencies and drop those too.

postgres=# drop extension mysql_fdw cascade;
NOTICE:  drop cascades to 12 other objects
DETAIL:  drop cascades to server mysql_svr
drop cascades to user mapping for public on server mysql_svr
drop cascades to foreign table "DEPT"
drop cascades to foreign table "EMP"
drop cascades to foreign table hrdb."DEPT"
drop cascades to foreign table hrdb."EMP"
drop cascades to foreign table payroll."DEPT"
drop cascades to foreign table payroll."EMP"
drop cascades to foreign table finance."DEPT"
drop cascades to foreign table finance."EMP"
drop cascades to foreign table sales."DEPT"
drop cascades to foreign table sales."EMP"
DROP EXTENSION
postgres=#

Conclusion

I should concede that the features offered by mysql_fdw are far fewer compared to postgres_fdw. Many of the features are not yet implemented, including column renaming. But the good news is that the key developer and maintainer of mysql_fdw is here with Percona! Hopefully, we will be able to put more effort into implementing some of the missing features. Even so, we can see here that the features implemented so far are powerful enough to support system integration. We can really make the two sing together!

Percona’s support for PostgreSQL

As part of our commitment to being unbiased champions of the open source database eco-system, Percona offers support for PostgreSQL – you can read more about that here.

The post PostgreSQL Accessing MySQL as a Data Source Using mysql_fdw appeared first on Percona Database Performance Blog.

In this tutorial, we will see, How To Connect Nodejs Application To MySQL Database. Node.js can be used in database applications, and one of the most popular databases is MySQL. We have used MongoDB almost time when we choose Node.js as a platform. That is why today’s example is for MySQL. For this first, you need to install MySQL server and then install the mysql package. You can download a free MySQL database at https://www.mysql.com/downloads/.

Connect Nodejs Application To MySQL Database

First, create a node.js project folder by the following command.

mkdir nodemysql
cd nodemysql

Now, create the package.json file by the following command.

npm init -y

Install the mysql package using the following command.

yarn add mysql

# or

npm install mysql --save

So you have downloaded the MySQL database driver and installed that driver successfully. Node.js can use this module to manage the MySQL database.

#Create a connection.

Create one file inside the root of the project called server.js and add the following code inside it.

// server.js

const mysql = require('mysql');

const con = mysql.createConnection({
  host: "localhost",
  user: "root",
  password: "root"
});

con.connect(function(err) {
  if (err) throw err;
  console.log("Connected!");
});

So, here we have imported the mysql module and called the createConnection() method and pass the three parameters. 

1. host
2. user
3. password

In my case, username and password is root. You also can use the environment variable for this because it is significant credentials. Save the file and type the following command.

node server

If your local MySQL database credentials are right then you can connect your MySQL database with the Node.js application and in your terminal, you can see that we have connected our node app to the MySQL database.

#Query a Database.

We can use the SQL statements to read from a MySQL database or write the new data into the database. This is also called “to query” the database. We have already created the connection object created in the example above, and that object has a method for querying the database.

Till now, we have connected our node app to the mysql database, but now we will create a database in MySQL using the query. So let us see how we can create the new database.

// server.js

const mysql = require('mysql');

const con = mysql.createConnection({
  host: "localhost",
  user: "root",
  password: "root"
});

con.connect(function(err) {
  if (err) throw err;
  console.log("Connected!");
  let sql = `CREATE DATABASE nodemysql`;
  con.query(sql, function (err, result) {
    if (err) throw err;
    console.log("The Database is created!!");
  });
});

Save the file, and if the database is not previously created, then it will create a new one. Otherwise, it will throw an error.

#Node.js MySQL Create Table

To create the table in MySQL, we can use the “CREATE TABLE” statement. You need to make sure you define the name of the database when you create the connection. So we need to set one more parameter, while we are connecting our Node app to the MySQL database.

// server.js

const con = mysql.createConnection({
  host: "localhost",
  user: "root",
  password: "root",
  database: "nodemysql"
});

The next thing is to write the following code inside the server.js file.

// server.js

const mysql = require('mysql');

const con = mysql.createConnection({
  host: "localhost",
  user: "root",
  password: "root",
  database: "nodemysql"
});

con.connect(function(err) {
  if (err) throw err;
  console.log("Connected!");
  let sql = `CREATE TABLE customers (name VARCHAR(255), address VARCHAR(255))`;
  con.query(sql, function (err, result) {
    if (err) throw err;
    console.log("The customers table is created!!");
  });
});

Save the file and restart the node.js server. We have not used the nodemon, so we need to restart the server manually.

node server

So in the console, we can see that the customers’ table is created. Now go to the MySQL client. You can either use phpMyAdmin or Sequel Pro.

So, we have successfully created the MySQL database and table.

That is it for the How To Connect Nodejs Application To MySQL Database. Thanks for taking.

The post How To Connect Nodejs Application To MySQL Database appeared first on AppDividend.

This tutorial is for you that is trying to import your current database into a Google Cloud SQL instance, replica, that will be setup for replication purposes.

According to the documentation, you will need to run:

mysqldump \
-h [MASTER_IP] -P [MASTER_PORT] -u [USERNAME] -p \
--databases [DBS] \
--hex-blob --skip-triggers --master-data=1 \
--order-by-primary --compact --no-autocommit \
--default-character-set=utf8 --ignore-table [VIEW] \
--single-transaction --set-gtid-purged=on | gzip | \
gsutil cp - gs://[BUCKET]/[PATH_TO_DUMP]

The mysqldump parameters are:

• -h the hostname or IPV4 address of the primary should replace [MASTER_IP]
• -P the port or the primary server, usually [MASTER_PORT] value will be 3306
• -u takes the username passed on [USERNAME]
• -p informs that a password will be given
• --databases a comma separated list of the databases to be imported. Keep in mind [DBS] should not include the sys, performance_schema, information_schema, and mysql schemas
• --hex-blob necessary for dumping binary columns which types could be BINARY, BLOB and others
• --skip-triggers recommended for the initial load, you can import the triggers at a later moment
• --master-data according to the documentation: “It causes the dump output to include a CHANGE MASTER TO statement that indicates the binary log coordinates (file name and position) of the dumped server”
• --order-by-primary it dumps the data in the primary key order
• --compact produces a more compact output, enabling several flags for the dump
• --no-autocommit encloses the table between a SET autocommit=0 and COMMIT statements
• --default-character-set informs the default character set
• --ignore-table must list the VIEW to be ignored on import, for multiple views, use this option multiple times. Views can be imported later on after promotion of the replica is done
• --single-transaction a START TRANSACTION is sent to the database so the dump will contain the data up to that point in time
• --set-gtid-purged writes the the state of the GTID information into the dump file and disables binary logging when the dump is loaded into the replica

After that the result is compressed in a GZIP file and uploaded to a bucket on Google Cloud Storage with gsutil cp - gs://[BUCKET]/[PATH_TO_DUMP] where [BUCKET] is the bucket you created on GCS and [PATH_TO_DUMP] will save the file in the desired path.

Be aware that no DDL operations should be performed in the database while the dump is being generated else you might find inconsistencies.

See something wrong in this tutorial? Please don’t hesitate to message me through the comments or the contact page.

Please join Percona’s Chief Evangelist, Colin Charles on Wednesday, August 29th, 2018, as he presents Databases in the Hosted Cloud at 7:00 AM PDT (UTC-7) / 10:00 AM EDT (UTC-4).

Nearly everyone today uses some form of database in the hosted cloud. You can use hosted MySQL, MariaDB, Percona Server, and PostgreSQL in several cloud providers as a database as a service (DBaaS).

In this webinar, Colin Charles explores how to efficiently deploy a cloud database configured for optimal performance, with a particular focus on MySQL.

You’ll learn the differences between the various public cloud offerings for Amazon RDS including Aurora, Google Cloud SQL, Rackspace OpenStack DBaaS, Microsoft Azure, and Alibaba Cloud, as well as the access methods and the level of control you have. Hosting in the cloud can be a challenge but after today’s webinar, we’ll make sure you walk away with a better understanding of how you can leverage the cloud for your business needs.

Topics include:

• Backup strategies
• Planning multiple data centers for availability
• Where to host your application
• How to get the most performance out of the solution
• Cost
• Monitoring
• Moving from one DBaaS to another
• Moving from a DBaaS to your own hosted platform

Register Now.

The post Webinar Wed 8/29: Databases in the Hosted Cloud appeared first on Percona Database Performance Blog.

Percona Monitoring and Management (PMM) provides an excellent solution for system monitoring. Sometimes, though, you’ll have the need for a metric that’s not present in the list of node_exporter metrics out of the box. In this post, we introduce a simple method and show how to extend the list of available metrics without modifying the node_exporter code. It’s based on the textfile collector.

Enable the textfile collector in pmm-client

This collector is not enabled by default in the latest version of pmm-client. So, first let’s enable the textfile collector.

# pmm-admin rm linux:metrics
OK, removed system pmm-client-hostname from monitoring.
# pmm-admin add linux:metrics -- --collectors.enabled=diskstats,filefd,filesystem,loadavg,meminfo,netdev,netstat,stat,time,uname,vmstat,textfile --collector.textfile.directory="/tmp"
OK, now monitoring this system.
# pmm-admin ls
pmm-admin 1.13.0
PMM Server      | 10.178.1.252  
Client Name     | pmm-client-hostname
Client Address  | 10.178.1.252  
Service Manager | linux-upstart
-------------- -------------------- ----------- -------- ------------ --------
SERVICE TYPE   NAME                 LOCAL PORT  RUNNING  DATA SOURCE  OPTIONS  
-------------- -------------------- ----------- -------- ------------ --------
linux:metrics  pmm-client-hostname  42000       YES      -

Notice that the whole list of default collectors has to be re-enabled. Also, don’t forget to specify the directory for reading files with the metrics (–collector.textfile.directory=”/tmp”). The exporter reads files with the extension .prom

Add a crontab task

The second step is to add a crontab task to collect metrics and place them into a file.

Here are the cron commands for collecting the number of running and stopping docker containers.

*/1 * * * *     root   echo -n "" > /tmp/docker_all.prom; /usr/bin/docker ps -a | sed -n '1!p'| /usr/bin/wc -l | sed -ne 's/^/node_docker_containers_total /p' >> /tmp/doc
ker_all.prom;
*/1 * * * *     root   echo -n "" > /tmp/docker_running.prom; /usr/bin/docker ps | sed -n '1!p'| /usr/bin/wc -l | sed -ne 's/^/node_docker_containers_running_total /p' >>
/tmp/docker_running.prom;

The result of the commands is placed into the files 

/tmp/docker_running.prom

/tmp/docker_running.prom

Adding the crontab tasks by using a script

Also, we have a few bash scripts that make it much easier to add crontab tasks.

The first one allows you to collect the logged-in users and the size of Innodb data files.

You may use the suggested names of files and metrics or set new ones.

The second script is more universal. It allows us to get the size of any directories or files. This script can be placed directly into a crontab task. You should just specify the list of monitored instances (e.g. /var/log /var/cache/apt /var/lib/mysql/ibdata1)

echo  "*/5 * * * * root bash  /root/object_sizes.sh /var/log /var/cache/apt /var/lib/mysql/ibdata1"  > /etc/cron.d/object_size

So, I hope this has provided useful insight into how to set up the collection of new PMM metrics without the need to write code. Please feel free to use the scripts or configure commands similar to the ones provided above.

More resources you might enjoy

If you are new to PMM, there is a great demo site of the latest version, showing you those out of the box metrics. Or how about our free webinar on monitoring Amazon RDS with PMM?

The post Extend Metrics for Percona Monitoring and Management Without Modifying Code appeared first on Percona Database Performance Blog.

This is a post based on recent tutorials I published, with the goal of discussing how to prepare your current MySQL instance to be configured as an External Primary Server with a Replica/Follower into Google Cloud Platform.

First, I want to talk about the jargon used here. I will be using primary to represent the external “master” server, and replica to represent the “slave” server. Personally, I prefer the terms leader/follower but primary/replica currently seems to be more common in the industry. At some point, the word slave will be used, but because it is the keyword embedded on the server to represent a replica.

The steps given will be in the context of a VM running a one-click install of WordPress acquired through the Google Marketplace (formerly known as Launcher) .

To help prepare for replication you need to configure your primary to meet some requirements.

1. server-id must be configured; it needs to have binary logging enabled; it needs to have GTID enabled, and GTID must be enforced. Tutorial.
2. A Replication User must exist on the primary, remembering you may need root to create it
3. A dump file must be generated using the mysqldump command with some information on it.

The steps above are also necessary if you are migrating from another cloud or on-prem.

Why split the application and database and use a service like Cloud SQL?

First, you will be able to use your application server to do what it was mainly designed for: serve requests of your WordPress application (and it doesn’t much matter for the purposes of this post if you are using nginx or Apache).

Databases are heavy, their deadly sin is gluttony, they tend to occupy as much memory as they can to make lookups fairly fast. Once you are faced with this reality, sharing resources with your application is not a good idea.

Next, you may say: I could use Kubernetes! Yes, you could, but just because you can do something doesn’t mean you should. Configuring stateful applications inside Kubernetes is a challenge, and the fact that pods can be killed at any moment may pose a threat to your data consistency if it happens mid transaction. There are solutions on the market that use MySQL on top of Kubernetes, but that would be a totally different discussion.

You also don’t need to use Cloud SQL, you can set up your
database replicas, or even the primary, on another VM (still wins when compared with putting the database and application together), but in this scenario you are perpetually risking hitting the limits of your finite hardware capabilities.

Finally, Cloud SQL has a 99.95% availability and it is curated by the SRE team of Google. That means you can focus your efforts on what really matters — developing your application — and not spend hours, or even days, setting up servers. Other persuasively convenient features include PITR (Point in Time Recovery) and High Availability in case a failover is necessary.

Setting up the replica on GCP

Accessing the menu SQL in your Google Cloud Console will give you a listing of your current Cloud SQL instances. From there execute the following:

1. Click on the Migrate Data button
2. Once you have familiarized yourself with the steps shown on the screen, click on Begin Migration
3. In the Data source details , fill the form out as follows:
   1. Name of data source: Any valid name for a Cloud SQL instance that will represent the primary server name
   2. Public IP address of source: The IP address of the primary
   3. Port number of source: The port number for the primary, usually 3306
   4. MySQL replication username: The username associated with the replication permissions on the primary
   5. MySQL replication password: The password for the replication username
   6. Database version: Choose between MySQL 5.6 and MySQL 5.7. If you are not sure which version you are running, execute SELECT @@version; in your primary server and you will have the answer.
   7. (Optional) Enable SSL/TLS certification: Upload or enter the Source CA Certificate
4. Click on Next

The next section Cloud SQL read replica creation, will allow you to choose:

1. Read replica instance ID: Any valid name for a Cloud SQL instance that will represent the replica server name
2. Location: choose the Region and then the Zone for which your instance will be provisioned.
3. Machine Type: Choose a Machine Type for your replica; This can be modified later! In some cases it is recommended to choose a higher instance configuration than what you will keep after replication synchronization finishes
4. Storage type: Choice between SSD and HDD. For higher performance choose SSD
5. Storage capacity: It can be from 10GB up to 10TB. The checkbox for Enable automatic storage increases means whenever you’re near capacity, space will be incrementally increased. All increases are permanent
6. SQL Dump File: Dump generated containing binary logging position and GTID information.
7. (Optional) More options can be configured by clicking on Show advanced options like Authorized networks, Database flags, and Labels.
8. Once you’ve filled out this information, click on Create.

The following section, Data synchronization, will display the previous selected options as well the Outgoing IP Address which must be added to your current proxy, firewall, white-list to be able to connect and fetch replication data. Once you are sure your primary can be accessed using the specified credentials, and the IP was white-listed, you can click on Next. After that replication will start.

Live demo

If you want to see this feature in action, please check this video from Google Cloud Next 2018:

This topic has been up in the air for a long time. What would be the best approach: get less large boxes or smaller for the same price? What would be pros and cons for every approach for building database layer infrastructure? What would be the best approach to spread write traffic, etc. Let’s think out loud.

First things first so let's talk about having single box versus few. This is really simple. Having one box for all reads and writes is the simplest architecture possible, but as every simplification it causes some inconveniences. Like outages. Nothing lasts forever and so will not a single box or even cloud instance (they are still utilizing bare metal somehow, right?), which means it will become unavailable at some point so if you used to have just a single box to serve the traffic you will have to wait for recovery or having fun restoring production from backup with downtime and most likely data loss. Not really exciting.

Here comes the second box and so MySQL replication which rise more questions than answers. Since we’re investing money in the second box should it serve live traffic or just be a small backup replica? Well, if master box dies and second is not powerful enough to serve all traffic we can still call it an outage. Not as severe as the one with the single box as we will likely not lose any data, but service is still interrupted which will not make business happy.

What would be the answer? The answer is to have enough capacity in the cluster. if you use two boxes you should have 50% spare capacity which means one completely idle box so in case first will die second will be able to handle full production traffic. Looks like we’ve just spent 50% of infrastructure budget for almost nothing. We can still use stand-by node for backups and to serve some ad-hoc queries, but still doesn’t look efficient, and still vulnerable for a double fault.

Simple math shows us that in the case of 3 boxes we only need 1/3 or 33.3% spare capacity and for four boxes this number is down to 25% which looks way better than half. If you have 10 boxes and one of them dies overall capacity will only drop by 10% so you may not even notice until you’ll get an alert. So what would be the best number? Well, it depends on your MySQL workload types, application requirements and environment you’re in. Cloud environment allows you to fire up instances quickly so you only have to wait for data to be copied to disk from latest backup which turns down turnaround to less than 24 hours (usually). For bare-metal installation, you have to consider delivery time for new boxes so you need to ensure you’ll have enough spare capacity to wait for new boxes to arrive or have the ability to route traffic to another data center.
In my experience I would recommend to start with at least three boxes:

• Active master (write traffic)
• Stand-by master (read traffic)
• Backup instance

Having master to serve write traffic and replication-sensitive queries let you utilize stand-by master for read traffic and keep it warm in case of an outage and automatic failover (can be done using MHA, Orchestrator, etc). The backup instance is usually used as delayed replica to prevent data loss in case of human mistake, application bug, etc, and can be scaled down in the cloud environment to keep overall ownership costs low. Three nodes is also a minimal recommended number of nodes for multi-writable MySQL-based technologists like MySQL InnoDB Cluster, MySQL Group Replication, Galera Cluster and XtraDB Cluster.

Next question is how to route traffic from application to DB cluster. There are several ways to do that. One of the approaches is to make the application aware of several hosts and make it to route traffic to different MySQL servers. This will include read-write split on the application side. Issue with this approach it that every time of the server is unavailable or overloaded application instance will be unable to keep working properly unless more or less complicated logic is implemented.

Another approach is to use third-party software to route traffic dynamically based on current cluster status, load, and roles. There is a number of such software available on the market you could find most of them including benchmarks here. Using third party tools giving you the ability to make your application infrastructure unaware and so avoid complicated non-user-oriented logic inside the application, but more importantly change the query routing dynamically without application deployment.

Speaking about routers there are some things which makes ProxySQL a great choice: it supports query level logic and able to understand query itself, user, schema and other details about the traffic. This will allow you to utilize a lot of ProxySQL features like query-based routing, load balancing, failover support, and many more. We usually recommend to setup ProxySQL instances on the application boxes for availability reasons. Typical architecture also includes monitoring host with graphing and failover tools. You can find it below on the infrastructure diagram.

If you have any questions please do not hesitate to contact us. Our performance and scalability experts will help you to analyze your infrastructure and help to build fast and reliable architecture. We also offer long term support and consulting for ProxySQL users.

Follow us on:

Authored by: Vlad Fedorkov

This release fixes a rare crash as well as a regression bug introduced in 13.1.0.

Bug Fixes:

–Fixed a regression bug introduced in 13.1.0, an error was returned on adding or editing the data in the ‘Result’ tab. The same steps also caused SQLyog to crash sometimes. This is fixed now.

The post SQLyog MySQL GUI 13.1.1 Released appeared first on SQLyog Blog.

A few months ago we faced a memory issue on some of our background jobs. Heroku was killing our dyno because it was exceeding its allowed memory. Thanks to our instrumentation of Sidekiq, it was easy to spot the culprit. The job was doing a fairly complex SQL request, and outputing the query’s result into a CSV file before archiving this file.

In this article, I’ll explain what happened and detail the method we used to solve the problem. I had never seen or used this technique before thus I thought it would be nice to share.

More context

We run a tiny framework, something more like a convention, to run SQL queries and archive the results. If I remove the noise of the framework, we had a code like:

rows = with_replica_database do
  ActiveRecord::Base.connection.select_rows(query)
end

CSV.generate do |csv|
  csv << header
  rows.each { |row| csv << row }
end

In this simplified example, there are:

• with_slave_database: a helper that helps us run a piece of code using a replica database,
• query: our SQL query, as a String, and
• header: a placeholder for the Array of our columns names.

We used select_rows as the results of the query didn’t really match any of our models. It is a reporting query that does too many join, group by, and subqueries. The query takes dozens of minutes to run. We could, and probably should, integrate that into our ETL but that’s not the point…

The resulting CSV file wasn’t that big, maybe a hundred megabytes.

The issue

The memory comsumption of this came from the many rows returned by the select_rows method. Each row is an array containing many entries as our CSV have many columns. Each entry could be a complex datatype converted by ActiveRecord into even more complex Ruby objects. We had many instances of Time with their TimeZone, BigDecimal, …

Since the query returns millions of rows, even while having a linear complexity, the memory consumption is too high.

An impossible approach

At first I thought about paginating the results much in the same way that find_each works. The problem with that was that for 10000 rows, if I paginatd by 1000, it would take 10 times the time of the same request without pagination.

Our query looked like this:

SELECT t.a, u.b, SUM(v.c) as c
FROM t
JOIN u ON u.id = t.u_id
JOIN v ON v.id = u.v_id
GROUP BY t.a, u.b

Just imagine t, u, v being subqueries with unions, OR conditions, other GROUP BYand more of poorly performing stuff. The sad part is the GROUP BY which required the engine to go through all results in order to group rows correctly. Using pagination on this would be something like:

SELECT t.a, u.b, SUM(v.c) as c
FROM t
JOIN u ON u.id = t.u_id
JOIN v ON v.id = u.v_id
GROUP BY 1, 2
ORDER BY 1, 2
LIMIT 10000
OFFSET 1000000

So the fewer entries on a page, the less memory used on the client-side but the more time spent in the database beacause more requests will be done. The more entries on a page, the more memory used on the client-side but the less time spent in the database because less requests will be done.

In the end, this approach wouldn’t have been future-proof.

Focusing more on the problem

It was easy to try to find solutions to the results does not fit in memory problem because it is a known one. It is common with Rails that long lists and association-preloading will cause you memory issues. The quick-fix is to use the find_each or in_batches methods.

I realized that I didn’t actually need to load everything in memory, I’m only interested in getting one line at a time in order to write it into the CSV and then forgotting about it, thanks to the garbage collector.

Solving the right problem

After acknowledging what the true issue was, it was possible to find something more efficient: streaming APIs.

CSV.generate do |csv|
  csv << header
  with_replica_database do
    mysql2 = ActiveRecord::Base.connection.instance_variable_get(:@connection)
    rows = mysql2.query(query, stream: true, cache_rows: false)
    rows.each { |row| csv << row }
  end
end

The idea was to bypass ActiveRecord and use the underlying MySQL client which was providing the stream option. I’m sure there are similar options for other databases.

With that implementation, we only do one request, so no pagination, but we won’t have all the results in memory. We never needed to have all those results in memory in the first place anyway.

Conclusion

I would be very interested to use this feature with ActiveRecord’s ability to return models rather than rows. Maybe it is already possible but I didn’t find it. If you have any further information on the subject, please let me know!

I hope you won’t have to use these lower level APIs. But, if you do encounter the same kind of memory issues, don’t throw money at it right away. Try this first ^^

And obviously, most of this could be avoided by tweaking the layout of data and their relations. In our case, denormalization could make this easier but we’re not ready to pay that cost - yet.

The MySQL Shell has a collection of commands and APIs that allow performing a variety of tasks on MySQL instances  interactively or through scripts.

Although there’s documentation for the JavaScript and Python APIs as well as the User Guide, it comes handy being able to get help about how to use specific elements of the application while working with it, this is what the centralized help system does.…

Pluggable Password Store

This feature which is enabled by default, allows the Shell using an external component “Secret Store”, to persist session passwords in a secure way. Whenever a new session is created while working with the shell in interactive mode, the option to persist the password for the session will be made available.…

Is increasing GP2 volumes size or increasing IOPS for IO1 volumes a valid method for scaling IO-Bound workloads? In this post I’ll focus on one question: how much can we improve performance if we use faster cloud volumes? This post is a continuance of previous cloud research posts:

• Saving With MyRocks in The Cloud
• Percona XtraDB Cluster on Amazon GP2 Volumes

To recap, in Amazon EC2 we can use gp2 and io1 volumes. gp2 performance can be scaled with size, i.e for gp2 volume size of 500GB we get 1500 iops; size 1000GB – 3000 iops; and for 3334GB – 10000 iops (maximal possible value). For io1 volumes we can “buy” throughput up to 30000 iops.

So I wanted to check how both InnoDB and RocksDB storage engines perform on these volumes with different throughput.

Benchmark Scenario

I will use the same datasize that I used in Saving With MyRocks in The Cloud, that is sysbench-tpcc, 50 tables, 100W each, about 500GB datasize in InnoDB and 100GB in RocksDB (compressed with LZ4).

Volumes settings: gp2 volumes from 500GB (1000GB for InnoDB) to 3400GB with 100GB increments (so each increment increases throughput by 300 iops); io1 volumes: 1TB in size, iops from 1000 to 30000 with 1000 increments.

Let’s take look at the results. I will use a slightly different format than usual, but hopefully it represents the results better. You will see density throughout the plots—a higher and narrower chart represents less variance in the throughput. The plot represents the distribution of the throughput.

Results on GP2 volumes:

It’s quite interesting to see how the result scales with better IO throughput. InnoDB does not improve its throughput after gp2 size 2600GB, while MyRocks continues to scale linearly. The problem with MyRocks is that there is a lot of variance in throughput (I will show a one second resolution chart).

Results on IO1 volumes

Here MyRocks again shows an impressive growth as as we add more IO capacity, but also shows a lot of variance on high capacity volumes.

Let’s compare how engines perform with one second resolution. GP2 volume, 3400GB:

IO1 volume, 30000 iops:

So for MyRocks there seems to be periodical background activity, which does not allow it to achieve a stable throughput.

Raw results, if you’d like to review them, can be found here: https://github.com/Percona-Lab-results/201808-rocksdb-cloudio

Conclusions

If you are looking to improve throughput in IO-bound workloads, either increasing GP2 volumes size or increasing IOPS for IO1 volumes is a valid method, especially for the MyRocks engine.

The post Scaling IO-Bound Workloads for MySQL in the Cloud appeared first on Percona Database Performance Blog.

This blog will show how to install the Percona XtraDB Cluster on three CentOS 7.3 servers, using the packages from Percona repositories. This is a step-by-step installation and configuration blog, We recommend Percona XtraDB Cluster for maximum availability / reliability and scale-out READ/WRITE optimally. We are an private-label independent and vendor neutral consulting, support, managed services and education solutions provider for MySQL, MariaDB, Percona Server and ClickHouse with core expertise in performance, scalability, high availability and database reliability engineering. All our blog posts are purely focussed on education and research across open source database systems infrastructure operations. To engage us for building and managing web-scale database infrastructure operations, Please contact us on contact@minervadb.com  

This cluster will be assembled of three servers/nodes:

node #1

hostname: PXC1

IP: 138.197.70.35

node #2

hostname: PXC2

IP: 159.203.118.230

node #3

hostname: PXC3

IP: 138.197.8.226

Prerequisites

• All three nodes have a CentOS 7.3 installation.
• Firewall has been set up to allow connecting to ports 3306, 4444, 4567 and 4568
• SELinux is disabled

Installing from Percona Repository on 138.197.70.35

• Install the Percona repository package:

$ sudo yum install http://www.percona.com/downloads/percona-release/redhat/0.1-4/percona-release-0.1-4.noarch.rpm

• You should see the following if successful:

Installed:

 percona-release.noarch 0:0.1-4

Complete!

• Check that the packages are available:

$ sudo yum list | grep Percona-XtraDB-Cluster-57

Percona-XtraDB-Cluster-57.x86_64          5.7.14-26.17.1.el7         percona-release-x86_64

Percona-XtraDB-Cluster-57-debuginfo.x86_64 5.7.14-26.17.1.el7         percona-release-x86_64

• Install the Percona XtraDB Cluster packages:

$ sudo yum install Percona-XtraDB-Cluster-57

• Start the Percona XtraDB Cluster server:

$ sudo service mysql start

• Copy the automatically generated temporary password for the superuser account:

$ sudo grep 'temporary password' /var/log/mysqld.log

• Use this password to login as root:

$ mysql -u root -p

• Change the password for the superuser account and log out. For example:

mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'root';

Query OK, 0 rows affected (0.00 sec)

mysql> exit

Bye

• Stop the mysql service:

$ sudo service mysql stop

Repeat the same Percona XtraDB Cluster installation process for 159.203.118.230 and 138.197.8.226

Configuring nodes

We have to configure separately the nodes 138.197.70.35, 159.203.118.230 and 138.197.8.226 for successfully implementing an fully operational Percona XtraDB Cluster ecosystem.

Configuring the node 138.197.70.35

Configuration file /etc/my.cnf for the first node should look like:

[mysqld]

datadir=/var/lib/mysql

user=mysql

# Path to Galera library

wsrep_provider=/usr/lib64/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3

wsrep_cluster_address=gcomm://138.197.70.35,159.203.118.230,138.197.8.226

# In order for Galera to work correctly binlog format should be ROW

binlog_format=ROW

# MyISAM storage engine has only experimental support

default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera

innodb_autoinc_lock_mode=2

# Node #1 address

wsrep_node_address=138.197.70.35

# SST method

wsrep_sst_method=xtrabackup-v2

# Cluster name

wsrep_cluster_name=pxc_cluster

# Authentication for SST method

wsrep_sst_auth="sstuser:sstuser"

The first node can be started with the following command:

# /etc/init.d/mysql bootstrap-pxc

We are using CentOS 7.3 so systemd bootstrap service should be used:

# systemctl start mysql@bootstrap.service

This command will start the cluster with initial wsrep_cluster_address set to gcomm://. This way the cluster will be bootstrapped and in case the node or MySQL have to be restarted later, there would be no need to change the configuration file.

After the first node has been started, cluster status can be checked by:

mysql> show status like 'wsrep%';

+------------------------------+------------------------------------------------------------+

| Variable_name               | Value                                                     |

+------------------------------+------------------------------------------------------------+

| wsrep_local_state_uuid      | 5ea977b8-0fc0-11e7-8f73-26f60f083bd5                      |

| wsrep_protocol_version      | 7                                                         |

| wsrep_last_committed        | 8                                                         |

| wsrep_replicated            | 4                                                         |

| wsrep_replicated_bytes      | 906                                                       |

| wsrep_repl_keys             | 4                                                         |

| wsrep_repl_keys_bytes       | 124                                                       |

| wsrep_repl_data_bytes       | 526                                                       |

| wsrep_repl_other_bytes      | 0                                                         |

| wsrep_received              | 9                                                         |

| wsrep_received_bytes        | 1181                                                      |

| wsrep_local_commits         | 0                                                         |

| wsrep_local_cert_failures   | 0                                                         |

| wsrep_local_replays         | 0                                                         |

| wsrep_local_send_queue      | 0                                                         |

| wsrep_local_send_queue_max  | 1                                                         |

| wsrep_local_send_queue_min  | 0                                                         |

| wsrep_local_send_queue_avg  | 0.000000                                                  |

| wsrep_local_recv_queue      | 0                                                         |

| wsrep_local_recv_queue_max  | 2                                                         |

| wsrep_local_recv_queue_min  | 0                                                         |

| wsrep_local_recv_queue_avg  | 0.111111                                                  |

| wsrep_local_cached_downto   | 3                                                         |

| wsrep_flow_control_paused_ns | 0                                                         |

| wsrep_flow_control_paused   | 0.000000                                                  |

| wsrep_flow_control_sent     | 0                                                         |

| wsrep_flow_control_recv     | 0                                                         |

| wsrep_flow_control_interval | [ 28, 28 ]                                                |

| wsrep_cert_deps_distance    | 1.000000                                                  |

| wsrep_apply_oooe            | 0.000000                                                  |

| wsrep_apply_oool            | 0.000000                                                  |

| wsrep_apply_window          | 1.000000                                                  |

| wsrep_commit_oooe           | 0.000000                                                  |

| wsrep_commit_oool           | 0.000000                                                  |

| wsrep_commit_window         | 1.000000                                                  |

| wsrep_local_state           | 4                                                         |

| wsrep_local_state_comment   | Synced                                                    |

| wsrep_cert_index_size       | 2                                                         |

| wsrep_cert_bucket_count     | 22                                                        |

| wsrep_gcache_pool_size      | 3128                                                      |

| wsrep_causal_reads          | 0                                                         |

| wsrep_cert_interval         | 0.000000                                                  |

| wsrep_incoming_addresses    | 159.203.118.230:3306,138.197.8.226:3306,138.197.70.35:3306 |

| wsrep_desync_count          | 0                                                         |

| wsrep_evs_delayed           |                                                           |

| wsrep_evs_evict_list        |                                                           |

| wsrep_evs_repl_latency      | 0/0/0/0/0                                                 |

| wsrep_evs_state             | OPERATIONAL                                               |

| wsrep_gcomm_uuid            | b79d90df-1077-11e7-9922-3a1b217f7371                      |

| wsrep_cluster_conf_id       | 3                                                         |

| wsrep_cluster_size          | 3                                                         |

| wsrep_cluster_state_uuid    | 5ea977b8-0fc0-11e7-8f73-26f60f083bd5                      |

| wsrep_cluster_status        | Primary                                                   |

| wsrep_connected             | ON                                                        |

| wsrep_local_bf_aborts       | 0                                                         |

| wsrep_local_index           | 2                                                         |

| wsrep_provider_name         | Galera                                                    |

| wsrep_provider_vendor       | Codership Oy <info@codership.com>                         |

| wsrep_provider_version      | 3.20(r7e383f7)                                            |

| wsrep_ready                 | ON                                                        |

+------------------------------+------------------------------------------------------------+

60 rows in set (0.01 sec)

This output above shows that the cluster has been successfully bootstrapped.

In order to perform successful State Snapshot Transfer using XtraBackup new user needs to be set up with proper privileges:

mysql@PXC1> CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'sstuser';

mysql@PXC1> GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';

mysql@PXC1> FLUSH PRIVILEGES;

Configuration file /etc/my.cnf on the second node (PXC2) should look like this:

[mysqld]

datadir=/var/lib/mysql

user=mysql

# Path to Galera library

wsrep_provider=/usr/lib64/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3

wsrep_cluster_address=gcomm://138.197.70.35,159.203.118.230,138.197.8.226

# In order for Galera to work correctly binlog format should be ROW

binlog_format=ROW

# MyISAM storage engine has only experimental support

default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera

innodb_autoinc_lock_mode=2

# Node #2 address

wsrep_node_address=159.203.118.230

# SST method

wsrep_sst_method=xtrabackup-v2

# Cluster name

wsrep_cluster_name=pxc_cluster

# Authentication for SST method

wsrep_sst_auth="sstuser:sstuser"

Second node can be started with the following command:

# systemctl start mysql

Cluster status can now be checked on both nodes. This is the example from the second node (PXC2):

mysql> show status like 'wsrep%';

+------------------------------+------------------------------------------------------------+

| Variable_name               | Value                                                     |

+------------------------------+------------------------------------------------------------+

| wsrep_local_state_uuid      | 5ea977b8-0fc0-11e7-8f73-26f60f083bd5                      |

| wsrep_protocol_version      | 7                                                         |

| wsrep_last_committed        | 8                                                         |

| wsrep_replicated            | 0                                                         |

| wsrep_replicated_bytes      | 0                                                         |

| wsrep_repl_keys             | 0                                                         |

| wsrep_repl_keys_bytes       | 0                                                         |

| wsrep_repl_data_bytes       | 0                                                         |

| wsrep_repl_other_bytes      | 0                                                         |

| wsrep_received              | 10                                                        |

| wsrep_received_bytes        | 1238                                                      |

| wsrep_local_commits         | 0                                                         |

| wsrep_local_cert_failures   | 0                                                         |

| wsrep_local_replays         | 0                                                         |

| wsrep_local_send_queue      | 0                                                         |

| wsrep_local_send_queue_max  | 1                                                         |

| wsrep_local_send_queue_min  | 0                                                         |

| wsrep_local_send_queue_avg  | 0.000000                                                  |

| wsrep_local_recv_queue      | 0                                                         |

| wsrep_local_recv_queue_max  | 1                                                         |

| wsrep_local_recv_queue_min  | 0                                                         |

| wsrep_local_recv_queue_avg  | 0.000000                                                  |

| wsrep_local_cached_downto   | 6                                                         |

| wsrep_flow_control_paused_ns | 0                                                         |

| wsrep_flow_control_paused   | 0.000000                                                  |

| wsrep_flow_control_sent     | 0                                                         |

| wsrep_flow_control_recv     | 0                                                         |

| wsrep_flow_control_interval | [ 28, 28 ]                                                |

| wsrep_cert_deps_distance    | 1.000000                                                  |

| wsrep_apply_oooe            | 0.000000                                                  |

| wsrep_apply_oool            | 0.000000                                                  |

| wsrep_apply_window          | 1.000000                                                  |

| wsrep_commit_oooe           | 0.000000                                                  |

| wsrep_commit_oool           | 0.000000                                                  |

| wsrep_commit_window         | 1.000000                                                  |

| wsrep_local_state           | 4                                                         |

| wsrep_local_state_comment   | Synced                                                    |

| wsrep_cert_index_size       | 2                                                         |

| wsrep_cert_bucket_count     | 22                                                        |

| wsrep_gcache_pool_size      | 2300                                                      |

| wsrep_causal_reads          | 0                                                         |

| wsrep_cert_interval         | 0.000000                                                  |

| wsrep_incoming_addresses    | 159.203.118.230:3306,138.197.8.226:3306,138.197.70.35:3306 |

| wsrep_desync_count          | 0                                                         |

| wsrep_evs_delayed           |                                                           |

| wsrep_evs_evict_list        |                                                           |

| wsrep_evs_repl_latency      | 0/0/0/0/0                                                 |

| wsrep_evs_state             | OPERATIONAL                                               |

| wsrep_gcomm_uuid            | 248e2782-1078-11e7-a269-4a3ec033a606                      |

| wsrep_cluster_conf_id       | 3                                                         |

| wsrep_cluster_size          | 3                                                         |

| wsrep_cluster_state_uuid    | 5ea977b8-0fc0-11e7-8f73-26f60f083bd5                      |

| wsrep_cluster_status        | Primary                                                   |

| wsrep_connected             | ON                                                        |

| wsrep_local_bf_aborts       | 0                                                         |

| wsrep_local_index           | 0                                                         |

| wsrep_provider_name         | Galera                                                    |

| wsrep_provider_vendor       | Codership Oy <info@codership.com>                         |

| wsrep_provider_version      | 3.20(r7e383f7)                                            |

| wsrep_ready                 | ON                                                        |

+------------------------------+------------------------------------------------------------+

60 rows in set (0.00 sec)

This output shows that the new node has been successfully added to the cluster.

MySQL configuration file /etc/my.cnf on the third node (PXC3) should look like this:

[mysqld]

datadir=/var/lib/mysql

user=mysql

# Path to Galera library

wsrep_provider=/usr/lib64/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3

wsrep_cluster_address=gcomm://138.197.70.35,159.203.118.230,138.197.8.226

# In order for Galera to work correctly binlog format should be ROW

binlog_format=ROW

# MyISAM storage engine has only experimental support

default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera

innodb_autoinc_lock_mode=2

# Node #3 address

wsrep_node_address=138.197.8.226

# SST method

wsrep_sst_method=xtrabackup-v2

# Cluster name

wsrep_cluster_name=pxc_cluster

# Authentication for SST method

wsrep_sst_auth="sstuser:sstuser"

Third node can now be started with the following command:

# systemctl start mysql

Percona XtraDB Cluster status can now be checked from the third node (PXC3):

mysql> show status like 'wsrep%';

+------------------------------+------------------------------------------------------------+

| Variable_name               | Value                                                     |

+------------------------------+------------------------------------------------------------+

| wsrep_local_state_uuid      | 5ea977b8-0fc0-11e7-8f73-26f60f083bd5                      |

| wsrep_protocol_version      | 7                                                         |

| wsrep_last_committed        | 8                                                         |

| wsrep_replicated            | 2                                                         |

| wsrep_replicated_bytes      | 396                                                       |

| wsrep_repl_keys             | 2                                                         |

| wsrep_repl_keys_bytes       | 62                                                        |

| wsrep_repl_data_bytes       | 206                                                       |

| wsrep_repl_other_bytes      | 0                                                         |

| wsrep_received              | 4                                                         |

| wsrep_received_bytes        | 529                                                       |

| wsrep_local_commits         | 0                                                         |

| wsrep_local_cert_failures   | 0                                                         |

| wsrep_local_replays         | 0                                                         |

| wsrep_local_send_queue      | 0                                                         |

| wsrep_local_send_queue_max  | 1                                                         |

| wsrep_local_send_queue_min  | 0                                                         |

| wsrep_local_send_queue_avg  | 0.000000                                                  |

| wsrep_local_recv_queue      | 0                                                         |

| wsrep_local_recv_queue_max  | 1                                                         |

| wsrep_local_recv_queue_min  | 0                                                         |

| wsrep_local_recv_queue_avg  | 0.000000                                                  |

| wsrep_local_cached_downto   | 6                                                         |

| wsrep_flow_control_paused_ns | 0                                                         |

| wsrep_flow_control_paused   | 0.000000                                                  |

| wsrep_flow_control_sent     | 0                                                         |

| wsrep_flow_control_recv     | 0                                                         |

| wsrep_flow_control_interval | [ 28, 28 ]                                                |

| wsrep_cert_deps_distance    | 1.000000                                                  |

| wsrep_apply_oooe            | 0.000000                                                  |

| wsrep_apply_oool            | 0.000000                                                  |

| wsrep_apply_window          | 1.000000                                                  |

| wsrep_commit_oooe           | 0.000000                                                  |

| wsrep_commit_oool           | 0.000000                                                  |

| wsrep_commit_window         | 1.000000                                                  |

| wsrep_local_state           | 4                                                         |

| wsrep_local_state_comment   | Synced                                                    |

| wsrep_cert_index_size       | 2                                                         |

| wsrep_cert_bucket_count     | 22                                                        |

| wsrep_gcache_pool_size      | 2166                                                      |

| wsrep_causal_reads          | 0                                                         |

| wsrep_cert_interval         | 0.000000                                                  |

| wsrep_incoming_addresses    | 159.203.118.230:3306,138.197.8.226:3306,138.197.70.35:3306 |

| wsrep_desync_count          | 0                                                         |

| wsrep_evs_delayed           |                                                           |

| wsrep_evs_evict_list        |                                                           |

| wsrep_evs_repl_latency      | 0/0/0/0/0                                                 |

| wsrep_evs_state             | OPERATIONAL                                               |

| wsrep_gcomm_uuid            | 3f51b20e-1078-11e7-8405-8e9b37a37cb1                      |

| wsrep_cluster_conf_id       | 3                                                         |

| wsrep_cluster_size          | 3                                                         |

| wsrep_cluster_state_uuid    | 5ea977b8-0fc0-11e7-8f73-26f60f083bd5                      |

| wsrep_cluster_status        | Primary                                                   |

| wsrep_connected             | ON                                                        |

| wsrep_local_bf_aborts       | 0                                                         |

| wsrep_local_index           | 1                                                         |

| wsrep_provider_name         | Galera                                                    |

| wsrep_provider_vendor       | Codership Oy <info@codership.com>                         |

| wsrep_provider_version      | 3.20(r7e383f7)                                            |

| wsrep_ready                 | ON                                                        |

+------------------------------+------------------------------------------------------------+

60 rows in set (0.03 sec)

This output confirms that the third node has joined the cluster.

Testing Replication

Creating the new database on the PXC1 node:

mysql> create database minervadb;

Query OK, 1 row affected (0.01 sec)

Creating the example table on the PXC2 node:

mysql> use minervadb;

Database changed

mysql> CREATE TABLE example (node_id INT PRIMARY KEY, node_name VARCHAR(30));

Query OK, 0 rows affected (0.01 sec)

Inserting records on the PXC3 node:

mysql> INSERT INTO minervadb.example VALUES (1, 'MinervaDB');

Query OK, 1 row affected (0.07 sec)

Retrieving all the rows from that table on the PXC1 node:

mysql> select * from minervadb.example;

+---------+-----------+

| node_id | node_name |

+---------+-----------+

|      1 | MinervaDB |

+---------+-----------+

1 row in set (0.00 sec)

The post Installation and configuration of Percona XtraDB Cluster on CentOS 7.3 appeared first on MySQL Consulting, Support and Remote DBA Services.

In April, I already posted an article on how to upgrade safely your MySQL InnoDB Cluster, let’s review this procedure now that MySQL 8.0.12 is out.

To upgrade all the members of a MySQL InnoDB Cluster (Group), you need to keep in mind the following points:

• upgrade all the nodes one by one
• always end by the Primary Master in case of Single Primary Mode
• after upgrading the binaries don’t forget to start MySQL without starting Group Replication (group_replication_start_on_boot=0)
• to run mysql_upgrade

Let’s see this in action on the video below:

As you could see, this is quick and easy.

Summary

This is an overview of the operations to run on each nodes, one by one:

mysql> set persist group_replication_start_on_boot=0;

# systemctl stop mysqld
# yum update mysql-community-server mysql-shell

# systemctl start mysqld
# mysql_upgrade

mysql> set persist group_replication_start_on_boot=1;
mysql> restart;

Enjoy MySQL InnoDB Cluster, good migration to 8.0.12 and don’t forget to register to Oracle Open World if you want to learn more about MySQL 8.0 and InnoDB Cluster !

This blog will show how to install the Percona XtraDB Cluster on three CentOS 7.3 servers, using the packages from Percona repositories. This is a step-by-step installation and configuration blog, We recommend Percona XtraDB Cluster for maximum availability / reliability and scale-out READ/WRITE optimally. We are an private-label independent and vendor neutral consulting, support, managed services and education solutions provider for MySQL, MariaDB, Percona Server and ClickHouse with core expertise in performance, scalability, high availability and database reliability engineering. All our blog posts are purely focussed on education and research across open source database systems infrastructure operations. To engage us for building and managing web-scale database infrastructure operations, Please contact us on contact@minervadb.com

This cluster will be assembled of three servers/nodes:

node #1

hostname: PXC1

IP: 138.197.70.35

node #2

hostname: PXC2

IP: 159.203.118.230

node #3

hostname: PXC3

IP: 138.197.8.226

Prerequisites

• All three nodes have a CentOS 7.3 installation.
• Firewall has been set up to allow connecting to ports 3306, 4444, 4567 and 4568
• SELinux is disabled

Installing from Percona Repository on 138.197.70.35

• Install the Percona repository package:

$ sudo yum install http://www.percona.com/downloads/percona-release/redhat/0.1-4/percona-release-0.1-4.noarch.rpm

• You should see the following if successful:

Installed:

 percona-release.noarch 0:0.1-4

Complete!

• Check that the packages are available:

$ sudo yum list | grep Percona-XtraDB-Cluster-57

Percona-XtraDB-Cluster-57.x86_64          5.7.14-26.17.1.el7         percona-release-x86_64

Percona-XtraDB-Cluster-57-debuginfo.x86_64 5.7.14-26.17.1.el7         percona-release-x86_64

• Install the Percona XtraDB Cluster packages:

$ sudo yum install Percona-XtraDB-Cluster-57

• Start the Percona XtraDB Cluster server:

$ sudo service mysql start

• Copy the automatically generated temporary password for the superuser account:

$ sudo grep 'temporary password' /var/log/mysqld.log

• Use this password to login as root:

$ mysql -u root -p

• Change the password for the superuser account and log out. For example:

mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'root';

Query OK, 0 rows affected (0.00 sec)

mysql> exit

Bye

• Stop the mysql service:

$ sudo service mysql stop

Repeat the same Percona XtraDB Cluster installation process for 159.203.118.230 and 138.197.8.226

Configuring nodes

We have to configure separately the nodes 138.197.70.35, 159.203.118.230 and 138.197.8.226 for successfully implementing an fully operational Percona XtraDB Cluster ecosystem.

Configuring the node 138.197.70.35

Configuration file /etc/my.cnf for the first node should look like:

[mysqld]

datadir=/var/lib/mysql

user=mysql

# Path to Galera library

wsrep_provider=/usr/lib64/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3

wsrep_cluster_address=gcomm://138.197.70.35,159.203.118.230,138.197.8.226

# In order for Galera to work correctly binlog format should be ROW

binlog_format=ROW

# MyISAM storage engine has only experimental support

default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera

innodb_autoinc_lock_mode=2

# Node #1 address

wsrep_node_address=138.197.70.35

# SST method

wsrep_sst_method=xtrabackup-v2

# Cluster name

wsrep_cluster_name=pxc_cluster

# Authentication for SST method

wsrep_sst_auth="sstuser:sstuser"

The first node can be started with the following command:

# /etc/init.d/mysql bootstrap-pxc

We are using CentOS 7.3 so systemd bootstrap service should be used:

# systemctl start mysql@bootstrap.service

This command will start the cluster with initial wsrep_cluster_address set to gcomm://. This way the cluster will be bootstrapped and in case the node or MySQL have to be restarted later, there would be no need to change the configuration file.

After the first node has been started, cluster status can be checked by:

mysql> show status like 'wsrep%';

+------------------------------+------------------------------------------------------------+

| Variable_name               | Value                                                     |

+------------------------------+------------------------------------------------------------+

| wsrep_local_state_uuid      | 5ea977b8-0fc0-11e7-8f73-26f60f083bd5                      |

| wsrep_protocol_version      | 7                                                         |

| wsrep_last_committed        | 8                                                         |

| wsrep_replicated            | 4                                                         |

| wsrep_replicated_bytes      | 906                                                       |

| wsrep_repl_keys             | 4                                                         |

| wsrep_repl_keys_bytes       | 124                                                       |

| wsrep_repl_data_bytes       | 526                                                       |

| wsrep_repl_other_bytes      | 0                                                         |

| wsrep_received              | 9                                                         |

| wsrep_received_bytes        | 1181                                                      |

| wsrep_local_commits         | 0                                                         |

| wsrep_local_cert_failures   | 0                                                         |

| wsrep_local_replays         | 0                                                         |

| wsrep_local_send_queue      | 0                                                         |

| wsrep_local_send_queue_max  | 1                                                         |

| wsrep_local_send_queue_min  | 0                                                         |

| wsrep_local_send_queue_avg  | 0.000000                                                  |

| wsrep_local_recv_queue      | 0                                                         |

| wsrep_local_recv_queue_max  | 2                                                         |

| wsrep_local_recv_queue_min  | 0                                                         |

| wsrep_local_recv_queue_avg  | 0.111111                                                  |

| wsrep_local_cached_downto   | 3                                                         |

| wsrep_flow_control_paused_ns | 0                                                         |

| wsrep_flow_control_paused   | 0.000000                                                  |

| wsrep_flow_control_sent     | 0                                                         |

| wsrep_flow_control_recv     | 0                                                         |

| wsrep_flow_control_interval | [ 28, 28 ]                                                |

| wsrep_cert_deps_distance    | 1.000000                                                  |

| wsrep_apply_oooe            | 0.000000                                                  |

| wsrep_apply_oool            | 0.000000                                                  |

| wsrep_apply_window          | 1.000000                                                  |

| wsrep_commit_oooe           | 0.000000                                                  |

| wsrep_commit_oool           | 0.000000                                                  |

| wsrep_commit_window         | 1.000000                                                  |

| wsrep_local_state           | 4                                                         |

| wsrep_local_state_comment   | Synced                                                    |

| wsrep_cert_index_size       | 2                                                         |

| wsrep_cert_bucket_count     | 22                                                        |

| wsrep_gcache_pool_size      | 3128                                                      |

| wsrep_causal_reads          | 0                                                         |

| wsrep_cert_interval         | 0.000000                                                  |

| wsrep_incoming_addresses    | 159.203.118.230:3306,138.197.8.226:3306,138.197.70.35:3306 |

| wsrep_desync_count          | 0                                                         |

| wsrep_evs_delayed           |                                                           |

| wsrep_evs_evict_list        |                                                           |

| wsrep_evs_repl_latency      | 0/0/0/0/0                                                 |

| wsrep_evs_state             | OPERATIONAL                                               |

| wsrep_gcomm_uuid            | b79d90df-1077-11e7-9922-3a1b217f7371                      |

| wsrep_cluster_conf_id       | 3                                                         |

| wsrep_cluster_size          | 3                                                         |

| wsrep_cluster_state_uuid    | 5ea977b8-0fc0-11e7-8f73-26f60f083bd5                      |

| wsrep_cluster_status        | Primary                                                   |

| wsrep_connected             | ON                                                        |

| wsrep_local_bf_aborts       | 0                                                         |

| wsrep_local_index           | 2                                                         |

| wsrep_provider_name         | Galera                                                    |

| wsrep_provider_vendor       | Codership Oy <info@codership.com>                         |

| wsrep_provider_version      | 3.20(r7e383f7)                                            |

| wsrep_ready                 | ON                                                        |

+------------------------------+------------------------------------------------------------+

60 rows in set (0.01 sec)

This output above shows that the cluster has been successfully bootstrapped.

In order to perform successful State Snapshot Transfer using XtraBackup new user needs to be set up with proper privileges:

mysql@PXC1> CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'sstuser';

mysql@PXC1> GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';

mysql@PXC1> FLUSH PRIVILEGES;

Configuration file /etc/my.cnf on the second node (PXC2) should look like this:

[mysqld]

datadir=/var/lib/mysql

user=mysql

# Path to Galera library

wsrep_provider=/usr/lib64/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3

wsrep_cluster_address=gcomm://138.197.70.35,159.203.118.230,138.197.8.226

# In order for Galera to work correctly binlog format should be ROW

binlog_format=ROW

# MyISAM storage engine has only experimental support

default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera

innodb_autoinc_lock_mode=2

# Node #2 address

wsrep_node_address=159.203.118.230

# SST method

wsrep_sst_method=xtrabackup-v2

# Cluster name

wsrep_cluster_name=pxc_cluster

# Authentication for SST method

wsrep_sst_auth="sstuser:sstuser"

Second node can be started with the following command:

# systemctl start mysql

Cluster status can now be checked on both nodes. This is the example from the second node (PXC2):

mysql> show status like 'wsrep%';

+------------------------------+------------------------------------------------------------+

| Variable_name               | Value                                                     |

+------------------------------+------------------------------------------------------------+

| wsrep_local_state_uuid      | 5ea977b8-0fc0-11e7-8f73-26f60f083bd5                      |

| wsrep_protocol_version      | 7                                                         |

| wsrep_last_committed        | 8                                                         |

| wsrep_replicated            | 0                                                         |

| wsrep_replicated_bytes      | 0                                                         |

| wsrep_repl_keys             | 0                                                         |

| wsrep_repl_keys_bytes       | 0                                                         |

| wsrep_repl_data_bytes       | 0                                                         |

| wsrep_repl_other_bytes      | 0                                                         |

| wsrep_received              | 10                                                        |

| wsrep_received_bytes        | 1238                                                      |

| wsrep_local_commits         | 0                                                         |

| wsrep_local_cert_failures   | 0                                                         |

| wsrep_local_replays         | 0                                                         |

| wsrep_local_send_queue      | 0                                                         |

| wsrep_local_send_queue_max  | 1                                                         |

| wsrep_local_send_queue_min  | 0                                                         |

| wsrep_local_send_queue_avg  | 0.000000                                                  |

| wsrep_local_recv_queue      | 0                                                         |

| wsrep_local_recv_queue_max  | 1                                                         |

| wsrep_local_recv_queue_min  | 0                                                         |

| wsrep_local_recv_queue_avg  | 0.000000                                                  |

| wsrep_local_cached_downto   | 6                                                         |

| wsrep_flow_control_paused_ns | 0                                                         |

| wsrep_flow_control_paused   | 0.000000                                                  |

| wsrep_flow_control_sent     | 0                                                         |

| wsrep_flow_control_recv     | 0                                                         |

| wsrep_flow_control_interval | [ 28, 28 ]                                                |

| wsrep_cert_deps_distance    | 1.000000                                                  |

| wsrep_apply_oooe            | 0.000000                                                  |

| wsrep_apply_oool            | 0.000000                                                  |

| wsrep_apply_window          | 1.000000                                                  |

| wsrep_commit_oooe           | 0.000000                                                  |

| wsrep_commit_oool           | 0.000000                                                  |

| wsrep_commit_window         | 1.000000                                                  |

| wsrep_local_state           | 4                                                         |

| wsrep_local_state_comment   | Synced                                                    |

| wsrep_cert_index_size       | 2                                                         |

| wsrep_cert_bucket_count     | 22                                                        |

| wsrep_gcache_pool_size      | 2300                                                      |

| wsrep_causal_reads          | 0                                                         |

| wsrep_cert_interval         | 0.000000                                                  |

| wsrep_incoming_addresses    | 159.203.118.230:3306,138.197.8.226:3306,138.197.70.35:3306 |

| wsrep_desync_count          | 0                                                         |

| wsrep_evs_delayed           |                                                           |

| wsrep_evs_evict_list        |                                                           |

| wsrep_evs_repl_latency      | 0/0/0/0/0                                                 |

| wsrep_evs_state             | OPERATIONAL                                               |

| wsrep_gcomm_uuid            | 248e2782-1078-11e7-a269-4a3ec033a606                      |

| wsrep_cluster_conf_id       | 3                                                         |

| wsrep_cluster_size          | 3                                                         |

| wsrep_cluster_state_uuid    | 5ea977b8-0fc0-11e7-8f73-26f60f083bd5                      |

| wsrep_cluster_status        | Primary                                                   |

| wsrep_connected             | ON                                                        |

| wsrep_local_bf_aborts       | 0                                                         |

| wsrep_local_index           | 0                                                         |

| wsrep_provider_name         | Galera                                                    |

| wsrep_provider_vendor       | Codership Oy <info@codership.com>                         |

| wsrep_provider_version      | 3.20(r7e383f7)                                            |

| wsrep_ready                 | ON                                                        |

+------------------------------+------------------------------------------------------------+

60 rows in set (0.00 sec)

This output shows that the new node has been successfully added to the cluster.

MySQL configuration file /etc/my.cnf on the third node (PXC3) should look like this:

[mysqld]

datadir=/var/lib/mysql

user=mysql

# Path to Galera library

wsrep_provider=/usr/lib64/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3

wsrep_cluster_address=gcomm://138.197.70.35,159.203.118.230,138.197.8.226

# In order for Galera to work correctly binlog format should be ROW

binlog_format=ROW

# MyISAM storage engine has only experimental support

default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera

innodb_autoinc_lock_mode=2

# Node #3 address

wsrep_node_address=138.197.8.226

# SST method

wsrep_sst_method=xtrabackup-v2

# Cluster name

wsrep_cluster_name=pxc_cluster

# Authentication for SST method

wsrep_sst_auth="sstuser:sstuser"

Third node can now be started with the following command:

# systemctl start mysql

Percona XtraDB Cluster status can now be checked from the third node (PXC3):

mysql> show status like 'wsrep%';

+------------------------------+------------------------------------------------------------+

| Variable_name               | Value                                                     |

+------------------------------+------------------------------------------------------------+

| wsrep_local_state_uuid      | 5ea977b8-0fc0-11e7-8f73-26f60f083bd5                      |

| wsrep_protocol_version      | 7                                                         |

| wsrep_last_committed        | 8                                                         |

| wsrep_replicated            | 2                                                         |

| wsrep_replicated_bytes      | 396                                                       |

| wsrep_repl_keys             | 2                                                         |

| wsrep_repl_keys_bytes       | 62                                                        |

| wsrep_repl_data_bytes       | 206                                                       |

| wsrep_repl_other_bytes      | 0                                                         |

| wsrep_received              | 4                                                         |

| wsrep_received_bytes        | 529                                                       |

| wsrep_local_commits         | 0                                                         |

| wsrep_local_cert_failures   | 0                                                         |

| wsrep_local_replays         | 0                                                         |

| wsrep_local_send_queue      | 0                                                         |

| wsrep_local_send_queue_max  | 1                                                         |

| wsrep_local_send_queue_min  | 0                                                         |

| wsrep_local_send_queue_avg  | 0.000000                                                  |

| wsrep_local_recv_queue      | 0                                                         |

| wsrep_local_recv_queue_max  | 1                                                         |

| wsrep_local_recv_queue_min  | 0                                                         |

| wsrep_local_recv_queue_avg  | 0.000000                                                  |

| wsrep_local_cached_downto   | 6                                                         |

| wsrep_flow_control_paused_ns | 0                                                         |

| wsrep_flow_control_paused   | 0.000000                                                  |

| wsrep_flow_control_sent     | 0                                                         |

| wsrep_flow_control_recv     | 0                                                         |

| wsrep_flow_control_interval | [ 28, 28 ]                                                |

| wsrep_cert_deps_distance    | 1.000000                                                  |

| wsrep_apply_oooe            | 0.000000                                                  |

| wsrep_apply_oool            | 0.000000                                                  |

| wsrep_apply_window          | 1.000000                                                  |

| wsrep_commit_oooe           | 0.000000                                                  |

| wsrep_commit_oool           | 0.000000                                                  |

| wsrep_commit_window         | 1.000000                                                  |

| wsrep_local_state           | 4                                                         |

| wsrep_local_state_comment   | Synced                                                    |

| wsrep_cert_index_size       | 2                                                         |

| wsrep_cert_bucket_count     | 22                                                        |

| wsrep_gcache_pool_size      | 2166                                                      |

| wsrep_causal_reads          | 0                                                         |

| wsrep_cert_interval         | 0.000000                                                  |

| wsrep_incoming_addresses    | 159.203.118.230:3306,138.197.8.226:3306,138.197.70.35:3306 |

| wsrep_desync_count          | 0                                                         |

| wsrep_evs_delayed           |                                                           |

| wsrep_evs_evict_list        |                                                           |

| wsrep_evs_repl_latency      | 0/0/0/0/0                                                 |

| wsrep_evs_state             | OPERATIONAL                                               |

| wsrep_gcomm_uuid            | 3f51b20e-1078-11e7-8405-8e9b37a37cb1                      |

| wsrep_cluster_conf_id       | 3                                                         |

| wsrep_cluster_size          | 3                                                         |

| wsrep_cluster_state_uuid    | 5ea977b8-0fc0-11e7-8f73-26f60f083bd5                      |

| wsrep_cluster_status        | Primary                                                   |

| wsrep_connected             | ON                                                        |

| wsrep_local_bf_aborts       | 0                                                         |

| wsrep_local_index           | 1                                                         |

| wsrep_provider_name         | Galera                                                    |

| wsrep_provider_vendor       | Codership Oy <info@codership.com>                         |

| wsrep_provider_version      | 3.20(r7e383f7)                                            |

| wsrep_ready                 | ON                                                        |

+------------------------------+------------------------------------------------------------+

60 rows in set (0.03 sec)

This output confirms that the third node has joined the cluster.

Testing Replication

Creating the new database on the PXC1 node:

mysql> create database minervadb;

Query OK, 1 row affected (0.01 sec)

Creating the example table on the PXC2 node:

mysql> use minervadb;

Database changed

mysql> CREATE TABLE example (node_id INT PRIMARY KEY, node_name VARCHAR(30));

Query OK, 0 rows affected (0.01 sec)

Inserting records on the PXC3 node:

mysql> INSERT INTO minervadb.example VALUES (1, 'MinervaDB');

Query OK, 1 row affected (0.07 sec)

Retrieving all the rows from that table on the PXC1 node:

mysql> select * from minervadb.example;

+---------+-----------+

| node_id | node_name |

+---------+-----------+

|      1 | MinervaDB |

+---------+-----------+

1 row in set (0.00 sec)

The post Installation and configuration of Percona XtraDB Cluster on CentOS 7.3 appeared first on MySQL Consulting, Support and Remote DBA Services.

One of the common ways to classify database workloads is whether it is  “read intensive” or “write intensive”. In other words, whether the workload is dominated by reads or writes.

Why should you care? Because recognizing if the workload is read intensive or write intensive will impact your hardware choices, database configuration as well as what techniques you can apply for performance optimization and scalability.

This question looks trivial on the surface, but as you go deeper—complexity emerges. There are different “levels” of reads and writes for you to consider. You can also choose to look at event counts or at the time it takes to do operations. These can provide very different responses, especially as the cost difference between a single read and a single write can be an order of magnitude.

Let’s examine the TPC-C Benchmark from this point of view, or more specifically its implementation in Sysbench. The illustrations below are taken from Percona Monitoring and Management (PMM) while running this benchmark.

Analyzing read/write workload by counts

At the highest level, you can think about queries that are sent to the database. In this case we can see about 30K of SELECT queries versus 20K of UPDATE+INSERT queries, making this benchmark slightly more read intensive by this measure.

Another way to look at the load is through actual operations at the row level – a single query may touch just one row or may touch millions. In this benchmark the difference between looking at workload from a SQL commands standpoint vs a row operation standpoint yields the same results, but it is not going to always be the case.

Let’s now look at the operating system level. We can see the amount of data written to the disk is 2x more than the amount of data being read from the disk. This workload is write intensive by this measure.

Yet another way to take a look at your workload is to take a look at it from the aspect of tables. This view shows us that tables are being mostly accessed for reads and writes. This in turn allows us to see whether a given table is getting more reads or writes. This is helpful, for example, if you are considering to move some of the tables to a different server and want to clearly understand how your workload will be impacted.

Analyzing Read/Write Workload by Response Time

As I mentioned already, the counts often do not reflect the time to respond, which is typically more representative of the real work being done. To look at timing information from query point of view, we want to look at query analytics.

The “Load” column here is a measure of such a combined response time, versus count which is reflective of query counts. Looking at this list we can see that three out of top five queries are SELECT queries. Looking at the numbers overall, we can see we have a read intensive application from this perspective.

In terms of row level operations, there is currently no easy way to see if reads or writes are dominating overall but  you can get an idea from the table operations dashboard:

This shows the load on a per table basis. It labels reads “Fetch” and breaks down writes in more detail—“Update”, “Delete”, “Inserts”—which is helpful. Not all writes are equal either.

If we want to look at a response time based view of read vs write on an operating system, we can check out this disk IO Load graph. You can see in this case it happens to match the IO activity graph, with storage taking more time to serve write requests versus read requests

Summary

As you can see, the question about whether a workload is read intensive or write intensive, while simple on the surface, can have many different answers. You might ask me “OK, so what should I use?” Well… it really depends.

Looking at query counts is a great way to understand the application’s demands on the database—you can’t really do anything to change the database size.  However by changing the database configuration and schema you may drastically alter the impact of these queries, both from the standpoint of the number of rows they crunch and in terms of the disk IO they require.

The response time based statistics, gathered from the impact your queries cause on the system or disk IO, provide a better representation of the load these queries currently generate.

Another thing to keep in mind—reads and writes are not created equal. My rule of thumb for InnoDB is that a single row write is about 10x more expensive than a single row read.

More resources that you might enjoy

If you found this post useful, you might also like to see some of Percona’s other resources.

For an introduction to PMM, our free and open source management and monitoring software, you might find value in my recorded webinar, MySQL Troubleshooting and Performance Optimization with PMM

While our white paper Performance at Scale could provide useful insight if you are at the planning or review stage.

The post Is It a Read Intensive or a Write Intensive Workload? appeared first on Percona Database Performance Blog.

This morning I received an update on a MySQL bug, someone added a comment on an issue I filed in November 2003, originally for MySQL 4.0.12 and 4.1.0. It’s MySQL bug#1956 (note the very low number!), “Parser allows naming of PRIMARY KEY”:

[25 Nov 2003 16:23] Arjen Lentz
Description:
When specifying a PRIMARY KEY, the parser still accepts a name for the index even though the MySQL server will always name it PRIMARY.
So, the parser should NOT accept this, otherwise a user can get into a confusing situation as his input is ignored/changed.

How to repeat:
CREATE TABLE pk (i INT NOT NULL);
ALTER TABLE pk ADD PRIMARY KEY bla (i);

'bla' after PRIMARY KEY should not be accepted.

Suggested fix:
Fix grammar in parser.

Most likely we found it during a MySQL training session, training days have always been a great bug catching source as students would be encouraged to try lots of quirky things and explore.

It’s not a critical issue, but one from the “era of sloppiness” as I think we may now call it, with the benefit of hindsight.  At the time, it was just regarded as lenient: the parser would silently ignore things it couldn’t process in a particular context.  Like creating a foreign key on a table using an engine that didn’t support foreign keys would see the foreign keys silently disappear, rather than reporting an error.  Many  if not most of those quirks have been cleaned up over the years.  Some are very difficult to get rid of, as people use them in code and thus essentially we’re stuck with old bad behaviour as otherwise we’d break to many applications.  I think that one neat example of that is auto-casting:

SELECT "123 apples" + 1;
+------------------+
| "123 apples" + 1 |
+------------------+
|              124 |
+------------------+
1 row in set, 1 warning (0.000 sec)

SHOW WARNINGS;
+---------+------+-----------------------------------------------+
| Level   | Code | Message                                       |
+---------+------+-----------------------------------------------+
| Warning | 1292 | Truncated incorrect DOUBLE value: '123 hello' |
+---------+------+-----------------------------------------------+

At least that one chucks a warning now, which is good as it allows developers to catch mistakes and thus prevent trouble.  A lenient parser (or grammar) can be convenient, but it tends to also enable application developers to be sloppy, which is not really a good thing.  Something that creates a warning may be harmless, or an indication of a nasty mistake: the parser can’t tell.  So it’s best to ensure that code is clean, free even of warnings.

Going back to the named PRIMARY KEY issue… effectively, a PRIMARY KEY has the name ‘PRIMARY’ internally.  So it can’t really have another name, and the parser should not accept an attempt to name it.  The name silently disappears so when you check back in SHOW CREATE TABLE or SHOW INDEXES, you won’t ever see it.
CREATE TABLE pkx (i INT PRIMARY KEY x) reports a syntax error. So far so good.
But, CREATE TABLE pkx (i INT, PRIMARY KEY x (i)) is accepted, as is ALTER TABLE pkx ADD PRIMARY KEY x (i).

MySQL 8.0 still allows these constructs, as does MariaDB 10.3.9 !

The bug is somewhere in the parser, so that’s sql/sql_yacc.yy.  I’ve tweaked and added things in there before, but it’s been a while and particularly in parser grammar you have to be very careful that changes don’t have other side-effects.  I do think it’s worthwhile fixing even these minor issues.