Join Percona Chief Evangelist Colin Charles as he covers happenings, gives pointers and provides musings on the open source database community.

Grading is underway for talks at Percona Live Europe 2018. I understand that by next week you will see the tutorial schedule released. As part of the program committee, I have enjoyed reviewing tutorials, and I reckon there is great competition for the schedule. I suggest you register now, and don’t forget to book your accommodation (need a discount?).

A video worth watching: How we Live Migrated Millions of BBM Users & its Infrastructure Across the Pacific (Cloud Next ’18). I was a big Blackberry Messenger (BBM) user in its heyday. They moved from their Canadian data center to Google Cloud. Blackberry used MySQL, migrated Oracle to PostgreSQL, as Oracle isn’t sanctioned to run on Google Cloud Platform. They also moved from MySQL to Google CloudSQL since the databases weren’t critical to running the application. Some cases write to BigQuery and have Tableau query them. They make use of Cassandra native replication. With PostgreSQL, there is master-slave replication: they got users onto the new master and then promoted the master. When they did the master promotion, there was minimal (5-10 minutes) downtime while this process was ongoing. This didn’t impact their key services, messages continued to work, and this was seen as acceptable.

Releases

• MariaDB 10.3.9 – InnoDB from 5.7.23, a new variable innodb_log_optimize_ddl for avoiding delay due to page flushing and allowing concurrent backup, and many fixes and improvements around ALTER TABLE too.
• Percona Server for MySQL 5.6.41-84.1 – full-text search index with InnoDB table bug fixed, as well as ensuring queries on a table with CHARSET=euckr COLLATE=euckr_bin always return the same results.
• Percona Server for MySQL 5.5.61-38.13 – bug fixes.

Link List

• From an SQL standards standpoint, Markus Winand has a What’s new in MariaDB 10.3: “Over the last two years, even the SQL dialects started to diverge notably. Treating MariaDB as a distinct product is the unavoidable consequence for me.” – meaning MariaDB Server is now distinct as a database in his eyes.
• Comparing Data-At-Rest Encryption Features for MariaDB, MySQL and Percona Server for MySQL

Upcoming Appearances

• db tech showcase Tokyo 2018 – 19-21 September 2018
• Open Source Summit Europe 2018 – 22-24 October 2018

Feedback

I look forward to feedback/tips via e-mail at colin.charles@percona.com or on Twitter @bytebot.

The post This Week in Data with Colin Charles 50: Percona Live Europe Sessions, PostgreSQL in Google Cloud appeared first on Percona Database Performance Blog.

One of the nice things about MySQL Connector/Python is that it is available in a pure Python implementation. This makes it very portable. Today I have been exploring the possibility to take advantage of that to make MySQL Connector/Python available on my iPad.

There are few Python interpreters available for iPad. The one I will be discussing today is Pythonista 3 which has support for both Python 2.7 and 3.6. One of the things that caught my interest is that it comes with libraries to work with iOS such as accessing the contact and photos as well as UI tools. This is a commercial program (AUD 15), but this far looks to be worth the money. There are other options and I hope to write about those another day.

MySQL Connector/Python is available from PyPi. This makes it easy to install the modules in a uniform way across platforms. Unfortunately, Pythonista 3 does not support the pip command out of the box. Instead there is a community contribution called StaSh that can be used to get a shell-like environment that can be used to execute pip. So, our first step is to install StaSh.

Install StaSh

StaSh is a project maintained by ywangd and can be found on GitHub. It is self-installing by executing a small Python program that can be found in the README.md file on the project repository page. You copy the source code into a new file in Pythonista 3 and execute it (using the “Play” button):

Then you need to restart Pythonista 3. At this point StaSh is installed.

Installing PyPi Package Using StaSh pip

In order to be able to use the pip command through StaSh, you need to launch the launch_stash.py program which was installed in the previous step. The program can be found in the This iPad folder:

Open the program and use the “Play” button again to execute it. This creates the shell. You can do other things than use the pip command, but for the purpose of installing MySQL Connector/Python that is all that is required. The command is:

pip install mysql-connector-python

The console output is:

That’s it. Now you can use the MySQL Connector/Python modules on your iPad just as in any other environment.

Example

To verify it is working, let’s create a simple example. For this to work, you need MySQL installed allowing external access which likely requires enabling access to port 3306 in your firewall.

A simple example that queries the city table in the world database for the city with ID = 130 is:

connect_args = {
    "host": "192.0.2.1",
    "port": 3306,
    "user": "testuser",
    "password": "password",
}

db = mysql.connector.connect(**connect_args)
cursor = db.cursor(named_tuple=True)

sql = "SELECT * FROM world.city WHERE ID = 130"
cursor.execute(sql)
print(cursor.fetchone())

cursor.close()
db.close()

Edit the connect_args dictionary with the connection arguments required for your MySQL instance.

Warning: The connection arguments are included inside the source code here to keep the example simple. Do not do this in real programs. It is unsafe to store the password in the source code and it makes the program hard to maintain.

When you run it, the details of Sydney, Australia is printed to the console:

This all looks great, but what about the X DevAPI? Unfortunately, there are some problems there.

X DevAPI

The X DevAPI is new in MySQL 8.0. It is included in MySQL Connector/Python in the mysqlx module. However, it does not work out of the box in Pythonista 3. When you try to execute the import mysqlx command, it fails:

>>> import mysqlx
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/private/var/mobile/Containers/Shared/AppGroup/BA51D6FF-34C8-460C-88B7-0078A16FBBD2/Pythonista3/Documents/site-packages-3/mysqlx/__init__.py", line 35, in <module>
    from .connection import Session
  File "/private/var/mobile/Containers/Shared/AppGroup/BA51D6FF-34C8-460C-88B7-0078A16FBBD2/Pythonista3/Documents/site-packages-3/mysqlx/connection.py", line 50, in <module>
    from .crud import Schema
  File "/private/var/mobile/Containers/Shared/AppGroup/BA51D6FF-34C8-460C-88B7-0078A16FBBD2/Pythonista3/Documents/site-packages-3/mysqlx/crud.py", line 34, in <module>
    from .statement import (FindStatement, AddStatement, RemoveStatement,
  File "/private/var/mobile/Containers/Shared/AppGroup/BA51D6FF-34C8-460C-88B7-0078A16FBBD2/Pythonista3/Documents/site-packages-3/mysqlx/statement.py", line 36, in <module>
    from .expr import ExprParser
  File "/private/var/mobile/Containers/Shared/AppGroup/BA51D6FF-34C8-460C-88B7-0078A16FBBD2/Pythonista3/Documents/site-packages-3/mysqlx/expr.py", line 34, in <module>
    from .protobuf import Message, mysqlxpb_enum
  File "/private/var/mobile/Containers/Shared/AppGroup/BA51D6FF-34C8-460C-88B7-0078A16FBBD2/Pythonista3/Documents/site-packages-3/mysqlx/protobuf/__init__.py", line 75, in <module>
    from . import mysqlx_connection_pb2
  File "/private/var/mobile/Containers/Shared/AppGroup/BA51D6FF-34C8-460C-88B7-0078A16FBBD2/Pythonista3/Documents/site-packages-3/mysqlx/protobuf/mysqlx_connection_pb2.py", line 8, in <module>
    from google.protobuf import reflection as _reflection
  File "/var/containers/Bundle/Application/743A0BDC-D77A-4479-A1D9-DFF16785FBC7/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/google/protobuf/reflection.py", line 69, in <module>
    from google.protobuf.internal import python_message
  File "/var/containers/Bundle/Application/743A0BDC-D77A-4479-A1D9-DFF16785FBC7/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/google/protobuf/internal/python_message.py", line 849
    except struct.error, e:
                       ^
SyntaxError: invalid syntax

So, the Protobuf module that comes with Pythonista 3 uses the old comma syntax for exception handling. This is not allowed with Python 3 (PEP 3110). An obvious thought is to try to use the StaSh pip command to update Protobuf, but StaSh does not touch the built-in modules. Installing a second copy of Protobuf does not help either. If I find a solution, I will provide an update. However, for the time being you will need to stick to the good old mysql.connector module.

Please join Percona’s CEO, Peter Zaitsev as he presents Choosing the Right Open Source Database on Wednesday, September 5th, 2018 at 11:00 AM PDT (UTC-7) / 2:00 PM EDT (UTC-4).

The world of open-source databases is overwhelming. There are dozens of types of databases – relational DBMS, time-series, graph, document, etc. – not to mention the dozens of software options within each of those categories. More and more, the strategies of enterprises involve open source software and open source database software. This allows companies to be more agile, quick to market, and cost-effective.

So how do you know which open source database to choose? Open-source database companies worldwide are competing for our attention. In this talk, we will distill the madness. I will give an overview of the types of open-source databases, use-cases, and key players in the ecosystem.

Register for this webinar on choosing the right open source database.

The post Webinar Wed 9/5: Choosing the Right Open Source Database appeared first on Percona Database Performance Blog.

Join Percona Chief Evangelist Colin Charles as he covers happenings, gives pointers and provides musings on the open source database community.

There has been a lot of talk around licenses in open source software, and it has hit the database world in the past weeks. Redis Labs relicensed some AGPL software to the Commons Clause (in their case, Apache + Commons Clause; so you can’t really call it Apache any longer). I’ll have more to say on this topic soon, but in the meantime you might enjoy reading Open-source licensing war: Commons Clause. This was the most balanced article I read about this move and the kerfuffle it has caused. We also saw this with Lerna (not database related), and here’s another good read: Open Source Devs Reverse Decision to Block ICE Contractors From Using Software.

Reviewing is under way for Percona Live Europe 2018 talks: the review of the tutorials is complete. We can expect to see a schedule by mid-September, so hang in there—I’ve received a lot of messages asking if talks are going to be approved or not.

Releases

• While not a new release, MySQL Shell 8.0.12 is worth spending some time with, especially since you might enjoy the pluggable password store.
• SqlKata for C# – SqlKata is an elegant Sql Query Builder for C#, it helps you to talk with your database engine with a higher order of freedom, it allows you to write complex queries in an Object Oriented Manner, helpful when you need. Works with MySQL, PostgreSQL, and more

Link List

• tkvdb – trie (radix trie) key-value database, similar to BDB/LevelDB/SQlite4 LSM
• Name that compaction algorithm
• Real world SSD wearout – featuring Redis and PostgreSQL.
• MySQL Connector/Python on iOS Using Pythonista 3 – as someone who enjoys using the iPad Pro with Smart Keyboard, this was a most interesting article!

Industry Updates

• Balazs Pocze is now a database SRE at Wikimedia Foundation. He has spoken at several Percona Live events too!

Upcoming Appearances

• London MySQL Meetup – 11 September 2018
• db tech showcase Tokyo 2018 – 19-21 September 2018
• Open Source Summit Europe 2018 – 22-24 October 2018

Feedback

I look forward to feedback/tips via e-mail at colin.charles@percona.com or on Twitter @bytebot.

The post This Week in Data With Colin Charles 51: Debates Emerging on the Relicensing of OSS appeared first on Percona Database Performance Blog.

In my previous blog post about millions of table in MySQL 8, I was able to create one million tables and test the performance of it. My next challenge is to create 40 million tables in MySQL 8 using shared tablespaces (one tablespace per schema). In this blog post I’m showing how to do it and what challenges we can expect.

Background

Once again – why do we need so many tables in MySQL, what is the use case? The main reason is: customer isolation. With the new focus on security and privacy (take GDPR for example) it is much easier and more beneficial to create a separate schema (or “database” in MySQL terms) for each customer. That creates a new set of challenges that we will need to solve. Here is the summary:

1. Too many files. For each table MySQL creates an FRM file. With MySQL 8.0, this is not the case for InnoDB tables (new data dictionary): it does not create FRM files, only creates IBD file.
2. Too much storage overhead. Just to create 40 million tables we will need to have ~4 – 5 Tb of space. The ZFS filesystem can help here a lot, through compression – see below.
3. MySQL does not work well with so many tables. We have observed a lot of overhead (MySQL needs to open/close table definition files) and contention (table definitions needs to be stored in memory to avoid performance penalty, which introduce mutex contention)

Challenges

When I approached the task of creating 40 million tables, my first challenge was disk space. Just to create them, I needed at least 5Tb of fast disk storage. The good news is: we have the ZFS filesystem which provides compression out of the box. With compression I was able to use just a 250G drive with ZFS – the compression ratio is > 10x:

# du -sh --apparent-size /var/lib/mysql-data
4.7T    /var/lib/mysql-data
# du -sh /var/lib/mysql-data
131G    /var/lib/mysql-data

The second challenge is how to create those tables in a reasonable amount of time. I created a script to “provision” the databases (create all 40 millions tables). The good new is that the performance regression in “create table” speed and scalability bug was fixed so I was able to use this script to create 40 million tables using shared tablespaces (one tablespace per schema):

#/bin/bash
function do_db {
        db_exist=$(mysql -A -s -Nbe "select 1 from information_schema.schemata where schema_name = '$db'")
        if [ "$db_exist" == "1" ]; then echo "Already exists $db"; return 0; fi;
        mysql -vvv -e "create database $db";
        mysql -vvv $db -e "CREATE TABLESPACE $db ADD DATAFILE '$db.ibd' engine=InnoDB;"
        for i in {1..100}
        do
                table="CREATE TABLE sbtest$i ( id int(10) unsigned NOT NULL AUTO_INCREMENT, k int(10) unsigned NOT NULL DEFAULT '0', c varchar(120) NOT NULL DEFAULT '', pad varchar(60) NOT NULL DEFAULT '', PRIMARY KEY (id), KEY k_1 (k) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 tablespace $db;"
                mysql $db -e "$table"
        done
}
c=0
for m in {1..4000000}
do
        for i in {1..40}
        do
                let c=$c+1
                echo $c
                db="sbtest_$c"
                do_db &
        done
        wait
        #if [ $c > 4000000 ]; then exit; fi
done

40 million tables in MySQL 8

Now it’s time for a real test. I’m using the latest MySQL 8 version (at the time of writing): 8.0.12. This implements the new data dictionary.

MySQL config file:

[mysqld]
datadir=/var/lib/mysql-data
socket=/var/lib/mysql-data/mysql.sock
datadir=/var/lib/mysql-data
log-error = /var/lib/mysql-log/error.log
server_id = 12345
log_bin = /var/lib/mysql-log/binlog
relay_log=/var/lib/mysql-log/relay-bin
skip-log-bin=1
innodb_log_group_home_dir = /var/lib/mysql-log
innodb_doublewrite = 0
innodb_flush_log_at_trx_commit=0
innodb_log_file_size=2G
innodb_buffer_pool_size=4G
tablespace_definition_cache = 524288
schema_definition_cache = 524288
table_definition_cache = 524288
table_open_cache=524288
open-files-limit=1000000

Sysbench shell script:

function run_sb() {
conn=" --db-driver=mysql --mysql-socket=/var/lib/mysql-data/mysql.sock  --mysql-db=sbtest_1 --mysql-user=sbtest --mysql-password=abc "
sysbench $conn --oltp_db_count=$db_count --oltp_tables_count=$table_count --oltp-table-size=10000 --report-interval=1 --num-threads=$num_threads --max-requests=0 --max-time=$max_time ./select_custom.lua run | tee -a sysbench_2.txt
}
let db_count=400000
table_count=100
max_time=10000
num_threads=32
run_sb

Sysbench lua script:

pathtest = "/usr/share/sysbench/tests/include/oltp_legacy/"
if pathtest then
   dofile(pathtest .. "common.lua")
else
   require("common")
end
function thread_init(thread_id)
   set_vars()
end
function event()
   local table_name
   local i
   local c_val
   local k_val
   local pad_val
   oltp_db_count = tonumber(oltp_db_count) or 1
   -- local oltp_db_count = 4
   table_name = "sbtest_" .. sb_rand(1, oltp_db_count)..".sbtest".. sb_rand(1, oltp_tables_count)
   k_val = sb_rand(1, oltp_table_size)
   c_val = sb_rand_str([[
###########-###########-###########-###########-###########-###########-###########-###########-###########-###########]])
   pad_val = sb_rand_str([[
###########-###########-###########-###########-###########]])
      rs = db_query("SELECT id FROM " .. table_name .." LIMIT 1")
end

Please note that the tables are empty – no data.

Now we can run the benchmark. Unfortunately, we have a serious mutex contention in the data dictionary. Here are the results:

[ 453s ] thds: 32 tps: 1203.96 qps: 1203.96 (r/w/o: 1203.96/0.00/0.00) lat (ms,95%): 41.10 err/s: 0.00 reconn/s: 0.00
[ 454s ] thds: 32 tps: 1202.32 qps: 1202.32 (r/w/o: 1202.32/0.00/0.00) lat (ms,95%): 42.61 err/s: 0.00 reconn/s: 0.00
[ 455s ] thds: 32 tps: 1196.74 qps: 1196.74 (r/w/o: 1196.74/0.00/0.00) lat (ms,95%): 41.10 err/s: 0.00 reconn/s: 0.00
[ 456s ] thds: 32 tps: 1197.18 qps: 1197.18 (r/w/o: 1197.18/0.00/0.00) lat (ms,95%): 41.10 err/s: 0.00 reconn/s: 0.00
[ 457s ] thds: 32 tps: 887.11 qps: 887.11 (r/w/o: 887.11/0.00/0.00) lat (ms,95%): 41.10 err/s: 0.00 reconn/s: 0.00
[ 458s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 459s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 460s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 461s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 462s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 463s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 464s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 465s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 466s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 467s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 468s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 469s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 470s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 471s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 472s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 473s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 474s ] thds: 32 tps: 403.96 qps: 403.96 (r/w/o: 403.96/0.00/0.00) lat (ms,95%): 16819.24 err/s: 0.00 reconn/s: 0.00
[ 475s ] thds: 32 tps: 1196.00 qps: 1196.00 (r/w/o: 1196.00/0.00/0.00) lat (ms,95%): 41.85 err/s: 0.00 reconn/s: 0.00
[ 476s ] thds: 32 tps: 1208.96 qps: 1208.96 (r/w/o: 1208.96/0.00/0.00) lat (ms,95%): 41.85 err/s: 0.00 reconn/s: 0.00
[ 477s ] thds: 32 tps: 1192.06 qps: 1192.06 (r/w/o: 1192.06/0.00/0.00) lat (ms,95%): 41.85 err/s: 0.00 reconn/s: 0.00
[ 478s ] thds: 32 tps: 1173.89 qps: 1173.89 (r/w/o: 1173.89/0.00/0.00) lat (ms,95%): 43.39 err/s: 0.00 reconn/s: 0.00

As we can see, for ~15 seconds no queries were processed: a complete MySQL stall. That situation – complete stall – happens constantly, every ~25-30 seconds.

Show engine innodb status query shows mutex contention:

SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 498635
--Thread 140456572004096 has waited at que0que.cc line 1072 for 0.00 seconds the semaphore:
Mutex at 0x34a3fe0, Mutex PARSER created pars0pars.cc:98, lock var 1
--Thread 140451898689280 has waited at que0que.cc line 1072 for 0.00 seconds the semaphore:
Mutex at 0x34a3fe0, Mutex PARSER created pars0pars.cc:98, lock var 1
--Thread 140451896919808 has waited at que0que.cc line 1072 for 0.00 seconds the semaphore:
Mutex at 0x34a3fe0, Mutex PARSER created pars0pars.cc:98, lock var 1
--Thread 140456571119360 has waited at que0que.cc line 1072 for 0.00 seconds the semaphore:
Mutex at 0x34a3fe0, Mutex PARSER created pars0pars.cc:98, lock var 1
--Thread 140457044215552 has waited at que0que.cc line 1072 for 0.00 seconds the semaphore:
Mutex at 0x34a3fe0, Mutex PARSER created pars0pars.cc:98, lock var 1
--Thread 140456572299008 has waited at que0que.cc line 1072 for 0.00 seconds the semaphore:
Mutex at 0x34a3fe0, Mutex PARSER created pars0pars.cc:98, lock var 1
--Thread 140457043035904 has waited at que0que.cc line 1072 for 0.00 seconds the semaphore:
Mutex at 0x34a3fe0, Mutex PARSER created pars0pars.cc:98, lock var 1
--Thread 140456571709184 has waited at que0que.cc line 1072 for 0.00 seconds the semaphore:
Mutex at 0x34a3fe0, Mutex PARSER created pars0pars.cc:98, lock var 1
--Thread 140451897214720 has waited at que0que.cc line 1072 for 0.00 seconds the semaphore:
Mutex at 0x34a3fe0, Mutex PARSER created pars0pars.cc:98, lock var 1
--Thread 140451896624896 has waited at que0que.cc line 1072 for 0.00 seconds the semaphore:
Mutex at 0x34a3fe0, Mutex PARSER created pars0pars.cc:98, lock var 1
--Thread 140457042740992 has waited at que0que.cc line 1072 for 0.00 seconds the semaphore:
Mutex at 0x34a3fe0, Mutex PARSER created pars0pars.cc:98, lock var 1
--Thread 140451899279104 has waited at que0que.cc line 1072 for 0.00 seconds the semaphore:
Mutex at 0x34a3fe0, Mutex PARSER created pars0pars.cc:98, lock var 1
--Thread 140457042446080 has waited at que0que.cc line 1072 for 0.00 seconds the semaphore:
Mutex at 0x34a3fe0, Mutex PARSER created pars0pars.cc:98, lock var 1
OS WAIT ARRAY INFO: signal count 89024
RW-shared spins 11216, rounds 14847, OS waits 3641

I’ve filed a  new MySQL bug: DICT_SYS mutex contention causes complete stall when running with 40 mill tables.

I’ve also tested with pareto distribution in sysbench, and even set the ratio to 0.05 (5%) and 0.01 (1%), and mutex contention is still an issue. I have used the following updated sysbench script:

function run_sb() {
conn=" --db-driver=mysql --mysql-socket=/var/lib/mysql-data/mysql.sock  --mysql-db=sbtest_1 --mysql-user=sbtest --mysql-password=abc "
sysbench $conn --rand-type=$rand_type --rand-pareto-h=$pareto_h --oltp_db_count=$db_count --oltp_tables_count=$table_count --oltp-table-size=10000 --report-interval=1 --num-threads=$num_threads --max-requests=0 --max-time=$max_time $test_name run | tee -a sysbench_2.txt
}
let db_count=400000
table_count=100
max_time=10000
num_threads=32
rand_type="pareto"
pareto_h=0.01
test_name="./select_custom.lua"
echo "Now running $rand_type for $max_time seconds, test=$test_name"
run_sb

And the results with 0.01 (1%) are the following:

[ 55s ] thds: 32 tps: 72465.29 qps: 72465.29 (r/w/o: 72465.29/0.00/0.00) lat (ms,95%): 0.53 err/s: 0.00 reconn/s: 0.00
[ 56s ] thds: 32 tps: 68641.04 qps: 68641.04 (r/w/o: 68641.04/0.00/0.00) lat (ms,95%): 0.61 err/s: 0.00 reconn/s: 0.00
[ 57s ] thds: 32 tps: 70479.82 qps: 70479.82 (r/w/o: 70479.82/0.00/0.00) lat (ms,95%): 0.57 err/s: 0.00 reconn/s: 0.00
[ 58s ] thds: 32 tps: 31395.55 qps: 31395.55 (r/w/o: 31395.55/0.00/0.00) lat (ms,95%): 0.49 err/s: 0.00 reconn/s: 0.00
[ 59s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 60s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 61s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 62s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 63s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 64s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 65s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 66s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 67s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 68s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 69s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 70s ] thds: 32 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 71s ] thds: 32 tps: 18879.04 qps: 18879.04 (r/w/o: 18879.04/0.00/0.00) lat (ms,95%): 0.75 err/s: 0.00 reconn/s: 0.00
[ 72s ] thds: 32 tps: 70924.82 qps: 70924.82 (r/w/o: 70924.82/0.00/0.00) lat (ms,95%): 0.48 err/s: 0.00 reconn/s: 0.00
[ 73s ] thds: 32 tps: 72395.57 qps: 72395.57 (r/w/o: 72395.57/0.00/0.00) lat (ms,95%): 0.47 err/s: 0.00 reconn/s: 0.00
[ 74s ] thds: 32 tps: 72483.22 qps: 72484.22 (r/w/o: 72484.22/0.00/0.00) lat (ms,95%): 0.58 err/s: 0.00 reconn/s: 0.00

ZFS

The ZFS filesystem provides compression, which helps tremendously in this case. When MySQL creates an InnoDB table it will create a new blank .ibd file and pre-allocate some pages, which will be blank. I have configured ZFS compression and can see > 10x compression ratio:

# zfs get all | grep compressratio
mysqldata                       compressratio         12.47x                      -
mysqldata                       refcompressratio      1.00x                       -
mysqldata/mysql                 compressratio         12.47x                      -
mysqldata/mysql                 refcompressratio      1.00x                       -
mysqldata/mysql/data            compressratio         12.51x                      -
mysqldata/mysql/data            refcompressratio      12.54x                      -
mysqldata/mysql/log             compressratio         2.79x                       -
mysqldata/mysql/log             refcompressratio      4.57x                       -

Conclusion

It is possible to create 40 million tables with MySQL 8.0 using shared tablespaces. ZFS provides an excellent compression ratio (with gzip) which can help by reducing the overhead of “schema per customer” architecture. Unfortunately, the new data dictionary in MySQL 8.0.12 suffers from the DICT_SYS mutex contention and causes constant “stalls”.

The post 40 million tables in MySQL 8.0 with ZFS appeared first on Percona Database Performance Blog.

Dear MySQL users,

MySQL Connector/Python 2.1.8 GA is a sixth GA version of 2.1 release
series of the pure Python database driver for MySQL. It can be used for
production environments.

MySQL Connector/Python version 2.1.8 GA is compatible with MySQL Server
versions 5.5 and greater. Python 2.6 and greater as well as Python 3.4
and greater are supported. Python 2.4, 2.5, and 3.1, 3.2, 3.3 are not
supported.

MySQL Connector/Python 2.1.8 is available for download from:

http://dev.mysql.com/downloads/connector/python/#downloads

MySQL Connector/Python 2.1.8 (Commercial) will be available for download
on the My Oracle Support (MOS) website. This release will be available
on eDelivery (OSDC) in next month’s upload cycle.

The ChangeLog file included in the distribution contains a brief summary
of changes in MySQL Connector/Python 2.1.8. For a more complete list of
changes, see below or online at:

http://dev.mysql.com/doc/relnotes/connector-python/en/

Enjoy!

Changes in MySQL Connector/Python 2.1.8 (2018-08-31, General Availability)

   Known limitation: MySQLConnection.cmd_change_user() with the C
   extension enabled does not establish a connection with the changed
   user. Instantiate a new MySQLConnection() connection as a workaround.

Functionality Added or Changed


     * MySQL Connector/Python now supports connections to MySQL
       accounts that use the caching_sha2_password authentication plugin
       (see Caching SHA-2 Pluggable Authentication
(http://dev.mysql.com/doc/refman/8.0/en/caching-sha2-pluggable-authentication.html)).
       This requires MySQL server version 8.0.3 or higher. It also
       requires use of a secure connection because Connector/Python does
       not support RSA encryption for password exchange.

Bugs Fixed

     * On Windows, the 32-bit MSI failed to install. The
       registry key path was updated to allow the CEXT
       prerequisite check to execute and pass. (Bug #28395599,
       Bug #28464866)

     * In pure Python mode, connecting to a MySQL server with
       default_authentication_plugin = caching_sha2_password would fail,
       even for accounts not using caching_sha2_password, because the
       authentication plugin was unknown to the Python connector.
       (Bug #27371245)

     * When using prepared statements, string columns were
       returned as bytearrays instead of strings. The returned value is
       now a string decoded using the connection's charset (defaults to
       'utf8'), or as a bytearray if this conversion fails.
       (Bug #27364914)

     * Connecting to a MySQL server configured to use TLS
       versions other than TLSv1, such as as TLSv1.1 and TLSv1.2), was
       not possible and failed with a "[SSL: WRONG_VERSION_NUMBER] wrong
       version number" error. This is because the connector restricted
       the connection to use TLSv1.  In addition, tsl_version support
       was added. (Bug #26484601, Bug #87121)

     * The result from JSON_TYPE() was returned as a bytearray
       instead of a string. The returned value is now a string decoded
       using the connection's charset (defaults to 'utf8'), or as a
       bytearray if this conversion fails. (Bug #24948205, Bug #83516)

     * JSON integer values were cast to bytes in Python instead
       of integers. (Bug #24948186, Bug #83513)

Enjoy and thanks for the support!

On behalf of the MySQL Release Team,
Nawaz Nazeer Ahamed

In his excellent blog post, Pavel Trukhanov showed the value of S.M.A.R.T. metric collections, so I wondered how hard would it be to enable their collection in Percona Monitoring and Management (PMM)

A quick search led me to the  text_collector plugin SmartMon, which can be easily integrated with any Prometheus Installation

For PMM, Vadim Yalovets recently showed how to do custom integrations based on text_collector

Let’s put those together:

1. Ensure you have the smartctl tool installed. It is available in repositories for most Linux distributions
2. Get  smartmon.sh and place it in /usr/local/bin or other location
3. Install the cron job
   

   echo  "*/5 * * * * root bash  /usr/local/bin/smartmon.sh > /tmp/smart_metrics.prom  " > /etc/cron.d/smartmon

4. Enable textfile_collector as described in this blog post

That’s it! You should get your data flowing. Now you can use Prometheus to query device information:

Or if you want to get a specific S.M.A.R.T value, such as media_wearout indicator:

If you would like to see a nicer visualization in Grafana, you can install the appropriate dashboard from the Grafana web site.

The number and kind of metrics you’re going to get depends on the storage device vendor and model. Here is an example list from one of my test systems:

# HELP smartmon_smartctl_version SMART metric smartctl_version
# TYPE smartmon_smartctl_version gauge
smartmon_smartctl_version{version="6.5"} 1
# HELP smartmon_current_pending_sector_raw_value SMART metric current_pending_sector_raw_value
# TYPE smartmon_current_pending_sector_raw_value gauge
smartmon_current_pending_sector_raw_value{disk="/dev/sda",type="sat",smart_id="197"} 0.000000e+00
# HELP smartmon_current_pending_sector_threshold SMART metric current_pending_sector_threshold
# TYPE smartmon_current_pending_sector_threshold gauge
smartmon_current_pending_sector_threshold{disk="/dev/sda",type="sat",smart_id="197"} 0
# HELP smartmon_current_pending_sector_value SMART metric current_pending_sector_value
# TYPE smartmon_current_pending_sector_value gauge
smartmon_current_pending_sector_value{disk="/dev/sda",type="sat",smart_id="197"} 100
# HELP smartmon_current_pending_sector_worst SMART metric current_pending_sector_worst
# TYPE smartmon_current_pending_sector_worst gauge
smartmon_current_pending_sector_worst{disk="/dev/sda",type="sat",smart_id="197"} 100
# HELP smartmon_device_info SMART metric device_info
# TYPE smartmon_device_info gauge
smartmon_device_info{disk="/dev/sda",type="sat",vendor="",product="",revision="",lun_id="",model_family="",device_model="Crucial_CT275MX300SSD1",serial_number="16431465B53F",firmware_version="M0CR031"} 1
# HELP smartmon_device_smart_available SMART metric device_smart_available
# TYPE smartmon_device_smart_available gauge
smartmon_device_smart_available{disk="/dev/sda",type="sat"} 1
# HELP smartmon_device_smart_enabled SMART metric device_smart_enabled
# TYPE smartmon_device_smart_enabled gauge
smartmon_device_smart_enabled{disk="/dev/sda",type="sat"} 1
# HELP smartmon_device_smart_healthy SMART metric device_smart_healthy
# TYPE smartmon_device_smart_healthy gauge
smartmon_device_smart_healthy{disk="/dev/sda",type="sat"} 1
# HELP smartmon_end_to_end_error_raw_value SMART metric end_to_end_error_raw_value
# TYPE smartmon_end_to_end_error_raw_value gauge
smartmon_end_to_end_error_raw_value{disk="/dev/sda",type="sat",smart_id="184"} 0.000000e+00
# HELP smartmon_end_to_end_error_threshold SMART metric end_to_end_error_threshold
# TYPE smartmon_end_to_end_error_threshold gauge
smartmon_end_to_end_error_threshold{disk="/dev/sda",type="sat",smart_id="184"} 0
# HELP smartmon_end_to_end_error_value SMART metric end_to_end_error_value
# TYPE smartmon_end_to_end_error_value gauge
smartmon_end_to_end_error_value{disk="/dev/sda",type="sat",smart_id="184"} 100
# HELP smartmon_end_to_end_error_worst SMART metric end_to_end_error_worst
# TYPE smartmon_end_to_end_error_worst gauge
smartmon_end_to_end_error_worst{disk="/dev/sda",type="sat",smart_id="184"} 100
# HELP smartmon_offline_uncorrectable_raw_value SMART metric offline_uncorrectable_raw_value
# TYPE smartmon_offline_uncorrectable_raw_value gauge
smartmon_offline_uncorrectable_raw_value{disk="/dev/sda",type="sat",smart_id="198"} 0.000000e+00
# HELP smartmon_offline_uncorrectable_threshold SMART metric offline_uncorrectable_threshold
# TYPE smartmon_offline_uncorrectable_threshold gauge
smartmon_offline_uncorrectable_threshold{disk="/dev/sda",type="sat",smart_id="198"} 0
# HELP smartmon_offline_uncorrectable_value SMART metric offline_uncorrectable_value
# TYPE smartmon_offline_uncorrectable_value gauge
smartmon_offline_uncorrectable_value{disk="/dev/sda",type="sat",smart_id="198"} 100
# HELP smartmon_offline_uncorrectable_worst SMART metric offline_uncorrectable_worst
# TYPE smartmon_offline_uncorrectable_worst gauge
smartmon_offline_uncorrectable_worst{disk="/dev/sda",type="sat",smart_id="198"} 100
# HELP smartmon_power_cycle_count_raw_value SMART metric power_cycle_count_raw_value
# TYPE smartmon_power_cycle_count_raw_value gauge
smartmon_power_cycle_count_raw_value{disk="/dev/sda",type="sat",smart_id="12"} 2.000000e+01
# HELP smartmon_power_cycle_count_threshold SMART metric power_cycle_count_threshold
# TYPE smartmon_power_cycle_count_threshold gauge
smartmon_power_cycle_count_threshold{disk="/dev/sda",type="sat",smart_id="12"} 0
# HELP smartmon_power_cycle_count_value SMART metric power_cycle_count_value
# TYPE smartmon_power_cycle_count_value gauge
smartmon_power_cycle_count_value{disk="/dev/sda",type="sat",smart_id="12"} 100
# HELP smartmon_power_cycle_count_worst SMART metric power_cycle_count_worst
# TYPE smartmon_power_cycle_count_worst gauge
smartmon_power_cycle_count_worst{disk="/dev/sda",type="sat",smart_id="12"} 100
# HELP smartmon_power_on_hours_raw_value SMART metric power_on_hours_raw_value
# TYPE smartmon_power_on_hours_raw_value gauge
smartmon_power_on_hours_raw_value{disk="/dev/sda",type="sat",smart_id="9"} 1.313300e+04
# HELP smartmon_power_on_hours_threshold SMART metric power_on_hours_threshold
# TYPE smartmon_power_on_hours_threshold gauge
smartmon_power_on_hours_threshold{disk="/dev/sda",type="sat",smart_id="9"} 0
# HELP smartmon_power_on_hours_value SMART metric power_on_hours_value
# TYPE smartmon_power_on_hours_value gauge
smartmon_power_on_hours_value{disk="/dev/sda",type="sat",smart_id="9"} 100
# HELP smartmon_power_on_hours_worst SMART metric power_on_hours_worst
# TYPE smartmon_power_on_hours_worst gauge
smartmon_power_on_hours_worst{disk="/dev/sda",type="sat",smart_id="9"} 100
# HELP smartmon_raw_read_error_rate_raw_value SMART metric raw_read_error_rate_raw_value
# TYPE smartmon_raw_read_error_rate_raw_value gauge
smartmon_raw_read_error_rate_raw_value{disk="/dev/sda",type="sat",smart_id="1"} 0.000000e+00
# HELP smartmon_raw_read_error_rate_threshold SMART metric raw_read_error_rate_threshold
# TYPE smartmon_raw_read_error_rate_threshold gauge
smartmon_raw_read_error_rate_threshold{disk="/dev/sda",type="sat",smart_id="1"} 0
# HELP smartmon_raw_read_error_rate_value SMART metric raw_read_error_rate_value
# TYPE smartmon_raw_read_error_rate_value gauge
smartmon_raw_read_error_rate_value{disk="/dev/sda",type="sat",smart_id="1"} 100
# HELP smartmon_raw_read_error_rate_worst SMART metric raw_read_error_rate_worst
# TYPE smartmon_raw_read_error_rate_worst gauge
smartmon_raw_read_error_rate_worst{disk="/dev/sda",type="sat",smart_id="1"} 100
# HELP smartmon_reallocated_sector_ct_raw_value SMART metric reallocated_sector_ct_raw_value
# TYPE smartmon_reallocated_sector_ct_raw_value gauge
smartmon_reallocated_sector_ct_raw_value{disk="/dev/sda",type="sat",smart_id="5"} 0.000000e+00
# HELP smartmon_reallocated_sector_ct_threshold SMART metric reallocated_sector_ct_threshold
# TYPE smartmon_reallocated_sector_ct_threshold gauge
smartmon_reallocated_sector_ct_threshold{disk="/dev/sda",type="sat",smart_id="5"} 10
# HELP smartmon_reallocated_sector_ct_value SMART metric reallocated_sector_ct_value
# TYPE smartmon_reallocated_sector_ct_value gauge
smartmon_reallocated_sector_ct_value{disk="/dev/sda",type="sat",smart_id="5"} 100
# HELP smartmon_reallocated_sector_ct_worst SMART metric reallocated_sector_ct_worst
# TYPE smartmon_reallocated_sector_ct_worst gauge
smartmon_reallocated_sector_ct_worst{disk="/dev/sda",type="sat",smart_id="5"} 100
# HELP smartmon_reported_uncorrect_raw_value SMART metric reported_uncorrect_raw_value
# TYPE smartmon_reported_uncorrect_raw_value gauge
smartmon_reported_uncorrect_raw_value{disk="/dev/sda",type="sat",smart_id="187"} 0.000000e+00
# HELP smartmon_reported_uncorrect_threshold SMART metric reported_uncorrect_threshold
# TYPE smartmon_reported_uncorrect_threshold gauge
smartmon_reported_uncorrect_threshold{disk="/dev/sda",type="sat",smart_id="187"} 0
# HELP smartmon_reported_uncorrect_value SMART metric reported_uncorrect_value
# TYPE smartmon_reported_uncorrect_value gauge
smartmon_reported_uncorrect_value{disk="/dev/sda",type="sat",smart_id="187"} 100
# HELP smartmon_reported_uncorrect_worst SMART metric reported_uncorrect_worst
# TYPE smartmon_reported_uncorrect_worst gauge
smartmon_reported_uncorrect_worst{disk="/dev/sda",type="sat",smart_id="187"} 100
# HELP smartmon_smartctl_run SMART metric smartctl_run
# TYPE smartmon_smartctl_run gauge
smartmon_smartctl_run{disk="/dev/sda",type="sat"} 1535666337
# HELP smartmon_temperature_celsius_raw_value SMART metric temperature_celsius_raw_value
# TYPE smartmon_temperature_celsius_raw_value gauge
smartmon_temperature_celsius_raw_value{disk="/dev/sda",type="sat",smart_id="194"} 3.100000e+01
# HELP smartmon_temperature_celsius_threshold SMART metric temperature_celsius_threshold
# TYPE smartmon_temperature_celsius_threshold gauge
smartmon_temperature_celsius_threshold{disk="/dev/sda",type="sat",smart_id="194"} 0
# HELP smartmon_temperature_celsius_value SMART metric temperature_celsius_value
# TYPE smartmon_temperature_celsius_value gauge
smartmon_temperature_celsius_value{disk="/dev/sda",type="sat",smart_id="194"} 69
# HELP smartmon_temperature_celsius_worst SMART metric temperature_celsius_worst
# TYPE smartmon_temperature_celsius_worst gauge
smartmon_temperature_celsius_worst{disk="/dev/sda",type="sat",smart_id="194"} 59
# HELP smartmon_udma_crc_error_count_raw_value SMART metric udma_crc_error_count_raw_value
# TYPE smartmon_udma_crc_error_count_raw_value gauge
smartmon_udma_crc_error_count_raw_value{disk="/dev/sda",type="sat",smart_id="199"} 0.000000e+00
# HELP smartmon_udma_crc_error_count_threshold SMART metric udma_crc_error_count_threshold
# TYPE smartmon_udma_crc_error_count_threshold gauge
smartmon_udma_crc_error_count_threshold{disk="/dev/sda",type="sat",smart_id="199"} 0
# HELP smartmon_udma_crc_error_count_value SMART metric udma_crc_error_count_value
# TYPE smartmon_udma_crc_error_count_value gauge
smartmon_udma_crc_error_count_value{disk="/dev/sda",type="sat",smart_id="199"} 100
# HELP smartmon_udma_crc_error_count_worst SMART metric udma_crc_error_count_worst
# TYPE smartmon_udma_crc_error_count_worst gauge
smartmon_udma_crc_error_count_worst{disk="/dev/sda",type="sat",smart_id="199"} 100

The post Monitoring S.M.A.R.T. Metrics with Prometheus and PMM appeared first on Percona Database Performance Blog.

A month ago, I wrote a blog on how you can configure the MySQL Shell prompt to suit your needs.  One thing I did not go into details with are the prompt templates prompt_256pl.json and prompt_256pl+aw.json. Common for both of these templates is that they require custom fonts to work.

In the two file names, pl stands for the PowerLine fonts and aw for the Awesome font. These fonts add symbols that are useful to create rich prompts, not only for MySQL Shell, but also for Bash, zsh, vim, etc. The symbols include an angle separator, a lock (that in MySQL Shell is used to symbolize an TLS/SSL connection), and so on.

This blog will show you how to install the necessary fonts to get an awesome MySQL Shell prompt on Oracle Linux 7, Red Hat Enterprise Linux (RHEL) 7, and CentOS 7. For other Linux distributions, the steps are the same or similar.

Specifying the Prompt Template

Before diving into installing the necessary fonts, let’s recall from the previous blog, how you choose how the prompt should look. Two ways to specify the prompt are:

• Set the MYSQLSH_PROMPT_THEME environment variable to the path to the file with the required template. For the prompt_256pl+aw.json template using the el7 MySQL Shell RPM, the full path is /usr/share/mysqlsh/prompt/prompt_256pl+aw.json. This method is useful while testing.
• Copy the prompt_256pl+aw.json file to ~/.mysqlsh/prompt.json. This way MySQL Shell will automatically pick up the new prompt definition. This is very useful when you have decided on your preferred prompt.

Awesome Terminal Fonts

If you want to use the prompt_256pl+aw.json template, then you will need to install the awesome font. One option is to use the patching-strategy branch of gabrielelana’s awesome-terminal-fonts project on GitHub. The steps to install the required font for the user are:

shell$ git clone https://github.com/gabrielelana/awesome-terminal-fonts.git
Cloning into 'awesome-terminal-fonts'...
remote: Counting objects: 329, done.
remote: Total 329 (delta 0), reused 0 (delta 0), pack-reused 329
Receiving objects: 100% (329/329), 2.77 MiB | 1.12 MiB/s, done.
Resolving deltas: 100% (186/186), done.

shell$ cd awesome-terminal-fonts

shell$ git checkout patching-strategy
Branch patching-strategy set up to track remote branch patching-strategy from origin.
Switched to a new branch 'patching-strategy'

shell$ mkdir -p ~/.local/share/fonts/

shell$ cp patched/SourceCodePro+Powerline+Awesome+Regular.* ~/.local/share/fonts/

shell$ fc-cache -fv ~/.local/share/fonts/
/home/myuser/.local/share/fonts: caching, new cache contents: 1 fonts, 0 dirs
/home/myuser/.local/share/fonts: caching, new cache contents: 1 fonts, 0 dirs
/var/cache/fontconfig: not cleaning unwritable cache directory
/home/myuser/.cache/fontconfig: cleaning cache directory
/home/myuser/.fontconfig: not cleaning non-existent cache directory
fc-cache: succeeded

This installs just the font required for MySQL Shell. You can however choose to install all of the fonts if you like.

It is necessary to restart the desktop environment for the fonts to be available. The resulting prompt (when logged in a default schema has been set) looks like:

Personally I am not entirely keen on how the separator looks, so I install another font to replace the separator.

Powerline Font

The Powerline fonts serve a similar purpose as the awesome font, but they are missing some of the symbols used by the prompt_256pl+aw.json template. It does however include the separator used by the template, and I personally prefer it over the Awesome Terminal Font separator. If you choose the prompt_256pl.json template, you can just install the Powerline Fonts and skip the Awesome Terminal Font.

The installation instructions in the manual are quite good. However, they assume you want to use the fonts for more than just MySQL Shell. For MySQL Shell the following steps are enough:

shell$ wget --directory-prefix="${HOME}/.local/share/fonts" \
            https://github.com/powerline/powerline/raw/develop/font/PowerlineSymbols.otf
...
2018-09-03 20:40:31 (6.51 MB/s) - ‘/home/myuser/.local/share/fonts/PowerlineSymbols.otf’ saved [2264/2264]

shell$ fc-cache -vf ~/.local/share/fonts/
/home/myuser/.local/share/fonts: caching, new cache contents: 2 fonts, 0 dirs                                                                                                                                                                
/home/myuser/.local/share/fonts: caching, new cache contents: 2 fonts, 0 dirs                                                                                                                                                                
/var/cache/fontconfig: not cleaning unwritable cache directory                                                                                                                                                                               
/home/myuser/.cache/fontconfig: cleaning cache directory                                                                                                                                                                                     
/home/myuser/.fontconfig: not cleaning non-existent cache directory                                                                                                                                                                          
fc-cache: succeeded

shell$ wget --directory-prefix="${HOME}/.config/fontconfig/conf.d" \
             https://github.com/powerline/powerline/raw/develop/font/10-powerline-symbols.conf
...
2018-09-03 20:41:45 (5.87 MB/s) - ‘/home/myuser/.config/fontconfig/conf.d/10-powerline-symbols.conf’ saved [2713/2713]

After restarting the desktop environment, the prompt looks like this:

You can of course also choose to play with the other symbols in the Powerline and Awesome Terminal fonts and make your own custom MySQL Shell prompt configuration file. Whichever is your preferred prompt, have fun playing with it.

Please join Percona’s Chief Technology Officer, Vadim Tkachenko, as he presents Running MySQL in Kubernetes on Thursday, September 6th, 2018, at 11:00 AM PDT (UTC-7) / 2:00 PM EDT (UTC-4).

Without question, Kubernetes is the most popular container orchestration platform. But can it handle databases? I think so, and in this webinar, I will show you how it does it. This talk presents a quick overview of the persistent storage options (the most critical part of data storage) and then what it takes to run highly available MySQL deployments with backup and recovery options.

By the end of the webinar, you will understand if Kubertenes is suitable for handling your databases. Moreover, you’ll understand how to approach using it if it is.

Register for this webinar on running MySQL in Kubernetes.

The post Upcoming Webinar Thurs 9/6: Running MySQL in Kubernetes appeared first on Percona Database Performance Blog.

Do you need to modify the metrics collected from Linux by Percona Monitoring and Management (PMM)? In this blog post we will see how to enable, disable, and update collected metrics on PMM’s linux:metrics exporter. 

We will assume that the PMM client packages are installed, and they are configured already.

Using a custom list of metrics

Let’s now suppose we are not yet collecting any metrics on our desired client server, and we want to enable only the following: diskstats, meminfo, netdev and vmstat. We can use the following command:

pmm-admin add linux:metrics -- -collectors.enabled=diskstats,meminfo,netdev,vmstat

So, in order to enable or disable the functionality we want, we need to modify the collectors.enabled list accordingly. In the following online documentation page, we are able to find all collectors supported:

https://www.percona.com/doc/percona-monitoring-and-management/section.exporter.node.html

In this way, by adding or removing items from the collectors.enabled list, we can choose which functionality will be set in our PMM linux:metrics collectors.

Checking list of metrics used

We can use the ps aux command to check the collectors list that applies at any given time. Let’s see this in practical terms. After running the previous command, we should expect to see the following:

shell> ps aux | grep node_exporter
...
root     20450 3.2  0.0 1660248 15924 ?       Sl 13:48 0:13 /usr/local/percona/pmm-client/node_exporter -collectors.enabled=diskstats,filefd,filesystem,loadavg,meminfo,netdev,netstat,stat,time,uname,vmstat,meminfo_numa -web.listen-address=10.10.8.141:42000 -web.auth-file=/usr/local/percona/pmm-client/pmm.yml -web.ssl-cert-file=/usr/local/percona/pmm-client/server.crt -web.ssl-key-file=/usr/local/percona/pmm-client/server.key -collectors.enabled=diskstats,meminfo,netdev,vmstat

Note: there is currently a bug where you will see duplicate entries when using custom collectors. We are tracking this under issue PMM-2857. For now, if you see two different lists on ps aux outputs, the one that applies is the last one (as seen above).

Updating the list of metrics

There is currently no way to enable (or disable) metrics dynamically (however, we are working on it). We will need to stop the collector, manually get and edit the metrics list, and re-add it with the updated collectors in place. For this, we will need to know the current list of metrics used from the ps aux outputs.

Continuing with the same example, suppose we are not happy with what netdev metrics offer, and we wish to disable them to avoid unnecessary overhead. Knowing the metrics list was:

-collectors.enabled=diskstats,meminfo,netdev,vmstat

We will need to run the following commands to have it removed from the collectors list:

pmm-admin stop linux:metrics
pmm-admin rm linux:metrics
pmm-admin add linux:metrics -- -collectors.enabled=diskstats,meminfo,vmstat

Links for reference

• Passing options to exporter: here and here
• Collector options for linux:metrics exporter: here

Did you find this post useful?

You might also enjoy some of our other resources. Check out some of these recent technical webinars on using PMM, presented by my colleagues from Percona:

• Migrating to AWS Aurora, Monitoring AWS Aurora with PMM a joint webinar with the team from Autodesk
• MySQL Troubleshooting and Performance Optimization with PMM presented by our CEO, Peter Zaitsev

The post Modifying List of Collected Metrics on PMM Linux Exporter appeared first on Percona Database Performance Blog.

Codership is pleased to announce the release of Galera Replication library 3.24, implementing wsrep API version 25.  The new release includes improved deadlock error management with foreign keys and security fixes. As always, Galera Cluster is now available as targeted packages and package repositories for a number of Linux distributions, including Ubuntu, Red Hat, Debian, CentOS, OpenSUSE and SLES, as well as FreeBSD. Obtaining packages using a package repository removes the need to download individual files and facilitates the easy deployment and upgrade of Galera Cluster nodes.

This release incorporates all changes up to MySQL 5.7.23, MySQL 5.6.41 and MySQL 5.5.61.

Galera Replication Library 3.24
New features and notable fixes in Galera replication since last binary release by Codership (3.23)

* A support for new certification key type was added to allow
more relaxed certification rules for foreign key references (galera#491).  Previous releases caused excessive (phantom) deadlock errors for transactions, which inserted rows into a table, which contained foreign key constraint. This anomaly has been optimized, and only true conflicts will result in deadlock error.

* New status variables were added to display the number of open transactions
and referenced client connections inside Galera provider (galera#492).

* GCache was sometimes cleared unnecessarily on startup if the recovered
state had smaller sequence number than the highest found from GCache.
Now only entries with sequence number higher than recovery point will be
cleared (galera#498).

* Non-primary configuration is saved into grastate.dat only when if the
node is in closing state (galera#499).

* Exception from GComm was not always handled properly resulting in
Galera to remain in half closed state. This was fixed by propagating the
error condition appropriately to upper layers (galera#500).

* A new status variable displaying the total weight of the cluster nodes
was added (galera#501).

* The value of pc.weight did not reflect the actual effective value after
setting it via wsrep_provider_options. This was fixed by making sure that
the new value is taken into use before returning the control back to
caller (galera#505, MDEV-11959)

* Use of ECHD algorithms with old OpenSSL versions was enabled (galera#511).

* Default port value is now used by garbd if the port is not explicitly
given in cluster address (MDEV-15531).

* Correct error handling for posix_fallocate().

* Failed causal reads are retried during configuration changes.

MySQL 5.7
New release of Galera Cluster for MySQL 5.7, consisting of MySQL-wsrep 5.7.23 and wsrep API version 25. Notable bug fixes in MySQL 5.7.23 and known issues:

* New configuration option wsrep_certification_rules to
enable more relaxed certification rules for foreign key references
on child table inserts. This option is effective only with Galera
version 3.24 or higher (galera#491).

* Resource leak in case of ROLLBACK TO SAVEPOINT followed
by COMMIT has been fixed (mysql-wsrep#318).

* FK constraint violation in applier after ALTER TABLE ADD FK was fixed
by adding both parent and child table keys into ALTER TOI write set
(mysql-wsrep#332).

* Possible node hang with conflicting inserts in FK child table was
fixed (mysql-wsrep#335).

* Memory leak with native MySQL replication when InnoDB was used
as a relay log info and master info repository has been fixed
(mysql-wsrep#329).

Known issues with this release:

* Server cannot be started using ‘service’ command on Debian Stretch.

* SST between 5.6 and 5.7 nodes is not supported

* The –wsrep-replication-bundle option has no effect and may be removed in
a future release

* InnoDB tablespaces outside of the data directory are not supported, as they
may not be copied over during SST

* Compilation with DTrace enabled may fail, so -DENABLE_DTRACE=:BOOL=OFF
may be given to cmake to disable DTrace

MySQL 5.6
New release of Galera Cluster for MySQL consisting of MySQL-wsrep 5.6.41 and wsrep API version 25. Notable bug fixes and known issues in MySQL 5.6.41:

* New configuration option wsrep_certification_rules to
enable more relaxed certification rules for foreign key references
on child table inserts. This option is effective only with Galera
version 3.24 or higher (galera#491).

* Fixed a resource leak in case of ROLLBACK TO SAVEPOINT which was followed
by COMMIT (mysql-wsrep#318).

* InnoDB undo tablespaces are now included in rsync SST (mysql-wsrep#337).

* FK constraint violation in applier after ALTER TABLE ADD FK was fixed
by adding both parent and child table keys into ALTER TOI write set
(mysql-wsrep#332).

* Memory leak with native MySQL replication when InnoDB was used
as a relay log info and master info repository has been fixed.
(mysql-wsrep#329).

Known issues with this release:

* If using the Ubuntu 16.04 Xenial package, the server can not be bootstrapped
using systemd. Please use the SysV init script with the ‘bootstap’ option to
bootstrap the node. Note that a server that has been started that way can not
be controlled via systemd and must be stopped using the SysV script. Normal
server startup and shutdown is possible via systemd.

* Server cannot be started using ‘service’ command on Debian Stretch.

MySQL 5.5
New release of Galera Cluster for MySQL consisting of MySQL-wsrep 5.5.61 and wsrep API version 25.This release incorporates all changes up to MySQL 5.5.61

* New configuration option wsrep_certification_rules to
enable more relaxed certification rules for foreign key references
on child table inserts. This option is effective only with Galera
version 3.24 or higher (galera#491).

* Resource leak in case of ROLLBACK TO SAVEPOINT followed
by COMMIT has been fixed (mysql-wsrep#318).

* FK constraint violation in applier after ALTER TABLE ADD FK was fixed
by adding both parent and child table keys into ALTER TOI write set
(mysql-wsrep#332).

Reminder: Changes to Repositories Structure
===========================================

With the new release the repository structure is changed
to allow for existence of all of the wsrep-patched mysql
versions currently supported: 5.5 through 5.7.
Thus the repository layout requires from the user to
adjust his or her repository configuration to accomodate
those changes. In order to have the WSREP and Galera
library installed, one would need to add the following
repositories:
1. Galera-3 repository for galera library:
e.g. http://releases.galeracluster.com/galera-3/<ldist>/
2. Corresponding mysql-wsrep repository:
e.g. http://releases.galeracluster.com/mysql-wsrep-<mversion>/<ldist>
here: *ldist* is Linux or BSD distribution (Ubuntu, Centos) and *mversion* is MySQL version, i.e.
5.5, 5.6, 5.7

How To Install
————–

Repositories contain dummy or meta packages, called mysql-wsrep-<mversion>
which are convenience packages for installation of the corresponding version
of WSREP. One can install the whole suite by running, for example:
`apt-get install mysql-wsrep-5.6 galera-3`

#### Quirks for Ubuntu Xenial and 5.6
Due to the peculiarities of how apt resolves packages and presence of 5.7
libraries in Xenial repositories the command above might require additional
steps/changes in order to succeed.

One would need to either configure apt pinning for codership repositories for
them to have priority over upstream packages or to specify mysql-common package
version explicitly as the one located in the WSREP repositories in order to get
things installed.

1. block cache size
2. number of background threads
3. compaction priority
4. dynamic leveled compaction
5. bloom filters

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and monitoring MySQL® and MongoDB® performance. You can run PMM in your own environment for maximum security and reliability. It provides thorough time-based analysis for MySQL® and MongoDB® servers to ensure that your data works as efficiently as possible.



We’ve included a plethora of visual improvements in this release, including:

• PostgreSQL Metrics Collection – Visualize PostgreSQL performance!
• Identify New Queries in Query Analytics
• New Dashboard: Compare System Parameters
• New Dashboard: PERFORMANCE_SCHEMA Wait Events Analysis
• Dashboard Updates – Advanced Data Exploration, MyRocks, TokuDB, InnoDB Metrics
• Disable SSL between Prometheus and Exporters
• Dashboards grouped by Folder – We’ve organized the Dashboard drop-down to present a cleaner interface

We addressed 16 new features and improvements, and fixed 20 bugs.

pmm-admin add mysql --disable-ssl ...

The post Percona Monitoring and Management (PMM) 1.14.0 Is Now Available appeared first on Percona Database Performance Blog.

CSV

Second Pass

Create Our Table

The Easy Way To Go From CSV to Relational Table

Tell the import wizard where the CSV is located

Start the table import wizard

The CSV is split up by fields and we could edit field types

And the data is loaded

The following statement usually is valid and the function returns 1. But sometimes it is invalid and sometimes the function returns 0.

CREATE FUNCTION f() RETURNS INT DETERMINISTIC
BEGIN
  DECLARE a CHAR DEFAULT 'a';
  IF a = 0 || a > 0 THEN RETURN 1; END IF;
  RETURN 0;
END;

Why?

First, consider that "||" is usually the same as "OR" because that's the default. But if sql_mode is 'ansi' and the DBMS is MySQL 8.0, then "||" is the operator for concatenating strings. So the meaning of the IF condition changes, and it becomes false.

Second, consider that the function is written with SQL/PSM syntax. But if sql_mode is 'oracle' and the DBMS is MariaDB 10.3, then the function has to be written with PL/SQL syntax. And the requirements differ as soon as the word "RETURNS" comes along, so the result is a syntax error.

Our lesson is: you can't know a statement's meaning if you don't know whether somebody said "SET sql_mode=..." earlier.

Usually SET sql_mode is a DIRECTIVE with the same sense as "pragma" in C, it is telling the compiler or interpreter how to treat the syntax of following statements. And in fact Oracle and SQLite actually use the word "pragma" in that sense. I use C pragmas despite a nagging feeling that somewhere there is a jeremiad blog post about how they're as bad as goto and #ifndef, and I'll recommended certain sql_mode values enthusiastically. The only strong warning is: be consistent, so you don't end up with idiocy like my example above.

(By the way, speaking of C, one of the possible settings is "treat warnings as errors", and if there were such a mode in MySQL/MariaDB then my example function wouldn't have worked. But there isn't an exact equivalent.)

The modes

The manuals' lists are more complete, but this list has more advice.

ANSI: Avoid. It won't deliver "American National Standards Institute" SQL. And try this:

SET @@sql_mode='';
CREATE TABLE x (s1 INT);
SHOW CREATE TABLE x;
SET @@sql_mode='ANSI';
SHOW CREATE TABLE x;

Compare the SHOW CREATE TABLE results. Before:


and after:


Notice that critical information is missing in the 'ANSI' result -- information that would be necessary to reproduce the table correctly. So, although using standard SQL is good, the way to do it is by setting other SQL-mode settings that don't have such ugly side effects.

TRADITIONAL: Don't avoid. This has very limited effect, it only "treats warnings as errors" if you're changing a database value, and it's only safe if you're using a storage engine that can do statement rollbacks, such as InnoDB. But most of the time it's good to disallow bad data, which is why it's "traditional" (which stands for "what everybody else has done for decades").

ALLOW_INVALID_DATES, Avoid. This made sense when the objective was to avoid "invalid data" errors because transactions were hard to roll back. That's less of a concern nowadays.

ANSI_QUOTES. Probably don't avoid. If you frequently have single quotes inside string literals, such as "O'Hara", it's understandable that you'd want ANSI_QUOTES to be off. With ANSI_QUOTES on, you have to say 'O''Hara'' and it's possible to lose the duplicated '. But ANSI_QUOTES are standard SQL.

HIGH_NOT_PRECEDENCE, NO_BACKSLASH_ESCAPES, NO_ZERO_DATE, NO_ZERO_IN_DATE, ONLY_FULL_GROUP_BY, PIPES_AS_CONCAT, REAL_AS_FLOAT. Don't avoid. These are standard SQL.

PAD_CHAR_TO_FULL_LENGTH, ERROR_FOR_DIVISION_BY_ZERO. Avoid in MySQL. They're deprecated.

IGNORE_SPACE. Avoid. The effect on naming is too great, because so many new reserved words appear.

NO_AUTO_VALUE_ON_ZERO. Avoid. This is a case where a non-standard feature can be treated in more than one way, so it's hard to care.

SIMULTANEOUS_ASSIGNMENT. See my earlier blog post, The simultaneous_assignment mode in MariaDB 10.3.5.

STRICT_ALL_TABLES + STRICT_TRANS_TABLES. Don't avoid. Perhaps you have some way of cleaning up messes after they're added, but setting is simpler.

TIME_TRUNCATE_FRACTIONAL. Avoid but not forever. I hear from reliable sources that there is some strange behaviour that will be corrected real soon.

DB2,MAXDB,MSSQL,MYSQL323,MYSQL40,ORACLE,POSTGRESQL, etc. Mostly avoid. These have disappeared in MySQL 8.0. They never delivered a serious amount of compatibility with other DBMSs. The only one that is worth any consideration is ORACLE in MariaDB 10.3, because it affects quite a few Oracle-related matters including PL/SQL support.

The statement

There are many variants.

SET sql_mode=value;
SET @sql_mode=value;
SET @@sql_mode=value;
SET SESSION | GLOBAL | PERSIST | PERSIST_ONLY sql_mode=value;
SET @@session.|@@global.|@@persist.|@@persist_only.sql_mode=value;

where value can be a string literal containing a mode name
or a series of mode names as in 'ansi,ansi', or a variable, or even
a keyword. When it's a keyword, it can be DEFAULT or
it can be a mode name -- this seems to be undocumented but
I've seen that one of the MariaDB developers likes to use

SET sql_mode=ORACLE;

The good news is that the setting is transient and local. By "transient" I mean its effect ends when the routine ends or the session ends. By "local" I mean changes won't affect other users whose sessions already started. And both these non-effects are good. Wouldn't it be awful if your SQL statements stopped working because you invoked a function that changed sql_mode, or because some other user on the system found a way to change it for everybody while they were online?

ocelotgui 1.0.7

Version 1.0.7 of our open-source ocelotgui MySQL/MariaDB client is out, and one of the features is that it recognizes all the current syntax of MySQL 8.0 and MariaDB 10.3, including the sql_mode bizarreness (though we can't get it right if the source value is a variable). That means that it won't get confused when parsing batches of SQL statements that include statements that change the dialect.

The major feature is that the debugger can now debug routines written in MySQL 8.0, and routines written in MariaDB 10.3 with sql_mode=oracle -- that is, with PL/SQL syntax.

As usual, download for various Linux distros and for Windows is via github.