After many months of heavy development and testing, Debian 8 (Jesse) and Ubuntu 15.04 (Vivid Vervet) were released recently.
For those who already upgraded their systems and/or plan do do it in close future, we prepared builds of WebScaleSQL :-)
You can also install it using PSCE repository.
In previous article, we have covered some errors and issues with using MySQL in “disk full” environment. Where there was no space left on device.(See here: Testing Disk Full Conditions)
Today’s scenario is -> Starting MySQL with GTID/binary log enabled, in 0 space left Linux(CentOS 6.5) environment.
If you hit a bug or problem, general rule for helping community to fix it is to provide as much information as possible. Especially useful is to give gdb output from coredump. To get coredump you can read this wonderful article Hunting-The-Core
Now let’s explore our situation. Because our segfault is detected while starting MySQL, it is not possible to attach PID to GDB and also using strace.
Our my.cnf file:
[mysqld] log_bin = /opt/mysql/datadir/mysql-bin log_bin_index = /opt/mysql/datadir/mysql-bin sync_binlog = 1 binlog_format = row gtid-mode = on log_slave_updates = 1 enforce-gtid-consistency = true sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES core-file [mysqld_safe] core_file_size=unlimited
Starting MySQL:
[root@localhost ~]# /opt/mysql/bin/mysqld_safe --defaults-file=/opt/mysql/my.cnf --user=mysql --datadir=/opt/mysql/datadir --socket=/opt/mysql/datadir/mysqld-new.sock --pid-file=/home/error_log_dir/mysqld-new.pid --port=3307 --log-error=/home/error_log_dir/error.err & [1] 2849 [root@localhost ~]# 150427 06:31:42 mysqld_safe Logging to '/home/error_log_dir/error.err'. 150427 06:31:42 mysqld_safe Starting mysqld daemon with databases from /opt/mysql/datadir /opt/mysql/bin/mysqld_safe: line 166: 3110 Segmentation fault (core dumped) nohup /opt/mysql/bin/mysqld --defaults-file=/opt/mysql/my.cnf --basedir=/opt/mysql --datadir=/opt/mysql/datadir --plugin-dir=/opt/mysql/lib/plugin --user=mysql --log-error=/home/error_log_dir/error.err --pid-file=/home/error_log_dir/mysqld-new.pid --socket=/opt/mysql/datadir/mysqld-new.sock --port=3307 < /dev/null >> /home/error_log_dir/error.err 2>&1 150427 06:32:07 mysqld_safe mysqld from pid file /home/error_log_dir/mysqld-new.pid ended
3110 Segmentation fault (core dumped) -> We have core dump.
Using gdb with coredump:
[root@localhost coredumps]# gdb /opt/mysql/bin/mysqld core.3110 BFD: Warning: /home/coredumps/core.3110 is truncated: expected core file size >= 803807232, found: 697573376. [New Thread 3110] [New Thread 3111] Cannot access memory at address 0x7f5ec00ff168 Cannot access memory at address 0x7f5ec00ff168 Cannot access memory at address 0x7f5ec00ff168 Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib64/ld-2.12.so.debug...done. done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Failed to read a valid object file image from memory. Core was generated by `/opt/mysql/bin/mysqld --defaults-file=/opt/mysql/my.cnf --basedir=/opt/mysql --'. Program terminated with signal 11, Segmentation fault. #0 0x00007f5ebfccd8ac in ?? () (gdb) bt #0 0x00007f5ebfccd8ac in ?? () Cannot access memory at address 0x7fffb0aa89b8 (gdb) bt full #0 0x00007f5ebfccd8ac in ?? () No symbol table info available. Cannot access memory at address 0x7fffb0aa89b8
Such information is not sufficient further analysis, that’s why you should try to run MySQL start command with-in GDB as follows:
[root@localhost error_log_dir]# gdb /opt/mysql/bin/mysqld
(gdb) run --defaults-file=/home/error_log_dir/my.cnf --basedir=/opt/mysql --datadir=/opt/mysql/datadir --plugin-dir=/opt/mysql/lib/plugin --user=mysql --log-error=/home/error_log_dir/error.err --pid-file=/home/error_log_dir/mysqld-new.pid --socket=/opt/mysql/datadir/mysqld-new.sock --port=3307
.
.
Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
.
.
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x0000000000ae6242 in my_printf_warning (format=0x1026f08 "Disk is full writing '%s' (Errcode: %d - %s). Waiting for someone to free space...")
at /root/mysql-5.6.24/mysys/my_error.c:260
#2 0x0000000000ac9052 in wait_for_free_space (filename=0x1fe7ee0 "/opt/mysql/datadir/mysql-bin.~rec~", errors=0) at /root/mysql-5.6.24/mysys/errors.c:115
#3 0x0000000000af1ac7 in my_write (Filedes=19, Buffer=0x1964d00 "/opt/mysql/datadir/mysql-bin.000003\n", Count=36, MyFlags=52)
at /root/mysql-5.6.24/mysys/my_write.c:89
#4 0x0000000000acd5ae in inline_mysql_file_write (src_file=0x1027708 "/root/mysql-5.6.24/mysys/mf_iocache.c", src_line=1788, file=19,
buffer=0x1964d00 "/opt/mysql/datadir/mysql-bin.000003\n", count=36, flags=52) at /root/mysql-5.6.24/include/mysql/psi/mysql_file.h:1141
#5 0x0000000000ad078c in my_b_flush_io_cache (info=0x183c1a8, need_append_buffer_lock=0) at /root/mysql-5.6.24/mysys/mf_iocache.c:1787
#6 0x0000000000a7132b in MYSQL_BIN_LOG::sync_purge_index_file (this=0x183b400) at /root/mysql-5.6.24/sql/binlog.cc:4420
#7 0x0000000000a6e206 in MYSQL_BIN_LOG::open_binlog (this=0x183b400, log_name=0x190d130 "/opt/mysql/datadir/mysql-bin", new_name=0x0,
io_cache_type_arg=WRITE_CACHE, max_size_arg=1073741824, null_created_arg=false, need_lock_index=true, need_sid_lock=true, extra_description_event=0x0)
at /root/mysql-5.6.24/sql/binlog.cc:3146
#8 0x000000000063ad8e in init_server_components () at /root/mysql-5.6.24/sql/mysqld.cc:5012
#9 0x000000000063b6e7 in mysqld_main (argc=19, argv=0x186de68) at /root/mysql-5.6.24/sql/mysqld.cc:5455
#10 0x000000000062fc74 in main (argc=10, argv=0x7fffffffe3d8) at /root/mysql-5.6.24/sql/main.cc:25
######################
(gdb) bt full
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#1 0x0000000000ae6242 in my_printf_warning (format=0x1026f08 "Disk is full writing '%s' (Errcode: %d - %s). Waiting for someone to free space...")
at /root/mysql-5.6.24/mysys/my_error.c:260
args = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7fffffffd8a0, reg_save_area = 0x7fffffffd7e0}}
wbuff = "Disk is full writing '/opt/mysql/datadir/mysql-bin.~rec~' (Errcode: 28 - No space left on device). Waiting for someone to free space...\000\330\327\377\377\377\177\000\000\210\341\377\367\377\177\000\000\340\267\376\367\377\177\000\000\271\332\024u\000\000\000\000Zn\336\367\377\177\000\000\000\000\000\000\000\000\000\000/\035!\366\377\177\000\000C\000\000\000\000\000\000\000f\223!\366\377\177\000\000\001\000\000\000\000\000\000\000P\031!\366\377\177\000\000LC_"...
_db_stack_frame_ = {func = 0x102c53c "my_write", file = 0x102c518 "/root/mysql-5.6.24/mysys/my_write.c", level = 2147483654, prev = 0x7fffffffd9f0}
#2 0x0000000000ac9052 in wait_for_free_space (filename=0x1fe7ee0 "/opt/mysql/datadir/mysql-bin.~rec~", errors=0) at /root/mysql-5.6.24/mysys/errors.c:115
errbuf = "No space left on device\000\325\024\257\000\035\000\000\000\230\324\002\001\000\000\000\000\272\324\002\001\000\000\000\000\000\331\377\377\034\000\000\000 ܆\001\000\000\000\000\034\000\000\000\006\000\000\000 ܆\001\000\000\000\000@\331\377\377\377\177\000\000[\346\257\000\000\000\000\000 ܆\001\000\000\000\000\000܆\001\023\000\000\000<\305\002\001\000\000\000\000\030\305\002\001\000\000\000"
#3 0x0000000000af1ac7 in my_write (Filedes=19, Buffer=0x1964d00 "/opt/mysql/datadir/mysql-bin.000003\n", Count=36, MyFlags=52)
at /root/mysql-5.6.24/mysys/my_write.c:89
writtenbytes = 18446744073709551615
sum_written = 0
errors = 0
initial_count = 36
_db_stack_frame_ = {func = 0x1027c7b "my_b_flush_io_cache", file = 0x1027708 "/root/mysql-5.6.24/mysys/mf_iocache.c", level = 2147483653,
prev = 0x7fffffffdb10}
From know developers should be able to see exact code line, for further examination.
For further reading(to see “bt full” full output) refer related BUG report: #76852
It is already verified.
The post Using GDB, investigating segmentation fault in MySQL appeared first on Azerbaijan MySQL UG.
So, on a large IBM POWER8 system I was recently running the newly coined “yesmark” benchmark, which is best translated as this:
Benchmark (N for concurrency): for i in {1..N}; do yes "DO 0;" | mysql > /dev/null & done
Live results: mysqladmin -ri 1 extended-status | grep Questions
Which sounds all fun until you realize that it’s *amazingly* close in results to a sysbench point select benchmark these days (well, with MySQL 5.7.7).
Since yesmark doesn’t use InnoDB though, MariaDB is back in the game.
I don’t think it matters between MariaDB and MySQL at this point for yesbench. With MySQL in a KVM guest on a shared 2 socket POWER8 I could get 754kQPS and on a larger system, I could get 1.3 million / sec.
1.3 Million queries / sec is probably the highest number anybody has ever seen out of MySQL or MariaDB, so that’s fairly impressive in itself.
What’s also impressive is that on this workload, mysqld was still only using 50% of CPU in the system. The mysql command line client was really heavy user.
Other users are: 8% completely idle, another 12% in linux scheduler (alarmingly high really). So out of all execution time, only about 44% spent in mysqld, 29% in mysql client.
It seems that the current issues scaling to two socked POWER8 machines are the same as with scaling to other large systems, when we go beyond about 20 POWER8 cores (SMT8), we start to find new and interesting challenges.
At VividCortex, the algorithm our agents use to digest queries needs to be fast and reliable. Accurately digesting queries into logical groups is important to providing a useful view of data in the UI - digest too few or too many and it becomes difficult to understand what the Top-Queries view is showing.
Though our algorithm is sophisticated, one of our PostgreSQL clients experienced an issue with query digestion. They had many queries of the form “select a,b,c from table where c = any(values (1), (2), (3)…)” which were not being collapsed correctly.
We recently improved our digest code to correctly digest queries of this form.
In both MySQL and PostgreSQL, “SELECT * FROM t WHERE a IN (SELECT aa FROM tt)” was incorrectly collapsed to SELECT * FROM t WHERE a IN (?). Now, the digest is the lowercased original query text, as there is nothing to collapse or conceal.
These are improvements that help our clients have clearer visibility into their database activity. We are always listening to customer requests, and our support brainiacs are constantly working to improve visibility and service. If you have not given us a try, sign up for a free trial today.
The mantra of the day is – performance, performance, performance! It is clear that everything needs to be totally available all of the time, function at the fastest speeds and performance as close to perfect as is possible. In all the business discussions swirling around us we hear the words; application performance, website performance, transaction performance, server performance and on and on in regards to virtually everything digital that we live and breath with. Performance is no longer just one of the items we need to be paying attention to, it is THE item we need to be critically aware of every minute of every day! No longer are the words “it’s good enough” acceptable. The impacts to business can be catastrophic if the digital performance of your business is not at the highest levels. So the question we are all challenged with is how can we be sure that the performance levels we need are actually what we are achieving? The answer is monitoring! The only way to measure this is to monitor everything and to monitor it from end to end constantly. It is the key that unlocks the door to success.
To be able to determine the real level of performance (and then improve and maintain it) we need to follow the principle of the 3 M’s – measure, monitor and manage. Let’s deal with some specifics that are definitive musts in regards to performance monitoring. To get where you need to be you will need to be monitoring across your IT world to include the following monitors: website, server, application, network, cloud, email, RUM and the custom monitors specific to your business and architecture.
Website – This needs to include the obvious of uptime for sure but that is just the tip of the iceberg. You will also need monitors to be checking the load times of your pages, to test the speed and completeness of your sites transaction and a load tester that can stress your site for you and let you see how much traffic you can really handle. It is a known fact that if your site slows down for any reason the impacts are fast and negative. A website (or transaction, that is slower than 3 seconds can cost you a very real loss of 20% of your customers and revenues.
Server – Your server monitoring needs to be able to capture the performance here of: CPUs, memory availability, storage and disk, network bandwidth, processes and services availability and speed, SNMP devices and it must address multiple TCP protocols, such as - UDP, SIP, IMAP, POP, SMTP, etc. It also should have agents for both Windows and Linux. With this in place you will be well positioned to see in real time the performance of your server and devices, will be able to isolate and clear troubles quicker and will be able to clearly anticipate additional server needs as your network demands increase.
Application - If your web application is up and running, everything must be fine, right? Not necessarily! Applications always come with the risk that while they may be running they are not running at optimal levels. At a minimum you will need application monitors for Email Round Trip, MySQL, Log, Oracle, Tomcat and Java/JMX. You will also want your monitoring system to have an open and easy to use API with the following:
Network – Your network monitoring should provide agent based and agentless monitoring for a wide range of network devices, including switches, phone systems, and Windows®, Linux® servers. It should also allow you to monitor such items as firewalls, switches, printers and VoIP devices.
Cloud – Day by day more and more of your world moves to the cloud so you had better be prepared to monitor this or it may quickly get out of control and you will have no idea. To not lose control make sure your cloud monitoring has features such as:
RUM – User experience is an area that has gained significant importance of late and it should. If your users aren’t having a fast and efficient experience you will lose them for good. RUM today must have the ability be an advanced end user monitoring system that collects valuable data about your users’ interactions with your website. It then presents it in a concise view on your dashboard so that you instantly see your site’s strong points, weak points and a wealth of other valuable information. Have the best site you can based on real information and significantly improve your revenues. Minimum features that is must have include:
This high level guide gives you an idea of the areas and disciplines that must be incorporated within your monitoring system. But there are additional key points that you should be keenly aware of, such as – how frequently does it run checks, how many locations does it test from around the globe, is it cloud based, do they offer a truly single view dashboard to make your tasks easier,can you get real time reports, does it offer 24/7 technical support and can it be customized to your exact business and IT needs? These are all just as critical as the items that preceded them and should be a must on your monitoring system shopping list.
To see exactly what Monitis can do for you and how we can help you monitor and make sure your performance levels are where you want them, visit our website and why not try it for yourself with a free 15 day trial.
I often talk with people who are very interested in the features of Percona XtraDB Cluster (PXC) such as synchronous and parallel replication, multi-node writing and high availability. However some get confused when operating a real PXC cluster because they do not fully realize the implications of these features. So here is a fun way to test your PXC knowledge: try to solve these 12 questions related to PXC! (you will find the answers at the end of the post.)
1. With Galera 3.x, support for MyISAM is experimental. When can we expect to have full MyISAM support?
a. This will never happen as Galera is designed for transactional storage engines.
b. This is planned for Galera 4.0.
2. Why aren’t all workloads a good fit for PXC?
a. Execution plans can change compared to a regular MySQL server, so performance is sometimes not as good as with a regular MySQL server.
b. Large transactions and write hotspots can create performance issues with Galera.
3. For workloads with a write hot spot, writing on all nodes to distribute the load is a good way to solve the issue.
a. True
b. False
4. Optimistic locking is used in a PXC cluster. What does it mean?
a. When a transaction starts on a node, locks are only set on this node but never on the remote nodes.
b. When a transaction starts on a node, locks are only set on the remote nodes but never on the local node.
c. Write conflict detection is built-in, so there is no need to set locks at all.
5. Galera implements virtually synchronous replication. What does it mean?
a. A transaction is first committed locally, and then it is committed on all remote nodes at the same exact point in time.
b. Transactions are replicated synchronously, but they are applied asynchronously on remote nodes.
c. Replication is actually asynchronous, but as it is faster than MySQL replication, so marketing decided to name it ‘virtually synchronous’.
6. When the receive queue of a node exceeds a threshold, the node sends flow control messages. What is the goal of these flow control messages?
a. They instruct the other nodes that they must pause processing writes for some time, to allow the slow node to catch up.
b. The other nodes trigger an election and if they have quorum they will evict the slow node.
c. The messages can be used by monitoring systems to detect a slow node, but they have no effect.
7. When you change the state of a node to Donor/Desynced, what happens?
a. The node stops receiving writes from the other nodes.
b. The node intentionally replicates writes at a slower pace, this is roughly equivalent to a delayed replica when using MySQL replication.
c. The node keeps working as usual, but it will not send flow control messages if its receive queue becomes large.
8. You should always use an odd number of nodes, because with an even number (say 4 or 6), the failure of one node will create a split-brain situation.
a. True
b. False
9. With a 3-node cluster, what happens if you gracefully stop 2 nodes?
a. The remaining node can process queries normally.
b. The remaining node is up but it stops processing queries as it does not have quorum.
10. If a node has been stopped for less than 5 minutes, it will always perform an IST.
a. True: SST is only performed after a node crash, never after a regular shutdown.
b. False: it depends on the gcache size.
11. Even with datasets under 5GB, the preferred SST method is xtrabackup-v2 not mysqldump.
a. True
b. False
12. Migration from a master-slave setup to a PXC cluster always involves a downtime to dump and reload the database.
a. True, because MySQL replication and Galera replication are incompatible.
b. False, one node of the PXC cluster can be set up as an asynchronous replica of the old master.
1. a 2. b 3. b
4. a 5. b 6. a
7. c 8. b 9. a
10. b 11. a 12. b
The post Test your knowledge: Percona XtraDB Cluster (PXC) quiz appeared first on MySQL Performance Blog.
Following up on popular "awesome-*" lists (e.g. awesome-python, awesome-golang etc.), I've created the awesome-mysql curated list.
This is a list of technologies (and resources) in and around MySQL, and means to serve as a place to find reliable software and info. I recently happened to notice there are some tools I'm familiar with that are unknown to others; tools unknown to me that are in good use.
The list is naturally and intentionally incomplete. I wish this to be a community based creation; so I put some categories and some tools. I left many out, deliberatey. Please assist by filling in the missing projects, tools, libraries! Additions gladly accepted via pull-requests. Do note the contribution guidelines (somewhat lengthy, I apologize).
I will moderate FUDs, promotional, commercials etc., and otherwise it may take some days for me to merge requests.
The work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.
Systemd is a management and configuration platform available in all major Linux distributions. It provides infrastructure for service start, stop, restart and several other novel functionalities to manage services. Systemd replaces SysV and upstart initialization systems and is the default init system in most modern Linux distributions including Red Hat Enterprise Linux, Oracle Linux, Debian, Ubuntu, Fedora, SLES and openSUSE.
Preliminary support for systemd was introduced in earlier versions of MySQL. However, it had the following limitations and disadvantages:
To overcome the above limitations and disadvantages, some minimal support should be provided within the server. With the release of MySQL 5.7.6, these limitations have been overcome and complete support for systemd is added on systemd based platforms Red Hat Enterprise Linux 7, Oracle Linux 7, CentOS 7, SLES 12, Fedora 20 & Fedora 21. Thus, MySQL 5.7.6 provides full native support for systemd, thereby leveraging the latest functionalities of all the major Linux distributions to their fullest capacity.
A new command-line option –-daemonize has been added which when enabled does a SysV style daemon initialization. This enables us to use a forking type service unit file for mysqld. Earlier on, we were using the simple type unit file for mysqld. This had the disadvantage that the mysqld service was had a state=running before the server was actually ready to handle client connections. This could cause problems by denying client connections from follow-up services that depend on the mysqld service. Also in some cases the server undergoes crash recovery—for which the completion time can vary wildly from sub-second times to many minutes—before the server is actually ready to handle connections. This resulted in the client connections of follow-up units to timeout and report failures.
With the forking type server (which uses the –-daemonize option), the disadvantage of using ping tricks to identify that mysqld is able to begin serving client connections is overcome. When mysqld is invoked with –-daemonize option, the parent process does a double fork and the grand child sets up the pid file and the listen socket initialization among other initialization steps. The grand child then notifies the parent to exit and this notifies systemd to set the service state to “running”.
We can now start, stop, restart and manage other functionalities of the MySQL server using systemd. Installing the rpm package for MySQL server will automatically enable the mysqld service. The systemctl command is the primary interface to control, query, and manage the functionality provided by systemd. After installing the MySQL server package, you can check to see whether mysqld is enabled or not, this way:
$ systemctl is-enabled mysqld enabled
This means that mysqld is automatically started at boot time or when you start the mysqld service after package install. The appropriate data directory creation and initialization is automatically taken care of as well.
We can start the mysqld service using:
$ systemctl start mysqld
We can see the status of mysqld with:
$ systemctl status mysqld
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled)
Active: active (running) since Mon 2015-04-06 06:56:56 BST; 13min ago
Process: 1030 ExecStart=/usr/sbin/mysqld --daemonize $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
Process: 957 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
Main PID: 1071 (mysqld)
CGroup: /system.slice/mysqld.service
└─1071 /usr/sbin/mysqld --daemonize
We can restart mysqld with:
$ systemctl restart mysqld
We can stop mysqld with:
$ systemctl stop mysqld
One of the major pieces of functionality offered by systemd is integrated process monitoring and automatic restarts in the event of a service failure/termination. Starting with MySQL 5.7.6, process monitoring and auto-restarts are now handled by systemd on systems that have it. If mysqld fails due to a restartable failure like a crash, then systemd automatically restarts mysqld. As an example, this can be verified by:
$ systemctl start mysqld
$ systemctl status mysqld
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled)
Active: active (running) since Mon 2015-04-06 07:18:51 BST; 11s ago
Process: 3010 ExecStart=/usr/sbin/mysqld --daemonize $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
Process: 2994 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
Main PID: 3014 (mysqld)
CGroup: /system.slice/mysqld.service
└─3014 /usr/sbin/mysqld --daemonize
$ kill -SEGV 3014
$ systemctl status mysqld
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled)
Active: active (running) since Mon 2015-04-06 07:19:53 BST; 2s ago
Process: 3057 ExecStart=/usr/sbin/mysqld --daemonize $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
Process: 3042 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
Main PID: 3061 (mysqld)
CGroup: /system.slice/mysqld.service
└─3061 /usr/sbin/mysqld --daemonize
mysqld_safe has a --malloc-lib option which allows you to specify a non-default library (e.g. jemalloc). mysqld_safe then sets the LD_PRELOAD environment variable and starts mysqld under this environment setting. On systemd based machines, the mysqld service unit file has the EnvironmentFile key set to /etc/sysconfig/mysql. You can then use an alternative malloc library by adding an entry in the /etc/sysconfig/mysqld file like this:
$ echo "LD_PRELOAD=/path/of/malloc/library.so" >> /etc/sysconfig/mysqld
You can also pass other custom options to mysqld by creating additional entries/lines in /etc/sysconfig/mysql using the MYSQLD_OPTS=”option” format. Lastly, you can also specify custom options and environment variables using the systemctl command:
$ systemctl set-environment MYSQLD_OPTS="--general_log=1"
The custom options can also be unset using:
$ systemctl unset-environment MYSQLD_OPTS
You must then restart the mysqld service for the new environment or configuration settings to take effect.
Two systemd service configuration files and one support file are shipped with MySQL 5.7.6 in order to enable support for systemd:
[Unit] Description=MySQL Server After=network.target After=syslog.target [Install] WantedBy=multi-user.target [Service] User=mysql Group=mysql Type=forking PIDFile=/var/run/mysqld/mysqld.pid # Execute pre and post scripts as root PermissionsStartOnly=true # Needed to create system tables ExecStartPre=/usr/bin/mysqld_pre_systemd # Start main service ExecStart=/usr/sbin/mysqld --daemonize $MYSQLD_OPTS # Use this to switch malloc implementation EnvironmentFile=-/etc/sysconfig/mysql Restart=on-failure RestartPreventExitStatus=1 PrivateTmp=false
d /var/run/mysqld 0755 mysql mysql -
--initialize option to create the data directory (this files is typically installed in /usr/bin). For more information on the new bootstrap process in MySQL 5.7, please see Joro’s blog post.MySQL 5.7.6 brings complete systemd support and is the first among the many variants of MySQL available in the marketplace to support systemd natively. We hope that you enjoy managing MySQL under systemd!
We also look forward to your feedback on this new work!. You can leave a comment here on the blog post or in a support ticket. If you feel that you encountered any related bugs, please do let us know via a bug report. Also you can find more details about managing MySQL 5.7.6 using systemd and the systemd related build options in the MySQL 5.7 docs here.
As always, THANK YOU for using MySQL!
There are plenty of storage engines out there, but none of them seem to offer fast and efficient time series storage and indexing. The existing options like RRDtool and Whisper aren’t very fast, and the fast options like LevelDB aren’t specifically made for time series and can lead to harsh operational issues. Instead of hacking on something like LevelDB to suit his needs, Preetam Jinka, one of the team’s brainiacs, decided to write his own storage engine.
Join us Tuesday, June 2nd at 2 PM EST (6 PM GMT), as Preetam Jinka covers the unique characteristics of time series data, time series indexing, and the basics of log-structured merge (LSM) trees and B-trees. After establishing some basic concepts, he will explain how Catena’s design is inspired by many of the existing systems today and why it works much better than its present alternatives.
Designing a storage engine is the easy part. Implementation is much more interesting. He will cover how Catena uses advanced concurrency optimizations like lock-free lists, atomics, and precise locking. He will also show some neat tricks to take advantage of CPU caches and prefetchers in subtle ways. All of these combined allow Catena to easily store and index over 800,000 time series points per second on an average laptop.
This webinar will help you understand the unique challenges of high-velocity time-series data in general, and VividCortex’s somewhat unique workload in particular. You’ll leave with an understanding of why commonly used technologies can’t handle even a fraction of VividCortex’s workload, and what we’re exploring as we investigate alternatives to our MySQL-backed time-series database.
We know you are busy, and checking the VividCortex interface is not always on the top of your priority list. That’s why we implemented our proactive reporting feature.
Each day, your team will receive a recap of the top ten queries of each environment from the day prior. You can see at a glance whether systems were running smoothly or if you need to look into performance.
If daily reports are creating too much noise, you can subscribe to weekly reports instead. These will show the top ten queries of each environment from the week prior.
Of course, if you do not wish to receive the reports, simply unsubscribe.
This is the first of many proactive features to bring database monitoring to everyone on your team. Stay tuned for more, and sign up for a free trial if you are interested in a monitoring solution that goes deeper to provide meaningful insight.
About 2 weeks ago Oracle published the MySQL 5.7.7-labs-json version which includes a very interesting feature called “Generated columns” (also know as Virtual or Computed columns). MariaDB has a similar feature as well: Virtual (Computed) Columns.
The idea is very simple: if we store a column
`FlightDate` date
in our table we may want to filter or group by year(FlightDate), month(FlightDate) or even dayofweek(FlightDate). The “brute-force” approach: use the above Date and Time MySQL functions in the query; however it will prevent MySQL from using an index (see below). Generated columns will allow you to declare a “Virtual”, non-stored column which is computed based on the existing field; you can then add index on that virtual column, so the query will use that index.
Here is the original example:
CREATE TABLE `ontime` ( `id` int(11) NOT NULL AUTO_INCREMENT, `FlightDate` date DEFAULT NULL, `Carrier` char(2) DEFAULT NULL, `OriginAirportID` int(11) DEFAULT NULL, `OriginCityName` varchar(100) DEFAULT NULL, `OriginState` char(2) DEFAULT NULL, `DestAirportID` int(11) DEFAULT NULL, `DestCityName` varchar(100) DEFAULT NULL, `DestState` char(2) DEFAULT NULL, `DepDelayMinutes` int(11) DEFAULT NULL, `ArrDelayMinutes` int(11) DEFAULT NULL, `Cancelled` tinyint(4) DEFAULT NULL, `CancellationCode` char(1) DEFAULT NULL, `Diverted` tinyint(4) DEFAULT NULL, PRIMARY KEY (`id`), KEY `FlightDate` (`FlightDate`) ) ENGINE=InnoDB
Now I want to find all flights on Sundays (in 2013) and group by airline.
mysql> EXPLAIN SELECT carrier, count(*) FROM ontime_sm
WHERE dayofweek(FlightDate) = 7 group by carrier
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: ontime_sm
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 151253427
Extra: Using where; Using temporary; Using filesort
Results:
32 rows in set (1 min 57.93 sec)The problem here is: MySQL will not be able to use index when you use a function which will “extract” something from the column. The standard approach is to “materialize” the column:
ALTER TABLE ontime_sm ADD Flight_dayofweek tinyint NOT NULL;
Then we will need to load data into that by running “UPDATE ontime_sm SET Flight_dayofweek = dayofweek(flight_date)”. After that we will also need to change the application to support that additional column or use a trigger to update the column. Here is the trigger example:
CREATE DEFINER = CURRENT_USER TRIGGER ontime_insert BEFORE INSERT ON ontime_sm_triggers FOR EACH ROW SET NEW.Flight_dayofweek = dayofweek(NEW.FlightDate);
One problem with the trigger is that it is slow. In my simple example it took almost 2x slower to “copy” the table using “insert into ontime_sm_copy select * from ontime_sm” when the trigger was on.
The Generated Columns from MySQL 5.7.7-labs-json version (only this version supports it on the time of writing) solves this problem. Here is the example which demonstrate its use:
CREATE TABLE `ontime_sm_virtual` ( `id` int(11) NOT NULL AUTO_INCREMENT, `FlightDate` date DEFAULT NULL, `Carrier` char(2) DEFAULT NULL, `OriginAirportID` int(11) DEFAULT NULL, `OriginCityName` varchar(100) DEFAULT NULL, `OriginState` char(2) DEFAULT NULL, `DestAirportID` int(11) DEFAULT NULL, `DestCityName` varchar(100) DEFAULT NULL, `DestState` char(2) DEFAULT NULL, `DepDelayMinutes` int(11) DEFAULT NULL, `ArrDelayMinutes` int(11) DEFAULT NULL, `Cancelled` tinyint(4) DEFAULT NULL, `CancellationCode` char(1) DEFAULT NULL, `Diverted` tinyint(4) DEFAULT NULL, `CRSElapsedTime` int(11) DEFAULT NULL, `ActualElapsedTime` int(11) DEFAULT NULL, `AirTime` int(11) DEFAULT NULL, `Flights` int(11) DEFAULT NULL, `Distance` int(11) DEFAULT NULL, `Flight_dayofweek` tinyint(4) GENERATED ALWAYS AS (dayofweek(FlightDate)) VIRTUAL, PRIMARY KEY (`id`), KEY `Flight_dayofweek` (`Flight_dayofweek`), ) ENGINE=InnoDB
Here we add Flight_dayofweek tinyint(4) GENERATED ALWAYS AS (dayofweek(FlightDate)) VIRTUAL column and index it.
Now MySQL can use this index:
mysql> EXPLAIN SELECT carrier, count(*) FROM ontime_sm_virtual WHERE Flight_dayofweek = 7 group by carrier
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: ontime_sm_virtual
partitions: NULL
type: ref
possible_keys: Flight_dayofweek
key: Flight_dayofweek
key_len: 2
ref: const
rows: 165409
filtered: 100.00
Extra: Using where; Using temporary; Using filesortTo further increase performance of this query we want to add a combined index on (Flight_dayofweek, carrier) so MySQL will avoid creating temporary table. However it is not currently supported:
mysql> alter table ontime_sm_virtual
add key comb(Flight_dayofweek, carrier);
ERROR 3105 (HY000): 'Virtual generated column combines with other columns to be indexed together' is not supported for generated columns.We can add an index on 2 generated columns thou, which is good. So a trick here will be to create a “dummy” virtual column on “carrier” and index 2 of those columns:
mysql> alter table ontime_sm_virtual add Carrier_virtual char(2) GENERATED ALWAYS AS (Carrier) VIRTUAL;
Query OK, 0 rows affected (0.43 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> alter table ontime_sm_virtual add key comb(Flight_dayofweek, Carrier_virtual);
Query OK, 999999 rows affected (36.79 sec)
Records: 999999 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT Carrier_virtual, count(*) FROM ontime_sm_virtual WHERE Flight_dayofweek = 7 group by Carrier_virtual
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: ontime_sm_virtual
partitions: NULL
type: ref
possible_keys: Flight_dayofweek,comb
key: comb
key_len: 2
ref: const
rows: 141223
filtered: 100.00
Extra: Using where; Using indexNow MySQL will use an index and completely avoid the filesort.
The last, but not the least: loading data to the table with generated columns is significantly faster compared to loading it into the same table with triggers:
mysql> insert into ontime_sm_triggers (id, YearD, FlightDate, Carrier, OriginAirportID, OriginCityName, OriginState, DestAirportID, DestCityName, DestState, DepDelayMinutes, ArrDelayMinutes, Cancelled, CancellationCode,Diverted, CRSElapsedTime, ActualElapsedTime, AirTime, Flights, Distance) select * from ontime_sm; Query OK, 999999 rows affected (27.86 sec) Records: 999999 Duplicates: 0 Warnings: 0 mysql> insert into ontime_sm_virtual (id, YearD, FlightDate, Carrier, OriginAirportID, OriginCityName, OriginState, DestAirportID, DestCityName, DestState, DepDelayMinutes, ArrDelayMinutes, Cancelled, CancellationCode,Diverted, CRSElapsedTime, ActualElapsedTime, AirTime, Flights, Distance) select * from ontime_sm; Query OK, 999999 rows affected (16.29 sec) Records: 999999 Duplicates: 0 Warnings: 0
Now the big disappointment: all operations with generated columns are not online right now.
mysql> alter table ontime_sm_virtual add Flight_year year GENERATED ALWAYS AS (year(FlightDate)) VIRTUAL, add key (Flight_year), lock=NONE; ERROR 1846 (0A000): LOCK=NONE is not supported. Reason: '%s' is not supported for generated columns.. Try LOCK=SHARED. mysql> alter table ontime_sm_virtual add key (Flight_year), lock=NONE; ERROR 1846 (0A000): LOCK=NONE is not supported. Reason: '%s' is not supported for generated columns.. Try LOCK=SHARED.
I hope it will be fixed in the future releases.
Conclusion
Generated columns feature is very useful. Imagine an ability to add a column + index for any “logical” piece of data without actually duplicating the data. And this can be any function: date/time/calendar, text (extract(), reverse(), metaphone()) or anything else. I hope this feature will be available in MySQL 5.7 GA. Finally, I wish adding a generated column and index can be online (it is not right now).
More information:
The post Generated (Virtual) Columns in MySQL 5.7 (labs) appeared first on MySQL Performance Blog.
Today, oCERT published advisory 2015-003 describing a TLS vulnerability in MySQL and derivative products. The content isn’t exactly news – it is documented legacy behavior and the subject of an earlier blog post describing how MySQL Server 5.7 solves the problem. That said, the efforts of Duo Security are certainly appreciated and welcomed – it provides a meaningful context to discuss how to properly harden existing MySQL 5.5 and 5.6 deployments, as well as frame a discussion on potential changes in these versions to increase security.
The vulnerability described in the advisory relies on the legacy behavior of the client –ssl option, which simply enables – but does not require – TLS connections. When connecting to a MySQL Server which has TLS enabled, this will normally result in TLS being used for the connection. When the server does not support TLS, though, the connection will be established without TLS. For versions of MySQL prior to 5.7.3, there is no means on the client side to require TLS connections. This allows an attacker with the ability to actively manipulate network packets to silently negotiate away TLS.
Such attacks require the ability to manipulate packets between the MySQL Server and client on the network in real time. There are several options to secure connections over untrusted networks where such activity may take place, but they were unfortunately not referenced in the advisory.
Secure communication has several aspects – not only does the data need to be encrypted, but identity needs to be established. It’s insufficient to know nobody other than the recipient can read data being sent if the identity of the recipient is unknown. A client may encrypt data, but if it’s being sent to somebody impersonating a MySQL Server, the data is compromised.
When an account is configured with the REQUIRE X509 option, the client must provide an acceptable certificate matching the criteria defined in the GRANT command – and doing so requires TLS connections from the client. In such cases, an nefarious proxy cannot negotiate away TLS.
Use of client-side key material is surely uncommon, but in a deployment where MySQL communication must be secured across an untrusted network, it’s a necessary step. More details about configuration to use client-side key material can be found in the manual, and don’t forget that you can use the new mysql_ssl_rsa_setup utility from MySQL 5.7 packages to easily produce client key material (obviously, properly-signed keys are preferred).
Please note that defining accounts with the REQUIRE SSL option is not sufficient to bypass this vulnerability – a nefarious proxy could easily establish a SSL connection between the proxy and the server, while leaving the proxy’s own connection to the client in plain-text.
If setting up client-side key material is too much of a burden, you can secure your communication by using a SSH tunnel instead of TLS. Tools such as MySQL Workbench make such connections even easier.
The easiest solution is to use clients from MySQL Server 5.7.3 or above. While MySQL 5.7 is not yet GA (it is in Release Candidate state as I write this), there’s no reason you cannot use just the client programs out of the 5.7 package which allow TLS to be required with the –ssl option – they will work just fine with 5.5 or 5.6 servers.
The legacy behavior that prevents clients from requiring TLS connections is clearly undesirable, but Oracle takes great pains to avoid behavioral changes in maintenance releases for already-GA products. For that reason, the –ssl option was redefined to require TLS only in 5.7, where it accompanies a change that makes TLS preferred by default. The latter change has potential significant impacts to users and third-party products, and cannot be reasonably back-ported to 5.5 or 5.6. However, we are considering back-porting the change which redefines –ssl to mean “TLS is required” to these versions. Below are some details and considerations – we would love to hear your feedback.
It’s my belief that most users who specify –ssl as a client option actually intend for the resulting connection to use TLS, and not simply allow TLS. Furthermore, most users who specify –ssl today would likely prefer – or even expect – connections to fail when TLS cannot be negotiated successfully. Back-porting the 5.7 changes would align behavior with most user expectations (as I perceive them).
There may be users who actually understand and rely on legacy behavior, and have specified –ssl even though target servers do not support TLS. If the 5.7 behavior is back-ported, such users will find that where previous client versions established non-secure connections without notice, new connections will fail until the –ssl option is removed.
Users who want legacy behavior may obtain that by specifying other TLS-related client options, such as –ssl-cert, but without –ssl. This would require changes to user scripts which rely on legacy –ssl behavior, though.
The –ssl option is a boolean configuration option, and as of MySQL 5.7, there are currently three different TLS “modes”:
In MySQL 5.7, TLS preferred is the default state; TLS required results from setting –ssl; TLS disabled can be triggered using –skip-ssl (among other boolean configuration variants). One option considered is to eliminate the –ssl option over time, and replace it with a multi-value option, such as –ssl-mode=[DISABLE|PREFER|REQUIRE]. This would need to co-exist with the –ssl option for a while, but at some point, the –ssl option would be deprecated and removed. This would result in requiring all scripts using –ssl to be modified, and likely causing pain for users whose scripts cannot deal with new deprecation warnings, even before functionality is fully removed.
The oCERT advisory highlights changes already made to MySQL Server 5.7 to improve security, and provides a context to discuss potential back-ports to MySQL 5.6 and 5.5. There are also meaningful steps which users of MySQL 5.5 or 5.6 can take today to eliminate exposure to this vulnerability.
I have recently seen several cases when performance for MySQL queries on a single table was terrible. The reason was simple: the wrong indexes were added and so the execution plan was poor. Here are guidelines to help you optimize various kinds of single-table queries.
Disclaimer: I will be presenting general guidelines and I do not intend to cover all scenarios. I am pretty confident that you can find examples where what I am writing does not work, but I am also confident that it will help you most of the time. Also I will not discuss features you can find in MySQL 5.6+ like Index Condition Pushdown to keep things simple. Be aware that such features can actually make a significant difference in query response time (for good or for bad).
An index can perform up to 3 actions: filter, sort/group and cover. While the first 2 actions are self-explanatory, not everyone may know what a ‘covering index’ is. Actually that’s very easy. The general workflow for a basic query is:
1. Use an index to find matching records and get the pointers to data.
2. Use the pointers to the corresponding data.
3. Return records
When a covering index can be used, the index already covers all fields requested in the query, so step #2 can be skipped and the workflow is now:
1. Use an index to find matching records
2. Return records
In many cases, indexes are small and can fit in memory while data is large and does not fit in memory: by using a covering index, you can avoid lots of disk operations and performance can be order of magnitudes better.
Let’s now look at different common scenarios.
This is the most basic scenario:
SELECT * FROM t WHERE c = 100
The idea is of course to add an index on (c). However note that if the criteria is not selective enough, the optimizer may choose to perform a full table scan that will certainly be more efficient.
Also note that a frequent variation of this query is when you only select a small subset of fields instead of all fields:
SELECT c1, c2 FROM t WHERE c = 100
Here it could make sense to create an index on (c, c1, c2) because it will be a covering index. Do not create an index on (c1, c2, c)! It will still be covering but it will not be usable for filtering (remember that you can only use a left-most prefix of an index to filter).
SELECT * FROM t WHERE c = 100 and d = 'xyz'
It is also very easy to optimize: just add an index on (c, d) or (d, c).
The main mistake here is to add 2 indexes: one on (c) and one on (d). Granted, MySQL is able to use both indexes with the index_merge algorithm, but it is almost always a very bad option.
SELECT * FROM t WHERE c > 100 and d = 'xyz'
Here we must be careful because as long as we are using a column with an inequality, this will prevent us from using further columns in the index.
Therefore if we create an index on (d, c), we will be able to filter both on c and d, this is good.
But if we create an index on (c, d), we will only be filtering on c, which is less efficient.
So unlike the situation when you have equalities, order of columns matters when inequalities are used.
SELECT * FROM t WHERE c > 100 and b < 10 and d = 'xyz'
As we have 2 inequalities, we already know that we will not be able to filter on both conditions (*). So we have to make a decision: will we filter on (d, b) or on (d, c)?
It is not possible to tell which option is better without looking at the data: simply choose the column where the inequality is the most selective. The main point is that you must put the column(s) with an equality first.
(*) Actually there is a way to ‘filter’ on both inequalites: partition on b and add an index on (d, c) or partition on c and add an index on (d, b). The details are out of the scope of this post but it might be an option for some situations.
SELECT * FROM t WHERE c = 100 and d = 'xyz' ORDER BY b
As mentioned in the first paragraph, an index can filter and sort so this query is easy to optimize. However like for inequalities, we must carefully choose the order of the columns in the index: the rule is that we will filter first, and then sort.
With that in mind, it is easy to know that (c, d, b) or (d, c, b) will be good indexes while (b, c, d) or (b, d, c) are not as good (they will sort but not filter).
And if we have:
SELECT c1, c2 FROM t WHERE c = 100 and d = 'xyz' ORDER BY b
We can create a super efficient index that will filter, sort and be covering: (c, d, b, c1, c2).
We have 2 main variations here. The first one is:
SELECT * FROM t WHERE c > 100 and d = 'xyz' ORDER BY b
Two options look reasonable in this case:
1. filter on d and sort by b.
2. filter on d and c.
Which strategy is more efficient? It will depend on your data, so you will have to experiment.
The second variation is:
SELECT * FROM t WHERE c > 100 ORDER BY b
This time we have no equality so we have to choose between filtering and sorting. Most likely you will choose filtering.
Not all cases have been covered in this post but you can already see that in some cases you will create poor MySQL indexes if you are not careful. In a future post, I will present a case that can look confusing at first sight but which is easy to understand if you already know everything mentioned here.
The post Indexing 101: Optimizing MySQL queries on a single table appeared first on MySQL Performance Blog.
I often talk with people who are very interested in the features of Percona XtraDB Cluster (PXC) such as synchronous and parallel replication, multi-node writing and high availability. However some get confused when operating a real PXC cluster because they do not fully realize the implications of these features. So here is a fun way to test your PXC knowledge: try to solve these 12 questions related to PXC! (you will find the answers at the end of the post.)
1. With Galera 3.x, support for MyISAM is experimental. When can we expect to have full MyISAM support?
a. This will never happen as Galera is designed for transactional storage engines.
b. This is planned for Galera 4.0.
2. Why aren’t all workloads a good fit for PXC?
a. Execution plans can change compared to a regular MySQL server, so performance is sometimes not as good as with a regular MySQL server.
b. Large transactions and write hotspots can create performance issues with Galera.
3. For workloads with a write hot spot, writing on all nodes to distribute the load is a good way to solve the issue.
a. True
b. False
4. Optimistic locking is used in a PXC cluster. What does it mean?
a. When a transaction starts on a node, locks are only set on this node but never on the remote nodes.
b. When a transaction starts on a node, locks are only set on the remote nodes but never on the local node.
c. Write conflict detection is built-in, so there is no need to set locks at all.
5. Galera implements virtually synchronous replication. What does it mean?
a. A transaction is first committed locally, and then it is committed on all remote nodes at the same exact point in time.
b. Transactions are replicated synchronously, but they are applied asynchronously on remote nodes.
c. Replication is actually asynchronous, but as it is faster than MySQL replication, so marketing decided to name it ‘virtually synchronous’.
6. When the receive queue of a node exceeds a threshold, the node sends flow control messages. What is the goal of these flow control messages?
a. They instruct the other nodes that they must pause processing writes for some time, to allow the slow node to catch up.
b. The other nodes trigger an election and if they have quorum they will evict the slow node.
c. The messages can be used by monitoring systems to detect a slow node, but they have no effect.
7. When you change the state of a node to Donor/Desynced, what happens?
a. The node stops receiving writes from the other nodes.
b. The node intentionally replicates writes at a slower pace, this is roughly equivalent to a delayed replica when using MySQL replication.
c. The node keeps working as usual, but it will not send flow control messages if its receive queue becomes large.
8. You should always use an odd number of nodes, because with an even number (say 4 or 6), the failure of one node will create a split-brain situation.
a. True
b. False
9. With a 3-node cluster, what happens if you gracefully stop 2 nodes?
a. The remaining node can process queries normally.
b. The remaining node is up but it stops processing queries as it does not have quorum.
10. If a node has been stopped for less than 5 minutes, it will always perform an IST.
a. True: SST is only performed after a node crash, never after a regular shutdown.
b. False: it depends on the gcache size.
11. Even with datasets under 5GB, the preferred SST method is xtrabackup-v2 not mysqldump.
a. True
b. False
12. Migration from a master-slave setup to a PXC cluster always involves a downtime to dump and reload the database.
a. True, because MySQL replication and Galera replication are incompatible.
b. False, one node of the PXC cluster can be set up as an asynchronous replica of the old master.
1. a 2. b 3. b
4. a 5. b 6. a
7. c 8. b 9. a
10. b 11. a 12. b
The post Test your knowledge: Percona XtraDB Cluster (PXC) quiz appeared first on MySQL Performance Blog.
About 2 weeks ago Oracle published the MySQL 5.7.7-labs-json version which includes a very interesting feature called “Generated columns” (also know as Virtual or Computed columns). MariaDB has a similar feature as well: Virtual (Computed) Columns.
The idea is very simple: if we store a column
`FlightDate` date
in our table we may want to filter or group by year(FlightDate), month(FlightDate) or even dayofweek(FlightDate). The “brute-force” approach: use the above Date and Time MySQL functions in the query; however it will prevent MySQL from using an index (see below). Generated columns will allow you to declare a “Virtual”, non-stored column which is computed based on the existing field; you can then add index on that virtual column, so the query will use that index.
Here is the original example:
CREATE TABLE `ontime` ( `id` int(11) NOT NULL AUTO_INCREMENT, `FlightDate` date DEFAULT NULL, `Carrier` char(2) DEFAULT NULL, `OriginAirportID` int(11) DEFAULT NULL, `OriginCityName` varchar(100) DEFAULT NULL, `OriginState` char(2) DEFAULT NULL, `DestAirportID` int(11) DEFAULT NULL, `DestCityName` varchar(100) DEFAULT NULL, `DestState` char(2) DEFAULT NULL, `DepDelayMinutes` int(11) DEFAULT NULL, `ArrDelayMinutes` int(11) DEFAULT NULL, `Cancelled` tinyint(4) DEFAULT NULL, `CancellationCode` char(1) DEFAULT NULL, `Diverted` tinyint(4) DEFAULT NULL, PRIMARY KEY (`id`), KEY `FlightDate` (`FlightDate`) ) ENGINE=InnoDB
Now I want to find all flights on Sundays (in 2013) and group by airline.
mysql> EXPLAIN SELECT carrier, count(*) FROM ontime_sm
WHERE dayofweek(FlightDate) = 7 group by carrier
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: ontime_sm
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 151253427
Extra: Using where; Using temporary; Using filesort
Results:
32 rows in set (1 min 57.93 sec)The problem here is: MySQL will not be able to use index when you use a function which will “extract” something from the column. The standard approach is to “materialize” the column:
ALTER TABLE ontime_sm ADD Flight_dayofweek tinyint NOT NULL;
Then we will need to load data into that by running “UPDATE ontime_sm SET Flight_dayofweek = dayofweek(flight_date)”. After that we will also need to change the application to support that additional column or use a trigger to update the column. Here is the trigger example:
CREATE DEFINER = CURRENT_USER TRIGGER ontime_insert BEFORE INSERT ON ontime_sm_triggers FOR EACH ROW SET NEW.Flight_dayofweek = dayofweek(NEW.FlightDate);
One problem with the trigger is that it is slow. In my simple example it took almost 2x slower to “copy” the table using “insert into ontime_sm_copy select * from ontime_sm” when the trigger was on.
The Generated Columns from MySQL 5.7.7-labs-json version (only this version supports it on the time of writing) solves this problem. Here is the example which demonstrate its use:
CREATE TABLE `ontime_sm_virtual` ( `id` int(11) NOT NULL AUTO_INCREMENT, `FlightDate` date DEFAULT NULL, `Carrier` char(2) DEFAULT NULL, `OriginAirportID` int(11) DEFAULT NULL, `OriginCityName` varchar(100) DEFAULT NULL, `OriginState` char(2) DEFAULT NULL, `DestAirportID` int(11) DEFAULT NULL, `DestCityName` varchar(100) DEFAULT NULL, `DestState` char(2) DEFAULT NULL, `DepDelayMinutes` int(11) DEFAULT NULL, `ArrDelayMinutes` int(11) DEFAULT NULL, `Cancelled` tinyint(4) DEFAULT NULL, `CancellationCode` char(1) DEFAULT NULL, `Diverted` tinyint(4) DEFAULT NULL, `CRSElapsedTime` int(11) DEFAULT NULL, `ActualElapsedTime` int(11) DEFAULT NULL, `AirTime` int(11) DEFAULT NULL, `Flights` int(11) DEFAULT NULL, `Distance` int(11) DEFAULT NULL, `Flight_dayofweek` tinyint(4) GENERATED ALWAYS AS (dayofweek(FlightDate)) VIRTUAL, PRIMARY KEY (`id`), KEY `Flight_dayofweek` (`Flight_dayofweek`), ) ENGINE=InnoDB
Here we add Flight_dayofweek tinyint(4) GENERATED ALWAYS AS (dayofweek(FlightDate)) VIRTUAL column and index it.
Now MySQL can use this index:
mysql> EXPLAIN SELECT carrier, count(*) FROM ontime_sm_virtual WHERE Flight_dayofweek = 7 group by carrier
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: ontime_sm_virtual
partitions: NULL
type: ref
possible_keys: Flight_dayofweek
key: Flight_dayofweek
key_len: 2
ref: const
rows: 165409
filtered: 100.00
Extra: Using where; Using temporary; Using filesortTo further increase performance of this query we want to add a combined index on (Flight_dayofweek, carrier) so MySQL will avoid creating temporary table. However it is not currently supported:
mysql> alter table ontime_sm_virtual
add key comb(Flight_dayofweek, carrier);
ERROR 3105 (HY000): 'Virtual generated column combines with other columns to be indexed together' is not supported for generated columns.We can add an index on 2 generated columns thou, which is good. So a trick here will be to create a “dummy” virtual column on “carrier” and index 2 of those columns:
mysql> alter table ontime_sm_virtual add Carrier_virtual char(2) GENERATED ALWAYS AS (Carrier) VIRTUAL;
Query OK, 0 rows affected (0.43 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> alter table ontime_sm_virtual add key comb(Flight_dayofweek, Carrier_virtual);
Query OK, 999999 rows affected (36.79 sec)
Records: 999999 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT Carrier_virtual, count(*) FROM ontime_sm_virtual WHERE Flight_dayofweek = 7 group by Carrier_virtual
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: ontime_sm_virtual
partitions: NULL
type: ref
possible_keys: Flight_dayofweek,comb
key: comb
key_len: 2
ref: const
rows: 141223
filtered: 100.00
Extra: Using where; Using indexNow MySQL will use an index and completely avoid the filesort.
The last, but not the least: loading data to the table with generated columns is significantly faster compared to loading it into the same table with triggers:
mysql> insert into ontime_sm_triggers (id, YearD, FlightDate, Carrier, OriginAirportID, OriginCityName, OriginState, DestAirportID, DestCityName, DestState, DepDelayMinutes, ArrDelayMinutes, Cancelled, CancellationCode,Diverted, CRSElapsedTime, ActualElapsedTime, AirTime, Flights, Distance) select * from ontime_sm; Query OK, 999999 rows affected (27.86 sec) Records: 999999 Duplicates: 0 Warnings: 0 mysql> insert into ontime_sm_virtual (id, YearD, FlightDate, Carrier, OriginAirportID, OriginCityName, OriginState, DestAirportID, DestCityName, DestState, DepDelayMinutes, ArrDelayMinutes, Cancelled, CancellationCode,Diverted, CRSElapsedTime, ActualElapsedTime, AirTime, Flights, Distance) select * from ontime_sm; Query OK, 999999 rows affected (16.29 sec) Records: 999999 Duplicates: 0 Warnings: 0
Now the big disappointment: all operations with generated columns are not online right now.
mysql> alter table ontime_sm_virtual add Flight_year year GENERATED ALWAYS AS (year(FlightDate)) VIRTUAL, add key (Flight_year), lock=NONE; ERROR 1846 (0A000): LOCK=NONE is not supported. Reason: '%s' is not supported for generated columns.. Try LOCK=SHARED. mysql> alter table ontime_sm_virtual add key (Flight_year), lock=NONE; ERROR 1846 (0A000): LOCK=NONE is not supported. Reason: '%s' is not supported for generated columns.. Try LOCK=SHARED.
I hope it will be fixed in the future releases.
Conclusion
Generated columns feature is very useful. Imagine an ability to add a column + index for any “logical” piece of data without actually duplicating the data. And this can be any function: date/time/calendar, text (extract(), reverse(), metaphone()) or anything else. I hope this feature will be available in MySQL 5.7 GA. Finally, I wish adding a generated column and index can be online (it is not right now).
More information:
The post Generated (Virtual) Columns in MySQL 5.7 (labs) appeared first on MySQL Performance Blog.