Quantcast
Channel: Planet MySQL
Viewing all 18822 articles
Browse latest View live

Percona Live Featured Session with Luís Soares: The New MySQL Replication Features in MySQL 8

$
0
0
Percona Live Featured Session

Percona Live Featured SessionWelcome to another post in the series of Percona Live featured session blogs! In these blogs, we’ll highlight some of the session speakers that will be at this year’s Percona Live conference. We’ll also discuss how these sessions can help you improve your database environment. Make sure to read to the end to get a special Percona Live 2017 registration bonus!

In this Percona Live featured session, we’ll meet Luís Soares, Principal Software Engineer at Oracle. His session is The New MySQL Replication Features in MySQL 8 (with fellow presenter Lars Thalmann, Development Director at Oracle). The most popular high availability (HA) techniques deployed are based on making services redundant, in particular by means of replication. This fits quite naturally in the MySQL universe, as MySQL server has provided a mature replication solution for over a decade now. Moreover, the new replication developments (and their roadmap) show that MySQL is also catering for the requirements posed by popular environments such as the cloud.

I had a chance to speak with Luís about MySQL 8.0 replication:

Percona: How did you get into database technology? What do you love about it?

Luis SoaresLuís: My background is in distributed systems, particularly in database replication, message passing technologies and fault-tolerance. It all started while I was taking my BSc in computer science. As I finished it, I felt very much drawn towards these subjects. That alone made me enroll in a Master’s course that allowed me to focus almost full time on database replication using group communication technologies. I continued to study this field for years, and further deepened my knowledge on this area. That was great fun and quite a learning experience!

Years went by, and eventually I ended up working at MySQL on the replication team. This happened after I came to a MySQL user conference to present some of the work that I was doing at the time.

These are very fond memories! But I digress!

Back to the point. In general, the thing I love about working on database replication is that I am constantly facing new and interesting problems. Data replication in itself is hard. Add to that the semantics and requirements of a database server, and complexity increases quite a bit. Also, building a generic database replication service that fits in a large set of use cases requires a lot of discipline and careful thinking when designing new features. And let’s not forget the Web itself, which is constantly changing. New technologies come and go at a fast pace. The volume of data that has to be handled, year after year, increases considerably. This poses scalability and integration challenges that need to be addressed.

All in all, these are very exciting times to work with high availability, data replication and data integration.

Now specifically about MySQL, I love the fact that I work on a popular database technology that embraced replication very early in its life cycle. Replication awareness runs deep in the product and in its ecosystem. Consequently, MySQL has an extensive user base exploring many different use case scenarios around replication. And this is extremely motivating, rewarding and exciting. I can honestly say that my day-to-day work is never boring!

Percona: Your talk is called The New MySQL Replication Features in MySQL 8. What are the key replication features in MySQL 8.0, and why are they important?

Luís: It was a huge amount of work to get the MySQL Group Replication plugin out with MySQL 5.7.17. Group Replication is a new plugin that gives the user some replication nice properties by resorting to group communication and state machine replication. This makes the system able to protect data against split brain situations, enables fault-tolerance and high availability and provides coordination between servers committing transactions that change the data.

In addition to Group Replication, the team has also invested quite a bit on core replication features. Some of these features were already released, and others will be released at some point in time in a MySQL DMR.

In the first 8.0 DMR (MySQL 8.0.0) replication has better instrumentation for row-based replication. The user can observe the row-based replication applier progress by querying performance schema tables. There is also an enhanced global transaction identifier.

GTIDs history management, as the user can set the variable GTID_PURGED in scenarios other than those where the server has an empty GTID execution history. And the user can now specify the stop condition when starting the relay log applier, even if there are multiple applier threads started.

All these features combined are of great help, since they reduce operations overhead through automation, better observability and coordination between servers.

Work continues on many fronts: performance, availability, scalability, efficiency and observability. Stay tuned!

Percona: How do these features make DBAs lives easier? What problems do they solve?

Luís: As mentioned above, the features in MySQL 8.0.0 take some of the operations burden from the DBA. Moreover, they allow the user to better observe what is happening inside the replication pipeline. This alone is quite interesting, since DBAs need to make decisions both when designing new deployments and when tackling issues, possibly having to meet very tight deadlines.

Simply put, these features will help DBAs to diagnose and fix problems faster.

Percona: What do you want attendees to take away from your session? Why should they attend?

Luís: Our session is primarily about the shiny new replication features already in MySQL 8. This is the first takeaway. To know, first hand, what is in MySQL 8 replication-wise. But there is another takeaway, and quite an interesting one. Part of the session is dedicated to presenting the overall ideas around MySQL replication. So attendees will get an overview of the roadmap, and will be able to participate and provide feedback along the way. They will learn more about the big picture, and we will bring together some of the hot MySQL technologies that we keep hearing about nowadays: Group Replication, InnoDB Clusters, Multi-Threaded Replication and more!

It will be fun.

Percona: What are you most looking forward to at Percona Live 2017?

Luís: As a conference participant, I look forward to doing some networking with the vibrant MySQL community. I must say, that I really enjoy engaging in nice technical discussions about my favorite topics: fault-tolerance, replication, dependability and distributed systems overall. The conference gives me a great opportunity to do this.

As a MySQL developer, and one that has been developing MySQL replication for quite some time now, I look forward to talking about the recent work that my team has done and getting all the feedback I can.

As a bystander, conferences like Percona Live make me realize how much MySQL has grown, and how much it has evolved. Replication, for instance, has had so many interesting features, release after release over the last eight or nine years. The community has embraced and deployed them, often worked/interacted with the developers to improve them by providing feedback, feature requests or contributions. And this means that they are part of the story too!

These conferences are always a great learning experience! After spending a week with the MySQL community, I always feel refreshed, energized, extra motivated and with lots of food for thought when I get back home.

Go MySQL!

Register for Percona Live Data Performance Conference 2017, and see Luís present his session on The New MySQL Replication Features in MySQL 8 (with fellow presenter Lars Thalmann, Development Director at Oracle). Use the code FeaturedTalk and receive $100 off the current registration price!

Percona Live Data Performance Conference 2017 is the premier open source event for the data performance ecosystem. It is the place to be for the open source community, as well as businesses that thrive in the MySQL, NoSQL, cloud, big data and Internet of Things (IoT) marketplaces. Attendees include DBAs, sysadmins, developers, architects, CTOs, CEOs, and vendors from around the world.

The Percona Live Data Performance Conference will be April 24-27, 2017 at the Hyatt Regency Santa Clara & The Santa Clara Convention Center.


SQL*Loader-704: Internal error: ulconnect:OCIServerAttache [0] ORA-12154: TNS:could not resolve the connect identifier specified

$
0
0

  When you see the above error , first thing come into mind is 

Is our tnsnames.ora file is exist , if its there used connection string is there

If you have it correctly and still issue is persist .. Then whats happens everything is good 

1.First thing you have to check if tnsnames is fine

sqlldr scott/HD@y45Hhds@orcl control=E:\test\table.ctl log=E:\test\table.log

Check if you have password having '@' symbol

2.Then change it as below , you will be able to load the millions of files into oracle table 

sqlldr scott/\"HD@y45Hhds\"@orcl control=E:\test\table.ctl log=E:\test\table.log

Make it useful ..

How to Install Cerb on CentOS 7

$
0
0
Cerb is a free and open source web-based application software for collaboration and email automation. It is written in PHP language and uses MySQL/MariaDB as a database. It is used for sending a large number of emails. Here, we will explain how to install Cerb on CentOS 7 server.

What to Do When the MySQL Optimizer Overestimates the Condition Filtering Effect

$
0
0

In my previous blog post, I showed an example of how the MySQL Optimizer found a better join order by taking into account the filtering effects of conditions. I also explained that for non-indexed columns the filtering estimate is just a guess, and that there is a risk for non-optimal query plans if the guess is off.

We have received a few bug reports on performance regressions when upgrading from 5.6 to 5.7 that are caused by the optimizer overestimating the filtering effect. In most cases, the cause of the regression is inaccurate filtering estimates for equality conditions on non-indexed columns with low cardinality. In this blog post, I will discuss three ways to handle such regressions:

  1. Create an index
  2. Use an optimizer hint to change the join order
  3. Disable condition filtering
First, I will show an example where overestimating the condition filtering effects gives a non-optimal query plan.

Example: DBT-3 Query 21

We will look at Query 21 in the DBT-3 benchmark:

SELECT s_name, COUNT(*) AS numwait
FROM supplier
JOIN lineitem l1 ON s_suppkey = l1.l_suppkey
JOIN orders ON o_orderkey = l1.l_orderkey
JOIN nation ON s_nationkey = n_nationkey
WHERE o_orderstatus = 'F'
AND l1.l_receiptdate > l1.l_commitdate
AND EXISTS (SELECT * FROM lineitem l2
WHERE l2.l_orderkey = l1.l_orderkey
AND l2.l_suppkey l1.l_suppkey)
AND NOT EXISTS (SELECT * FROM lineitem l3
WHERE l3.l_orderkey = l1.l_orderkey
AND l3.l_suppkey l1.l_suppkey
AND l3.l_receiptdate > l3.l_commitdate)
AND n_name = 'JAPAN'
GROUP BY s_name ORDER BY numwait DESC, s_name LIMIT 100;

Query 21 is called Suppliers Who Kept Orders Waiting Query. In MySQL 5.7, Visual EXPLAIN shows the following query plan for Query 21:

The four tables of the join are joined from left-to-right, starting with a full table scan of the orders table. There are also two dependent subqueries on the lineitem table that will be executed for each row of the outer lineitemtable. The execution time for this query plan is almost 25 seconds on a scale factor 1 DBT-3 database. This is more than ten times as long as the query plan used in MySQL 5.6!

The filtered column of tabular EXPLAIN shows the optimizer's estimates for the condition filter effects (some of the columns have been removed to save space):

idselect_typetabletypekeyrowsfilteredExtra
1PRIMARYordersALLNULL150000010.00Using where; Using temporary; Using filesort
1PRIMARYl1refPRIMARY433.33Using where
1PRIMARYsuppliereq_refPRIMARY1100.00Using index condition
1PRIMARYnationALLNULL254.00Using where; Using join buffer (Block Nested Loop)
3DEPENDENT SUBQUERYl3refPRIMARY430.00Using where
2DEPENDENT SUBQUERYl2refPRIMARY490.00Using where

This shows that the optimizer assumes that the condition o_orderstatus = 'F' is satisfied by 10% of the rows in the orders table. Hence, the optimizer thinks that it will be possible to filter out a lot of orders early by starting with the orders table. However, the truth is that almost 50% of the rows have the requested order status. In other words, by overestimating the filtering effect for orders, query plans that start with the orders table will appear to be less costly than is actually the case.

We will now look at how we can influence the optimizer to pick a better query plan for this query.

Option 1: Create an Index

As mentioned, the optimizer does not have any statistics on non-indexed columns. So one way to improve the optimizer's precision is to create an index on the column. For Query 21, since the filtering estimate for o_orderstatus is way off, we can try to see what happens if we create an index on this column:

CREATE INDEX i_o_orderstatus ON orders(o_orderstatus);
With this index, the query plan has changed:
idselect_typetabletypekeyrowsfilteredExtra
1PRIMARYnationALLNULL2510.00Using where; Using temporary; Using filesort
1PRIMARYsupplierrefi_s_nationkey400100.00NULL
1PRIMARYl1refi_l_suppkey60033.33Using where
1PRIMARYorderseq_refPRIMARY150.00Using where
3DEPENDENT SUBQUERYl3refPRIMARY430.00Using where
2DEPENDENT SUBQUERYl2refPRIMARY490.00Using where

We see from the EXPLAIN output that the estimated filtering effect for orders is now 50%. Given that, the optimizer prefers a different join order, starting with the nationtable. This is the same join order as one got in MySQL 5.6, and the execution time with this plan is 2.5 seconds. Instead of accessing 50% of all orders, the query will now just access orders for suppliers in Japan. However, this improvement comes at the cost of having to maintain an index that will probably never be used!

Looking at Query 21, there is also an equality condition on another column without an index; n_name of the nationtable. For this column, 10% is actually a too high estimate. There are 25 nations in the table. Hence, the correct estimate should be 4%. What if we, instead, create an index on this column?

DROP INDEX i_o_orderstatus ON orders;
CREATE INDEX i_n_name ON nation(n_name);
Then we get this query plan:
idselect_typetabletypekeyrowsfilteredExtra
1PRIMARYnationrefi_n_name1100.00Using index; Using temporary; Using filesort
1PRIMARYsupplierrefi_s_nationkey400100.00NULL
1PRIMARYl1refi_l_suppkey60033.33Using where
1PRIMARYorderseq_refPRIMARY110.00Using where
3DEPENDENT SUBQUERYl3refPRIMARY430.00Using where
2DEPENDENT SUBQUERYl2refPRIMARY490.00Using where

In this case, our new index is actually used! Since scanning a table with 25 rows takes a neglible part of the total execution time, the savings for Query 21 are insignificant, but there might be other queries where such an index could be more useful.

Option 2: Join Order Hint

Instead of trying to improve statistics to get a better query plan, we can use hints to influence the optimizer's choice of query plan. The STRAIGHT_JOIN hint can be used to change the join order. It comes in two flavors:

We will use the second variant and specify that nationshould be processed before orders:
SELECT s_name, COUNT(*) AS numwait
FROM supplier
JOIN lineitem l1 ON s_suppkey = l1.l_suppkey
JOIN nation ON s_nationkey = n_nationkey
STRAIGHT_JOIN orders ON o_orderkey = l1.l_orderkey
WHERE o_orderstatus = 'F'
AND l1.l_receiptdate > l1.l_commitdate
AND EXISTS (SELECT * FROM lineitem l2
WHERE l2.l_orderkey = l1.l_orderkey
AND l2.l_suppkey l1.l_suppkey)
AND NOT EXISTS (SELECT * FROM lineitem l3
WHERE l3.l_orderkey = l1.l_orderkey
AND l3.l_suppkey l1.l_suppkey
AND l3.l_receiptdate > l3.l_commitdate)
AND n_name = 'JAPAN'
GROUP BY s_name ORDER BY numwait DESC, s_name LIMIT 100;
This way we force the optimizer to pick a query plan where nation comes before orders, and the resulting query plan is the "good one":
idselect_typetabletypekeyrowsfilteredExtra
1PRIMARYnationALLNULL2510.00Using where; Using temporary; Using filesort
1PRIMARYsupplierrefi_s_nationkey400100.00NULL
1PRIMARYl1refi_l_suppkey60033.33Using where
1PRIMARYorderseq_refPRIMARY110.00Using where
3DEPENDENT SUBQUERYl3refPRIMARY430.00Using where
2DEPENDENT SUBQUERYl2refPRIMARY490.00Using where

In order to user STRAIGHT_JOIN we had to rearrange the tables in the FROM clause. This is a bit cumbersome, and to avoid this, we have in MySQL 8.0 introduced new join order hintsthat uses the new optimizer hint syntax. Using this syntax, we can add hints right after SELECT and avoid editing the rest of the query. In the case of Query 21, we can add hints like

SELECT /*+ JOIN_PREFIX(nation) */ …
or
SELECT /*+ JOIN_ORDER(nation, orders) */ …
to achieve the desired query plan.

Option 3: Disable Condition Filtering

Many optimizer features can be disabled by setting the optimizer_switch variable. The following statement will make the optimizer not use condition filtering estimates:
SET optimizer_switch='condition_fanout_filter=off';
Looking that the query plan as presented by EXPLAIN, we see that filtering is no longer taken into account:
idselect_typetabletypekeyrowsfilteredExtra
1PRIMARYnationALLNULL25100.00Using where; Using temporary; Using filesort
1PRIMARYsupplierrefi_s_nationkey400100.00NULL
1PRIMARYl1refi_l_suppkey600100.00Using where
1PRIMARYorderseq_refPRIMARY1100.00Using where
3DEPENDENT SUBQUERYl3refPRIMARY4100.00Using where
2DEPENDENT SUBQUERYl2refPRIMARY4100.00Using where

Note that you can set optimizer_switch at session level. Hence, it is possible to disable condition filtering for individual queries. However, this requires extra round-trips to the server to set optimizer_switch before and after the execution of the query.

(Option 4: Wait for Histograms)

We are working to improve the statistics available to the optimizer by introducing histograms. A histogram provides more detailed information about the data distribution in a table column. With histograms, the optimizer will be able to estimate pretty accurately the filtering effects also for conditions on non-indexed columns. Until then, you will have to resort to one of options presented above to improve bad query plans caused by inaccurate filtering estimates.

Common Pitfalls When Using database/sql in Go

$
0
0

Updated 3/23/2017

Here at VividCortex, we’re huge fans of the Go language and its database access library, database/sql. As you’ve probably seen firsthand, the surface area of database/sql is pretty small, but there’s a lot you can do with it. That includes plenty of risk for error and deceptive mistakes. This blog post is dedicated to some of the past mistakes we’ve made ourselves, in hopes that you won’t also make them when the time comes.


Pitfalls Img.jpg
Image Credit


Common Pitfalls

  • Deferring inside a loop. A long-lived function with a query inside a loop, and defer rows.Close() inside the loop, will cause both memory and connection usage to grow without bounds.
  • Opening many db objects. Make a global sql.DB, and don’t open a new one for, say, every incoming HTTP request your API server should respond to. Otherwise you’ll be opening and closing lots of TCP connections to the database. It’s a lot of latency, load, and TCP connections in TIME_WAIT status.
  • Not doing rows.Close() when done. Forgetting to close the rows variable means leaking connections. Combined with growing load on the server, this likely means running into max_connections errors or similar. Run rows.Close() as soon as you can, even if it’ll later be run again (it’s harmless). Chain db.QueryRow() and .Scan() together for the same reason.
  • Single-use prepared statements. If a prepared statement isn’t going to be used more than once, consider whether it makes sense to assemble the SQL with fmt.Sprintf() and avoid parameters and prepared statements. This could save two network round-trips, a lot of latency, and potentially wasted work. Update: Be wary, however, of any SQL injection vulnerabilities. An alternative to replacing prepared statements is https://golang.org/pkg/database/sql/#DB.Query.

Download and read our free ebook 
The Ultimate Guide to Building Database-Driven Apps with Go

  • Prepared statement bloat. If code will be run at high concurrency, consider whether prepared statements are the right solution, since they are likely to be reprepared multiple times on different connections when connections are busy.
  • Cluttering the code with strconv or casts. Scan into a variable of the type you want, and let .Scan() convert behind the scenes for you.
  • Cluttering the code with error-handling and retry. Let database/sql handle connection pooling, reconnecting, and retry logic for you.
  • Forgetting to check errors after rows.Next(). Don’t forget that the rows.Next() loop can exit abnormally.
  • Using db.Query() for non-SELECT queries. Don’t tell Go that you want to iterate over a result set if there won’t be one, or you’ll leak connections.
  • Assuming that subsequent statements use the same connection. Run two statements one after another and they’re likely to run on two different connections. Run LOCK TABLES tbl1 WRITE followed by SELECT * FROM tbl1 and you’re likely to block and wait. If you need a guarantee of a single statement being used, you need to use a sql.Tx.
  • Accessing the db while working with a tx. A sql.Tx is bound to a transaction, but the db is not, so access to it will not participate in the transaction.
  • Being surprised by a NULL. You can’t scan a NULL into a variable unless it is one of the NullXXX types provided by the database/sql package (or one of your own making, or provided by the driver). Examine your schema carefully, because if a column can be NULL, someday it will be, and what works in testing might blow up in production.
  • Passing a uint64 as a parameter. For some reason the Query(), QueryRow(), and Exec() methods don’t accept parameters of type uint64 with the most significant bit set. If you start out small and eventually your numbers get big, they could start failing unexpectedly. Convert them to strings with fmt.Sprint() to avoid this.

Watch the webinar
How Indexes Work in MySQL


What are some other pitfalls you've encountered when using database/sql in Go? Share them in the comments below.

Percona Server 5.6.35-81.0 is Now Available

$
0
0
Percona Server for MySQL 5.6.35-81.0

percona server 5.6.35-81.0Percona announces the release of Percona Server 5.6.35-81.0 on March 24, 2017. Download the latest version from the Percona web site or the Percona Software Repositories. You can also run Docker containers from the images in the Docker Hub repository.

Based on MySQL 5.6.35, and including all the bug fixes in it, Percona Server 5.6.35-81.0 is the current GA release in the Percona Server 5.6 series. Percona Server is open-source and free – this is the latest release of our enhanced, drop-in replacement for MySQL. Complete details of this release are available in the 5.6.35-81.0 milestone on Launchpad.

New Features:
  • Percona Server has implemented new mysqldump --order-by-primary-desc option. This feature tells mysqldump to take the backup by descending primary key order (PRIMARY KEY DESC) which can be useful if storage engine is using reverse order column for a primary key.
Bugs Fixed:
  • When innodb_ft_result_cache_limit was exceeded by internal memory allocated by InnoDB during the FT scan not all memory was released which could lead to server assertion. Bug fixed #1634932 (upstream #83648).
  • Log tracking initialization did not find last valid bitmap data correctly, potentially resulting in needless redo log retracking or hole in the tracked LSN range. Bug fixed #1658055.
  • If Audit Log Plugin was unable to create file pointed by audit_log_file, the server would crash during the startup. Bug fixed #1666496.
  • A DROP TEMPORARY TABLE ... for a table created by a CREATE TEMPORARY TABLE ... SELECT ... would get logged in the binary log on a disconnect with mixed mode replication. Bug fixed #1671013.
  • TokuDB did not use an index with even if cardinality was good. Bug fixed #1671152.
  • Row-based replication events were not reflected in Rows_updated fields in the User Statistics INFORMATION_SCHEMA tables. Bug fixed #995624.
  • A long-running binary log commit would block SHOW STATUS, which in turn could block a number of other operations such as client connects and disconnects. Bug fixed #1646100.
  • It was impossible to use column compression dictionaries with partitioned InnoDB tables. Bug fixed #1653104.
  • Diagnostics for OpenSSL errors have been improved. Bug fixed #1660339 (upstream #75311).
  • When DuplicateWeedout strategy was used for joins, use was not reported in the query plan info output extension for the slow query log. Bug fixed #1592694.

Other bugs fixed: #1650321, #1650322, #1654501, #1663251, #1666213, #1652912, #1659548, #1663452, #1670834, #1672871, #1626545, #1644174, #1658006, #1658021, #1659218, #1659746, #1660239, #1660243, #1660255, #1660348, #1662163 upstream (#81467), #1664219, #1664473, #1671076, and #1671123.

Release notes for Percona Server 5.6.35-81.0 are available in the online documentation. Please report any bugs on the launchpad bug tracker.

Group Replication GCS Troubleshooting

$
0
0
In the last post I shared the simple set of steps to configure a Group Replication setup using SQL commands, and a few in the configuration file too. Indeed, it can be simple.  But then there are times where there are more requirements and configurations need more attention. Maybe the OS environment we use for MySQL setups has never impacted us… Read More »

Percona Server for MySQL 5.7.17-12 is Now Available

$
0
0
Percona Server for MySQL 5.7.17-12

Percona Server for MySQL 5.7.17-12Percona announces the GA release of Percona Server for MySQL 5.7.17-12 on March 24, 2017. Download the latest version from the Percona web site or the Percona Software Repositories. You can also run Docker containers from the images in the Docker Hub repository.

Based on MySQL 5.7.17, including all the bug fixes in it, Percona Server for MySQL 5.7.17-12 is the current GA release in the Percona Server for MySQL 5.7 series. Percona’s provides completely open-source and free software. Find release details in the 5.7.17-12 milestone at Launchpad.

New Features:
  • Percona Server has implemented new mysqldump --order-by-primary-desc option. This feature tells mysqldump to take the backup by descending primary key order (PRIMARY KEY DESC) which can be useful if storage engine is using reverse order column family for a primary key.
  • mysqldump will now detect when MyRocks is installed and available by seeing if there is a session variable named rocksdb_skip_fill_cache and setting it to 1 if it exists.
  • mysqldump will now automatically enable session variable rocksdb_bulk_load if it is supported by the target server.
Bugs Fixed:
  • If the variable thread_handling was set to pool-of-threads in the MySQL configuration file, the server couldn’t be gracefully shut down. Bug fixed #1537554.
  • When innodb_ft_result_cache_limit was exceeded by internal memory allocated by InnoDB during the FT scan not all memory was released which could lead to server assertion. Bug fixed #1634932 (upstream #83648).
  • Executing the FLUSH LOGS on a read-only slave with a user that doesn’t have the SUPER privilege would result in Error 1290. Bug fixed #1652852 (upstream #84350).
  • FLUSH LOGS was disabled with read_only and super_read_only variables. Bug fixed #1654682 (upstream #84437).
  • If SHOW BINLOGS or PERFORMANCE_SCHEMA.GLOBAL_STATUS query, and a transaction commit would run in parallel, they could deadlock. Bug fixed #1657128.
  • A long-running binary log commit would block SHOW STATUS, which in turn could block a number of other operations such as client connects and disconnects. Bug fixed #1646100.
  • Log tracking initialization did not find last valid bitmap data correctly. Bug fixed #1658055.
  • A query using range scan with a complex range condition could lead to a server crash. Bug fixed #1660591 (upstream #84736).
  • Race condition between buffer pool page optimistic access and eviction could lead to a server crash. Bug fixed #1664280.
  • If Audit Log Plugin was unable to create file pointed by audit_log_file, the server would crash during the startup. Bug fixed #1666496.
  • A DROP TEMPORARY TABLE ... for a table created by a CREATE TEMPORARY TABLE ... SELECT ... would get logged in the binary log on a disconnect with mixed mode replication. Bug fixed #1671013.
  • TokuDB did not use an index with even if cardinality was good. Bug fixed #1671152.
  • Row-based replication events were not reflected in Rows_updated fields in the User Statistics INFORMATION_SCHEMA tables. Bug fixed #995624.
  • When DuplicateWeedout strategy was used for joins, use was not reported in the query plan info output extension for the slow query log. Bug fixed #1592694.
  • It was impossible to use column compression dictionaries with partitioned InnoDB tables. Bug fixed #1653104.
  • Diagnostics for OpenSSL errors have been improved. Bug fixed #1660339 (upstream #75311).

Other bugs fixed: #1665545, #1650321, #1654501, #1663251, #1659548, #1663452, #1670834, #1672871, #1626545, #1658006, #1658021, #1659218, #1659746, #1660239, #1660243, #1660348, #1662163 (upstream #81467), #1664219, #1664473, #1671076, and #1671123.

The release notes for Percona Server for MySQL 5.7.17-12 are available in the online documentation. Please report any bugs on the launchpad bug tracker.


MySQL 8.0 roles

$
0
0

One of the most interesting features introduced in MySQL 8.0 is roles or the ability of defining a set of privileges as a named role and then granting that set to one or more users. The main benefits are more clarity of privileges and ease of administration. Using roles we can assign the same set of privileges to several users, and eventually modify or revoke all privileges at once.

Roles nutshell 2

Roles in a nutshell

Looking at the manual, we see that using roles is a matter of several steps.

(1) Create a role. The statement is similar to CREATE USER though the effects are slightly different (we will see it in more detail later on.)

mysql [localhost] {root} ((none)) > CREATE ROLE r_lotr_dev;
Query OK, 0 rows affected (0.02 sec)

(2) Grant privileges to the role. Again, this looks like granting privileges to a user.

mysql [localhost] {root} ((none)) > GRANT ALL on lotr.* TO r_lotr_dev;
Query OK, 0 rows affected (0.01 sec)

(3) Create a user. This is the same that we've been doing until version 5.7.

mysql [localhost] {root} (mysql) > create user aragorn identified by 'lotrpwd';
Query OK, 0 rows affected (0.01 sec)

Notice that the role is in the mysql.user table, and looks a lot like a user.

mysql [localhost] {root} ((none)) > select host, user, authentication_string from mysql.user where user not like '%sandbox%;
+-----------+-------------+-------------------------------------------+
| host | user | authentication_string |
+-----------+-------------+-------------------------------------------+
| % | r_lotr_dev | |
| % | aragorn | *3A376D0203958F6EB9E6166DC048EC04F84C00B9 |
| localhost | mysql.sys | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE |
| localhost | root | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
+-----------+-------------+-------------------------------------------+

(4) Grant the role to the user. Instead of granting single privileges, we grant the role. Note that when we use this syntax we can't specify the ON xxx clause, because it is already implicit in the role definition.

mysql [localhost] {root} (mysql) > grant r_lotr_dev to aragorn;
Query OK, 0 rows affected (0.03 sec)

The relationship between user and role is recorded in a new table in the mysql database:

mysql [localhost] {root} (mysql) > select * from mysql.role_edges;
+-----------+------------+---------+---------+-------------------+
| FROM_HOST | FROM_USER | TO_HOST | TO_USER | WITH_ADMIN_OPTION |
+-----------+------------+---------+---------+-------------------+
| % | r_lotr_dev | % | aragorn | N |
+-----------+------------+---------+---------+-------------------+

(5) Finally we set the default role. Until this point, the role is assigned to the user, but not active. We either set the default role permanently (as done below) or we let the user activate the role.

mysql [localhost] {root} (mysql) > alter user aragorn default role r_lotr_dba;
Query OK, 0 rows affected (0.01 sec)

If a default role has been set, it is recorded in another new table, mysql.default_roles.

mysql [localhost] {root} (mysql) > select * from mysql.default_roles;
+------+---------+-------------------+-------------------+
| HOST | USER | DEFAULT_ROLE_HOST | DEFAULT_ROLE_USER |
+------+---------+-------------------+-------------------+
| % | aragorn | % | r_lotr_dba |
+------+---------+-------------------+-------------------+

Common gotchas

If we follow all the steps described above, using roles would not feel any different than using old style grants. But it is easy to go astray if we skip something. Let's see an example. First, we create an user and grant the same role as the one given to user aragorn:

mysql [localhost] {root} (mysql) > create user legolas identified by 'lotrpwd';
Query OK, 0 rows affected (0.03 sec)

mysql [localhost] {root} (mysql) > grant r_lotr_dev to legolas;
Query OK, 0 rows affected (0.01 sec)

Then we connect using user legolas:

mysql [localhost] {legolas} ((none)) > use lotr;
ERROR 1044 (42000): Access denied for user 'legolas'@'%' to database 'lotr'
mysql [localhost] {legolas} ((none)) > show grants;
+-----------------------------------------+
| Grants for legolas@% |
+-----------------------------------------+
| GRANT USAGE ON *.* TO `legolas`@`%` |
| GRANT `r_lotr_dev`@`%` TO `legolas`@`%` |
+-----------------------------------------+
2 rows in set (0.00 sec)

mysql [localhost] {legolas} ((none)) > show grants for legolas using r_lotr_dev;
+---------------------------------------------------+
| Grants for legolas@% |
+---------------------------------------------------+
| GRANT USAGE ON *.* TO `legolas`@`%` |
| GRANT ALL PRIVILEGES ON `lotr`.* TO `legolas`@`%` |
| GRANT `r_lotr_dev`@`%` TO `legolas`@`%` |
+---------------------------------------------------+
3 rows in set (0.00 sec)

The role is assigned to the user, but it is not active, as no default role was defined. To use a role, the user must activate one:

mysql [localhost] {legolas} ((none)) > select current_role();
+----------------+
| current_role() |
+----------------+
| NONE |
+----------------+
1 row in set (0.01 sec)

There is no default role for legolas. We need to assign one.

mysql [localhost] {legolas} ((none)) > set role r_lotr_dev;
Query OK, 0 rows affected (0.00 sec)

mysql [localhost] {legolas} ((none)) > select current_role();
+------------------+
| current_role() |
+------------------+
| `r_lotr_dev`@`%` |
+------------------+
1 row in set (0.00 sec)

Now the role is active, and all its privileges kick in:

mysql [localhost] {legolas} ((none)) > use lotr
Database changed
mysql [localhost] {legolas} (lotr) > show tables;
Empty set (0.00 sec)

mysql [localhost] {legolas} (lotr) > create table t1 (i int not null primary key);
Query OK, 0 rows affected (0.15 sec)

Note that the role activation is volatile. If the user reconnects, the role activation goes away:

mysql [localhost] {legolas} ((none)) > connect;
Connection id: 33
Current database: *** NONE ***

mysql [localhost] {legolas} ((none)) > select current_role();
+----------------+
| current_role() |
+----------------+
| NONE |
+----------------+
1 row in set (0.01 sec)

For a permanent assignment, the user can use the SET DEFAULT ROLE statement:

mysql [localhost] {legolas} ((none)) > set default role r_lotr_dev to legolas;
Query OK, 0 rows affected (0.01 sec)

The above statement corresponds to ALTER USER ... DEFAULT ROLE .... Every user can set its own role with this statement, without ending additional privileges.

mysql [localhost] {legolas} ((none)) > alter user legolas default role r_lotr_dev;
Query OK, 0 rows affected (0.01 sec)

Now if the user reconnects, the role persists:

mysql [localhost] {legolas} ((none)) > connect
Connection id: 34
Current database: *** NONE ***

mysql [localhost] {legolas} ((none)) > select current_role();
+------------------+
| current_role() |
+------------------+
| `r_lotr_dev`@`%` |
+------------------+
1 row in set (0.01 sec)

However, if an user sets its own default role using ALTER USER, the change will be available only in the next session, or after calling SET ROLE.

Let's try:

mysql [localhost] {legolas} ((none)) > set default role none to legolas;
Query OK, 0 rows affected (0.02 sec)

mysql [localhost] {legolas} ((none)) > select current_role();
+------------------+
| current_role() |
+------------------+
| `r_lotr_dev`@`%` |
+------------------+
1 row in set (0.00 sec)

The role stays what it was before. This is similar to what happens when using SET GLOBAL var_name vs SET SESSION var_name. In the first case the effect persists, but it is not activated immediately, while a session set will be immediately usable, but will not persist after a new connection.

mysql [localhost] {legolas} ((none)) > connect
Connection id: 35
Current database: *** NONE ***

select current_role();
+----------------+
| current_role() |
+----------------+
| NONE |
+----------------+
1 row in set (0.00 sec)

It's worth mentioning that SET DEFAULT ROLE implies an hidden update of a mysql table (default_roles), similar to what happens with SET PASSWORD. In both cases, a user without explicit access to the mysql database will be unable to check the effects of the operation.

Advanced role management.

Dealing with one or two roles is no big deal. Using the statements seen above, we can easily see what privileges were granted to a role or an user. When we have many roles and many users, the overview become more difficult to achieve.

Before we see the complex scenario, let's have a deeper look at what constitutes a role.

mysql [localhost] {root} (mysql) > create role role1;
Query OK, 0 rows affected (0.05 sec)

mysql [localhost] {root} (mysql) > create user user1 identified by 'somepass';
Query OK, 0 rows affected (0.01 sec)

mysql [localhost] {root} (mysql) > select host, user, authentication_string , password_expired , account_locked from user where user in ('role1', 'user1');
+------+-------+-------------------------------------------+------------------+----------------+
| host | user | authentication_string | password_expired | account_locked |
+------+-------+-------------------------------------------+------------------+----------------+
| % | role1 | | Y | Y |
| % | user1 | *13883BDDBE566ECECC0501CDE9B293303116521A | N | N |
+------+-------+-------------------------------------------+------------------+----------------+
2 rows in set (0.00 sec)

The main difference between user and role is that a role is created with password_expired and account_locked. Apart from that, an user could be used as a role and vice versa.

mysql [localhost] {root} (mysql) > alter user role1 identified by 'msandbox';
Query OK, 0 rows affected (0.01 sec)

mysql [localhost] {root} ((none)) > alter user role1 account unlock;
Query OK, 0 rows affected (0.02 sec)

Now role1 can access the database as any other user.

mysql [localhost] {root} ((none)) > grant root@'localhost' to user1;
Query OK, 0 rows affected (0.03 sec)

And user1 inherits all privileges from root, but it can access the server from any host.

Now let's see some complex usage of roles. In a typical organisation, we would define several roles to use the lotr database:

CREATE ROLE r_lotr_observer;
CREATE ROLE r_lotr_tester;
CREATE ROLE r_lotr_dev;
CREATE ROLE r_lotr_dba;

GRANT SELECT on lotr.* TO r_lotr_observer;
GRANT SELECT, INSERT, UPDATE, DELETE on lotr.* TO r_lotr_tester;
GRANT ALL on lotr.* TO r_lotr_dev;
GRANT ALL on *.* TO r_lotr_dba;

And then assign those roles to several users:

CREATE USER bilbo     IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER frodo IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER sam IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER pippin IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER merry IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER boromir IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER gimli IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER aragorn IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER legolas IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER gollum IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER galadriel IDENTIFIED BY 'msandbox' PASSWORD EXPIRE;
CREATE USER gandalf IDENTIFIED BY 'msandbox';

GRANT r_lotr_observer TO pippin, merry, boromir, gollum;
SET DEFAULT ROLE r_lotr_observer to pippin, merry, boromir, gollum;

GRANT r_lotr_tester TO sam, bilbo, gimli;
SET DEFAULT ROLE r_lotr_tester to sam, bilbo, gimli;

GRANT r_lotr_dev to frodo, aragorn, legolas;
SET DEFAULT ROLE r_lotr_dev to frodo, aragorn, legolas;

GRANT r_lotr_dba TO gandalf, galadriel;
SET DEFAULT ROLE r_lotr_dba to gandalf, galadriel;

Now we have 12 users with 4 different roles. Looking at the user table, we don't get a good overview:

mysql [localhost] {root} (mysql) > select host, user, authentication_string from mysql.user where user not like '%sandbox%';
+-----------+-----------------+-------------------------------------------+
| host | user | authentication_string |
+-----------+-----------------+-------------------------------------------+
| % | aragorn | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | bilbo | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | boromir | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | frodo | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | galadriel | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | gandalf | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | gimli | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | gollum | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | legolas | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | merry | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | pippin | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| % | r_lotr_dba | |
| % | r_lotr_dev | |
| % | r_lotr_observer | |
| % | r_lotr_tester | |
| % | sam | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
| localhost | mysql.sys | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE |
| localhost | root | *6C387FC3893DBA1E3BA155E74754DA6682D04747 |
+-----------+-----------------+-------------------------------------------+

And even the roles_edges table does show a clear picture:

mysql [localhost] {root} (mysql) > select * from role_edges;
+-----------+-----------------+---------+-----------+-------------------+
| FROM_HOST | FROM_USER | TO_HOST | TO_USER | WITH_ADMIN_OPTION |
+-----------+-----------------+---------+-----------+-------------------+
| % | r_lotr_dba | % | galadriel | N |
| % | r_lotr_dba | % | gandalf | N |
| % | r_lotr_dev | % | aragorn | N |
| % | r_lotr_dev | % | frodo | N |
| % | r_lotr_dev | % | legolas | N |
| % | r_lotr_observer | % | boromir | N |
| % | r_lotr_observer | % | gollum | N |
| % | r_lotr_observer | % | merry | N |
| % | r_lotr_observer | % | pippin | N |
| % | r_lotr_tester | % | bilbo | N |
| % | r_lotr_tester | % | gimli | N |
| % | r_lotr_tester | % | sam | N |
+-----------+-----------------+---------+-----------+-------------------+

Here's a better use of that table's data. Which users are using the dev role?

select to_user as users from role_edges where from_user = 'r_lotr_dev';
+---------+
| users |
+---------+
| aragorn |
| frodo |
| legolas |
+---------+
3 rows in set (0.00 sec)

And the testers?

select to_user as users from role_edges where from_user = 'r_lotr_tester';
+-------+
| users |
+-------+
| bilbo |
| gimli |
| sam |
+-------+
3 rows in set (0.00 sec)

Or, even better, we could see all the roles at once:

select from_user as role, count(*) as how_many_users, group_concat(to_user) as users from role_edges group by role;
+-----------------+----------------+-----------------------------+
| role | how_many_users | users |
+-----------------+----------------+-----------------------------+
| r_lotr_dba | 2 | galadriel,gandalf |
| r_lotr_dev | 3 | aragorn,frodo,legolas |
| r_lotr_observer | 4 | boromir,gollum,merry,pippin |
| r_lotr_tester | 3 | bilbo,gimli,sam |
+-----------------+----------------+-----------------------------+
4 rows in set (0.01 sec)

Similarly, we could list the default role for several users:

select default_role_user as default_role, group_concat(user) as users from default_roles group by default_role;
+-----------------+-----------------------------+
| default_role | users |
+-----------------+-----------------------------+
| r_lotr_dba | galadriel,gandalf |
| r_lotr_dev | aragorn,frodo,legolas |
| r_lotr_observer | boromir,gollum,merry,pippin |
| r_lotr_tester | bilbo,gimli,sam |
+-----------------+-----------------------------+

The latest two queries should be good candidates for information_schema views.

Another candidate for an information_schema view is the list of roles, for which there is no satisfactory workaround now. The best we could think of is a list of users with password_expired and account_locked:

select host, user from mysql.user where password_expired='y' and account_locked='y';
+------+-----------------+
| host | user |
+------+-----------------+
| % | r_lotr_dba |
| % | r_lotr_dev |
| % | r_lotr_observer |
| % | r_lotr_tester |
+------+-----------------+
4 rows in set (0.00 sec)

Until a feature to differentiate users and roles is developed, it is advisable to use a name format that helps identify roles without help from the server. In this article I am using r_ as a prefix, which makes role listing easier.

mysql [localhost] {root} (mysql) > select host, user from mysql.user where user like 'r_%';
+------+-----------------+
| host | user |
+------+-----------------+
| % | r_lotr_dba |
| % | r_lotr_dev |
| % | r_lotr_observer |
| % | r_lotr_tester |
+------+-----------------+
4 rows in set (0.00 sec)

Known bugs

  • bug#85562 Dropping a role does not remove privileges from active users.
  • bug#85561 Users can be assigned non-existing roles as default.
  • bug#85559 Dropping a role does not remove the associated default roles.
  • bug#84244 Distinguish roles and plain users in mysql.user.
  • bug#82987 SHOW CREATE USER doesn't show default role.

Summing up

Roles are a great addition to the MySQL feature set. The usability could be improved with some views in information_schema (usersforrole, defaultroleforuser) and functions in sys schema (defaultroleforuser, is_role.)

Perhaps some commands to activate or deactivate roles could make administration easier. The current bugs don't affect the basic functionality but decrease usability. I think that another run of bug fixing and user feedback would significantly improve this feature.

More about roles in my talk at PerconaLive 2017.


Fun with Bugs #51 - My Bug Reports that Oracle doesn't Want to Fix

$
0
0
This week I noticed (yet another) customer issue related to the output produced by mysqladmin debug command (or when mysqld process gets SIGHUP signal). I mean the output generated by the mysql_print_status() function. In this issue the content of the output was misinterpreted. I've seen this in the past more than once, and requested to document the output properly, but it never happened for a reason that there is an internal feature request to put this information elsewhere, in Performance Schema or Information Schema. The bug ended up with "Won't fix" status.

Surely I complained in a comment and on Facebook, and then decided to check if there are any other my bug reports and documentation request that Oracle explicitly decided not to fix after accepting the fact that there is a problem.

I've ended up with the following short list:
  • Bug #69399 - "Inconsistency in crash report". Here I've got a perfect reason to keep things as they are currently implemented. Functions called from signal handlers must be async signal safe, and time() is like that, but it always outputs in UTC. It would be great to print time in UTC in some messages as well, so that timezone difference is 100% clear, but it's truly not a big deal.
  • Bug #71300 - "Manual does not explain that statement/abstract/* instruments appeared in 5.6.15". This change in naming that happened in 5.6.15 is, indeed, explained in the manual, even if there is no highlighted statements about incompatible change etc. I can live with that.
  • Bug #71303 - "Manual page on P_S build configuration does not provide enough details". I really missed the details at that time on how to instrument individual buffer's mutexes/rwlocks, after getting a hint during my talk that I had no chance to see real most important waits in Bug #68079  with Performance Schema without recompiling it properly. I've got a useful comment at the end of the bug report, but I truly do not understand why this detail ("The only way to enable it is by removing the line which defines PFS_SKIP_BUFFER_MUTEX_RWLOCK in storage/innobase/include/sync0sync.h. Seems to be no compiler flags to enable or disable the above mentioned symbol.") was not added to the small enough page as a note.
  • Bug #71304 - "Manual does not provide enough details about automatic sizing of P_S parameters ". Here my suggestions were refused. Go figure yourself, check the output of mysqld --verbose --help 2>/dev/null | grep performance | grep "\-1" to find out what parameters are auto-sized and go read the code to find out how exactly, if you care. They don't.
  • Bug #71274 - "Manual does not provide enough details about background threads in P_S.threads". All I've got in reply is: "The purpose of the page is to describe the table structure, not enumerate the (subject to change) set of background threads." You may be satisfied with this remark, but I am not.
  • Bug #71346 - "Manual does not provide details on mysqladmin debug output". As you can check here, even for MySQL 8 the command is still there, but all we have about the output is: "Tell the server to write debug information to the error log. Format and content of this information is subject to change. This includes information about the Event Scheduler."
  • Bug #75366 - "mysql_install_db requires explicit --lc-messages-dir if non-default PREFIX used". This was reported at early MySQL 5.7.x development stage, and I've got a recommendation to use mysqld --initialize instead. I do so now, but sometimes some related problem still happen, see Bug #84173 and Bug #80351. I think that even deprecated commands must be properly documented, including any incompatible changes in behavior, options, binaries location etc, until they still exist in GA and supported versions of MySQL.
  • Bug #78822 - "Materialized table from semijoin may change join order and lead to bad plan". In MySQL 5.7 the problem must be fixed, and for 5.6 the following obvious workaround was suggested: set optimizer_switch="semijoin=off";
  • Bug #80601 - "Manual is wrong/not clear while explaining kill flag check for ALTER TABLE".  Even though it was stated that "this is an implementation detail subject to change", some clarifications happened in the manual.
To summarize, out of 310 bug reports I've created since 2005, Oracle decided not to fix just 9, and in many cases provided proper explanations about the reasons to do this, or made some changes in the manual. The remaining cases all are related to MySQL manual and mostly happened in 2014, when nice people found a way to shut me up (temporary) on the topic of MySQL bugs...

Video: 9 Tips to Building a Stable MySQL Replication Environment

$
0
0

This video walks you through nine tips you should consider when deploying and managing a MySQL Replication environment and how utilizing ClusterControl can help make deploying, managing, monitoring, and scaling MySQL easy.

Though MySQL replication is one of the easier HA deployments to set up, it is also easy to break and time-consuming to troubleshoot.  ClusterControl’s point-and-click interface lets you securely automate deployment and scaling of MySQL replication setups to get production-ready environments up and running in a very short time. No need for guesswork, or time-consuming experimentation with different open source utilities.

ClusterControl and MySQL Replication

ClusterControl provides advanced deployment, management, monitoring, and scaling functionality to get your MySQL instances up-and-running using proven methodologies that you can depend on to work.  ClusterControl makes MySQL replication easy and secure with point-and click interfaces and no need to have specialized knowledge about the technology or multiple tools. It covers all aspects one might expect for a production-ready replication setup.

Ongoing maintenance and troubleshooting MySQL Replication is easier with ClusterControl because it removes the complexity that is often introduced when using multiple external tools and ClusterControl lets you monitor all your MySQL deployments from a single interface.

To learn more check out the following resources…

Tags: 

Test a Flask App with Selenium WebDriver – Part 2

$
0
0

This is the second and final part of a tutorial on how to test a Python/Flask web app with Selenium webdriver. We are testing Project Dream Team, an existing CRUD web app. Part One introduced Selenium WebDriver as a web browser automation tool for browser-based tests. By the end of Part One, we had written tests for registration, login, performing CRUD operations on departments and roles, as well as assigning departments and roles to employees.

In Part Two, we will write tests to ensure that protected pages can only be accessed by authorised users. We will also integrate our app with CircleCI, a continuous integration and delivery platform. I have included a demo video showing all the tests running, so be sure to check it out!

Permissions Tests

Recall that in the Dream Team app, there are two kinds of users: regular users, who can only register and login as employees, and admin users, who can access departments and roles and assign them to employees. Non-admin users should not be able to access the departments, roles, and employees pages. We will therefore write tests to ensure that this is the case.

In your tests/test_front_end.py file, add the following code:

# tests/test_front_end.py

class TestPermissions(CreateObjects, TestBase):

    def test_permissions_admin_dashboard(self):
        """
        Test that non-admin users cannot access the admin dashboard
        """
        # Login as non-admin user
        self.login_test_user()

        # Navigate to admin dashboard
        target_url = self.get_server_url() + url_for('home.admin_dashboard')
        self.driver.get(target_url)

        # Assert 403 error page is shown
        error_title = self.driver.find_element_by_css_selector("h1").text
        self.assertEqual("403 Error", error_title)
        error_text = self.driver.find_element_by_css_selector("h3").text
        assert "You do not have sufficient permissions" in error_text

    def test_permissions_list_departments_page(self):
        """
        Test that non-admin users cannot access the list departments page
        """
        # Login as non-admin user
        self.login_test_user()

        # Navigate to admin dashboard
        target_url = self.get_server_url() + url_for('admin.list_departments')
        self.driver.get(target_url)

        # Assert 403 error page is shown
        error_title = self.driver.find_element_by_css_selector("h1").text
        self.assertEqual("403 Error", error_title)
        error_text = self.driver.find_element_by_css_selector("h3").text
        assert "You do not have sufficient permissions" in error_text

    def test_permissions_add_department_page(self):
        """
        Test that non-admin users cannot access the add department page
        """
        # Login as non-admin user
        self.login_test_user()

        # Navigate to admin dashboard
        target_url = self.get_server_url() + url_for('admin.add_department')
        self.driver.get(target_url)

        # Assert 403 error page is shown
        error_title = self.driver.find_element_by_css_selector("h1").text
        self.assertEqual("403 Error", error_title)
        error_text = self.driver.find_element_by_css_selector("h3").text
        assert "You do not have sufficient permissions" in error_text

    def test_permissions_list_roles_page(self):
        """
        Test that non-admin users cannot access the list roles page
        """
        # Login as non-admin user
        self.login_test_user()

        # Navigate to admin dashboard
        target_url = self.get_server_url() + url_for('admin.list_roles')
        self.driver.get(target_url)

        # Assert 403 error page is shown
        error_title = self.driver.find_element_by_css_selector("h1").text
        self.assertEqual("403 Error", error_title)
        error_text = self.driver.find_element_by_css_selector("h3").text
        assert "You do not have sufficient permissions" in error_text

    def test_permissions_add_role_page(self):
        """
        Test that non-admin users cannot access the add role page
        """
        # Login as non-admin user
        self.login_test_user()

        # Navigate to admin dashboard
        target_url = self.get_server_url() + url_for('admin.add_role')
        self.driver.get(target_url)

        # Assert 403 error page is shown
        error_title = self.driver.find_element_by_css_selector("h1").text
        self.assertEqual("403 Error", error_title)
        error_text = self.driver.find_element_by_css_selector("h3").text
        assert "You do not have sufficient permissions" in error_text

    def test_permissions_list_employees_page(self):
        """
        Test that non-admin users cannot access the list employees page
        """
        # Login as non-admin user
        self.login_test_user()

        # Navigate to admin dashboard
        target_url = self.get_server_url() + url_for('admin.list_employees')
        self.driver.get(target_url)

        # Assert 403 error page is shown
        error_title = self.driver.find_element_by_css_selector("h1").text
        self.assertEqual("403 Error", error_title)
        error_text = self.driver.find_element_by_css_selector("h3").text
        assert "You do not have sufficient permissions" in error_text

    def test_permissions_assign_employee_page(self):
        """
        Test that non-admin users cannot access the assign employee page
        """
        # Login as non-admin user
        self.login_test_user()

        # Navigate to admin dashboard
        target_url = self.get_server_url() + url_for('admin.assign_employee', id=1)
        self.driver.get(target_url)

        # Assert 403 error page is shown
        error_title = self.driver.find_element_by_css_selector("h1").text
        self.assertEqual("403 Error", error_title)
        error_text = self.driver.find_element_by_css_selector("h3").text
        assert "You do not have sufficient permissions" in error_text

We begin by creating a class TestPermissions, which inherits from the CreateObjects and TestBase classes that we wrote in Part One. In each of the test methods inside the class, we login as a non-admin user, and then attempt to access a protected page. First, we test the departments pages (list and add), then the roles pages (list and add), and finally the employees pages (list and assign). In each method, we test that the 403 error page is shown by asserting that the appropriate page title ("403 Error") and text ("You do not have sufficient permissions to access this page") are shown on the page.

Take note of the difference between the assertEqual method and the assert ... in statement. The former checks that two things are exactly the same, whereas the latter checks that the first thing is contained in the second. In the case of our tests, "403 Error" and the error page title are exactly the same, so we can use assertEqual. For the second assertion, we are merely checking that the words "You do not have sufficient permissions" are contained in the error page text. The assert ... in statement is ideal when you don't want to check for identicalness, but rather that a certain important word or phrase is contained in the element in question.

Let's run our tests now:

$ nose2
......................................
----------------------------------------------------------------------
Ran 38 tests in 168.981s

OK

Continuous Integration and Continuous Delivery

You may have heard of continuous integration (CI), but you may not be very clear on what exactly it is or how to implement it in your development workflow. Well, CI refers to a software development pratice of integrating project code into a shared repository frequently, typically multiple times a day. CI usually goes hand-in-hand with automated builds and automated testing, such that each time code is pushed into the shared repo, the code is run and tested automatically to ensure it has no errors.

The idea is that small changes in the code are integrated to the main repo frequently, which makes it easier to catch errors should they occur and troubleshoot them. This is in contrast to a scenario where integration is done less often and with more code, making it more difficult to detect which change was responsible if an error was to occur.

Martin Fowler, Chief Scientist at ThoughtWorks, put it well when he said:

Continuous Integration doesn’t get rid of bugs, but it does make them dramatically easier to find and remove.

Continuous delivery entails building and handling your code in such a way that it can be released into production at any time. Practising continuous delivery means not having any code in your main repo that you wouldn't want to deploy. Sometimes, this even means that any code that is pushed to the main repo is automatically put in production if the build is successful and all tests pass. This is called continuous deployment.

Introducing CircleCI

Now that you're up to speed with continuous integration and continuous delivery, let's get familiar with one of the most popular continuous integration and delivery platforms today: CircleCI. CircleCI is quick and easy to set up. It automates software builds and testing, and also supports pushing code to many popular hosts such as Heroku and Google Cloud Platform.

To start using CircleCI, sign up by authenticating your GitHub or Bitbucket account. Once you login, navigate to the Projects page where you can add your project repository. Select Build Project next to your repository name, and CircleCI will start the build.

Uh oh! The first build fails. You'll notice the disconcerting red colour all over the page, the multiple error messages, and even the disheartening red favicon in your browser, all of which denote failure. First of all, congratulations on your first failed build! :) Secondly, don't worry; we haven't configured CircleCI or our app yet, so it's no wonder the build failed! Let's get to work setting things up to turn the red to green.

Environment Variables

We'll start by adding some important environment variables to CircleCI. Because we won't be reading from the instance/config.py file, we'll need to add those variables to CircleCI. On the top right of the build page on CircleCI, click the cog icon to access the Project Settings. In the menu on the left under Build Settings, click on Environment Variables. You can now go ahead and add the following variables:

  1. SECRET_KEY. You can copy this from your instance/config.py file.

  1. SQLALCHEMY_DATABASE_URI. We will use CircleCI's default circle_test database and ubuntu user, so our SQLALCHEMY_DATABASE_URI will be mysql://ubuntu@localhost/circle_test.

You should now have all two environment variables:

The circle.yml File

Next, create a circle.yml file in your root folder and in it, add the following:

machine:
  python:
    version: 2.7.10
test:
  override:
    - nose2 

We begin by indicating the Python version for our project, 2.7.10. We then tell CircleCI to run our tests using the nose2 command. Note that we don't need to explicitly tell CircleCI to install the software dependencies because it automatically detects the requirements.txt file in Python projects and installs the requirements.

The create_app Method

Next, edit the create_app method in the app/__init__.py file as follows:

# app/__init__.py

def create_app(config_name):
    # modify the if statement to include the CIRCLECI environment variable
    if os.getenv('FLASK_CONFIG') == "production":
        app = Flask(__name__)
        app.config.update(
            SECRET_KEY=os.getenv('SECRET_KEY'),
            SQLALCHEMY_DATABASE_URI=os.getenv('SQLALCHEMY_DATABASE_URI')
        )
    elif os.getenv('CIRCLECI'):
        app = Flask(__name__)
        app.config.update(
            SECRET_KEY=os.getenv('SECRET_KEY')
        )
    else:
        app = Flask(__name__, instance_relative_config=True)
        app.config.from_object(app_config[config_name])
        app.config.from_pyfile('config.py')

This checks for CircleCI's built-in CIRCLECI environment variable, which returns True when on CircleCI. This way, when running the tests on CircleCI, Flask will not load from the instance/config.py file, and will instead get the value of the SECRET_KEY configuration variable from the environment variable we set earlier.

The Test Files

Now edit the create_app method in the tests/test_front_end.py file as follows:

# tests/test_front_end.py

# update imports
import os

class TestBase(LiveServerTestCase):

    def create_app(self):
        config_name = 'testing'
        app = create_app(config_name)
        if os.getenv('CIRCLECI'):
            database_uri = os.getenv('SQLALCHEMY_DATABASE_URI')
        else:
            database_uri = 'mysql://dt_admin:dt2016@localhost/dreamteam_test',
        app.config.update(
            # Specify the test database
            SQLALCHEMY_DATABASE_URI=database_uri,
            # Change the port that the liveserver listens on
            LIVESERVER_PORT=8943
        )
        return app

This ensures that when the tests are running on CircleCI, Flask will get the SQLALCHEMY_DATABASE_URI from the environment variable we set earlier rather than using the test database we have locally.

Finally, do the same for the create_app method in the tests/test_back_end.py file:

# tests/test_back_end.py

# update imports
import os

class TestBase(TestCase):

    def create_app(self):
        config_name = 'testing'
        app = create_app(config_name)
        if os.getenv('CIRCLECI'):
            database_uri = os.getenv('SQLALCHEMY_DATABASE_URI')
        else:
            database_uri = 'mysql://dt_admin:dt2016@localhost/dreamteam_test',
        app.config.update(
            # Specify the test database
            SQLALCHEMY_DATABASE_URI=database_uri
        )
        return app

Push your changes to your repository. You'll notice that as soon as you push your code, CircleCI will automatically rebuild the project. It'll take a few minutes, but the build should be successful this time. Good job!

Status Badge

CircleCI porvides a status badge for use on your project repository or website to display your build status. To get your badge, click on the Status Badges link in the menu on the left under Notifications. You can get the status badge in a variety of formats, including image and MarkDown.

Conclusion

You are now able to write a variety of front-end tests for a Flask application with Selenium WebDriver. You also have a good understanding of continuous integration and continuous delivery, and can set up a project on CircleCI. I hope you've enjoyed this tutorial! I look forwrad to hearing your feedback and experiences in the comment section below.

For more information on continuous integration in Python with CircleCI, you may refer to this Scotch tutorial by Elizabeth Mabishi.

Howto Encrypt MySQL Backups on S3

$
0
0

TwinDB Backup supports encrypted backup copies since version 2.11.0. As usual the tool supports natively backup and restore operations, if backup copies are encrypted the tool takes care of decryption.

Installing TwinDB Packages repository

I will work with CentOS 7 system to show the example, but there are also packages for Ubuntu trusty and Debian jessie.

We host our packages in PackageCloud which provides a great installation guide if you need to install the repo via puppet, chef etc. The manual way is pretty straightforward as well. A PackageCloud script installs and configures the repository.

curl -s https://packagecloud.io/install/repositories/twindb/main/script.rpm.sh | sudo bash

Installing twindb-backup

Once the repository is ready it’s time to install the tool.

yum install twindb-backup

Let’s review what files the tool actually installs.

# rpm -ql twindb-backup
/opt
/opt/twindb-backup
...
/opt/twindb-backup/bin
...
/opt/twindb-backup/bin/twindb-backup
...

The RPM installs the files in opt because we use OmniBus to package twindb-backup. We package with the tool itself its own python, dependencies. That way we make sure there are no conflicts, no surprises due to different modules versions etc.

The post installation script also creates a cron config and a sample tool configuration file.

# cat /etc/cron.d/twindb-backup
@hourly  root twindb-backup backup hourly
@daily   root twindb-backup backup daily
@weekly  root twindb-backup backup weekly
@monthly root twindb-backup backup monthly
@yearly  root twindb-backup backup yearly

# cat /etc/twindb/twindb-backup.cfg
# NOTE: don't quote option values
# What to backup
[source]
backup_dirs=/etc /root /home
backup_mysql=no

# Destination
[destination]
# backup destination can be ssh or s3
backup_destination=ssh
keep_local_path=/var/backup/local


[s3]

# S3 destination settings

AWS_ACCESS_KEY_ID=XXXXX
AWS_SECRET_ACCESS_KEY=YYYYY
AWS_DEFAULT_REGION=us-east-1
BUCKET=twindb-backups

[ssh]

# SSH destination settings

backup_host=127.0.0.1
backup_dir=/tmp/backup
ssh_user=root
ssh_key=/root/.ssh/id_rsa

[mysql]

# MySQL

mysql_defaults_file=/etc/twindb/my.cnf

full_backup=daily

[retention]

# Remote retention policy

hourly_copies=24
daily_copies=7
weekly_copies=4
monthly_copies=12
yearly_copies=3

[retention_local]

# Local retention policy

hourly_copies=1
daily_copies=1
weekly_copies=0
monthly_copies=0
yearly_copies=0

[intervals]

# Run intervals

run_hourly=yes
run_daily=yes
run_weekly=yes
run_monthly=yes
run_yearly=yes

Preparing Encryption Key

We use GPG to encrypt the backups. The tool doesn’t manage the keys so it’s all user responsibility to create and save a backup copy of the key.

Let’s generate the key first.

# gpg --gen-key
gpg (GnuPG) 2.0.22; Copyright (C) 2013 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: directory `/root/.gnupg' created
gpg: new configuration file `/root/.gnupg/gpg.conf' created
gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/root/.gnupg/secring.gpg' created
gpg: keyring `/root/.gnupg/pubring.gpg' created
Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is 2048 bits
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y

GnuPG needs to construct a user ID to identify your key.

Real name: Aleksandr Kuzminsky
Email address: backups@twindb.com
Comment: Key for encrypting MySQL backups
You selected this USER-ID:
"Aleksandr Kuzminsky (Key for encrypting MySQL backups) <backups@twindb.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O

You don't want a passphrase - this is probably a *bad* idea!
I will do it anyway. You can change your passphrase at any time,
using this program with the option "--edit-key".

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 8564B88A marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
pub 2048R/8564B88A 2017-03-28
Key fingerprint = 441E 4B7A FD92 C0D5 4C6B 0C89 4AE0 849C 8564 B88A
uid Aleksandr Kuzminsky (Key for encrypting MySQL backups) <backups@twindb.com>
sub 2048R/0CE02576 2017-03-28

We don’t use passphrase for the key.

Preparing twindb-backup configuration

We need to change default config. Let’s review the changes.

[source]
backup_dirs=/etc
backup_mysql=yes

It’s always nice to save backup copies of /etc. If you don’t want to backup directories, comment out backup_dirs.

# Destination
[destination]
# backup destination can be ssh or s3
backup_destination=s3
keep_local_path=/var/backup/local

We store backups in s3 and we will also keep a local copy (for faster restore time).

[s3]

# S3 destination settings

AWS_ACCESS_KEY_ID=XXXXX
AWS_SECRET_ACCESS_KEY=YYYYY
AWS_DEFAULT_REGION=us-east-1
BUCKET=twindb-backups

We will store backups in S3, so change these options to your key and bucket values.

[mysql]

# MySQL
mysql_defaults_file=/etc/twindb/my.cnf
full_backup=daily

The tool uses a defaults file to connect to MySQL, so specify it here.

# cat /etc/twindb/my.cnf
[client]
user=root

Don’t forget to chmod 600 /etc/twindb/my.cnf.

The config also tells how often to take daily full copies. The hourly copies will be the difference between the last full copy and the current state. It’s so-called differential backups.

To encrypt the backup copies add a [gpg] section

[gpg]
keyring = /root/.gnupg/pubring.gpg
secret-keyring = /root/.gnupg/secring.gpg
recipient = backups@twindb.com

It specifies where GnuPG can find private and public keys of the recipient.

Optionally you may want to change local and remote retention policies, but the defaults should be good enough.

Test backup run

Now let’s run the tool manually to see how it works.

# twindb-backup backup daily

The tool should produce no output unless there is an error.

Listing available backup copies

The tool can tell you what backup copies are available now.

# twindb-backup ls
2017-03-28 05:32:40,412: INFO: ls.list_available_backups():22: Local copies:
/var/backup/local/d312b5e3a877/status
/var/backup/local/d312b5e3a877/daily/files/_etc-2017-03-28_05_32_26.tar.gz
/var/backup/local/d312b5e3a877/daily/mysql/mysql-2017-03-28_05_32_30.xbstream.gz
2017-03-28 05:32:40,417: INFO: ls.list_available_backups():33: hourly copies:
2017-03-28 05:32:41,087: INFO: ls.list_available_backups():33: daily copies:
s3://twindb-backup-test-0/d312b5e3a877/daily/files/_etc-2017-03-28_05_32_26.tar.gz.gpg
s3://twindb-backup-test-0/d312b5e3a877/daily/mysql/mysql-2017-03-28_05_32_30.xbstream.gz.gpg
2017-03-28 05:32:41,687: INFO: ls.list_available_backups():33: weekly copies:
2017-03-28 05:32:42,269: INFO: ls.list_available_backups():33: monthly copies:
2017-03-28 05:32:42,831: INFO: ls.list_available_backups():33: yearly copies:

The encrypted copies have .gpg suffix. Note the local copies are not encrypted.

Restore MySQL from backup

Now we have a backup copy s3://twindb-backup-test-0/d312b5e3a877/daily/mysql/mysql-2017-03-28_05_32_30.xbstream.gz.gpg. Let’s restore MySQL database from it.

# twindb-backup restore mysql s3://twindb-backup-test-0/d312b5e3a877/daily/mysql/mysql-2017-03-28_05_32_30.xbstream.gz.gpg --dst restored
...
170328 05:39:49  innobackupex: completed OK!
2017-03-28 05:39:49,566: INFO: restore.restore_from_mysql():354: Successfully restored s3://twindb-backup-test-0/d312b5e3a877/daily/mysql/mysql-2017-03-28_05_32_30.xbstream.gz.gpg in restored.
2017-03-28 05:39:49,566: INFO: restore.restore_from_mysql():356: Now copy content of restored to MySQL datadir: cp -R restored/* /var/lib/mysql/
2017-03-28 05:39:49,566: INFO: restore.restore_from_mysql():357: Fix permissions: chown -R mysql:mysql /var/lib/mysql/
2017-03-28 05:39:49,566: INFO: restore.restore_from_mysql():359: Make sure innodb_log_file_size and innodb_log_files_in_group in restored/backup-my.cnf and in /etc/my.cnf are same.
2017-03-28 05:39:49,566: INFO: restore.restore_from_mysql():362: Original my.cnf is restored in restored/_config.
2017-03-28 05:39:49,566: INFO: restore.restore_from_mysql():364: Then you can start MySQL normally.

Now we have a restored database in restored directory that we can copy to /var/lib/mysql

# ls -la restored/
total 30756
drwxr-xr-x 6 root root     4096 Mar 28 05:39 .
dr-xr-x--- 5 root root     4096 Mar 28 05:39 ..
drwxr-xr-x 3 root root     4096 Mar 28 05:39 _config
-rw-r----- 1 root root      262 Mar 28 05:39 backup-my.cnf
-rw-r--r-- 1 root root  5242880 Mar 28 05:39 ib_logfile0
-rw-r--r-- 1 root root  5242880 Mar 28 05:39 ib_logfile1
-rw-r----- 1 root root 18874368 Mar 28 05:39 ibdata1
drwx------ 2 root root     4096 Mar 28 05:39 mysql
drwx------ 2 root root     4096 Mar 28 05:39 performance_schema
drwx------ 2 root root     4096 Mar 28 05:39 test
-rw-r----- 1 root root       89 Mar 28 05:39 xtrabackup_checkpoints
-rw-r----- 1 root root      562 Mar 28 05:39 xtrabackup_info
-rw-r----- 1 root root  2097152 Mar 28 05:39 xtrabackup_logfile

The post Howto Encrypt MySQL Backups on S3 appeared first on Backup and Data Recovery for MySQL.

MySQL Tutorial - Troubleshooting MySQL Replication Part 1

$
0
0

Replication is one of the most common ways to achieve high availability for MySQL and MariaDB. It has become much more robust with the addition of GTIDs, and is thoroughly tested by thousands and thousands of users. MySQL Replication is not a ‘set and forget’ property though, it needs to be monitored for potential issues and maintained so it stays in good shape.  In this blog post, we’d like to share some tips and tricks on how to maintain, troubleshoot and fix issues with MySQL replication.

How to determine if MySQL replication is in a good shape?

This is hands down the most important skill that anyone taking care of a MySQL replication setup has to possess. Let’s take a look at where to look for information about the state of replication. There is a slight difference between MySQL and MariaDB and we will discuss this as well.

SHOW SLAVE STATUS

This is hands down the most common method of checking the state of replication on a slave host - it’s with us since always and it’s usually the first place where we go if we expect that there is some issue with replication.

mysql> SHOW SLAVE STATUS\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 10.0.0.101
                  Master_User: rpl_user
                  Master_Port: 3306
                Connect_Retry: 10
              Master_Log_File: binlog.000002
          Read_Master_Log_Pos: 767658564
               Relay_Log_File: relay-bin.000002
                Relay_Log_Pos: 405
        Relay_Master_Log_File: binlog.000002
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 767658564
              Relay_Log_Space: 606
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
                  Master_UUID: 5d1e2227-07c6-11e7-8123-080027495a77
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set: 5d1e2227-07c6-11e7-8123-080027495a77:1-394233
                Auto_Position: 1
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)

Some details may differ between MySQL and MariaDB but the majority of the content will look the same. Changes will be visible in the GTID section as MySQL and MariaDB do it in a different way. From SHOW SLAVE STATUS, you can derive some pieces of information - which master is used, which user and which port is used to connect to the master. We have some data about the current binary log position (not that important anymore as we can use GTID and forget about binlogs) and the state of SQL and I/O replication threads. Then you can see if and how filtering is configured. You can also find some information about errors, replication lag, SSL settings and GTID. The example above comes from MySQL 5.7 slave which is in a healthy state. Let’s take a look at some example where replication is broken.

MariaDB [test]> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 10.0.0.104
                  Master_User: rpl_user
                  Master_Port: 3306
                Connect_Retry: 10
              Master_Log_File: binlog.000003
          Read_Master_Log_Pos: 636
               Relay_Log_File: relay-bin.000002
                Relay_Log_Pos: 765
        Relay_Master_Log_File: binlog.000003
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 1032
                   Last_Error: Could not execute Update_rows_v1 event on table test.tab; Can't find record in 'tab', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log binlog.000003, end_log_pos 609
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 480
              Relay_Log_Space: 1213
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 1032
               Last_SQL_Error: Could not execute Update_rows_v1 event on table test.tab; Can't find record in 'tab', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log binlog.000003, end_log_pos 609
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
               Master_SSL_Crl:
           Master_SSL_Crlpath:
                   Using_Gtid: Slave_Pos
                  Gtid_IO_Pos: 0-1-73243
      Replicate_Do_Domain_Ids:
  Replicate_Ignore_Domain_Ids:
                Parallel_Mode: conservative
1 row in set (0.00 sec)

This sample is taken from MariaDB 10.1, you can see changes at the bottom of the output to make it work with MariaDB GTID’s. What’s important for us is the error - you can see that something is not right in the SQL thread:

Last_SQL_Error: Could not execute Update_rows_v1 event on table test.tab; Can't find record in 'tab', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log binlog.000003, end_log_pos 609

We will discuss this particular problem later, for now it’s enough that you will see how you can check if there are any errors in the replication using SHOW SLAVE STATUS.

Another important information that comes from SHOW SLAVE STATUS is - how badly our slave lags. You can check it in “Seconds_Behind_Master” column. This metric is especially important to track if you know your application to be sensitive when it comes to stale reads.

In ClusterControl you can track this data in the “Overview” section:

We made visible all of the most important pieces of information from SHOW SLAVE STATUS command. You can check the status of the replication, who is master, if there is a replication lag or not, binary log positions. You can also find retrieved and executed GTID’s.

Performance Schema

Another place you can look for the information about replication is the performance_schema. This applies only to Oracle’s MySQL 5.7 - earlier versions and MariaDB don’t collect this data.

mysql> SHOW TABLES FROM performance_schema LIKE 'replication%';
+---------------------------------------------+
| Tables_in_performance_schema (replication%) |
+---------------------------------------------+
| replication_applier_configuration           |
| replication_applier_status                  |
| replication_applier_status_by_coordinator   |
| replication_applier_status_by_worker        |
| replication_connection_configuration        |
| replication_connection_status               |
| replication_group_member_stats              |
| replication_group_members                   |
+---------------------------------------------+
8 rows in set (0.00 sec)

Below you can find some examples of data available in some of those tables.

mysql> select * from replication_connection_status\G
*************************** 1. row ***************************
             CHANNEL_NAME:
               GROUP_NAME:
              SOURCE_UUID: 5d1e2227-07c6-11e7-8123-080027495a77
                THREAD_ID: 32
            SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 1
 LAST_HEARTBEAT_TIMESTAMP: 2017-03-17 19:41:34
 RECEIVED_TRANSACTION_SET: 5d1e2227-07c6-11e7-8123-080027495a77:715599-724966
        LAST_ERROR_NUMBER: 0
       LAST_ERROR_MESSAGE:
     LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
1 row in set (0.00 sec)
mysql> select * from replication_applier_status_by_worker\G
*************************** 1. row ***************************
         CHANNEL_NAME:
            WORKER_ID: 0
            THREAD_ID: 31
        SERVICE_STATE: ON
LAST_SEEN_TRANSACTION: 5d1e2227-07c6-11e7-8123-080027495a77:726086
    LAST_ERROR_NUMBER: 0
   LAST_ERROR_MESSAGE:
 LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
1 row in set (0.00 sec)

As you can see, we can verify the state of the replication, last error, received transaction set and some more data. What’s important - if you enabled multi-threaded replication, in replication_applier_status_by_worker table, you will see the state of every single worker - this helps you understand the state of replication for each of the worker threads.

ClusterControl
Single Console for Your Entire Database Infrastructure
Find out what else is new in ClusterControl

Replication Lag

Lag is definitely one of the most common problems you’ll be facing when working with MySQL replication. Replication lag shows up when one of the slaves is unable to keep up with the amount of write operations performed by the master. Reasons could be different - different hardware configuration, heavier load on the slave, high degree of write parallelization on master which has to be serialized (when you use single thread for the replication) or the writes cannot be parallelized to the same extent as it has been on the master (when you use multi-threaded replication).

How to detect it?

There are couple of methods to detect the replication lag. First of all, you may check “Seconds_Behind_Master” in the SHOW SLAVE STATUS output - it will tell you if the slave is lagging or not. It works well in most of the cases but in more complex topologies, when you use intermediate masters, on hosts somewhere low in the replication chain, it may be not precise. Another, better, solution is to rely on external tools like pt-heartbeat. Idea is simple - a table is created with, amongst others, a timestamp column. This column is updated on the master at regular intervals. On a slave, you can then compare the timestamp from that column with current time - it will tell you how far behind the slave is.

Regardless of the way you calculate the lag, make sure your hosts are in sync time-wise. Use ntpd or other means of time syncing - if there is a time drift, you will see “false” lag on your slaves.

How to reduce lag?

This is not an easy question to answer. In short, it depends on what is causing the lag, and what became a bottleneck. There are two typical patterns - slave is I/O bound, which means that its I/O subsystem can’t cope with amount of write and read operations. Second - slave is CPU-bound, which means that replication thread uses all CPU it can (one thread can use only one CPU core) and it’s still not enough to handle all write operations.

When CPU is a bottleneck, the solution can be as simple as to use multi-threaded replication. Increase the number of working threads to allow higher parallelization. It is not always possible though - in such case you may want to play a bit with group commit variables (for both MySQL and MariaDB) to delay commits for a slight period of time (we are talking about milliseconds here) and, in this way, increase parallelization of commits.

If the problem is in the I/O, the problem is a bit harder to solve. Of course, you should review your InnoDB I/O settings - maybe there is room for improvements. If my.cnf tuning won’t help, you don’t have too many options - improve your queries (wherever it’s possible) or upgrade your I/O subsystem to something more capable.

Most of the proxies (for example, all proxies which can be deployed from ClusterControl: ProxySQL, HAProxy and MaxScale) give you a possibility to remove a slave out of rotation if replication lag crosses some predefined threshold. This is by no means a method to reduce lag but it can be helpful to avoid stale reads and, as a side effect, to reduce the load on a slave which should help it to catch up.

Of course, query tuning can be a solution in both cases - it’s always good to improve queries which are CPU or I/O heavy.

Errant Transactions

Errant transactions are transactions which have been executed on a slave only, not on the master. In short, they make a slave inconsistent with the master. When using GTID-based replication, this can cause serious troubles if the slave is promoted to a master. We have an in-depth post on this topic and we encourage you to look into it and get familiar with how to detect and fix issues with errant transactions. We also included there information how ClusterControl detects and handles errant transactions.

No Binlog File on the Master

How to identify the problem?

Under some circumstances, it may happen that a slave connects to a master and asks for non-existing binary log file. One reason for this could be the errant transaction - at some point in time, a transaction has been executed on a slave and later this slave becomes a master. Other hosts, which are configured to slave off that master, will ask for that missing transaction. If it was executed a long time ago, there is a chance that binary log files have already been purged.

Another, more typical, example - you want to provision a slave using xtrabackup. You copy the backup on a host, apply the log, change the owner of MySQL data directory - typical operations you do to restore a backup. You execute

SET GLOBAL gtid_purged=

based on the data from xtrabackup_binlog_info and you run CHANGE MASTER TO … MASTER_AUTO_POSITION=1 (this is in MySQL, MariaDB has a slightly different process), start the slave and then you end up with an error like:

                Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.'

in MySQL or:

                Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Could not find GTID state requested by slave in any binlog files. Probably the slave state is too old and required binlog files have been purged.'

in MariaDB.

This basically means that the master doesn’t have all binary logs needed to execute all missing transactions. Most likely, the backup is too old and the master already purged some of binary logs created between the time when backup was created and when the slave was provisioned.

How to solve this problem?

Unfortunately, there’s not much you can do in this particular case. If you have some MySQL hosts which store binary logs for longer time than the master, you can try to use those logs to replay missing transactions on the slave. Let’s take a look how it can be done.

First of all, let’s take a look at the oldest GTID in the master’s binary logs:

mysql> SHOW BINARY LOGS\G
*************************** 1. row ***************************
 Log_name: binlog.000021
File_size: 463
1 row in set (0.00 sec)

So, ‘binlog.000021’ is the latest (and only) file. Let’s check what’s the first GTID entry in this file:

root@master:~# mysqlbinlog /var/lib/mysql/binlog.000021
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#170320 10:39:51 server id 1  end_log_pos 123 CRC32 0x5644fc9b     Start: binlog v 4, server v 5.7.17-11-log created 170320 10:39:51
# Warning: this binlog is either in use or was not closed properly.
BINLOG '
d7HPWA8BAAAAdwAAAHsAAAABAAQANS43LjE3LTExLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAEzgNAAgAEgAEBAQEEgAAXwAEGggAAAAICAgCAAAACgoKKioAEjQA
AZv8RFY=
'/*!*/;
# at 123
#170320 10:39:51 server id 1  end_log_pos 194 CRC32 0x5c096d62     Previous-GTIDs
# 5d1e2227-07c6-11e7-8123-080027495a77:1-1106668
# at 194
#170320 11:21:26 server id 1  end_log_pos 259 CRC32 0xde21b300     GTID    last_committed=0    sequence_number=1
SET @@SESSION.GTID_NEXT= '5d1e2227-07c6-11e7-8123-080027495a77:1106669'/*!*/;
# at 259

As we can see, the oldest binary log entry that’s available is: 5d1e2227-07c6-11e7-8123-080027495a77:1106669

We need also to check what’s the last GTID covered in the backup:

root@slave1:~# cat /var/lib/mysql/xtrabackup_binlog_info
binlog.000017    194    5d1e2227-07c6-11e7-8123-080027495a77:1-1106666

It is: 5d1e2227-07c6-11e7-8123-080027495a77:1-1106666 so we lack two events:
5d1e2227-07c6-11e7-8123-080027495a77:1106667-1106668

Let’s see if we can find those transactions on other slave.

mysql> SHOW BINARY LOGS;
+---------------+------------+
| Log_name      | File_size  |
+---------------+------------+
| binlog.000001 | 1074130062 |
| binlog.000002 |  764366611 |
| binlog.000003 |  382576490 |
+---------------+------------+
3 rows in set (0.00 sec)

It seems that ‘binlog.000003’ is the latest binary log. We need to check if our missing GTID’s can be found in it:

slave2:~# mysqlbinlog /var/lib/mysql/binlog.000003 | grep "5d1e2227-07c6-11e7-8123-080027495a77:110666[78]"
SET @@SESSION.GTID_NEXT= '5d1e2227-07c6-11e7-8123-080027495a77:1106667'/*!*/;
SET @@SESSION.GTID_NEXT= '5d1e2227-07c6-11e7-8123-080027495a77:1106668'/*!*/;

Please keep in mind that you may want to copy binlog files outside of the production server as processing them can add some load. As we verified that those GTID’s exist, we can extract them:

slave2:~# mysqlbinlog --exclude-gtids='5d1e2227-07c6-11e7-8123-080027495a77:1-1106666,5d1e2227-07c6-11e7-8123-080027495a77:1106669' /var/lib/mysql/binlog.000003 > to_apply_on_slave1.sql

After a quick scp, we can apply those events on the slave

slave1:~# mysql -ppass < to_apply_on_slave1.sql

Once done, we can verify if those GTID’s have been applied by looking into the output of SHOW SLAVE STATUS:

                Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.'
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
                  Master_UUID: 5d1e2227-07c6-11e7-8123-080027495a77
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp: 170320 10:45:04
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set: 5d1e2227-07c6-11e7-8123-080027495a77:1-1106668

Executed_GTID_set looks good therefore we can start slave threads:

mysql> START SLAVE;
Query OK, 0 rows affected (0.00 sec)

Let’s check if it worked fine. We will, again, use SHOW SLAVE STATUS output:

           Master_SSL_Crlpath:
           Retrieved_Gtid_Set: 5d1e2227-07c6-11e7-8123-080027495a77:1106669
            Executed_Gtid_Set: 5d1e2227-07c6-11e7-8123-080027495a77:1-1106669

Looks good, it’s up and running!

Another method of solving this problem will be to take a backup one more time and provision the slave again, using fresh data. This will quite likely be faster and definitely more reliable. It is not often that you have different binlog purge policies on master and on slaves)

We will continue discussing other types of replication issues in the next blog post.

MySQL 8.0 Collations: The devil is in the details.

$
0
0

One of the challenges of language specific collations, is making sure they are accurate in the edge-cases of sometimes lesser-used language features. Since I am Norwegian, let me use the Danish collation (which is identical to Norwegian collation) as an example:

Most Scandinavian people know that in Danish (and Norwegian), we have three extra letters: ‘Æ’, ‘Ø’ and ‘Å’ and they follow after ‘Z’ in that order.…


What’s Next for SQL Databases?

$
0
0
SQL Databases

SQL DatabasesIn this blog, I’ll go over my thoughts on what we can expect in the world of SQL databases.

After reading Baron’s prediction on databases, here:

https://www.xaprb.com/blog/defining-moments-in-database-history/

I want to provide my own view on what’s coming up next for SQL databases. I think we live in interesting times, when we can see the beginning of the next-generation of RDBMSs.

There are defining characteristics of such databases:

  1. Auto-scaling. The ability to add and use resources depending on the current load and database size. This is done transparently for users and DBAs.
  2. Auto-healing. The automatic handling of node failures.
  3. Multi-regional, cloud-agnostic, geo-distributed. The ability to support multiple data centers and multiple clouds, in different parts of the world.
  4. Transactional. All the above, with the ability to support multi-statements transactional workloads.
  5. Strong consistency. The full definition of strong consistency is pretty involved. For simplicity, let’s say it means that reads (in the absence of ongoing writes) will return the same data, despite what region or data center you are getting it from. A simple counter-example is the famous MySQL asynchronous replication, where (with the slave delay) reading the data on a slave can return very outdated data. I am focusing on reads, because in a distributed environment the consistent reads performance will be affected. This is where network latency (often limited by the speed of light) will define performance.
  6. SQL language. SQL, despite being old and widely criticized, is not going anywhere. This is a universal language for app developers to access data.

With this, I see following interesting projects:

  • Google Cloud Spanner (https://cloud.google.com/spanner/). Recently announced and still in the Beta stage. Definitely an interesting projects, with the obvious limitation of running only in Google Cloud.
  • FaunaDB (https://fauna.com/). Also very recently announced, so it is hard to say how it performs. The major downside I see is that it does not provide SQL access, but uses a custom language.
  • Two open source projects:
    • CockroachDB (https://www.cockroachlabs.com/). This is still in the Beta stage, but definitely an interesting project to follow. Initially, the project planned to support only key-value access, but later they made a very smart decision to provide SQL access via a PostgreSQL-compatible protocol.
    • TiDB (https://github.com/pingcap/tidb). Right now in RC stages, and the target is to provide SQL access over a MySQL compatible protocol (and later PostgreSQL protocol).

Protocol compatibility is a wise approach, although not strictly necessary. It lowers an entry barrier for the existing applications.

Both CockroachDB and TiDB, at the moment of this writing, still have rough edges and can’t be used in serious deployments (from my experience). I expect both projects will make a big progress in 2017.

What shared characteristics can we expect from these systems?

As I mentioned above, we may see that the read performance is degraded (as latency increases), and often it will be defined more by network performance than anything else. Storage IO and CPU cycles will be secondary factors. There will be more work on how to understand and tune the network traffic.

We may need to get used to the fact that point or small range selects become much slower. Right now, we see very fast point selects for traditional RDBM (MySQL, PostgreSQL, etc.).

Heavy writes will be problematic. The problem is that all writes will need to go through the consistency protocol. Write-optimized storage engines will help (both CockroachDB and TiDB use RocksDB in the storage layer).

The long transactions (let’s say changing 100000 or more rows) also will be problematic. There is just too much network round-trips and housekeeping work on each node, making long transactions an issue for distributed systems.

Another shared property (at least between CockroachDB and TiDB) is the active use of the Raft protocol to achieve consistency. So it will be important to understand how this protocol works to use it effectively. You can find a good overview of the Raft protocol here: http://container-solutions.com/raft-explained-part-1-the-consenus-problem/.

There probably are more NewSQL technologies than I have mentioned here, but I do not think any of them captured critical market- or mind-share. So we are at the beginning of interesting times . . .

What about MySQL? Can MySQL become the database that provides all these characteristics? It is possible, but I do not think it will happen anytime soon. MySQL would need to provide automatic sharding to do this, which will be very hard to implement given the current internal design. It may happen in the future, though it will require a lot of engineering efforts to make it work properly.

Troubleshooting Issues with MySQL Character Sets Q & A

$
0
0
MySQL Character Sets

MySQL Character SetsIn this blog, I will provide answers to the Q & A for the Troubleshooting Issues with MySQL Character Sets webinar.

First, I want to thank everybody for attending the March 9 MySQL character sets troubleshooting webinar. The recording and slides for the webinar are available here. Below is the list of your questions that I wasn’t able to answer during the webinar, with responses:

Q: We’ve had some issues converting tables from

utf8
  to
utf8mb4
. Our issue was that the collation we wanted to use –
utf8mb4_unicode_520_ci
 – did not distinguish between spaces and ideographic (Japanese) spaces, so we were getting unique constraint violations for the 
varchar
 fields when two entries had the same text with different kinds of spaces. Have you seen this problem and is there a workaround? We were wondering if this was related to the mother-child character bug with this collation.

A: Unfortunately this issue exists for many languages. For example, in Russian you cannot distinguish “е” and “ё” if you use

utf8
 or
utf8mb4
. However, there is hope for Japanese: Oracle announced that they will implement new language-specific
utf8mb4
 collations in MySQL 8.0. I already see 21 new collations in my 8.0.0 installation.
mysql> show collation like '%0900%';
+----------------------------+---------+-----+---------+----------+---------+
| Collation                  | Charset | Id  | Default | Compiled | Sortlen |
+----------------------------+---------+-----+---------+----------+---------+
| utf8mb4_0900_ai_ci         | utf8mb4 | 255 |         | Yes      |       8 |
| utf8mb4_cs_0900_ai_ci      | utf8mb4 | 266 |         | Yes      |       8 |
| utf8mb4_da_0900_ai_ci      | utf8mb4 | 267 |         | Yes      |       8 |
| utf8mb4_de_pb_0900_ai_ci   | utf8mb4 | 256 |         | Yes      |       8 |
| utf8mb4_eo_0900_ai_ci      | utf8mb4 | 273 |         | Yes      |       8 |
| utf8mb4_es_0900_ai_ci      | utf8mb4 | 263 |         | Yes      |       8 |
| utf8mb4_es_trad_0900_ai_ci | utf8mb4 | 270 |         | Yes      |       8 |
| utf8mb4_et_0900_ai_ci      | utf8mb4 | 262 |         | Yes      |       8 |
| utf8mb4_hr_0900_ai_ci      | utf8mb4 | 275 |         | Yes      |       8 |
| utf8mb4_hu_0900_ai_ci      | utf8mb4 | 274 |         | Yes      |       8 |
| utf8mb4_is_0900_ai_ci      | utf8mb4 | 257 |         | Yes      |       8 |
| utf8mb4_la_0900_ai_ci      | utf8mb4 | 271 |         | Yes      |       8 |
| utf8mb4_lt_0900_ai_ci      | utf8mb4 | 268 |         | Yes      |       8 |
| utf8mb4_lv_0900_ai_ci      | utf8mb4 | 258 |         | Yes      |       8 |
| utf8mb4_pl_0900_ai_ci      | utf8mb4 | 261 |         | Yes      |       8 |
| utf8mb4_ro_0900_ai_ci      | utf8mb4 | 259 |         | Yes      |       8 |
| utf8mb4_sk_0900_ai_ci      | utf8mb4 | 269 |         | Yes      |       8 |
| utf8mb4_sl_0900_ai_ci      | utf8mb4 | 260 |         | Yes      |       8 |
| utf8mb4_sv_0900_ai_ci      | utf8mb4 | 264 |         | Yes      |       8 |
| utf8mb4_tr_0900_ai_ci      | utf8mb4 | 265 |         | Yes      |       8 |
| utf8mb4_vi_0900_ai_ci      | utf8mb4 | 277 |         | Yes      |       8 |
+----------------------------+---------+-----+---------+----------+---------+
21 rows in set (0,03 sec)

In 8.0.1 they promised new case-sensitive and Japanese collations. Please see this blog post for details. The note about the planned Japanese support is at the end.

Meanwhile, I can only suggest that you implement your own collation as described here. You may use

utf8_russian_ci
 collation from Bug #51976 as an example.

Although the user manual does not list

utf8mb4
 as a character set for which it’s possible to create new collations, you can actually do it. What you need to do is add a record about the character set
utf8mb4
 and the new collation into
Index.xml
, then restart the server.
<charset name="utf8mb4">
<collation name="utf8mb4_russian_ci" id="1033">
 <rules>
    <reset>u0415</reset><p>u0451</p><t>u0401</t>
  </rules>
</collaiton>
</charset>
mysql> show collation like 'utf8mb4_russian_ci';
+--------------------+---------+------+---------+----------+---------+
| Collation          | Charset | Id   | Default | Compiled | Sortlen |
+--------------------+---------+------+---------+----------+---------+
| utf8mb4_russian_ci | utf8mb4 | 1033 |         |          |       8 |
+--------------------+---------+------+---------+----------+---------+
1 row in set (0,03 sec)
mysql> create table test_yo(gen varchar(100) CHARACTER SET utf8mb4, yo varchar(100) CHARACTER SET utf8mb4 collate utf8mb4_russian_ci) engine=innodb default character set=utf8mb4;
Query OK, 0 rows affected (0,25 sec)
mysql> set names utf8mb4;
Query OK, 0 rows affected (0,02 sec)
mysql> insert into test_yo values('ел', 'ел'), ('ель', 'ель'), ('ёлка', 'ёлка');
Query OK, 3 rows affected (0,05 sec)
Records: 3  Duplicates: 0  Warnings: 0
mysql> insert into test_yo values('Ел', 'Ел'), ('Ель', 'Ель'), ('Ёлка', 'Ёлка');
Query OK, 3 rows affected (0,06 sec)
Records: 3  Duplicates: 0  Warnings: 0
mysql> select * from test_yo order by gen;
+----------+----------+
| gen      | yo       |
+----------+----------+
| ел       | ел       |
| Ел       | Ел       |
| ёлка     | ёлка     |
| Ёлка     | Ёлка     |
| ель      | ель      |
| Ель      | Ель      |
+----------+----------+
6 rows in set (0,00 sec)
mysql> select * from test_yo order by yo;
+----------+----------+
| gen      | yo       |
+----------+----------+
| ел       | ел       |
| Ел       | Ел       |
| ель      | ель      |
| Ель      | Ель      |
| ёлка     | ёлка     |
| Ёлка     | Ёлка     |
+----------+----------+
6 rows in set (0,00 sec)

Q: If receiving

utf8
 on
latin1
 charset it will be corrupted. Just want to confirm that you can reformat as
utf8
 and un-corrupt the data? Also, is there a time limit on how quickly this needs to be done?

A: It will be corrupted only if you store

utf8
 data in the 
latin1
 column. For example, if you have a table, defined as:
create table latin1(
  f1 varchar(100)
) engine=innodb default charset=latin1;

And then insert a word in

utf8
 format into it that contains characters that are not in the 
latin1
 character set:
mysql> set names utf8;
Query OK, 0 rows affected (0,00 sec)
mysql> set sql_mode='';
Query OK, 0 rows affected, 1 warning (0,00 sec)
mysql> insert into latin1 values('Sveta'), ('Света');
Query OK, 2 rows affected, 1 warning (0,04 sec)
Records: 2  Duplicates: 0  Warnings: 1

The data in

UTF8
 will be corrupted and can never be recovered:
mysql> select * from latin1;
+-------+
| f1    |
+-------+
| Sveta |
| ????? |
+-------+
2 rows in set (0,00 sec)
mysql> select f1, hex(f1) from latin1;
+-------+------------+
| f1    | hex(f1)    |
+-------+------------+
| Sveta | 5376657461 |
| ????? | 3F3F3F3F3F |
+-------+------------+
2 rows in set (0,01 sec)

However, if your data is stored in the 

UTF8
 column and you use
latin1
 for a connection, you will only get a corrupted result set. The data itself will be left untouched:
mysql> create table utf8(f1 varchar(100)) engine=innodb character set utf8;
Query OK, 0 rows affected (0,18 sec)
mysql> insert into utf8 values('Sveta'), ('Света');
Query OK, 2 rows affected (0,15 sec)
Records: 2  Duplicates: 0  Warnings: 0
mysql> set names latin1;
Query OK, 0 rows affected (0,00 sec)
mysql> select f1, hex(f1) from utf8;
+-------+----------------------+
| f1    | hex(f1)              |
+-------+----------------------+
| Sveta | 5376657461           |
| ????? | D0A1D0B2D0B5D182D0B0 |
+-------+----------------------+
2 rows in set (0,00 sec)
mysql> set names utf8;
Query OK, 0 rows affected (0,00 sec)
mysql> select f1, hex(f1) from utf8;
+------------+----------------------+
| f1         | hex(f1)              |
+------------+----------------------+
| Sveta      | 5376657461           |
| Света      | D0A1D0B2D0B5D182D0B0 |
+------------+----------------------+
2 rows in set (0,00 sec)

Q: Can you discuss how charsets affect mysqldump? Specifically, how do we dump a database containing tables with different default charsets?

A: Yes, you can. MySQL can successfully convert data that uses different character sets, so your only job is to specify option

--default-character-set
 for
mysqldump
. In this case, strings in any character set you use can be converted to the character set specified. For example, if you use
cp1251
 and
latin1
, you may set option
--default-character-set
 to
cp1251
,
utf8
 and 
utf8mb4
. However, you cannot set it to
latin1
 because Cyrillic characters exist in the 
cp1251
 character set, but do not exist in
latin1
.

The default value for

mysqldump
 is
utf8
. You only need to change this default if you use values that are outside of the range supported by 
utf8
 (for example, the smileys in
utf8mb4
).

Q: But if you use the 

--single-transaction
 option for
mysqldump
, you can only specify one character set in the default?

A: Yes, and this is OK: all data will be converted into this character set. And then, when you will restore the dump, it will be converted back to the character set specified in column definitions.

Q: I noticed that MySQL doesn’t support case-sensitive

UTF-8
 character sets. What do you recommend for implementing case-sensitive
UTF-8
, if it’s at all possible?

A: In the link I provided earlier, Oracle promises to implement case-sensitive collations for

utf8mb4
 in version 8.0.1. Before that happens, I recommend you to implement your own case-sensitive collation.

Q: How are tools like

pt-table-checksum
 affected by charsets? Is it safe to use a 4-byte charset (like
utf8mb4
) as the default charset for all comparisons? Assuming our tables are a mix of
latin1
 ,
utf8
 and
utf8mb4
.

A: With this combination, you won’t have any issues:

pt-table-checksum
 uses a complicated set of functions that joins columns and calculates a 
crc32
 checksum on them. In your case, all data will be converted to
utf8mb4
 and no conflicts will happen.

However, if you use incompatible character sets in a single table, you may get the error

"Illegal mix of collations for operation 'concat_ws' "
:
mysql> create table cp1251(f1 varchar(100) character set latin1, f2 varchar(100) character set cp1251) engine=innodb;
Query OK, 0 rows affected (0,32 sec)
mysql> set names utf8;
Query OK, 0 rows affected (0,00 sec)
mysql> insert into cp1251 values('Sveta', 'Света');
Query OK, 1 row affected (0,07 sec)
sveta@Thinkie:~/build/mysql-8.0/mysql-test$ ~/build/percona-toolkit/bin/pt-table-checksum h=127.0.0.1,P=13000,u=root,D=test
Diffs cannot be detected because no slaves were found.  Please read the --recursion-method documentation for information.
03-18T03:51:58 Error executing EXPLAIN SELECT COUNT(*) AS cnt, COALESCE(LOWER(CONV(BIT_XOR(CAST(CRC32(CONCAT_WS('#', `f1`, `f2`, CONCAT(ISNULL(`f1`), ISNULL(`f2`)))) AS UNSIGNED)), 10, 16)), 0) AS crc FROM `db1`.`cp1251` /*explain checksum table*/: DBD::mysql::st execute failed: Illegal mix of collations for operation 'concat_ws' [for Statement "EXPLAIN SELECT COUNT(*) AS cnt, COALESCE(LOWER(CONV(BIT_XOR(CAST(CRC32(CONCAT_WS('#', `f1`, `f2`, CONCAT(ISNULL(`f1`), ISNULL(`f2`)))) AS UNSIGNED)), 10, 16)), 0) AS crc FROM `db1`.`cp1251` /*explain checksum table*/"] at /home/sveta/build/percona-toolkit/bin/pt-table-checksum line 11351.
03-18T03:51:58 Error checksumming table db1.cp1251: Error executing checksum query: DBD::mysql::st execute failed: Illegal mix of collations for operation 'concat_ws' [for Statement "REPLACE INTO `percona`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT ?, ?, ?, ?, ?, ?, COUNT(*) AS cnt, COALESCE(LOWER(CONV(BIT_XOR(CAST(CRC32(CONCAT_WS('#', `f1`, `f2`, CONCAT(ISNULL(`f1`), ISNULL(`f2`)))) AS UNSIGNED)), 10, 16)), 0) AS crc FROM `db1`.`cp1251` /*checksum table*/" with ParamValues: 0='db1', 1='cp1251', 2=1, 3=undef, 4=undef, 5=undef] at /home/sveta/build/percona-toolkit/bin/pt-table-checksum line 10741.
TS ERRORS  DIFFS     ROWS  CHUNKS SKIPPED    TIME TABLE
03-18T03:51:58      2      0        0       1       0   0.003 db1.cp1251
03-18T03:51:58      0      0        2       1       0   0.167 db1.latin1
03-18T03:51:58      0      0        6       1       0   0.198 db1.test_yo
...

The tool continues working, and will process the rest of your tables. I reported this behavior as Bug #1674266.

Thanks for attending the Troubleshooting Issues with MySQL Character Sets webinar.

Network attacks on MySQL, Part 4: SSL hostnames

$
0
0

In my previous blogs I told you to enable SSL/TLS and configure it to check the CA. So I followed my advice and did all that. Great!

So the --ssl-mode setting was used a few times as a solution. And it has a setting we didn't use yet: VERIFY_IDENTITY. In older MySQL versions you can use --ssl-verify-server-cert. Both turn on hostname verification.

The attack

Get any certificate which is trusted by the configured CA, this can for example be a certificate from a development machine. And use that with a man-in-the-middle proxy.

Then the client:

  1. Checks if SSL is uses (--ssl-mode=REQUIRED)
  2. Verify if the certificate is signed by a trusted CA (--ssl-mode=VERIFY_CA)

Both checks succeed. But the certificate might be for testhost01.example.com and the database server might be prod-websitedb-123.example.com.

Browsers by default verify hostnames, MySQL does not.

Turning on hostname validation

So use --ssl-mode=VERIFY_IDENTITY and everything should be fine?

Well that might work for simple setups, but would probably fail for more complex setups.

This is because you might have a master-slave setup with loadbalancer in front of it. So your webapp connect to mydb-prod-lb.example.com which might be served by mydb1.example.com (master) or mydb2.example.com (slave). There might or might not be any automatic read/write splitting.

So then just configure the loadbalancer be the endpoint of the SSL connection? Well no, because most loadbalancers don't know how to speak the mysql protocol, which is needed to setup the SSL connection.

Ok, then just configure both servers with the certificate for mydb-prod-lb.example.com and everything should work. And it does!

But then you want to change the replication connection to also use SSL, but now the certificates and hostnames don't match anymore as they connect directly.

The same might be true for mysqldump or mysqlbinlog instances running on a separate backup server.

But there is a X.509 extension available which can be used: 'SubjectAlternativeName' a.k.a. SAN. (Not to be confused with Storage Area Networking). This allows you to have a certificate with multiple hostnames.

So for both hosts put their own hostname and the loadbalancer hostname in there.

But unfortunately that doesn't work yet. MySQL doesn't support this.

See Bug #68052: SSL Certificate Subject ALT Names with IPs not respected with --ssl-verify-serve for more details.

So yes, do enable hostname verification, but probably not everywhere yet.

MySQL Tutorial - Troubleshooting MySQL Replication Part 2

$
0
0

In the previous post, we discussed how to verify that MySQL Replication is in good shape. We also looked at some of the typical problems. In this post, we will have a look at some more issues that you might see when dealing with MySQL replication.

Missing or Duplicated Entries

This is something which should not happen, yet it happens very often - a situation in which an SQL statement executed on the master succeeds but the same statement executed on one of slaves fails. Main reason is slave drift - something (usually errant transactions but also other issues or bugs in the replication) causes the slave to differ from its master. For example, a row which existed on the master does not exist on a slave and it cannot be deleted or updated. How often this problem shows up depends mostly on your replication settings. In short, there are three ways in which MySQL stores binary log events. First, “statement”, means that SQL is written in plain text, just as it has been executed on a master. This setting has the highest tolerance on slave drift but it’s also the one which cannot guarantee slave consistency - it’s hard to recommend to use it in production. Second format, “row”, stores the query result instead of query statement. For example, an event may look like below:

### UPDATE `test`.`tab`
### WHERE
###   @1=2
###   @2=5
### SET
###   @1=2
###   @2=4

This means that we are updating a row in ‘tab’ table in ‘test’ schema where first column has a value of 2 and second column has a value of 5. We set first column to 2 (value doesn’t change) and second column to 4. As you can see, there’s not much room for interpretation - it’s precisely defined which row is used and how it’s changed. As a result, this format is great for slave consistency but, as you can imagine, it’s very vulnerable when it comes to data drift. Still it is the recommended way of running MySQL replication.

Finally, the third one, “mixed”, works in a way that those events which are safe to write in the form of statements use “statement” format. Those which could cause data drift will use “row” format.

How do you detect them?

As usual, SHOW SLAVE STATUS will help us identify the problem.

               Last_SQL_Errno: 1032
               Last_SQL_Error: Could not execute Update_rows event on table test.tab; Can't find record in 'tab', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log binlog.000021, end_log_pos 970
               Last_SQL_Errno: 1062
               Last_SQL_Error: Could not execute Write_rows event on table test.tab; Duplicate entry '3' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log binlog.000021, end_log_pos 1229

As you can see, errors are clear and self-explanatory (and they are basically identical between MySQL and MariaDB.

How do you fix the issue?

This is, unfortunately the complex part. First of all, you need to identify a source of truth. Which host contains the correct data? Master or slave? Usually you’d assume it’s the master but don’t assume it by default - investigate! It could be that after failover, some part of the application still issued writes to the old master, which now acts as a slave. It could be that read_only hasn’t been set correctly on that host or maybe the application uses superuser to connect to database (yes, we’ve seen this in production environments). In such case, the slave could be the source of truth - at least to some extent.

Depending on which data should stay and which should go, the best course of action would be to identify what’s needed to get replication back in sync. First of all, replication is broken so you need to attend to this. Log into the master and check the binary log even that caused replication to break.

           Retrieved_Gtid_Set: 5d1e2227-07c6-11e7-8123-080027495a77:1106672
            Executed_Gtid_Set: 5d1e2227-07c6-11e7-8123-080027495a77:1-1106671

As you can see, we miss one event: 5d1e2227-07c6-11e7-8123-080027495a77:1106672. Let’s check it in the master’s binary logs:

mysqlbinlog -v --include-gtids='5d1e2227-07c6-11e7-8123-080027495a77:1106672' /var/lib/mysql/binlog.000021
#170320 20:53:37 server id 1  end_log_pos 1066 CRC32 0xc582a367     GTID    last_committed=3    sequence_number=4
SET @@SESSION.GTID_NEXT= '5d1e2227-07c6-11e7-8123-080027495a77:1106672'/*!*/;
# at 1066
#170320 20:53:37 server id 1  end_log_pos 1138 CRC32 0x6f33754d     Query    thread_id=5285    exec_time=0    error_code=0
SET TIMESTAMP=1490043217/*!*/;
SET @@session.pseudo_thread_id=5285/*!*/;
SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/;
SET @@session.sql_mode=1436549152/*!*/;
SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;
/*!\C utf8 *//*!*/;
SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=8/*!*/;
SET @@session.lc_time_names=0/*!*/;
SET @@session.collation_database=DEFAULT/*!*/;
BEGIN
/*!*/;
# at 1138
#170320 20:53:37 server id 1  end_log_pos 1185 CRC32 0xa00b1f59     Table_map: `test`.`tab` mapped to number 571
# at 1185
#170320 20:53:37 server id 1  end_log_pos 1229 CRC32 0x5597e50a     Write_rows: table id 571 flags: STMT_END_F

BINLOG '
UUHQWBMBAAAALwAAAKEEAAAAADsCAAAAAAEABHRlc3QAA3RhYgACAwMAAlkfC6A=
UUHQWB4BAAAALAAAAM0EAAAAADsCAAAAAAEAAgAC//wDAAAABwAAAArll1U=
'/*!*/;
### INSERT INTO `test`.`tab`
### SET
###   @1=3
###   @2=7
# at 1229
#170320 20:53:37 server id 1  end_log_pos 1260 CRC32 0xbbc3367c     Xid = 5224257
COMMIT/*!*/;

We can see it was an insert which sets first column to 3 and second to 7. Let’s verify how our table looks like now:

mysql> SELECT * FROM test.tab;
+----+------+
| id | b    |
+----+------+
|  1 |    2 |
|  2 |    4 |
|  3 |   10 |
+----+------+
3 rows in set (0.01 sec)

Now we have two options, depending on which data should prevail. If correct data is on the master, we can simply delete row with id=3 on the slave. Just make sure you disable binary logging to avoid introducing errant transactions. On the other hand, if we decided that the correct data is on the slave, we need to run REPLACE command on the master to set row with id=3 to correct content of (3, 10) from current (3, 7). On the slave, though, we will have to skip current GTID (or, to be more precise, we will have to create an empty GTID event) to be able to restart replication.

Deleting a row on a slave is simple:

SET SESSION log_bin=0; DELETE FROM test.tab WHERE id=3; SET SESSION log_bin=1;

Inserting an empty GTID is almost as simple:

mysql> SET @@SESSION.GTID_NEXT= '5d1e2227-07c6-11e7-8123-080027495a77:1106672';
Query OK, 0 rows affected (0.00 sec)
mysql> BEGIN;
Query OK, 0 rows affected (0.00 sec)
mysql> COMMIT;
Query OK, 0 rows affected (0.00 sec)
mysql> SET @@SESSION.GTID_NEXT=automatic;
Query OK, 0 rows affected (0.00 sec)

Another method of solving this particular issue (as long as we accept the master as a source of truth) is to use tools like pt-table-checksum and pt-table-sync to identify where the slave is not consistent with its master and what SQL has to be executed on the master to bring the slave back in sync. Unfortunately, this method is rather on the heavy side - lots of load is added to master and a bunch of queries are written into the replication stream which may affect lag on slaves and general performance of the replication setup. This is especially true if there is a significant number of rows which need to be synced.

Finally, as always, you can rebuild your slave using data from the master - in this way you can be sure that the slave will be refreshed with the freshest, up-to-date data. This is, actually, not necessarily a bad idea - when we are talking about large number of rows to sync using pt-table-checksum/pt-table-sync, this comes with significant overhead in replication performance, overall CPU and I/O load and man-hours required.

ClusterControl allows you to rebuild a slave, using a fresh copy of the master data.

Consistency checks

As we mentioned in the previous chapter, consistency can become a serious issue and can cause lots of headaches for users running MySQL replication setups. Let’s see how you can verify that your MySQL slaves are in sync with the master and what you can do about it.

How to detect an inconsistent slave

Unfortunately, the typical way an user gets to know that a slave is inconsistent is by running into one of the issues we mentioned in the previous chapter. To avoid that proactive monitoring of slave consistency is required. Let’s check how it can be done.

We are going to use a tool from Percona Toolkit: pt-table-checksum. It is designed to scan replication cluster and identify any discrepancies.

We built a custom scenario using sysbench and we introduced a bit of inconsistency on one of the slaves. What’s important (if you’d like to test it like we did), you need to apply a patch below to force pt-table-checksum to recognize ‘sbtest’ schema as non-system schema:

--- pt-table-checksum    2016-12-15 14:31:07.000000000 +0000
+++ pt-table-checksum-fix    2017-03-21 20:32:53.282254794 +0000
@@ -7614,7 +7614,7 @@

    my $filter = $self->{filters};

-   if ( $db =~ m/information_schema|performance_schema|lost\+found|percona|percona_schema|test/ ) {
+   if ( $db =~ m/information_schema|performance_schema|lost\+found|percona|percona_schema|^test/ ) {
       PTDEBUG && _d('Database', $db, 'is a system database, ignoring');
       return 0;
    }

At first, we are going to execute pt-table-checksum in following way:

master:~# ./pt-table-checksum  --max-lag=5 --user=sbtest --password=sbtest --no-check-binlog-format --databases='sbtest'
            TS ERRORS  DIFFS     ROWS  CHUNKS SKIPPED    TIME TABLE
03-21T20:33:30      0      0  1000000      15       0  27.103 sbtest.sbtest1
03-21T20:33:57      0      1  1000000      17       0  26.785 sbtest.sbtest2
03-21T20:34:26      0      0  1000000      15       0  28.503 sbtest.sbtest3
03-21T20:34:52      0      0  1000000      18       0  26.021 sbtest.sbtest4
03-21T20:35:34      0      0  1000000      17       0  42.730 sbtest.sbtest5
03-21T20:36:04      0      0  1000000      16       0  29.309 sbtest.sbtest6
03-21T20:36:42      0      0  1000000      15       0  38.071 sbtest.sbtest7
03-21T20:37:16      0      0  1000000      12       0  33.737 sbtest.sbtest8

Couple of important notes on how we invoked the tool. First of all, user that we set has to exists on all slaves. If you want, you can also use ‘--slave-user’ to define other, less privileged user to access slaves. Another thing worth explaining - we use row-based replication which is not fully compatible with pt-table-checksum. If you have row-based replication, what happens is pt-table-checksum will change binary log format on a session level to ‘statement’ as this is the only format supported. The problem is that such change will work only on a first level of slaves which are directly connected to a master. If you have intermediate masters (so, more than one level of slaves), using pt-table-checksum may break the replication. This is why, by default, if the tool detects row-based replication, it exits and prints error:

“Replica slave1 has binlog_format ROW which could cause pt-table-checksum to break replication. Please read "Replicas using row-based replication" in the LIMITATIONS section of the tool's documentation. If you understand the risks, specify --no-check-binlog-format to disable this check.”

We used only one level of slaves so it was safe to specify “--no-check-binlog-format” and move forward.

Finally, we set maximum lag to 5 seconds. If this threshold will be reached, pt-table-checksum will pause for a time needed to bring the lag under the threshold.

As you could see from the output,

03-21T20:33:57      0      1  1000000      17       0  26.785 sbtest.sbtest2

an inconsistency has been detected on table sbtest.sbtest2.

By default, pt-table-checksum stores checksums in percona.checksums table. This data can be used for another tool from Percona Toolkit, pt-table-sync, to identify which parts of the table should be checked in detail to find exact difference in data.

ClusterControl
Single Console for Your Entire Database Infrastructure
Find out what else is new in ClusterControl

How to fix inconsistent slave

As mentioned above, we will use pt-table-sync to do that. In our case we are going to use data collected by pt-table-checksum although it is also possible to point pt-table-sync to two hosts (the master and a slave) and it will compare all data on both hosts. It is definitely more time- and resource-consuming process therefore, as long as you have already data from pt-table-checksum, it’s much better to use it. This is how we executed it to test the output:

master:~# ./pt-table-sync --user=sbtest --password=sbtest --databases=sbtest --replicate percona.checksums h=master --print
REPLACE INTO `sbtest`.`sbtest2`(`id`, `k`, `c`, `pad`) VALUES ('1', '434041', '61753673565-14739672440-12887544709-74227036147-86382758284-62912436480-22536544941-50641666437-36404946534-73544093889', '23608763234-05826685838-82708573685-48410807053-00139962956') /*percona-toolkit src_db:sbtest src_tbl:sbtest2 src_dsn:h=10.0.0.101,p=...,u=sbtest dst_db:sbtest dst_tbl:sbtest2 dst_dsn:h=10.0.0.103,p=...,u=sbtest lock:1 transaction:1 changing_src:percona.checksums replicate:percona.checksums bidirectional:0 pid:25776 user:root host:vagrant-ubuntu-trusty-64*/;

As you can see, as a result some SQL has been generated. Important to note is --replicate variable. What happens here is we point pt-table-sync to table generated by pt-table-checksum. We also point it to master.

To verify if SQL makes sense we used --print option. Please note SQL generated is valid only at the time it’s generated - you cannot really store it somewhere, review it and then execute. All you can do is to verify if the SQL makes any sense and, immediately after, reexecute tool with --execute flag:

master:~# ./pt-table-sync --user=sbtest --password=sbtest --databases=sbtest --replicate percona.checksums h=10.0.0.101 --execute

This should make slave back in sync with the master. We can verify it with pt-table-checksum:

root@vagrant-ubuntu-trusty-64:~# ./pt-table-checksum  --max-lag=5 --user=sbtest --password=sbtest --no-check-binlog-format --databases='sbtest'
            TS ERRORS  DIFFS     ROWS  CHUNKS SKIPPED    TIME TABLE
03-21T21:36:04      0      0  1000000      13       0  23.749 sbtest.sbtest1
03-21T21:36:26      0      0  1000000       7       0  22.333 sbtest.sbtest2
03-21T21:36:51      0      0  1000000      10       0  24.780 sbtest.sbtest3
03-21T21:37:11      0      0  1000000      14       0  19.782 sbtest.sbtest4
03-21T21:37:42      0      0  1000000      15       0  30.954 sbtest.sbtest5
03-21T21:38:07      0      0  1000000      15       0  25.593 sbtest.sbtest6
03-21T21:38:27      0      0  1000000      16       0  19.339 sbtest.sbtest7
03-21T21:38:44      0      0  1000000      15       0  17.371 sbtest.sbtest8

As you can see, there are no diffs anymore in sbtest.sbtest2 table.

We hope you found this blog post informative and useful. Click here to learn more about MySQL Replication. If you have any questions or suggestions, feel free to reach us through comments below.

Webinar Thursday 3/30: MyRocks Troubleshooting

$
0
0
MyRocks Troubleshooting

MyRocks TroubleshootingPlease join Percona’s Principal Technical Services Engineer Sveta Smirnova, and Senior Software Engineer George Lorch, MariaDB’s Query Optimizer Developer Sergei Petrunia and Facebook’s Database Engineer Yoshinori Matsunobu as they present MyRocks Troubleshooting on March 30, 2017 at 11:00 am PDT / 2:00 pm EDT (UTC-7).

MyRocks is an alternative storage engine designed for flash storage. It provides great write workload performance and space efficiency. Like any other powerful engine, it has its own specific configuration scenarios that require special troubleshooting solutions.

This webinar will discuss how to deal with:

  • Data corruption issues
  • Inconsistent data
  • Locks
  • Slow performance

We will use well-known instruments and tools, as well as MyRocks-specific tools, and demonstrate how they work with the MyRocks storage engine.

Register for this webinar here.

Viewing all 18822 articles
Browse latest View live