In this post, I’ll discuss how database support is good for your enterprise.

Years ago when I worked for the MySQL Support organization at the original MySQL AB, we spoke about MySQL Support as insurance and focused on a value proposition similar to that of car insurance. You must purchase car insurance before the incident happens, or insurance won’t cover the damage. In fact, most places around the world require automobile insurance. Similarly, many organizations that leverage production-use technology have their own “insurance” by means of 24/7 support.

In my opinion, this is a very one-sided view that does not capture the full value (and ROI) that a database support contract with Percona provides. With a Percona support contract, you are assured that your database environment (virtual, physical, or in the cloud) is fully covered – whether it’s one server or many.

Increasingly – especially with the introduction of cloud-based database environments – database servers are being spun up and torn down on a day-to-day basis. However briefly these databases exist, they need support. One of the challenges modern businesses face is providing support for a changing database infrastructure, while still maintaining a viable cost structure.

Let’s look at the different dimensions of value offered by Percona Support based on the different support cases we have received throughout the years.

Reduce and Prevent Downtime

If your database goes down, the time to recover will be significantly shorter with a support agreement than without it. The cost of downtime varies widely between organizations. A recent IBM-sponsored Ponemon Institute research study found that the average data security breach could cost upwards of $7.01M (in the United States).

With our clients, we’ve found preventing even one significant downtime event a year justifies support costs. Even when the client’s in-house team is very experienced, our help is often invaluable as we are exposed to a great variety of incidents from hundreds of companies. It is much more likely we have encountered the same incident before and have a solution ready. Helping to recover from downtime quickly is a reactive part of support – you can realize even more value by proactively working with support to get advice on your HA options as well as ensure that you’re following the best database backup and security practices.

Better Security

Having a database support contract by itself is not enough to prevent all security incidents. Databases are only one of the component attack vectors, and it takes a lot of everyday work to stay secure. There is nothing that can guarantee complete security. Database support, however, can be a priceless resource for your security team. It can apply security and compliance practices to your database environment and demonstrate how to avoid typical mistakes.

The cost of data breaches can be phenomenal, as well as impact business reputations much more than downtime or performance issues. Depending on the company size and market, costs vary. Different studies estimate costs ranging in average from $1.6M to 7.01M in direct costs. Everyone agrees leaving rising security risks and costs unchecked is a recipe for disaster.

Fix Database Software Bugs

While you might have great DBAs on your team who are comfortable with best practices and downtime recovery, most likely you do not have a development team comfortable with fixing bugs in the database kernel or supporting tools. Getting up-to-date software fixes reduces downtime. It also limits other issues as well, such as ensuring efficient development and operation teams, avoiding using complex workarounds, etc.

Reduce Resources

We deal with a large number of performance-related questions. When we address such problems, we provide a better user experience, save costs, and minimize environmental impact by reducing resource use.

Savings vary depending on your application scale and how optimized the environment is already. In the best cases, our support team helped customers make applications more than 10x more efficient. In most cases, we can help make things at least 30% more efficient. If you’re spending $100K or more on your database environment, this benefit alone makes a support agreement well worth it.

Efficient Developers

You cannot minimize the importance of development efficiency. Too often customers don’t give their developers support access, even though they critically help realize application’s full value. Developers make database decisions about schema design all the time. These include query writing, stored procedures, triggers, sharding, document storage, or foreign keys. Without a database support contract, developers often have resort to “Google University” to find an answer – and often end up with inapplicable, outdated or simply wrong information. Combined with this, they often apply or resort to time-consuming trial and error.

With the help of a Percona Support team, developers can learn proven practices that apply to their specific situation. This saves a lot of time and gets better applications to market faster. Even with a single US-based developer intensively working within the database environment, a support agreement might justify the cost based on increased developer efficiency alone. Larger development teams simply cannot afford to not have support.

Efficient Operations

Your operations staff (DBAs, DevOps, Sysadmins) are in the same boat – if your database environment is significant, chances are you are always looking for ways to save time, make operations more efficient and reduce mistakes. Our support team can provide you with specific actionable advice for your challenges.

Chances are we have seen environments similar to yours and know which software, approaches and practices work well (and which do not). This knowledge helps prevent and reduce downtime. It also helps with team efficiency. Percona Support’s help allows you to handle operations with a smaller team, or address issues with a less experienced staff.

Better Applications

Percona Support access helps developers not only be more productive, but results in better application quality because application database interface design, schema, queries, etc. best practices are followed. The Percona team supports many applications, for many years. We often  think about problems before you might think about them, such as:

• “How will this design play with replication or sharding?”
• “Will it scale with large amounts of users or data?”
• “How flexible is such a design when the  application will inevitably be evolving over years?”

While a better application is hard to quantify, it really is quite important.

Faster Time to Market

Yet another benefit that comes from developers having access to a database support team is faster time-to-market. For many agile applications, being able to launch new features faster is even more important than cost savings – this is how businesses succeed against the competition. At Percona, we love helping businesses succeed.

Conclusion

As you see, there are a lot of ways Percona Support can contribute to the success of your business. Support is much more than “insurance” that you should consider purchasing for compliance reasons. Percona Support provides a great return on investment. It allows you to minimize risks and costs while delivering the highest quality applications or services. Our flexible plans can cover your database environment, even if it is an ever-changing one, while still allowing you to plan your operations costs.

Liquibase

Liquibase is a versioning tool for databases. Currently is in the 3.5 version and it is installed as a JAR. It has been in the market since 2006, and recently completed its 10th anniversary. In it's feature list we have:

• Code branching and merging
• Multiple database types
• Supports XML, YAML, JSON and SQL formats
• Supports context-dependent logic
• Generate Database change documentation
• Generate Database "diffs"
• Run through your build process, embedded in your application or on demand
• Automatically generate SQL scripts for DBA code review
• Does not require a live database connection

Why you need it?

Some frameworks comes with built-in solutions out of the box like Eloquent and Doctrine. There is nothing wrong in using that when you have only DB per project, but when you have multiple systems, it starts to get complicated.

Since Liquibase works as a versioning tool, you can branch like a normal code on github and merge as needed. You have contexts, which means changes can be applied to specific environments only and tagging capabilities to be able to rollback.

Rollback is a tricky thing, you can either do an automatically rollback or define a script, useful when deal with MySQL for instance where DDL changes are NOT transactional.

Guidelines for changelogs and migrations

• MUST be written using the JSON format. Exceptions are changes/legacy/base.xml and changes/legacy/base_procedures_triggers.sql.
• MUST NOT be edited. If a new column is to be added, a new migration file must be created and the file MUST be added AFTER the las run transaction.

Branching

There could be 3 main branches:

• production (master)
• staging
• testing

Steps:

1. Create your changelog branch;
2. Merge into testing;
3. When the feature ready to staging, merge into staging;
4. When the feature is ready, merge into production.

Example:

Rules:

• testing, staging and production DO NOT merge amongst themselves in any capacity;
• DO NOT rebase the main branches;
• Custom branch MUST be deleted after merged into production.

The downside of this approach is the diverging state between the branches. Current process is to from time to time compare the branches and manually check the diffs for unplanned discrepancies.

Procedures for converting a legacy database to Liquibase migrations

Some projects are complete monoliths. More than one application connects to it, and this is not a good practice, if you do that I recommend you treating the database sourcing as in its own repository and not together with your application.

Writing migrations

This is a way I found for keeping the structure in reasonable sense. Suggestions are welcome.

1. Create the property file

Should be in the root of the project and be named liquibase.properties:

driver: com.mysql.jdbc.Driver
classpath: /usr/share/java/mysql-connector-java.jar:/usr/share/java/snakeyaml.jar
url: jdbc:mysql://localhost:3306/mydb
username: root
password: 123

The JAR files in the classpath can be manually downloaded or installed though the server package manager.

Create the Migration file

You can choos e between different formats, I chose to use JSON, in this instance I will be running this SQL:

Which will translate to this:

It is verbose? Yes, completely, but then you have a tool to show you what the SQL will look like and be able to manage the rollbacks.

Save the file as:

.
  /changes
    - changelog.json
    - create_mydb_users.json

Where changelog.json looks like this:

For each new change you add it to the end of the databaseChangeLog array.

Run it

To run, just simply do:

$ liquibase --changeLogFile=changes/changelog.json migrate

Don't worry if you run it twice, the change only happens once.

Next post is how to add a legacy DB into Liquibase.

In the previous blog post we explained how to create schemas and collections. In this one we are going to explain how to work with collections: adding, updating and deleting documents.

The following code demonstrates how to add a single document to an existing collection:

var mysqlx = require('mysqlx');
mysqlx.getSession({
  host: 'host',
  port: '33060',
  dbUser: 'root',
  dbPassword: 'my pass'
}).then(function (session) {
  var schema = session.getSchema('mySchema');
  var coll = schema.getCollection('myColl');
  var newDoc = { name: 'Test Name', description: 'Test Description' };

  coll.add(newDoc).execute().then(function (added) {
    console.log('Document(s) added: '
                + added.getAffectedItemsCount());
    session.close();
  })
  .catch(function (err) {
    console.log(err.message);
    console.log(err.stack);
  });
}).catch(function (err) {
  console.log(err.message);
  console.log(err.stack);
});

In the previous code, first we get the objects that represent the schema (schema) and the collection (coll) that we want to work with. An object using JSON (newDoc) is created and is passed as a parameter to the coll object’s add method. Calling the execute method the document is added to the collection. Once the execute method has finished, we receive an object (added) that contains information about the document added. To verify if the document was added, we can call the added object’s getAffectedItemsCount method, which will return how many documents were added.

But, what if you want to add multiple documents?

You can do it with almost no changes in your code. The following code adds two documents at the same time:

var mysqlx = require('mysqlx');
mysqlx.getSession({
  host: 'host',
  port: '33060',
  dbUser: 'root',
  dbPassword: 'my pass'
}).then(function (session) {
  var schema = session.getSchema('mySchema');
  var coll = schema.getCollection('myColl');
  var newDoc = { name: 'Test Name', description: 'Test Description' };
  var newDoc2 = { name: 'Test Name 2', description: 'Test Description 2' };

  coll.add(newDoc , newDoc2).execute().then(function (added) {
    console.log('Document(s) added: '
                + added.getAffectedItemsCount());
    session.close();
  })
  .catch(function (err) {
    console.log(err.message);
    console.log(err.stack);
  });
}).catch(function (err) {
  console.log(err.message);
  console.log(err.stack);
});

As you can see, the previous code is almost identical to the first example. We just add an extra line to declare the new document (newDoc2), and we add the new document as a parameter in the coll object’s add method. At the end we call added object’s getAffectedItemsCount method to verify we added the two documents.

Now we know how to add multiples documents to a collection using multiple variables and passing them as parameters, but we can also do the same using an Array object. In the following code example we create an array object to use it to add new documents to a collection.

var mysqlx = require('mysqlx');
mysqlx.getSession({
  host: 'host',
  port: '33060',
  dbUser: 'root',
  dbPassword: 'my pass'
}).then(function (session) {
  var schema = session.getSchema('mySchema');
  var coll = schema.getCollection('myColl');
  var newDocs = [{ name: 'Test Name', description: 'Test Description' }, 
                 { name: 'Test Name 2', description: 'Test Description 2' }];

  coll.add(newDocs).execute().then(function (added) {
    console.log('Document(s) added: '
                + added.getAffectedItemsCount());
    session.close();
  })
  .catch(function (err) {
  console.log(err.message);
  console.log(err.stack);
  });
}).catch(function (err) {
  console.log(err.message);
  console.log(err.stack);
});

The previous code is almost identical to the first example; the difference is that we pass an array object as a parameter instead of a JSON object. The rest of the code is the same. This could be useful if you receive an array of objects from the client or if you load the data from a JSON file, you just pass the whole array to upload it to the collection.

Updating a field in a document is also very easy to do. The following code is an example of how to do it:

var mysqlx = require('mysqlx');
mysqlx.getSession({
  host: 'host',
  port: '33060',
  dbUser: 'root',
  dbPassword: 'my pass'
}).then(function (session) {
  var schema = session.getSchema('mySchema');
  var coll = schema.getCollection('myColl');
  var query = "$._id == 'f0a743ac-a052-d615-1f8c-ef65ebc4'";

  coll.modify(query)
  .set('$.name', 'New Name')
  .set('$.description', 'New Description')
  .execute()
  .then(function (updated) {
    console.log('Document(s) updated: '
                + updated.getAffectedItemsCount());
    session.close();
  })
  .catch(function (err) {
    console.log(err.message);
    console.log(err.stack);
  });
}).catch(function (err) {
  console.log(err.message);
  console.log(err.stack);
});

In the previous code, first we get the objects that represent the schema (schema) and the collection (coll) we want to work. Then we declare the query variable which contains the where clause for our update. Next, we call the coll object’s modify method that receives the query variable as a parameter. Chained to the modify method is the set method, which receives a pair of objects; the first one is the field to update and the second one the new value to be set. As we did in our previous examples we call the execute method to perform the action requested. When the execute method finishes we receive an object (updated) with information about the update. To know how many documents were updated we call the updated object’s getAffectedItemsCount method.

Now that we know how to add and update documents in a collection, we are going to explain how to remove them. The following code demonstrates it.

var mysqlx = require('mysqlx');
mysqlx.getSession({
  host: 'host',
  port: '33060',
  dbUser: 'root',
  dbPassword: 'my pass'
}).then(function (session) {
  var schema = session.getSchema('mySchema');
  var coll = schema.getCollection('myColl');
  var query = "$._id == 'f0a743ac-a052-d615-1f8c-ef65ebc4'";

  coll.remove(query).execute().then(function (deleted) {
    console.log('Document(s) deleted: ' 
                + deleted.getAffectedItemsCount());
    session.close();
  })
  .catch(function (err) {
    console.log(err.message);
    console.log(err.stack);
  });
}).catch(function (err) {
  console.log(err.message);
  console.log(err.stack);
});

The previous code defines the objects that represent the schema (schema) and the collection (coll) where we want to work. Then, we define the query variable again to contain the where clause for our operation, the remove in this case. To remove a document we call the coll object’s remove method followed by the execute method. Once the execute method is completed, we receive an object (deleted) with the information about the operation that has finished. By calling the deleted object’s getAffectedItemsCount method, we know how many documents were removed from the collection.

Now we are going to see how to get documents from a collection. In the following example, we are retrieving the document that match the _id that we want:

var mysqlx = require('mysqlx');
mysqlx.getSession({
  host: 'host',
  port: '33060',
  dbUser: 'root',
  dbPassword: 'my pass'
}).then(function (session) {
  var schema = session.getSchema('mySchema');
  var coll = schema.getCollection('myColl');
  var query = "$._id == 'f0a743ac-a052-d615-1f8c-ef65ebc4'";

  coll.find(query).execute(function (doc) {
    console.log(doc);
  })
  .catch(function (err) {
    console.log(err.message);
    console.log(err.stack);
  });
  session.close();
}).catch(function (err) {
  console.log(err.message);
  console.log(err.stack);
});

The previous code defines the objects that represent the schema (schema) and the collection (coll) we want to work with. Then the query variable is defined and the where clause is set to it. Then we call the coll object’s execute method to perform the query. When the execute method completes, we receive the document that match our search criteria and is send to the console to view it.

But, what if we want all the records from a collection? Well that is simple; we just need to remove the search criteria from the find method. The updated code would look like the following:

var mysqlx = require('mysqlx');
mysqlx.getSession({
  host: 'host',
  port: '33060',
  dbUser: 'root',
  dbPassword: 'my pass'
}).then(function (session) {
  var schema = session.getSchema('mySchema');
  var coll = schema.getCollection('myColl');

  coll.find().execute(function (doc) {
    console.log(doc);
  })
  .catch(function (err) {
    console.log(err.message);
    console.log(err.stack);
  });
  session.close();
}).catch(function (err) {
  console.log(err.message);
  console.log(err.stack);
});

Now we know how to search a specific document and get all the documents from a collection. What if we want to get just some number of documents that match query criteria? The next example shows the code to do it:

var mysqlx = require('mysqlx');
mysqlx.getSession({
  host: 'host',
  port: '33060',
  dbUser: 'root',
  dbPassword: 'my pass'
}).then(function (session) {
  var schema = session.getSchema('mySchema');
  var coll = schema.getCollection('myColl');
  var query = "$.name like '%Test%'";

  coll.find(query).limit(3).execute(function (doc) {
    console.log(doc);
  })
  .catch(function (err) {
    console.log(err.message);
    console.log(err.stack);
  });
  session.close();
}).catch(function (err) {
  console.log(err.message);
  console.log(err.stack);
});

The previous code looks very similar to the example that returns one document with a specific _id, the difference here is that our query is performing a like and we are adding a call to the coll object’s limit method. Note that the query statement is case sensitive; this means that if we have documents that have  ‘test’ in the field ‘name’, those documents will not be returned because we are searching for ‘Test’ names.

See you in the next blog post.

Recently I upgraded some MySQL databases from 5.6 to 5.7, but -- for boring reasons unique to my environment -- I had to leave one replica on version 5.6. I knew there was a chance that the 5.7 -> 5.6 replication wouldn't work, but I decided to try it out to see if (and why) it would fail. Once I upgraded the master, replication failed, so I checked the error log on the replica and found these messages:

[ERROR] Slave I/O: Found a Gtid_log_event or Previous_gtids_log_event when @@GLOBAL.GTID_MODE = OFF. Error_code: 1784 [ERROR] Slave I/O: Relay log write failure: could not queue event from master, Error_code: 1595

The error surprised me a little bit since I'm not using GTIDs in that replication topology. I asked around a bit, and Kenny Gryp hypothesized that I might be experiencing MySQL bug #74683, which was fixed in version 5.6.23. Since my replica was on 5.6.22, I decided to do an incremental 5.6 upgrade to see if that resolved the issue. I upgraded to 5.6.31 and replication started working.

YMMV and there are certainly bound to be 5.7 -> 5.6.31 replication scenarios that don't work, but this was a simple fix for me. In hindsight it makes sense that replicating from a new major version to an older major version is more likely to work with a more recent minor version, so if I have to do this in the future I'll make sure the replica is running the latest minor version before upgrading the master.

P.S. Thanks Kenny!

Pardon me if my memory fails a bit, I am trying to recount the events as they happened, but my memory is not my strongest suit.

I think I had a bit of different path than most PHP Software Developers. My first job was at a company that had their own CRM and they wanted a web version of some parts of the system. At that time, I had only “experience” with ASP and Microsoft Access (I know, tough). They wanted it to be with PHP, the difference I think came when they said they wanted the integration to run directly into the database. The web app would write directly into the DB. The CRM system was written using Delphi and Firebird. So I learned the PHP and my first database wasn’t MySQL (I don’t count MS Access as a DB). After that I got a job which MySQL was used, it was a bit weird at the time learning that MyISAM (I was really fresh on MySQL, and I didn’t know about the engines and such) didn’t had foreign keys for instance.

After that I got a job in a huge multinational where they had this project of migrating every Excel spreadsheet to a PHP program, VBA was heavily used there and they had entire programs running into that, what they didn't tell us was that was cheaper for them having a whole team of PHP developers doing an internal system then have the features built into their ERP. But for “security” reasons, no business logic could be inside the PHP code, so I had to do tons of Stored Procedures. They also had integrations with MS SQL Server, the workflow system used it together with a PHP tool called Scriptcase.

Another job I had was with another multinational where I had to do a program to read from various sources and store in a DB for later reports and all, that would be a “scratch” of an ETL (Extract, Transform, Load) but at the time I wasn’t really well versed and data warehouse techniques. For that job I used PostgreSQL. In the same company we later on did Magento white label stores for our clients (other big companies) and it had to have integration to our ERP, differently from my first job, the integration was through a Service Bus written in Java and the ERP had Oracle as DB.

I could go on and on to every job I had, so when I said my path was a bit different, I say that because most of my jobs were at companies considered corporate ones, and that put me into contact with many flavours of Relational Databases and NoSQL too (MongoDB, Dynamo and Cassandra being the main ones).

In the end, that kind of exposure made me be the “DB” person among my fellow engineers, and everything considered more than a CRUD to it would fall into my lap. And I would be happy with it, I noticed that other developers didn't like much doing database tasks. One of my employers noticed my interest on it and how quickly I became familiarised with the schema and our data architecture. Our main relational database had millions and millions of records, terabytes and they created a Data Team to make it more efficient. We would create workers to sync data into Elasticsearch for instance. I was still a PHP Developer officially, but mainly curating the DB and doing workers with NodeJS (we needed the async and the indexing time was crucial for business).

That’s when I discovered that I wanted to work with data, I didn’t have any idea of what kind of title that would be, I knew I didn’t want to be a DBA, too much infrastructure involved on it and it didn’t sound fun for me. Once the opportunity to work officially as a Data Engineer appeared, I grabbed it with my both hands and I knew it was right.

Last week I wrote about aDNS discovery feature in Etcd. As a step in the whole process we need to create DNS records in twindb.com zone. CloudFlare provides rich API to work with it. We wrapped it into a Python module twindb_cloudflare and opensourced it:

• Source code: https://github.com/twindb/twindb_cloudflare
• Documentation: https://twindb-cloudflare.readthedocs.io
• PyPi package: https://pypi.python.org/pypi/twindb_cloudflare

In the post I will show how to use twindb_cloudflare module.

CloudFlare API credentials

First of all you need to get credentials to work with CloudFlare. Visit https://www.cloudflare.com/a/account/my-account and get “API Key”.

CLOUDFLARE_EMAIL = "aleks@twindb.com"
CLOUDFLARE_AUTH_KEY = "dbb4a7ae063347a306e9ad8c5bda58a7a3cfa"

Installing twindb_cloudflare

The module is available in PyPi. You can install it with pip:

$ pip install twindb_cloudflare
Collecting twindb_cloudflare
  Using cached twindb_cloudflare-0.1.1-py2.py3-none-any.whl
Installing collected packages: twindb-cloudflare
Successfully installed twindb-cloudflare-0.1.1

Creating A record

To create an A record

import socket
import time
from twindb_cloudflare.twindb_cloudflare import CloudFlare, CloudFlareException

CLOUDFLARE_EMAIL = "aleks@twindb.com"
CLOUDFLARE_AUTH_KEY = "dbb4a7ae063347a306e9ad8c5bda58a7a3cfa"

cf = CloudFlare(CLOUDFLARE_EMAIL, CLOUDFLARE_AUTH_KEY)

try:
    cf.create_dns_record('blogtest.twindb.com', 'twindb.com', '10.10.10.10')
    # The new record isn't available right away
    wait_until = time.time() + 600
    while time.time() < wait_until:
        try:
            ip = socket.gethostbyname('blogtest.twindb.com')
            print(ip)
            exit(0)
        except socket.gaierror:
            time.sleep(1)
    print("New record isn't available after 600 seconds")
    exit(-1)
except CloudFlareException as err:
    print(err)
    exit(-1)

The script runs a minute or two:

[16:41:51 aleks@Aleksandrs-MacBook-Pro mydns]$ python create.py
10.10.10.10
[16:43:59 aleks@Aleksandrs-MacBook-Pro mydns]$

Updating A record

Updating a record is pretty straightforward either. In real applications you have to take into account that a DNS response may be cached and CloudFlare changes aren’t available momentarily.

import socket
import time
from twindb_cloudflare.twindb_cloudflare import CloudFlare, CloudFlareException

CLOUDFLARE_EMAIL = "aleks@twindb.com"
CLOUDFLARE_AUTH_KEY = "dbb4a7ae063347a306e9ad8c5bda58a7a3cfa"

cf = CloudFlare(CLOUDFLARE_EMAIL, CLOUDFLARE_AUTH_KEY)

try:
    cf.update_dns_record('blogtest.twindb.com', 'twindb.com', '10.20.20.20')
    # The new record isn't available right away
    wait_until = time.time() + 600
    while time.time() < wait_until:
        try:
            ip = socket.gethostbyname('blogtest.twindb.com')
            if ip == '10.20.20.20':
                print(ip)
                exit(0)
            else:
                time.sleep(1)
        except socket.gaierror:
            time.sleep(1)
    print("New record isn't updated after 600 seconds")
    exit(-1)
except CloudFlareException as err:
    print(err)
    exit(-1)

The change is visible after a minute

[16:54:02 aleks@Aleksandrs-MacBook-Pro mydns]$ python myupdate.py
10.20.20.20
[16:55:03 aleks@Aleksandrs-MacBook-Pro mydns]$

Deleting A record

And finally let’s delete the record we’ve created. Here I won’t wait until the change is propagated and will assume that if there was no exception then we are good.

from twindb_cloudflare.twindb_cloudflare import CloudFlare, CloudFlareException

CLOUDFLARE_EMAIL = "aleks@twindb.com"
CLOUDFLARE_AUTH_KEY = "dbb4a7ae063347a306e9ad8c5bda58a7a3cfa"

cf = CloudFlare(CLOUDFLARE_EMAIL, CLOUDFLARE_AUTH_KEY)

try:
    cf.delete_dns_record('blogtest.twindb.com', 'twindb.com')
except CloudFlareException as err:
    print(err)
    exit(-1)

Final notes

ClaudFlare API provides many more actions than the module implements. However I wanted to start with something small and incrementally implementing more feature as they are needed.

I encourage you to file bugs and feature requests on https://github.com/twindb/twindb_cloudflare/issues.
Also, pull requests are welcomed

The post Working with CloudFlare DNS in python appeared first on Backup and Data Recovery for MySQL.

Percona announces the GA release of Percona XtraBackup 2.4.4 on July 25th, 2016. You can download it from our download site and from apt and yum repositories.

Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies with large data sets and mission-critical applications that cannot tolerate long periods of downtime. Offered free as an open source solution, Percona XtraBackup drives down backup costs while providing unique features for MySQL backups.

New Features:

• Percona XtraBackup has been rebased on MySQL 5.7.13.

Bugs Fixed:

• Percona XtraBackup reported the difference in the actual size of the system tablespace and the size which was stored in the tablespace header. This check is now skipped for tablespaces with autoextend support. Bug fixed #1550322.
• Because Percona Server 5.5 and MySQL 5.6 store the LSN offset for large log files at different places inside the redo log header, Percona XtraBackup was trying to guess which offset is better to use by trying to read from each one and compare the log block numbers and assert lsn_chosen == 1 when both LSNs looked correct, but they were different. Fixed by improving the server detection. Bug fixed #1568009.
• Percona XtraBackup didn’t correctly detect when tables were both compressed and encrypted. Bug fixed #1582130.
• Percona XtraBackup would crash if the keyring file was empty. Bug fixed #1590351.
• Backup couldn’t be prepared when the size in cache didn’t match the physical size. Bug fixed #1604299.
• Free Software Foundation address in copyright notices was outdated. Bug fixed #1222777.
• Backup process would fail if the datadir specified on the command-line was not the same as one that is reported by the server. Percona XtraBackup now allows the datadir from my.cnf override the one from SHOW VARIABLES. xtrabackup prints a warning that they don’t match, but continues. Bug fixed #1526467.
• With upstream change of maximum page size from 16K to 64K, the size of incremental buffer became 1G. Which increased the requirement to 1G of RAM in order to prepare the backup. While in fact there is no need to allocate such a large buffer for smaller pages. Bug fixed #1582456.
• Backup process would fail on MariaDB Galera cluster operating in GTID mode if binary logs were in non-standard directory. Bug fixed #1517629.

Other bugs fixed: #1583717, #1583954, and #1599397.

Release notes with all the bugfixes for Percona XtraBackup 2.4.4 are available in our online documentation. Please report any bugs to the launchpad bug tracker.

As work on WarpSQL (Shard-Query 3) progresses, it has outgrown MySQL proxy.  MySQL proxy is a very useful tool, but it requires LUA scripting, and it is an external daemon that needs to be maintained.  The MySQL proxy module for Shard-Query works well, but to make WarpSQL into a real distributed transaction coordinator, moving the proxy logic inside of the server makes more sense.

The main benefit of MySQL proxy is that it allows a script to "inject" queries between the client and server, intercepting the results and possibly sending back new results to the client.  I would like similar functionality, but inside of the server.

For example, I would like to implement new SHOW commands, and these commands do not need to be implemented as actual MySQL SHOW commands under the covers.

For example, for this blog post I made a new example command called "SHOW PASSWORD"

Example "injection" which adds SHOW PASSWORD functionality to the server

mysql> select user();
+----------------+
| user()         |
+----------------+
| root@localhost |
+----------------+
1 row in set (0.00 sec)

-- THIS COMMAND DOES NOT EXIST
mysql> show password;
+-------------------------------------------+
| password_hash                             |
+-------------------------------------------+
| *00A51F3F48415C7D4E8908980D443C29C69B60C9 |
+-------------------------------------------+
1 row in set (0.00 sec)

Important - This isn't a MySQL proxy plugin.  There is C++ code in the SERVER to answer that query, but it isn't the normal SHOW command code.  This "plugin" (I put it in quotes because my plan is for a pluggable interface but it isn't added to the server yet) doesn't access the mysql.user table using normal internal access methods. It runs actual SQL inside of the server, on the same THD as the client connection, in the same transaction as the client connection, to get the answer!

Problem #1 - Running SQL in the server

The MySQL C client API doesn't have any methods for connecting to the server from inside of the server, except to connect to the normally available socket interfaces, authenticate, and then issue queries like a normal client.  While it is perfectly possible to connect to the server as a client in this manner, it is sub-optimal for a number of reasons.  First, it requires a second connection to the server, second, it requires that you authenticate again (which requires you have the user's password), and lastly, any work done in the second connection is not party to transactional changes in the first, and vice-versa.

The problem is communication between the client and server, which uses a mechanism called VIO.  There was work done a long time ago for external stored procedures, which never made it into the main server that would have alleviated this problem by implementing a in-server VIO layer, and making the parser re-entrant.  That work was done on MySQL 5.1 though.

It is possible to run queries without using VIO though.  You simply can't get results back, except to know if the query succeeded or not.  This means it is perfectly acceptable for any command that doesn't need a resultset, basically anything other than SELECT.  There is a loophole however, in that any changes made to the THD stay made to that THD.  Thus, if the SQL executed sets any user variables, then those variables are of course visible after query execution.

Solution  - encapsulate arbitrary SQL resultsets through a user variable

Since user variables are visible after query execution, the goal is to get the complete results of a query into a user variable, so that the resultset can be accessed from the server.  To accomplish this, first a method to get the results into the variable must be established, and then some data format for communication that is amenable to that method has to be decided upon so that the resultset can be accessed conveniently..

With a little elbow grease MySQL can convert any SELECT statement into CSV resultset.  To do so, the following are used:

1. SELECT ... INTO @user_variable


2. A subquery in the FROM clause (for the original query)


3. CONCAT, REPLACE, IFNULL, GROUP_CONCAT (to encode the resultset data)

Here is the SQL that the SHOW PASSWORD command uses to get the correct password:

select authentication_string as pw,
       user 
  from mysql.user 
 where concat(user,'@',host) = USER() 
    or user = USER() 
LIMIT 1

Here is the "injected" SQL that the database generates to encapsulate the SQL resultset as CSV:

select 
  group_concat( 
    concat('"',
           IFNULL(REPLACE(REPLACE(`pw`,'"','\\"'),"\n","\\n"),"\N"),
           '"|"',
           IFNULL(REPLACE(REPLACE(`user`,'"','\\"'),"\n","\\n"),"\N"),
           '"'
    ) 
  separator "\n"
  ) 
from 
  ( select authentication_string as pw,
           user 
      from mysql.user 
      where concat(user,'@',host) = USER() 
        OR user = USER() 
    LIMIT 1
  ) the_query 
into @sql_resultset ;
Query OK, 1 row affected (0.00 sec)

Here is the actual encapsulated resultset.  If there were more than one row, they would be newline separated.

mysql> select @sql_resultset;
+----------------+
| @sql_resultset |
+----------------+
| ""|"root"      |
+----------------+
1 row in set (0.00 sec)

Injecting SQL in the server

With the ability to encapsulate resultsets into CSV in user variables, it is possible to create a cursor over the resultset data and access it in the server.  The MySQL 5.7 pre-parse rewrite plugins, however,  still run inside the parser.  The THD is not "clean" with respect to being able to run a second query.  The parser is not re-entrant.  Because I desire to run (perhaps many) queries between the time a user enters a query and the server actually answers the query (perhaps with a different query than the user entered!) the MySQL 5.7 pre-parse rewrite plugin infrastructure doesn't work for me.

I modified the server, instead, so that there is a hook in do_command() for query injections.  I called it conveniently query_injection_point() and the goal is to make it a new plugin type, but I haven't written that code yet.  Here is the current signature for query_injection_point():

bool query_injection_point(
  THD* thd, COM_DATA *com_data, enum enum_server_command command,
  COM_DATA* new_com_data, enum enum_server_command* new_command );

It has essentially the same signature as dispatch_command(), but it provides the ability to replace the command, or keep it as is.  It returns true when the command has been replaced.

Because it is not yet pluggable, here is the code that I placed in the injection point:

/* TODO: make this pluggable */
bool query_injection_point(THD* thd, COM_DATA *com_data, enum enum_server_command command,
 COM_DATA* new_com_data, enum enum_server_command* new_command)
{
 /* example rewrite rule for SHOW PASSWORD*/
 if(command != COM_QUERY)
 { return false; }
 
 /* convert query to upper case */
 std::locale loc;
 std::string old_query(com_data->com_query.query,com_data->com_query.length);
 for(unsigned int i=0;i<com_data->com_query.length;++i) {
   old_query[i] = std::toupper(old_query[i], loc);
 } 
   
 if(old_query == "SHOW PASSWORD")
 {
   std::string new_query;
   SQLClient conn(thd);
   SQLCursor* stmt;
   SQLRow* row;

   if(conn.query("pw,user",
    "select authentication_string as pw,user from mysql.user " \
    "where concat(user,'@',host) = USER() or user = USER() LIMIT 1", &stmt))
   {
     if(stmt != NULL)
     {
       if((row = stmt->next()))
       {
          new_query = "SELECT '" + row->at(0) + "' as password_hash";
       }
       } else
       {
         return false;
       }
     } else {
       return false;
     }

     /* replace the command sent to the server */
     if(new_query != "")
     {
       Protocol_classic *protocol= thd->get_protocol_classic();
       protocol->create_command(
         new_com_data, COM_QUERY, 
         (uchar *) strdup(new_query.c_str()), 
         new_query.length()
       );
       *new_command = COM_QUERY;
     } else {
       if(stmt) delete stmt;
       return false;
     }
     if(stmt) delete stmt;
     return true;
   }
 }

 /* don't replace command */
 return false;
}

SQLClient

You will notice that the code access the mysql.user table using SQL, using the SQLClient, SQLCursor, and SQLRow objects.  These are the objects that wrap around encapsulating the SQL into a CSV resultset, and actually accessing the result set.  The interface is very simple, as you can see from the example.  You create a SQLClient for a THD (one that is NOT running a query already!) and then you simply run queries and access the results.

The SQLClient uses a stored procedure to methodically encapsulate the SQL into CSV and then provides objects to access and iterate over the data that is buffered in the user variable.  Because MySQL 5.7 comes with the sys schema, I placed the stored procedure into it, as there is no other available default database that allows the creation of stored procedures.  I called it sys.sql_client().

Because the resultset is stored as text data, the SQLRow object returns all column values as std::string.

What's next?

I need to add a proper plugin type for "SQL injection plugins".  Then I need to work on a plugin for parallel queries.  Most of the work for that is already done, actually, at least to get it into an alpha quality state.  There is still quite a bit of work to be done though.

You can find the code in the internal_client branch of my fork of MySQL 5.7:

http://github.com/greenlion/warpsql-server

• The Billing team is responsible for the financially critical data in the company. The data we generate on a daily basis for subscription charges, gift cards, credits, chargebacks, etc. is rolled up to finance and is reported into the Netflix accounting. We have stringent SLAs on our daily processing to ensure that the revenue gets booked correctly for each day. We cannot tolerate delays in processing pipelines.
• Billing has zero tolerance for data loss.
• For most parts, the existing data was structured with a relational model and necessitates use of transactions to ensure an all-or-nothing behavior. In other words we need to be ACID for some operations. But we also had use-cases where we needed to be highly available across regions with minimal replication latencies.
• Billing integrates with the DVD business of the company, which has a different architecture than the Streaming component, adding to the integration complexity.
• The Billing team also provides data to support Netflix Customer Service agents to answer any member billing issues or questions. This necessitates providing Customer Support with a comprehensive view of the data.

• 2 Oracle databases in the Data Center - One storing the customer subscription information and other storing the invoice/payment data.
• Multiple REST-based applications - Serving calls from the www.netflix.com and Customer support applications. These were essentially doing the CRUD operations
• 3 Batch applications -

• Subscription Renewal - A daily job that looks through the customer base to determine the customers to be billed that day and the amount to be billed by looking at their subscription plans, discounts, etc.
• Order & Payment Processor - A series of  batch jobs that create an invoice to charge the customer to be renewed and process the invoice through various stages of the invoice lifecycle.
• Revenue Reporting - A daily job that looks through billing data and generates reports for the Netflix Finance team to consume.

• One Billing Proxy application (in the Cloud) - used to route calls from rest of Netflix applications in the Cloud to the Data Center.
• Weblogic queues with legacy formats being used for communications between processes.

• Act I - Launch new countries directly in the Cloud on the billing side while syncing the data back to the Data Center for legacy batch applications to continue to work.
• Act II - Model the user-facing data, which could live with eventual consistency and does not need to be ACID, to persist to Cassandra (Cassandra gave us the ability to perform writes in one region and make it available in the other regions with very low latency. It also gives us high-availability across regions).
• Act III - Finally move the SQL databases to the Cloud.

• Disable the non-GET apis for the country under migration. (This would not impact members, but delay any updates to subscriptions in billing)
• Use Sqoop jobs to get the data from Oracle to S3 and Hive.
• Transform it to the Cassandra format using Pig.
• Insert the records for all members for that country into Cassandra.
• Enable the non-GET apis to now serve data from the Cloud for the country that was migrated.

The end of a two-part series on the tech stack that Uber Engineering uses to make transportation as reliable as running water, everywhere, for everyone, as of spring 2016.

The post The Uber Engineering Tech Stack, Part II: The Edge and Beyond appeared first on Uber Engineering Blog.

Uber Engineering explains the technical reasoning behind its switch in database technologies, from Postgres to MySQL.

The post Why Uber Engineering Switched from Postgres to MySQL appeared first on Uber Engineering Blog.

This blog post will detail the results of Samsung storage in

tpcc-mysql

I had an opportunity to test different Samsung storage devices under tpcc-mysql benchmark powered by Percona Server 5.7. You can find a summary with details here https://github.com/Percona-Lab-results/201607-tpcc-samsung-storage/blob/master/summary-tpcc-samsung.md

I have in my possession:

• Samsung 850 Pro, 2TB: This is a SATA device and is positioned as consumer-oriented, something that you would use in a high-end user desktop. As of this post, I estimate the price of this device as around $430/TB.
• Samsung SM863, 1.92TB: this device is also a SATA, and positioned for a server usage. The current price is about $600/TB. 
• Samsung PM1725, 800GB: This is an NVMe device, in a 2.5″ form factor, but it requires a connection to a PCIe slot, which I had to allocate in my server. The device is high-end, oriented for server-side and demanding workloads. The current price is about $1300/TB.

I am going to use 1000 warehouses in the 

tpcc-mysql

This benchmark varies the

innodb_buffer_pool_size

All buffer pool sizes in the middle of the interval correspond to resulting IO reads and writes. For example, we can see the read to write ratio on the chart below (received for the PM1725 device) with different buffer pool sizes:

We can see that for the 5GB buffer pool size we have 56000 read IOPs operations and 32000 write IOPs. For 115GB, the reads are minimal at about 300 IOPS and the background writes are at the 20000 IOPs level. Reads gradually decline with the increasing buffer pool size.

The charts are generated with the Percona Monitoring and Management tools.

Results

Let’s review the results. The first chart shows measurements taken every one second, allowing us to see the trends and stalls.

If we take averages, the results are:

In table form (the results are in new order transactions per minute (NOTPM)):

Conclusion

The Samsung 850 obviously can’t keep with the more advanced SM863 and PM1725. The PM1725 shows a greater benefit with smaller buffer pool sizes. In cases using large amounts of memory, there is practically no difference with SM863. The reason is that with big buffer pool sizes, MySQL does not push IO subsystem much to use all the PM1725 performance.

For the reference, my.cnf file is

[mysqld]
datadir=/var/lib/mysql
socket=/tmp/mysql.sock
ssl=0
symbolic-links=0
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
# general
thread_cache_size=2000
table_open_cache = 200000
table_open_cache_instances=64
back_log=1500
query_cache_type=0
max_connections=4000
# files
innodb_file_per_table
innodb_log_file_size=15G
innodb_log_files_in_group=2
innodb_open_files=4000
innodb_io_capacity=10000
loose-innodb_io_capacity_max=12000
innodb_lru_scan_depth=1024
innodb_page_cleaners=32
# buffers
innodb_buffer_pool_size= 200G
innodb_buffer_pool_instances=8
innodb_log_buffer_size=64M
# tune
innodb_doublewrite= 1
innodb_support_xa=0
innodb_thread_concurrency=0
innodb_flush_log_at_trx_commit= 1
innodb_flush_method=O_DIRECT_NO_FSYNC
innodb_max_dirty_pages_pct=90
join_buffer_size=32K
sort_buffer_size=32K
innodb_use_native_aio=0
innodb_stats_persistent = 1
# perf special
innodb_adaptive_flushing = 1
innodb_flush_neighbors = 0
innodb_read_io_threads = 16
innodb_write_io_threads = 8
innodb_purge_threads=4
innodb_adaptive_hash_index=0
innodb_change_buffering=none
loose-innodb-log_checksum-algorithm=crc32
loose-innodb-checksum-algorithm=strict_crc32
loose-innodb_sched_priority_cleaner=39
loose-metadata_locks_hash_instances=256

In this blog, we’ll discuss monitoring MongoDB with Nagios.

There is a significant amount of talk around graphing MongoDB metrics using things like Prometheus, Data Dog, New Relic, and Ops Manager from MongoDB Inc. However, I haven’t noticed a lot of talk around “What MongoDB alerts should I be setting up?”

While building out Percona’s remote DBA service for MongoDB, I looked at Prometheus’s AlertManager. After reviewing it, I’m not sure it’s quite ready to be used exclusively. We needed to decide quickly if there are better Nagios checks on the market, or did I need to write my own?

In the end, we settled on a hybrid approach. There are some good frameworks, but we need to create or tweak some of the things needed for an “SEV 1-” or “SEV 2-” type issue (which are most important to me). One of the most common problems for operations, Ops, DevOps, DBA teams and most engineering is alert spam. As such I wanted to be very careful to only alert on the things pointing to immediate dangers or current outages. As a result, we have now added

pmp-check-mongo.py

As part of this push, I want to give you some real world examples of how you might use this tool. There are many options available to you, and Nagios is still a bit green in regards to making those options as user-friendly as our tools are.

Usage: pmp-check-mongo.py [options]
Options:
  -h, --help                         show this help message and exit
  -H HOST, --host=HOST               The hostname you want to connect to
  -P PORT, --port=PORT               The port mongodb is running on
  -u USER, --user=USER               The username you want to login as
  -p PASSWD, --password=PASSWD       The password you want to use for that user
  -W WARNING, --warning=WARNING      The warning threshold you want to set
  -C CRITICAL, --critical=CRITICAL   The critical threshold you want to set
  -A ACTION, --action=ACTION         The action you want to take. Valid choices are
                                     (check_connections, check_election, check_lock_pct,
                                     check_repl_lag, check_flushing, check_total_indexes,
                                     check_balance, check_queues, check_cannary_test,
                                     check_have_primary, check_oplog, check_index_ratio,
                                     check_connect) Default: check_connect
  -s SSL, --ssl=SSL                  Connect using SSL
  -r REPLICASET, --replicaset=REPLICASET    Connect to replicaset
  -c COLLECTION, --collection=COLLECTION    Specify the collection in check_cannary_test
  -d DATABASE, --database=DATABASE          Specify the database in check_cannary_test
  -q QUERY, --query=QUERY                   Specify the query, only used in check_cannary_test
  --statusfile=STATUS_FILENAME      File to current store state data in for delta checks
  --backup-statusfile=STATUS_FILENAME_BACKUP    File to previous store state data in for delta checks
  --max-stale=MAX_STALE             Age of status file to make new checks (seconds)

There seems to be a huge amount going on here, but let’s break it down into a few categories:

• Connection options
• Actions
• Action options
• Status options

Hopefully, this takes some of the scariness out of the script above.

Connection options

• Host / Port Number
  • Pretty simple, this is just the host you want to connect to and what TCP port it is listening on.
• Username and Password
  • Like with Host/Port, this is some of your normal and typical Mongo connection field options. If you do not set both the username and password, the system will assume auth was disabled.
• SSL
  • This is mostly around the old SSL support in Mongo clients (which was a boolean). This tool needs updating to support the more modern SSL connection options. Use this as a “deprecated” feature that might not work on newer versions.
• ReplicaSet
  • Very particular option that is only used for a few checks and verifies that the connection uses a replicaset connection. Using this option lets the tool automatically find a primary node for you, and is helpful to some checks specifically around replication and high availability (HA):
    • check_election
    • check_repl_lag
    • check_cannary_test
    • chech_have_primary
    • check_oplog

Actions and what they mean

• check_connections
  • This parameter refers to memory usage, but beyond that you need to know if your typical connections suddenly double. This indicates something unexpected happened in the application or database and caused everything to reconnect. It often takes up to 10 minutes for those old connections to go away.
• check_election
  • This uses the status file options we will cover in a minute, but it checks to see if the primary from the last check differs from the current found primary. If so, it alerts. This check should only have a threshold of one before it alarms (as an alert means an HA event occurred).
• check_lock_pct
  • MMAP only, this engine has a write lock on the whole collection/database depending on the version. This is a crucial metric to determine if MMAP writes are blocking reads, meaning you need to scale the DB layer in some way.
• check_repl_lag
  • Checks the replication stream to understand how lagged a given node is the primary. To accomplish this, it uses a fake record in the test DB to cause a write. Without this, a read-only system would look lagged artificially as no new oplog entries get created.
• check_flushing
  • A common issue with MongoDB is very long flush times, causing a system halt. This is a caused by your disk subsystem not keeping up, and then the DB having to wait on flushing to make sure writes get correctly journaled.
• check_total_indexes
  • The more indexes you have, the more the planner has to work to determine which index is a good fit. This increases the risk that the recovery of a failure will take a long time. This is due to the way a restore builds indexes and how MongoDB can only make one index at a time.
• check_balance
  • While MongoDB should keep things in balance across a cluster, many things can happen: jumbo chunks, a disabled balancer being, constantly attempting to move the same chunk but failing, and even adding/removing sharding. This alert is for these cases, as an imbalance means some records might get served faster than others. It is purely based on the chunk count that the MongoDB balancer is also based on, which is not necessarily the same as disk usage.
• check_queues
  • No matter what engine you have selected, a backlog of sustained reads or writes indicates your DB layer is unable to keep up with demand. It is important in these cases to send an alert if the rate is maintained. You might notice this is also in our Prometheus exporter for graphics as both trending and alerting are necessary to watch in a MongoDB system.
• check_cannary_test
  • This is a typical query for the database and then used to set critical/warning levels based on the latency of the returned query. While not as accurate as full synthetic transactions, queries through the application are good to measure response time expectations and SLAs.
• check_have_primary
  • If we had an HA event but failed to get back up quickly, it’s important to know if a new primary is causing writes to error on the system. This check simply determines if the replica set has a primary, which means it can handle reads and writes.
• check_oplog
  • This check is all about how much oplog history you have. This is much like measuring how much history you have in MySQL blogs. The reason this is important is when recovering from a backup and performing a point in time recovery, you can use the current oplog if the oldest timestamp in the oplog is newer than the backup timestamp. As a result, this is normal three times the backup interval you use to guarantee that you have plenty of time to find the newest recovery and then do the recovery.
• check_index_ratio
  • This is an older metric that modern MongoDB versions will not find useful, but in the past, it was a good way to understand the percentage of queries not handled by an index.
• check_connect
  • A very basic check to ensure it can connect (and optionally login) to MongoDB and verify the server is working.

Status File options

These options rarely need to be changed but are present in case you want to store the status on an SHM mount point to avoid actual disk writes.

• statusfile
  • This is where a copy of the current rs.status, serverStatus and other command data is stored
• backup-statusfile
  • Like status_file, but status_file is moved here when a new check is done. These two objects can then be compared to find the delta between two checkpoints.
• max-stale
  • This is the amount of age for which an old file is still valid. Deltas older then this aren’t allowed and exist to protect the system from will assumption when a statusfile is hours or days old.

If you have any questions on how to use these parameters, feel free to let us know. In the code, there is also a defaults dictionary for most of these options so that in many cases setting warning and critical level are not needed.