• MySQL-4.x
• write the transaction in the binlog file
• commit the transaction in InnoDB or other storage engine
• MySQL-5.0:
• prepare the transaction in InnoDB or other storage engines
• write the transaction in the binlog file - this is considered as the commit point
• commit the transaction in InnoDB or other storage engines

• commit the transaction
• wait for the replica databases acknowledge that they already received the transaction - this step has a timeout
• tell the client that the commit has been processed

• transaction failure timeouts
• transactions without going through semi-synchronous replication
• network timeouts

• the master database need to know when to wait for the reply from the slave database; right now, the master database never waits
• the slave database need to know when it should send reply message to the master master database
• we can not do the ping-pong process for every replication event; it can only work for one transaction to minimize the network overhead

• on the slave database side:
• a slave database calls safe_connect() to login to the master database
• COM_BINLOG_DUMP command is sent to the master database to request for binlogs with the following information: binlog_filename, binlog_pos, binlog_flag, server_id
• on the master database side:
• COM_BINLOG_DUMP is handled to recognize the requested dump information
• mysql_binlog_send() is called to send the requested binlog events

• a transaction acquires LOCK_log before writing transaction events to a binlog
• the transaction releases LOCK_log after committing and flushing the binlog to the file system
• replication thread acquires LOCK_log before reading each event and release the lock afterwards

• innobase_repl_semi_cond: this variable is signaled when enough binlog has been sent to slave, so that a waiting transaction can return the 'ok' message to the client for a commit
• innobase_repl_semi_cond_mutex: the mutex that is associated with the above condition variable

• write all binlog events, append the transaction-end event and flush the file to the filesystem
• commit the transaction inside InnoDB
• acquire innobase_repl_semi_cond_mutex
• while true:
• if semi-synchronous replication has been disabled by timeout:
• update the asynchronous transaction counter
• release innobase_repl_semi_cond_mutex and return from the commit
• check the current binlog sending status
• if the binlog sending status is ahead of my transaction's waiting position
• release innobase_repl_semi_cond_mutex and return from the commit
• set my binlog waiting position to my commited transaction position
• wait for innobase_repl_semi_cond with a timeout
• if timeout occurs with waiting innobase_repl_semi_cond or if semi-synchronous replication is disabled after wake-up
• print the error message
• update failed timeout counter
• disable the semi-synchronous replication until the replication thread enables it again
• release innobase_repl_semi_cond_mutex and return from the commit

• if the replication is not the semi-synchronous target, then do nothing and simply return
• if the most recent sent event is NOT an transaction-end event, then do nothing and simply return
• wait for the confirmation from the slave database with a network timeout
• remember whether network timeout occurs
• acquire innobase_repl_semi_cond_mutex
• if the network timeout occurs:
• update failed timeout counter
• disable the semi-synchronous replication until the replication thread enables it again
• release innobase_repl_semi_cond_mutex and return
• if the semi-synchronous replication is disabled, then enable the semi-synchronous replication again
• check whether any session is waiting for the current sending position
• if there exist such sessions, wake them up through innobase_repl_semi_cond
• release innobase_repl_semi_cond_mutex and return

• Binlog_events - number of replication events written to the binlog
• Binlog_largest_event - larget event in the current binlog
• Denied_connections - number of connection attempts that fail because of the max_connections limit
• Malloc_sbrk_bytes_alloc, Malloc_chunks_free, Malloc_mmap_chunks_alloc, Malloc_mmap_bytes_alloc, Malloc_bytes_used, Malloc_bytes_free - values reported from mallinfo()
• Gettimeofday_errors - errors for gettimeofday calls (yes, this happens)
• Sort_filesort_old - number of times the old filesort algorithm is used
• Sort_filesort_new - number of times the new filesort algorithm is used

• Replication_fail_io_connections - on a slave, number of times the IO thread has disconnected from the master because of an error
• Replication_total_io_connections - number of connections made by the IO thread to the master
• Replication_last_event_buffered - on a slave, time when last replication event received
• Replication_last_event_done - on a slave, time when last replication event replayed

• Rpl_semi_sync_clients - number of semi-sync clients connected to a master
• Rpl_semi_sync_net_avg_wait_time(us) - average time to wait for an acknowledgement of a replication event from a semi-sync slave
• Rpl_semi_sync_net_wait_time - total time waiting for acknowledgement
• Rpl_semi_sync_net_waits
• Rpl_semi_sync_no_times  
• Rpl_semi_sync_no_tx - number of transactions not acknowledged by semi-sync slaves
• Rpl_semi_sync_status - indicates whether semi-sync is enabled
• Rpl_semi_sync_slave_status 
• Rpl_semi_sync_timefunc_failures
• Rpl_semi_sync_tx_avg_wait_time(us) - average time a sessions waits for commit to finish
• Rpl_semi_sync_tx_wait_time
• Rpl_semi_sync_tx_waits
• Rpl_semi_sync_wait_pos_backtraverse
• Rpl_semi_sync_wait_sessions
• Rpl_semi_sync_yes_tx - number of transactions acknowledged by semi-sync slaves
• Rpl_transaction_support

• Innodb_dict_size - number of bytes used for the InnoDB dictionary
• Innodb_have_atomic_builtins - indicates whether InnoDB uses atomic memory operations in place of pthreads synchronization functions
• Innodb_heap_enabled - indicates  whether the InnoDB malloc heap was enabled -- see bug 38531
• Innodb_long_lock_wait - set when there is a long lock wait on an internal lock. These usually indicate an InnoDB bug. They also occur because the adaptive hash latch is not always released when it should be (such as during an external sort).
• Innodb_long_lock_waits - incremented once for each internal long lock wait
• Innodb_os_read_requests - from SHOW INNODB STATUS
• Innodb_os_write_requests - from SHOW INNODB STATUS
• Innodb_os_pages_read - from SHOW INNODB STATUS
• Innodb_os_pages_written - from SHOW INNODB STATUS
• Innodb_os_read_time - from SHOW INNODB STATUS
• Innodb_os_write_time - from SHOW INNODB STATUS
• Innodb_time_per_read - average microseconds per read
• Innodb_time_per_write - average microseconds per write
• Innodb_deadlocks - application deadlocks, detected automatically
• Innodb_transaction_count - from SHOW INNODB STATUS
• Innodb_transaction_purge_count - from SHOW INNODB STATUS
• Innodb_transaction_purge_lag - count of work to be done by the InnoDB purge thread, see this post
• Innodb_active_transactions - from SHOW INNODB STATUS
• Innodb_summed_transaction_age - from SHOW INNODB STATUS
• Innodb_longest_transaction_age - from SHOW INNODB STATUS
• Innodb_lock_wait_timeouts - count of lock wait timeouts
• Innodb_lock_waiters - from SHOW INNODB STATUS
• Innodb_summed_lock_wait_time - from SHOW INNODB STATUS
• Innodb_longest_lock_wait - from SHOW INNODB STATUS
• Innodb_pending_normal_aio_reads - from SHOW INNODB STATUS
• Innodb_pending_normal_aio_writes - from SHOW INNODB STATUS
• Innodb_pending_ibuf_aio_reads - from SHOW INNODB STATUS
• Innodb_pending_log_ios - from SHOW INNODB STATUS
• Innodb_pending_sync_ios - from SHOW INNODB STATUS
• Innodb_os_reads - from SHOW INNODB STATUS
• Innodb_os_writes - from SHOW INNODB STATUS
• Innodb_os_fsyncs - from SHOW INNODB STATUS
• Innodb_ibuf_inserts - from SHOW INNODB STATUS
• Innodb_ibuf_size - counts work to be done by the insert buffer, see here
• Innodb_ibuf_merged_recs - from SHOW INNODB STATUS
• Innodb_ibuf_merges - from SHOW INNODB STATUS
• Innodb_log_ios_done - from SHOW INNODB STATUS
• Innodb_buffer_pool_hit_rate - from SHOW INNODB STATUS

• list transactions last in the output in case the output was too long and truncated by InnoDB
• report average and worst-case IO latencies

• srv_master_thread_loops - counts work done by main background thread
• spinlock delay displays the number of milliseconds that the spinlock will spin before going to sleep
• fsync callers displays the source of calls to fsync()

----------
BACKGROUND THREAD
----------
srv_master_thread loops: 28488 1_second, 28487 sleeps, 2730 10_second, 1182 background, 761 flush
srv_master_thread log flush: 29146 sync, 2982 async
srv_wait_thread_mics 0 microseconds, 0.0 seconds
spinlock delay for 5 delay 20 rounds is 5 mics
fsync callers: 1034231 buffer pool, 39227 other, 73053 checkpoint, 10737 log aio, 80994 log sync, 0 archive

• lock wait timeouts counter
• number of spinlock rounds per OS wait for a mutex

----------
SEMAPHORES
----------
Lock wait timeouts 0
...
Spin rounds per wait: 2.90 mutex, 1.27 RW-shared, 3.04 RW-excl

• number of pages read/written
• number of read/write system calls used to read/write those pages
• time in milliseconds to complete the IO requests

--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread) reads 24 writes 0 requests 14 io secs 0.033997 io msecs/request 2.428357 max_io_wait 18.598000
I/O thread 1 state: waiting for i/o request (log thread) reads 0 writes 10737 requests 10737 io secs 30.626824 io msecs/request 2.852456 max_io_wait 710.588000
I/O thread 2 state: waiting for i/o request (read thread) reads 136659 writes 0 requests 118296 io secs 3093.412099 io msecs/request
26.149761 max_io_wait 2631.029000
I/O thread 3 state: waiting for i/o request (read thread) reads 91262 writes 0 requests 71709 io secs 1900.155508 io msecs/request 26.498145 max_io_wait 1626.209000
I/O thread 6 state: waiting for i/o request (write thread) reads 0 writes 1847360 requests 7065434 io secs 1063.904923 io msecs/request 0.150579 max_io_wait 2569.244000

• frequency at which the main background IO thread runs
• IO latency for each background IO thread
• per-file IO statistics
• insert buffer prefetch reads
• statistics on checkpoint related IO
• statistics on prefetches
• statistics on sources of background IO

• srv_master_thread loops - number of iterations of the main background loop including the tasks per second (1_second) and the tasks per 10 seconds (10_second).
• Seconds in background IO thread: number of seconds performing different background IO tasks

BACKGROUND THREAD
----------
srv_master_thread loops: 1623 1_second, 1623 sleeps, 162 10_second, 1 background, 1 flush
srv_master_thread log flush: 1785 sync, 1 async
srv_wait_thread_mics 0 microseconds, 0.0 seconds
spinlock delay for 5 delay 20 rounds is 2 mics
Seconds in background IO thread: 5.10 insert buffer, 49.02 buffer pool, 0.00 adaptive checkpoint, 52.34 purge
fsync callers: 0 buffer pool, 189 other, 1323 checkpoint, 263 log aio, 5179 log sync, 0 archive

• reads, writes - number of pages read and written
• requests - number of pwrite/pread system calls. There may be fewer of these than reads and writes because of request merging.
• msecs/r - average number of milliseconds per *request*. For the *io:* section this is the time for the pwrite/pread system call. For the *svc:* section this is the time from when the page is submitted to the background thread until it is completed.
• secs - total seconds for all pread/pwrite calls
• old - number of pages for which the service time is greater than 2 seconds
• Sync reads, Sync writes - IO operations done synchronously. These share code with the background IO threads, but the IO calls are done directly rather than begin put in the request array and handled by a background IO thread.

FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread) reads 2177 writes 0 io: requests 125 secs 2.99 msecs/r 23.90 max msecs 82.07 svc: 106.58 msecs/r 48.96 max msecs 128.76 old 0
I/O thread 1 state: waiting for i/o request (log thread) reads 0 writes 263 io: requests 263 secs 0.13 msecs/r 0.49 max msecs 30.43 svc: secs 0.14 msecs/r 0.54 max msecs 30.48 old 0
I/O thread 2 state: doing file i/o (read thread) reads 116513 writes 0 io: requests 35777 secs 564.96 msecs/r 15.79 max msecs 251.04 svc: secs 7643.21 msecs/r 65.60 max msecs 2492.18 old 111 ev set
I/O thread 6 state: waiting for i/o request (write thread) reads 0 writes 391586 io: requests 256597 secs 1169.16 msecs/r 4.56 max msecs 336.70 svc: secs 104498.79 msecs/r 266.86 max msecs 3001.04 old 169
Sync reads: requests 10126259, pages 10126278, bytes 165912465408, seconds 171656.02, msecs/r 16.95
Sync writes: requests 2849234, pages 3029512, bytes 11289789952, seconds 77.81, msecs/r 0.03

• pages - number of pages read or written
• requests - number of pwrite/pread system calls. There may be fewer of these than reads and writes because of request merging
• msecs/r - average number of milliseconds per request
• secs - total seconds for all pread/pwrite calls

File IO statistics
  ./test/warehouse.ibd 10 -- read: 3 requests, 3 pages, 0.01 secs, 4.36 msecs/r, write: 30 requests, 30 pages, 0.11 secs, 3.70 msecs/r
  ./ibdata1 0 -- read: 1123 requests, 3349 pages, 22.97 secs, 20.46 msecs/r, write: 2662 requests, 86526 pages, 32.86 secs, 12.34 msecs/r
  ./test/orders.ibd 29 -- read: 26301 requests, 28759 pages, 450.63 secs, 17.13 msecs/r, write: 82089 requests, 101564 pages, 425.44 secs, 5.18 msecs/r
  ./test/customer.ibd 28 -- read: 333186 requests, 338048 pages, 5955.39 secs, 17.87 msecs/r, write: 185378 requests, 200494 pages, 883.61 secs, 4.77 msecs/r
  ./test/stock.ibd 27 -- read: 902675 requests, 1179864 pages, 16036.91 secs, 17.77 msecs/r, write: 577970 requests, 790063 pages, 2473.27 secs, 4.28 msecs/r
  ./test/order_line.ibd 25 -- read: 74232 requests, 92644 pages, 1217.65 secs, 16.40 msecs/r, write: 141432 requests, 274155 pages, 643.97 secs, 4.55 msecs/r
  ./test/new_orders.ibd 22 -- read: 4642 requests, 4960 pages, 81.02 secs, 17.45 msecs/r, write: 11482 requests, 60368 pages, 103.86 secs, 9.05 msecs/r
  ./test/history.ibd 21 -- read: 8006 requests, 11323 pages, 123.86 secs, 15.47 msecs/r, write: 24640 requests, 52809 pages, 119.01 secs, 4.83 msecs/r
  ./test/district.ibd 18 -- read: 14 requests, 14 pages, 0.14 secs, 10.35 msecs/r, write: 39 requests, 249 pages, 0.43 secs, 10.96 msecs/r
  ./test/item.ibd 16 -- read: 2892 requests, 3033 pages, 51.96 secs, 17.97 msecs/r, write: 0 requests, 0 pages, 0.00 secs, 0.00 msecs/r
  ./ib_logfile0 4294967280 -- read: 6 requests, 9 pages, 0.00 secs, 0.02 msecs/r, write: 314701 requests, 316680 pages, 6.73 secs, 0.02 msecs/w

• Ibuf read pages - number of requested and actual prefetch reads done to merge insert buffer records. InnoDB chooses entries to merge at random. If the number requested is much higher than the actual number, then the random algorithm is inefficient.
• Ibuf merge - rate at which work is done for the insert buffer

INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 3776, free list len 1895, seg size 5672,
984975 inserts, 454561 merged recs, 58782 merges
Ibuf read pages: 32960 requested, 36280 actual
Ibuf merge: 1229.9 requested_io/s, 39269.4 records_in/s, 19639.5 records_out/s, 2210.6 page_reads/s

• Foreground (Background) page flushes - page flushes done synchronously (asynchronously) by user sessions to maintain a small number of clean buffer pool pages. 

LOG
---
Foreground page flushes:  sync 0 async 0
Background adaptive page flushes: 0
Foreground flush margins: sync 3025130459 async 2823455095
Space to flush margin:     sync 3000381113 async 2798705749
Current_LSN - Min_LSN     24749346
Checkpoint age            25858470
Max checkpoint age        3226805822

• LRU_old pages - number of *old* pages on the LRU list
• Total writes - sources of pages for dirty page writes
• Write sources - callers from which dirty page write requests are submitted
• Foreground flushed dirty - number of dirty page writes submitted from a user session because the main background IO thread was too slow
• Read ahead - number of prefetch read requests submitted because random or sequential access to an extent was detected
• Pct_dirty - percent of pages in the buffer pool that are dirty

BUFFER POOL AND MEMORY
----------------------
LRU_old pages      48109
Total writes: LRU 23271, flush list 1491363, single page 0
Write sources: free margin 23271, bg dirty 552374, bg lsn 0, bg extra 2742, recv 0, preflush 0
Foreground flushed dirty 935903
Read ahead: 44312 random, 355282 sequential
Pct_dirty 25.83

• MysqlHttp - we added an http server to mysql for exporting monitoring. This was work by Nick Burrett
• InnodbAsyncIo - this explains perf improvements we made for InnoDB
• InnoDbIoTuning - explains more perf improvements we made for InnoDB

• http_enable - start the embedded HTTP demon when ON, see MysqlHttp
• http_port - port on which HTTP listens, see MysqlHttp
• innodb_max_merged_io - max number of IO requests merged into one large request by a background IO thread
• innodb_read_io_threads, innodb_write_io_threads - number of background IO threads for prefetch reads and dirty page writes, see InnodbAsyncIo
• show_command_compatible_mysql4 - make output from some SHOW commands match that used by MySQL4
• show_default_global - make SHOW STATUS use global statistics
• global_status_update_interval - the interval at which per-thread stats are read for SHOW STATUS. When SHOW STATUS is run more frequently cached values are used rather than locking and reading data from each thread.
• google_profile[=name] - enable profiling using Google Perftools and write output to this file. Server must have been compiled to use Google Perftools.
• equality_propagation - enables use of equality propagation in the optimizer because the overhead was too much in a few releases (bug filed & fixed)
• trim_trailing_blanks - trim trailing blanks on varchar fields when set
• allow_view_trigger_sp_subquery - allow use of views, triggers, stored procedures and subqueries when set
• allow_delayed_write - allow use of delayed insert and replace statements
• local-infile-needs-file - LOAD DATA LOCAL INFILE requires the FILE privilege when set  
• audit_log[=name] - log logins, queries against specified tables, and startup
• audit_log_tables=name - log queries that use these tables to the audit log (comma separated)
• log_root - log DML done by users with the SUPER privilege
• repl_port[=#] - extra port on which mysqld listens for connections from users with SUPER and replication privileges
• rpl_always_reconnect_on_error - slave IO thread always tries to reconnect on error when set
• rpl_always_enter_innodb - slave SQL thread always enter innodb when set regardless of innodb concurrency ticket count
• rpl_event_buffer_size=# - size of the per-connection buffer used on the master to copy events to a slave. Avoids allocating/deallocating a buffer for each event.
• reserved_super_connections=# - number of reserved connections for users with SUPER privileges.
• rpl_always_begin_event - always add a BEGIN event at the beginning of each transaction block written to the binlog. This fixes a bug.
• rpl_semi_sync_enabled - enable semisync replication on a master
• rpl_semi_sync_slave_enabled - semisync replication on a slave
• rpl_semi_sync_timeout - timeout in milliseconds for semisync replication in the master
• rpl_semi_sync_trace_level - trace level for debugging for semisync replication
• rpl_transaction_enabled - use transactional replication on a slave
• innodb_crash_if_init_fails - crash if InnoDB initialization fails
• innodb_io_capacity - number of disk IOPs the server can do, see InnodbIoTuning
• innodb_extra_dirty_writes - flush dirty buffer pages when dirty pct is less than max dirty pct
• connect_must_have_super - only connections with SUPER_ACL, REPL_SLAVE_ACL or REPL_CLIENT_ACL are accepted (yes, this is dynamic)
• readonly_databases - prevents writes to any DB except for mysql
• readonly_mysql - prevents writes to mysql DB will fail.
• fixup_binlog_end_pos - fix for MySQL bug 23171 which updates the end_log_pos of  binlog events as they are written to the  bin log
• log_slave_connects - log connect and disconnect messages for replication slaves
• mapped_users - use the mapped_user table to map users to roles
• xa_enabled - enable support for XA transactions (I like to disable this)

• mapped user - one row in mysql.mapped_user
• role - one row in mysql.user referenced by at least one row in mysql.mapped_user

• multiple passwords per account
• manual password expiration
• roles
• transparent to users (mysql -uuser -ppassword works regardless of whether authentication is done using entries in mysql.mapped_user or mysql.user)

• User - the name for this mapped user
• Role - the name of the account in mysql.user from which this account gets its privileges
• Password - the password hash for authenticating a connection
• PasswordChanged - the timestamp when this entry was last updated or created. This is intended to support manual password expiration via a script that deletes all entries where PasswordChanged less than the cutoff.
• ssl_type, ssl_cipher, x509_issuer, x509_subject - values for SSL authentication, note that code has yet to be added in the server to handle these values

CREATE TABLE mapped_user (
  User char(16) binary DEFAULT '' NOT NULL,
  Role char(16) binary DEFAULT '' NOT NULL,
  Password char(41) character set latin1 collate latin1_bin DEFAULT '' NOT NULL,
  PasswordChanged Timestamp DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP NOT NULL,
  ssl_type enum('','ANY','X509','SPECIFIED') character set utf8 NOT NULL default '',
  ssl_cipher blob NOT NULL,
  x509_issuer blob NOT NULL,
  x509_subject blob NOT NULL,
  PRIMARY KEY (User, Role, Password)
) engine=MyISAM
CHARACTER SET utf8 COLLATE utf8_bin
comment='Mapped users';

• the values of mysql.user.User and mysql.user.Host are used to search the other privilege tables
• the global privileges stored in mysql.user for the matching entry are used

• Role is the empty string
• User is the empty string
• Password is the empty string

• error messages related to access control
• select current_user()
• select user()
• show user_statistics
• show processlist

• Add a unique index on (User, Password)
• Add an email column to mysql.mapped_user
• Inherit limits (hostname/IP address from which connections are allowed, connection limits, max queries per minute limit) from the mysql.user table.
• Implement support for SSL -- the mysql.mapped_user table has columns for SSL authentication. Code has not been added to the server to handle them.

• v1 patch published in 2007
• v2 patch with all of our changes for MySQL 5.0.37
• v3 patch with all of our changes for MySQL 5.0.37 as of May 6, 2009. This adds global transaction IDs, row-change logging and more InnoDB SMP performance fixes.
• v4 patch [http://google-mysql-tools.googlecode.com/svn/trunk/old/mysql-as of June 1, 2009
• semisync v1 patch published in 2007
• mutexstats patch MySQL 5.1.26
• SMP perf patch for MySQL 5.0.67. This has two changes:
• use atomic memory instructions for the InnoDB mutex and rw-mutex. This is only done for x86 platforms that use a recent (>= 4.1) version of GCC.
• disable the InnoDB memory heap. This is done for all platforms
• SMP plugin for the InnoDB 1.0.1 plugin in MySQL 5.1
• Patch to enable/disable IO to InnoDB files for MySQL 5.0.37
• Patch to use pthread_mutex_t instead of mutex_t for rw_lock_struct::mutex in InnoDB
• Patch for global transaction IDs and binlog event checksums stand-alone patch] extracted out of the big V3 patch and ported to mysql-5.0.68 as of 

• Wei Li
• Gene Pang
• Eric Rollins
• Ben Handy
• Justin Tolmer
• Larry Zhou
• Yuan Wei
• Robert Banz
• Chip Turner
• Steve Gunn
• Mark Callaghan

• SqlChanges
• SemiSyncReplication
• InnodbSmp
• NewShowStatus
• NewShowInnodbStatus
• NewConfiguration
• UserTableMonitoring
• TransactionalReplication
• MysqlRoles
• MysqlRateLimiting
• MoreLogging
• InnodbAsyncIo
• FastMasterPromotion
• MirroredBinlogs
• InnodbSampling
• NewSqlFunctions
• InnodbStatus
• LosslessFloatDump
• MysqlHttp
• InnodbIoTuning
• MutexContentionStats
• FastMutexes
• InnodbFreeze

• GlobalTransactionIds
• OnlineDataDrift
• BatchKeyAccess
• InnodbMutexContention2
• BinlogEventChecksums

• InnodbIoPerformance

• MysqlThreadPool

• one thread uses synchronous IO to write pages to disk. When write latency is significant (because O_DIRECT is used, SATA write cache is disabled, network attached storage is used, ext2 is used) then this thread becomes a bottleneck. This was fixed in the v2 Google patch and is also fixed in MySQL 5.4 and Percona builds.
• rate limits are too small. InnoDB has a background thread that schedules writes for 100 dirty pages at a time when there are too many dirty pages. The limit of 100 is reasonable for a single-disk server. It must be larger for a high IOPs server. The v2 Google patch, MySQL 5.4 (maybe) and Percona branches use the innodb_io_capacity my.cnf variable to determine the number of pages per second that should be written for this case and the amount of IO that should be done in other cases. All work is expressed as a fraction of this variable, rather than as a fixed number of IO operations.
• request arrays are too small. On my servers, each array has 256 slots. For a server that can do 1000 IOPs, this is too small. The v4 patch makes the size of the array a function of the value of innodb_io.capacity
• user sessions are not constrained. InnoDB delay's user sessions when the purge thread gets too far behind. Otherwise, not much is done to delay a user session. The v4 patch adds code to force user session's to stop and flush dirty pages when the maximum number of dirty pages has been exceeded. Hopefully, this code does nothing as the background thread is more likely to keep up given other changes in the v4 patch.

• more efficient when processing IO requests
• more likely to use available IO capacity
• better at balancing different IO tasks
• easier to monitor

• Percona Performance Conference slides from me. (edit - who remembers the motivation for this event?)
• insert buffering - now this is change buffering
• blog post on InnoDB IO lag
• another blog post on InnoDB IO performance
• details on new my.cnf parameters for InnoDB IO performance
• more details on my.cnf parameters for more background IO threads]

• Changes the computation of the percentage of dirty buffer pool pages. Before this change the percentage exluded pages borrowed from the buffer pool for other uses. While that may be more accurate, it also requires the caller to lock/unlock a hot mutex. It also made the percentage vary a bit too much as the insert buffer grew and shrank. The v4 patch doesn't exclude the borrowed pages. As most of the borrowed pages should be used in the insert buffer and the insert buffer should be smaller (thanks to ibuf_max_pct_of_buffer), this is probably a good thing.
• (edit removed many links to other project pages)

• checksum computation and verification - the v4 patch does not make this faster. Using -O3 rather than -O2 with gcc makes this faster. On a server that does 10,000 IOPs, this will consume a lot of CPU time. Domas wrote about this. We may need to consider alternative checksum algorithms and machine-specific optimizations.
• request array iteration - InnoDB maintains requests for IO in an array. It frequently iterates on the array and called a function to get the next element in the array. That has been changed to use pointer arithmetic. This makes a big difference when the array is large.
• request merging - InnoDB merges requests for adjacent blocks so that one large IO operation is done instead of several page size operations. Up to 64 page requests can be merged into one large (1MB) request. The merging algorithm was O(N*N) on the size of the request array and has been changed to be O(N). This will merge fewer requests but use much less CPU. A better change might be to replace each array with two lists: one that maintains requests in file order and the other in arrival order. But that must wait for another day.

• work done by several people to fix the InnoDB rw-lock for SMP
• other work for InnoDB SMP performance
• user, table, index monitoring
• crash safe replication state
• parser changes for new features
• semisync replication and design doc
• SHOW STATUS
• SHOW INNODB STATUS
• my.cnf
• adding roles to mysql
• make user delayed
• the Google patch for MySQL 5.0
• InnoDB IO performance

• user, table, index monitoring, also see this doc
• InnoDB Async IO
• Batch Key Access
• Binlog Event Checksum
• Fast Master Promotion
• InnoDB IO tuning
• InnoDB IO configuration
• InnoDB optimizer stats sampling
• InnoDB SMP
• Making mysqldump lossless for float & double
• Mirrored binlogs - this turned out to be a problem in production. I won't say more.
• More logging
• SHOW INNODB STATUS
• SHOW STATUS
• New SQL functions
• Online data drift
• Patch for MySQL 4.0
• QPS Rate Limits
• Semisync doc and design doc
• Transactional replication
• Global transactions IDs - work by Justin Tolmer at Google to enable failover automation. This was impressive. It was done years ahead of upstream. We couldn't wait. 
• Roles
• SMP Performance
• Mutex statistics

Author: Robert Agar

The Internet of Things (IoT) has grown from an interesting concept to a paradigm that is changing the way individuals and businesses operate in the 21st Century. It is based on connecting IP-capable devices so they can communicate with each other in a variety of ways. They range from automated industrial assembly lines to smart appliances that promise to make life easier and more convenient for consumers.

A common aspect of all IoT implementations is that they make use of large amounts of data collected from network-connected devices. As with most data-centric applications, IoT systems rely on databases to store and process the accumulated information that drives them. MySQL is a valid choice in database platforms when you are designing a system that interacts with the IoT.

Fundamental Aspects of IoT Systems

The point of the Internet of Things is to gather or exchange data to facilitate physical processes or gain insight into complex abstract or tangible systems. IoT implementations accomplish this by deploying network-connected devices that have the ability to measure and collect relevant information.

Attaching devices to a network is nothing new. Factors that differentiate IoT systems from traditional networks are the intelligence and capabilities that are built into the connected devices. Complicated tasks can be carried out without direct human intervention through communication between IoT sensors, monitors, and actuators in industrial applications. Consumers can talk to their home and have it respond to them in a variety of ways. Marketers obtain data that helps them fine-tune their offerings and increase the efficiency of their sales initiatives.

Behind the enhanced functionality that the IoT affords when compared to previous networks is an incredible amount of data. The concepts of Big Data and the IoT are intertwined. Storing and manipulating the tremendous amounts of data generated by IoT devices effectively is vitally important to the productivity of the system. The whole point of an IoT implementation is to generate and make use of information gathered from its sensors.

Why MySQL Makes Sense as Your Platform of Choice

The selection of the backend database for IoT systems is critical to their eventual success. You need a platform that can handle the demands of processing large amounts of data efficiently. Most IoT implementations also require high availability and excellent performance. Choosing the wrong database can have detrimental effects on the whole system.

MySQL is one of the most popular database platforms in the world and is used in a wide variety of business and scientific solutions. It has several advantages that make it a viable solution for IoT implementations. Here is a list of features that make it attractive:

• Data security is critical for many types of applications including IoT systems. MySQL is well known as an exceptionally secure database platform that is used by many popular websites.

• Scalability is another factor that is important when designing an IoT system. As the system evolves it may need to expand in unexpected directions. MySQL has a proven track record of handling the large amounts of data generated by the IoT.

• High performance and availability are hallmarks of MySQL that make it a perfect fit as the backend of an IoT system. It provides excellent performance for demanding applications and can easily meet the data processing demands of the IoT. Maintaining availability in industrial IoT systems is of paramount importance. One of the reasons for the popularity of MySQL is its high availability solutions.

• Open source flexibility means that MySQL can be tailored in whatever ways are needed to address the requirements of an IoT system. Its wide adoption throughout the IT world also ensures that skilled technical professionals are available to support the implementation.

A Tool for Managing Your MySQL Instances

SQL Diagnostic Manager for MySQL is the perfect tool for managing the MySQL databases powering your IoT systems. It provides the capability to perform real-time monitoring that helps you quickly pinpoint issues and resolve them before they become serious problems. The tool comes with over 600 pre-built monitors that continuously check the health of your MySQL servers and send alerts when defined thresholds are exceeded. Customizable dashboards and charts provide visibility into your systems that allow you to take proactive measures to avoid running low on capacity.

If you choose MySQL as the database engine for your IoT system, you should strongly consider SQL Diagnostic Manager for MySQL as your management solution. It’s a valuable addition to your DBA’s toolbox and will quickly become their favorite tool when working with MySQL databases.

The post Implementing the Internet of Things with MySQL appeared first on Monyog Blog.

• MySQL for High-Scale, MySQL Ignite
• MyRocks Overview, MyRocks meetup
• RocksDB: from faster to better, CMU with Coach Pavlo
• MySQL is Web Scale, Istanbul with Monty Program
• Scaling InnoDB, O'Reilly MySQL 2008 (no slides/video yet)
• This is not a web app, O'Reilly MySQL 2009
• Running MySQL at Scale, FB Tech Talk 2010
• High Throughput MySQL, O'Reilly MySQL 2010
• Success with MySQL, MySQL Sunday 2010
• MySQL at Facebook, O'Reilly MySQL 2010
• NoSQL, Hot Storage 2011
• MySQL for the datacenter, O'Reilly MySQL 2011
• High Value Transaction Processing, Percona Live SF 2011 - this has a list of bugs and fixes
• Success with MySQL, North East Database Day 2012
• What comes next for MySQL, Percona Live 2012
• InnoDB Buffer Pool Notes, 2012
• MySQL at Facebook, China 2013
• MyRocks vs Something Else, video and slides, Highload 2013
• Storage Economics, Tech Talk 2013
• Open Source at Scale, OpenWest 2013
• Web scale OLTP, 2014
• Social Graph OLTP, 2014
• MySQL at Scale, Dropbox 2014
• Running MyRocks in Production, 2015
• Storage Engine Diversity, MongoDB World 2015
• Web-Scale History, DevOps Day Camp at OSUOSL 2015
• The Case for MyRocks, 2015
• Web Scale Data Management, Percona Live Europe 2015
• Open Problems for RocksDB, CWI 2015
• Read, Write and Space Efficiency: Pick 2, ACM Applicative 2016
• Making the case for write-optimized, Yandex 2016
• Making the case for write-optimized: video and slides, Highload 2016
• Performance and Efficiency with RocksDB, CodeMesh 2016
• Updated case for MyRocks, MongoRocks and RocksDB, RocksDB Meetup 2016
• MyRocks, MongoRocks and MongoDB, Percona Live 2016
• MyRocks and MongoRocks: Performance & Efficiency, Percona Live Europe 2016
• Features we need, features that scale, Percona Live Europe 2016
• RocksDB Read Cache, MongoDB World 2016
• An Update on MyRocks, Percona Live 2017
• MongoRocks, WiredTiger and mmapv1, Percona Live 2017
• MongoRocks Improvements, Percona Live Dublin 2017
• MyRocks, FOSDEM 2017
• How to talk to me at conferences, maybe HPTS gong show

One of the benefits of MySQL GTIDs is that each server remembers all GTID entries ever executed. Normally these would be ranges, e.g. 0041e600-f1be-11e9-9759-a0369f9435dc:1-3772242 or multi-ranges, e.g. 24a83cd3-e30c-11e9-b43d-121b89fcdde6:1-103775793, 2efbcca6-7ee1-11e8-b2d2-0270c2ed2e5a:1-356487160, 46346470-6561-11e9-9ab7-12aaa4484802:1-26301153, 757fdf0d-740e-11e8-b3f2-0a474bcf1734:1-192371670, d2f5e585-62f5-11e9-82a5-a0369f0ed504:1-10047.

One of the common problems in asynchronous replication is the issue of consistent reads. I've just written to the master. Is the data available on a replica yet? We have iterated on this, from reading on master, to heuristically finding up-to-date replicas based on heartbeats (see presentation and slides) via freno, and now settled, on some parts of our apps, to using GTID.

GTIDs are reliable as any replica can give you a definitive answer to the question: have you applied a given transaction or not?. Given a GTID entry, say f7b781a9-cbbd-11e9-affb-008cfa542442:12345, one may query for the following on a replica:

mysql> select gtid_subset('f7b781a9-cbbd-11e9-affb-008cfa542442:12345', @@global.gtid_executed) as transaction_found;
+-------------------+
| transaction_found |
+-------------------+
|                 1 |
+-------------------+

mysql> select gtid_subset('f7b781a9-cbbd-11e9-affb-008cfa542442:123450000', @@global.gtid_executed) as transaction_found;
+-------------------+
| transaction_found |
+-------------------+
|                 0 |
+-------------------+

Getting OWN_GTID

This is all well, but, given some INSERT or UPDATE on the master, how can I tell what's the GTID associated with that transaction? There\s good news and bad news.

• Good news is, you may SET SESSION session_track_gtids = OWN_GTID. This makes the MySQL protocol return the GTID generated by your transaction.
• Bad news is, this isn't a standard SQL response, and the common MySQL drivers offer you no way to get that information!

At GitHub we author our own Ruby driver, and have implemented the functionality to extract OWN_GTID, much like you'd extract LAST_INSERT_ID. But, how does one solve that without modifying the drivers? Here's a poor person's solution which gives you an inexact, but good enough, info. Following a write (insert, delete, create, ...), run:

select gtid_subtract(concat(@@server_uuid, ':1-1000000000000000'), gtid_subtract(concat(@@server_uuid, ':1-1000000000000000'), @@global.gtid_executed)) as master_generated_gtid;

The idea is to "clean" the executed GTID set from irrelevant entries, by filtering out all ranges that do not belong to the server you've just written to (the master). The number 1000000000000000 stands for "high enough value that will never be reached in practice" - set to your own preferred value, but this value should take you beyond 300 years assuming 100,000 transactions per second.

The value you get is the range on the master itself. e.g.:

mysql> select gtid_subtract(concat(@@server_uuid, ':1-1000000000000000'), gtid_subtract(concat(@@server_uuid, ':1-1000000000000000'), @@global.gtid_executed)) as master_generated_gtid;
+-------------------------------------------------+
| master_generated_gtid                           |
+-------------------------------------------------+
| dc103953-1598-11ea-82a7-008cfa5440e4:1-35807176 |
+-------------------------------------------------+

You may further parse the above to extract dc103953-1598-11ea-82a7-008cfa5440e4:35807176 if you want to hold on to the latest GTID entry. Now, this entry isn't necessarily your own. Between the time of your write and the time of your GTID query, other writes will have taken place. But the entry you get is either your own or a later one. If you can find that entry on a replica, that means your write is included on the replica.

One may wonder, why do we need to extract the value at all? Why not just select @@global.gtid_executed? Why filter only the master's UUID? Logically, the answer is the same if you do that. But in practice, your query may be unfortunate enough to return some:

select @@global.gtid_executed \G

e71f0cdb-b8ef-11e9-9361-008cfa542442:1-83331,
e742d87f-dea7-11e9-be6d-008cfa542c9e:1-18485,
e7880c0e-ac54-11e9-865a-008cfa544064:1-7331973,
e82043c6-c7d9-11e9-9413-008cfa5440e4:1-61692,
e902678b-b046-11e9-a281-008cfa542c9e:1-83108,
e90d7ff9-e35e-11e9-a9a0-008cfa544064:1-18468,
e929a635-bb40-11e9-9c0d-008cfa5440e4:1-139348,
e9351610-ef1b-11e9-9db4-008cfa5440e4:1-33460918,
e938578d-dc41-11e9-9696-008cfa542442:1-18232,
e947f165-cd53-11e9-b7a1-008cfa5440e4:1-18480,
e9733f37-d537-11e9-8604-008cfa5440e4:1-18396,
e97a0659-e423-11e9-8433-008cfa542442:1-18237,
e98dc1f7-e0f8-11e9-9bbd-008cfa542c9e:1-18482,
ea16027a-d20e-11e9-9845-008cfa542442:1-18098,
ea1e1aa6-e74a-11e9-a7f2-008cfa544064:1-18450,
ea8bc1bd-dd06-11e9-a10c-008cfa542442:1-18203,
eae8c750-aaca-11e9-b17c-008cfa544064:1-85990,
eb1e41e9-af81-11e9-9ceb-008cfa544064:1-86220,
eb3c9b3b-b698-11e9-b67a-008cfa544064:1-18687,
ec6daf7e-b297-11e9-a8a0-008cfa542c9e:1-80652,
eca4af92-c965-11e9-a1f3-008cfa542c9e:1-18333,
ecd110b9-9647-11e9-a48f-008cfa544064:1-24213,
ed26890e-b10b-11e9-a79d-008cfa542c9e:1-83450,
ed92b3bf-c8a0-11e9-8612-008cfa542442:1-18223,
eeb60c82-9a3d-11e9-9ea5-008cfa544064:1-1943152,
eee43e06-c25d-11e9-ba23-008cfa542442:1-105102,
eef4a7fb-b438-11e9-8d4b-008cfa5440e4:1-74717,
eefdbd3b-95b3-11e9-833d-008cfa544064:1-39415,
ef087062-ba7b-11e9-92de-008cfa5440e4:1-9726172,
ef507ff0-98b3-11e9-8b15-008cfa5440e4:1-928030,
ef662471-9a3b-11e9-bd2e-008cfa542c9e:1-954800,
f002e9f7-97ee-11e9-bed0-008cfa542c9e:1-5180743,
f0233228-e9a1-11e9-a142-008cfa542c9e:1-18583,
f04780c4-a864-11e9-9f28-008cfa542c9e:1-83609,
f048acd9-b1d2-11e9-a0b6-008cfa544064:1-70663,
f0573d8c-9978-11e9-9f73-008cfa542c9e:1-85642135,
f0b0a37c-c89c-11e9-804c-008cfa5440e4:1-18488,
f0cfe1ac-e5af-11e9-bc09-008cfa542c9e:1-18552,
f0e4997c-cbc9-11e9-9179-008cfa542442:1-1655552,
f24e481c-b5c4-11e9-aff0-008cfa5440e4:1-83015,
f4578c4b-be6d-11e9-982e-008cfa5440e4:1-132701,
f48bce80-e99f-11e9-94f4-a0369f9432f4:1-18460,
f491adf1-9b04-11e9-bc71-008cfa542c9e:1-962823,
f5d3db74-a929-11e9-90e8-008cfa5440e4:1-75379,
f6696ba7-b750-11e9-b458-008cfa542c9e:1-83096,
f714cb4c-dab7-11e9-adb9-008cfa544064:1-18413,
f7b781a9-cbbd-11e9-affb-008cfa542442:1-18169,
f81f7729-b10d-11e9-b29b-008cfa542442:1-86820,
f88a3298-e903-11e9-88d0-a0369f9432f4:1-18548,
f9467b29-d78c-11e9-b1a2-008cfa5440e4:1-18492,
f9c08f5c-e4ea-11e9-a76c-008cfa544064:1-1667611,
fa633abf-cee3-11e9-9346-008cfa542442:1-18361,
fa8b0e64-bb42-11e9-9913-008cfa542442:1-140089,
fa92234c-cc90-11e9-b337-008cfa544064:1-18324,
fa9755eb-e425-11e9-907d-008cfa542c9e:1-1668270,
fb7843d5-eb38-11e9-a1ff-a0369f9432f4:1-1668957,
fb8ceae5-dd08-11e9-9ed3-008cfa5440e4:1-18526,
fbf9970e-bc07-11e9-9e4f-008cfa5440e4:1-136157,
fc0ffaee-98b1-11e9-8574-008cfa542c9e:1-940999,
fc9bf1e4-ee54-11e9-9ce9-008cfa542c9e:1-18189,
fca4672f-ac56-11e9-8a83-008cfa542442:1-82014,
fcebaa05-dab5-11e9-8356-008cfa542c9e:1-18490,
fd0c88b1-ad1b-11e9-bf3a-008cfa5440e4:1-75167,
fd394feb-e4e4-11e9-bd09-008cfa5440e4:1-18574,
fd687577-b048-11e9-b429-008cfa542442:1-83479,
fdb18995-a79f-11e9-a28d-008cfa542442:1-82351,
fdc72b7f-b696-11e9-ade9-008cfa544064:1-57674,
ff1f3b6b-c967-11e9-ae04-008cfa544064:1-18503,
ff6fe7dc-c186-11e9-9bb4-008cfa5440e4:1-103192,
fff9dd94-ed95-11e9-90b7-008cfa544064:1-911039

This can happen when you fail over to a new master, multiple times; it happens when you don't recycle UUIDs, when you provision new hosts and let MySQL pick their UUID. Returning this amount of data per query is an excessive overhead, hence why we extract the master's UUID only, which is guaranteed to be limited in size.

To avoid infinite replication loops MySQL doesn’t allow you to have log_slave_updates and replicate-same-server-id.

When using GTIDs that may lead to something not expected that you may not be aware of.

In this scenario, we have 2 MySQL servers using GTID. The sever uuid part of the GTID has been modified in the illustration to make it more clear. Both servers have log_slave_updates enabled too:

So far nothing unusual. So let’s write data on the master (MySQL A):

We can see that this first transaction is identified by its GTID where the uuid matches MySQL A and the sequence number is 1. Let’s write some data again:

All good.

Now let’s take a backup on the replica (MySQL B):

Backup is consistent and matches the data on both servers.

Now let’s write again some data:

Of course the backup that was taken earlier does not change.

All suddenly, MySQL A crashes and goes away… We promote MySQL B as new writer and we use it to write again some data:

We can notice that the GTID changed to use the uuid of MySQL B. (This information is contained in the variable gtid_executed).

It’s time to restore our backup on MySQL A:

And we configure MySQL A to become replica of MySQL B:

Wow ! All the transactions happened after the backup on MySQL A have been ignored !

In fact, this is again to protect our user to have problems like circular replication and infinite loop.

To be able to replicate the missing transaction is to change the server_id to a new unique value after the restore and before starting replication. Then replication will work as expected:

In summary, if you have enabled log_slave_updates and you want to recreate a master from a backup taken on a replica, you must change the server_id. Even if you use GTIDs, server_id is for the moment still to be taken in consideration.

If you plan to write back on MySQL A, it’s always safer to also change its server_uuid to avoid any split-brain situation.

In my road trip of the Open Source projects using MySQL, after having tested WordPress, Drupal and Joomla, let’s try to install Magento using MySQL 8.0 !

In Magento’s manual, we can see that the project requires MySQL 5.6 and supports 5.7.x since versoin 2.1.2.

In my test, I will use Magento 2.3.3, the latest stable when writing this article.

The manual stipulates that we should use ROW based replication but not GTID because Magento 2 is using CREATE TEMPORARY TABLE inside transactions. In fact, this limitation doesn’t exist anymore since MySQL 8.0.13.

From MySQL 8.0.13, when binlog_format is set to ROW or MIXED, CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE statements are allowed inside a transaction, procedure, function, or trigger when GTIDs are in use. The statements are not written to the binary log and are therefore not replicated to slaves. The use of row-based replication means that the slaves remain in sync without the need to replicate temporary tables. If the removal of these statements from a transaction results in an empty transaction, the transaction is not written to the binary log. See the MySQL Manual.

So this is not a limitation anymore. And for this installation I will of course use the latest version of MySQL 8.0 available right now, MySQL 8.0.18.

Setup the database

So MySQL 8.0.18 is installed and now we need to create the database/schema and setup the credentials to install Magento:

mysql> CREATE DATABASE magento2;

mysql> CREATE USER magento2 IDENTIFIED BY 'magento2';

mysql> GRANT ALL ON magento2.* TO magento2;

We can see that we are using MySQL 8.0 and the default authenticatio plugin:

mysql> SELECT Host, User, plugin, @@version FROM mysql.user 
       WHERE user='magento2';
 +------+----------+-----------------------+-----------+
 | Host | User     | plugin                | @@version |
 +------+----------+-----------------------+-----------+
 | %    | magento2 | caching_sha2_password | 8.0.18    |
 +------+----------+-----------------------+-----------+

As in the future we will use any kind of replication, we will also enable GTID:

mysql> SET persist enforce_gtid_consistency=on;
 
mysql> SET persist_only gtid_mode=on;

mysql> RESTART;

Magento Installation

We can now start the installation wizard of Magento:

The first checks are not related to MySQL and all necessary packages are installed. FYI, this is Oracle Linux 8:

Now it’s time to insert the MySQL information:

And when we press on Next, we have our first small issue:

This is exactly what the error message is telling. The new authentication plugin is now supported. We can also see that in the php error log:

[11-Dec-2019 20:30:49 UTC] PHP Fatal error:  Uncaught PDOException: PDO::__construct(): 
The server requested authentication method unknown to the client [caching_sha2_password]
in /var/www/html/vendor/magento/zendframework1/library/Zend/Db/Adapter/Pdo/Abstract.php:128

The PHP version is 7.2.11 which doesn’t support MySQL 8’s new default secure authentication plugin: caching_sha2_password.

It was supported in 7.2.8 but removed on 7.2.11, for more info check:

• https://lefred.be/content/php-7-2-8-mysql-8-0/
• https://lefred.be/content/mysql-8-0-17-and-drupal-8-7/

So now, we just need to modify the authentication method of our user:

mysql> ALTER USER magento2 IDENTIFIED 
       WITH 'mysql_native_password' BY 'magento2';

And we can press Next again in Magento’s installation wizard and fill all the next Steps until Step 6:

Let’s go an press Install Now !

The installation process seems to hang at 58%… and when we check the Console Log we can see the following:

We followed the manual, but it seems our user needs more privileges. But let’s try to not provide that and try the second option:

mysql> SET PERSIST log_bin_trust_function_creators=1;

We can refresh the page and click again on Install Now.

The process continues to 91% and then another strange error:

I checked the status of the table and nothing seems wrong, so I refreshed the table and restart the process that resumes and goes to the end successfully:

And of course, we can now access the Site we just deployed:

Summary

In summary, Magento works fine with MySQL 8.0, you have very few changes to perform, they are related to the user Magento that connect to MySQL and one global variable:

mysql> CREATE USER magento2 IDENTIFIED 
       WITH 'mysql_native_password' BY 'magento2';
mysql> GRANT ALL ON magento2.* TO magento2;
mysql> SET PERSIST log_bin_trust_function_creators=1;

And of course, there is no more reason to not use GTIDs !

MySQL and PHP is a love story that started long time ago. However the love story with MySQL 8.0 was a bit slower to start… but don’t worry it rules now !

The support of MySQL 8.0’s new default authentication method in PHP took some time and was added in PHP 7.2.8 but removed in PHP 7.2.11.

Now it’s fully supported in PHP 7.4 !

If you have installed PHP 7.4, you can see that the new plugin auth_plugin_caching_sha2_passwordis now available:

# php -i | grep "Loaded plugins|PHP Version " | tail -n2
PHP Warning:  Module 'mysql_xdevapi' already loaded in Unknown on line 0
PHP Version => 7.4.0
Loaded plugins => mysqlnd,debug_trace,auth_plugin_mysql_native_password,
                  auth_plugin_mysql_clear_password,
                  auth_plugin_caching_sha2_password,
                  auth_plugin_sha256_password

So no need to create a user with mysql_native_passwordas authentication method in MySQL 8.0 as explained in the following posts:

• https://lefred.be/content/mysql-8-0-17-and-drupal-8-7/
• https://lefred.be/content/migrating-to-mysql-8-0-for-wordpress-episode-1/
• https://lefred.be/content/drupal-and-mysql-8-0-11-are-we-there-yet/
• https://lefred.be/content/joomla-and-mysql-8-0-12/
• https://lefred.be/content/mysql-8-0-and-magento/

In summary, if you want to use a more secure method to connect to your MySQL 8.0 form your PHP application, make sure you upgrade to PHP 7.4

Introduction In this article, we are going to see how a CROSS JOIN works, and we will also make use of this SQL join type to build a poker card game. Database table model For our poker card game application, we have created the ranks and suits database tables: The ranks table defines the ranking of cards, as well as the name and symbol used for each card rank: The suits table describes the four possible categories used by the French playing cards: Cartesian product In the set theory, the Cartesian product... Read More

The post A beginner’s guide to SQL CROSS JOIN appeared first on Vlad Mihalcea.

Database administrators are responsible for maintaining the privacy and integrity of data. When the data contains confidential information, your company has a legal obligation to ensure that privacy is maintained. Even so, being able to access the information contained in that dataset, for example for testing or reporting purposes, has great value so what to do? MySQL Enterprise Edition offers data masking and de-identification, so I decided to contribute similar functionality to Percona Server for MySQL. In this post, I provide some background context and information on how to use these new functions in practice.

Some context

One of the most important assets of any company is data. Having good data allows engineers to build better systems and user experiences.

Even through our most trivial activities, we continuously generate and share great volumes of data. I’m walking down the street and if I take a look at my phone it’s quite straightforward to get recommendations for a place to have lunch. The platform knows that it’s almost lunch time and that I have visited this nearby restaurant, or a similar one, a few times in the past. Sounds cool, right?

But this process could be more manual than we might think at first. Even if the system has implemented things like AI or Machine Learning, a human will have validated the results; they might have taken a peek to ensure that everything is fine; or perhaps they are developing some new cool feature that must be tested… And this means that someone, somewhere has the ability to access my data. Or your data.

Now, that is not so great, is it?

In the last decade or so, governments around the world have taken this challenge quite seriously. They have enforced a series of rules to guarantee that the data is not only safely stored, but also safely used. I’m sure you will have heard terms like PCI, GDPR or HIPAA. They contain mandatory guidelines for how our data can be used, for primary or secondary purposes, and if it can be used at all.

Data masking and de-identification

One of the most basic safeguarding rules is that if the data is to be used for secondary purposes – such as for data analytics – it has to be de-identified in a way that it would make impossible identify the original individual.

Let’s say that the company ACME is storing employee data.

We will use the example database of employees that’s freely available.

Employee number
First name
Last name
Birth date
Gender
Hire date
Gross salary
Salary from date
Salary to date

We can clearly see that all those fields can be classified as private information. Some of these directly identify the original individual, like employee number or first + last name. Others could be used for indirect identification: I could ask my co-workers their birthday and guess the owner of that data using

birth date

So, here is where de-identification and data-masking come into play. But what are the differences?

De-identification transforms the original data into something different that could look more or less real. For example, I could de-identify

birth date

However, this method would make that information unusable if I want to see the relationship between salary and employee’s age.

On the other hand, data-masking transforms the original data leaving some part untouched. I could mask

birth date

Of course, if the dataset I’m working with is not big enough, certain methods of data-masking would be inappropriate as I could still deduce who the data belonged to.

MySQL data masking

Oracle’s MySQL Enterprise Edition offers a de-identification and data-masking solution for MySQL, using a flexible set of functions that cover most of our needs.

Percona Server for MySQL 8.0.17 introduces that functionality as an open source plugin, and is compatible with Oracle’s implementation. You no longer need to code slow and complicated stored procedures to achieve data masking, and you can migrate the processes that were written for the MySQL Enterprise Edition to Percona Server for MySQL. Go grab a cup of coffee and contribute something cool to the community with all that time you have got back. ☺

In the lab

Put on your thinking cap and let’s see how it works.

First we need an instance of Percona MySQL Server 8.0.17 or newer. I think containers are the most flexible way to test new stuff so I will be using that, but you could use a virtual server or just a traditional setup. Let’s download the latest version of Percona MySQL Server in a ready to run container:

docker pull percona:8.0.17-1

Eventually that command should work but sadly, Percona hadn’t built this version of the docker image when this article was written. Doing it yourself is quite simple, though, and by the time you read this it will likely be already there.

Once in place, Running an instance of Percona MySQL Server has never been so easy:

docker run --name ps -e MYSQL_ROOT_PASSWORD=secret -d percona:8.0.17-8

We’ll logon to the new container:

docker exec -ti ps mysql -u root -p

Now is the time to download the test database employees from GitHub and load it into our Percona Server. You can follow the official instructions in the project page.

Next step is to enable the data de-identification and masking feature. Installing the data masking module in Percona MySQL Server is easier than in Oracle.

mysql> INSTALL PLUGIN data_masking SONAME 'data_masking.so';
Query OK, 0 rows affected (0.06 sec)

This automatically defines a set of global functions in our MySQL instance, so we don’t need to do anything else.

A new concept: Dictionaries

Sometimes we will like to generate new data selecting values from a predefined collection. For example we could want to have

first name

For this task we have dictionaries. They are nothing more than text files containing a value per line that are loaded into MySQL memory. You need to be aware that the contents of the file are fully loaded into memory and that the dictionary only exists while MySQL is running. So keep this in mind before loading any huge file or after restarting the instance.

For our lab we will load two dictionaries holding first and last names. You can use these files or create different ones: first names and last names

Store the files in a folder of your database server (or container) readable by the

mysqld

wget https://raw.githubusercontent.com/philipperemy/name-dataset/master/names_dataset/first_names.all.txt
docker cp first_names.all.txt ps:/tmp/
wget https://raw.githubusercontent.com/philipperemy/name-dataset/master/names_dataset/last_names.all.txt
docker cp last_names.all.txt ps:/tmp/

Once the files are in our server we can map them as MySQL dictionaries.

mysql> select gen_dictionary_load('/tmp/first_names.all.txt', 'first_names');
+----------------------------------------------------------------+
| gen_dictionary_load('/tmp/first_names.all.txt', 'first_names') |
+----------------------------------------------------------------+
| Dictionary load success                                        |
+----------------------------------------------------------------+
1 row in set (0.04 sec)

mysql> select gen_dictionary_load('/tmp/last_names.all.txt', 'last_names');
+--------------------------------------------------------------+
| gen_dictionary_load('/tmp/last_names.all.txt', 'last_names') |
+--------------------------------------------------------------+
| Dictionary load success                                      |
+--------------------------------------------------------------+
1 row in set (0.03 sec)

Masking some data

Now let’s take another look at our

employees

mysql> show columns from employees;
+------------+---------------+------+-----+---------+-------+
| Field      | Type          | Null | Key | Default | Extra |
+------------+---------------+------+-----+---------+-------+
| emp_no     | int(11)       | NO   | PRI | NULL    |       |
| birth_date | date          | NO   |     | NULL    |       |
| first_name | varchar(14)   | NO   |     | NULL    |       |
| last_name  | varchar(16)   | NO   |     | NULL    |       |
| gender     | enum('M','F') | NO   |     | NULL    |       |
| hire_date  | date          | NO   |     | NULL    |       |
+------------+---------------+------+-----+---------+-------+

Ok, it’s very likely we will want to de-identify everything in this table. You can apply different methods to achieve your security requirements, but I will create a view with the following transformations:

• emp_no: get a random value from 900.000.000 to 999.999.999
• birth_date: set it to January 1st of the original year
• first_name: set a random first name from a list of names that we have in a text file
• last_name: set a random last name from a list of names that we have in a text file
• gender: no transformation
• hire_date: set it to January 1st of the original year

CREATE VIEW deidentified_employees
AS
SELECT
  gen_range(900000000, 999999999) as emp_no,
  makedate(year(birth_date), 1) as birth_date,
  gen_dictionary('first_names') as first_name,
  gen_dictionary('last_names') as last_name,
  gender,
  makedate(year(hire_date), 1) as hire_date
FROM employees;

Let’s check how the data looks in our de-identified view.

mysql> SELECT * FROM employees LIMIT 10;
+--------+------------+------------+-----------+--------+------------+
| emp_no | birth_date | first_name | last_name | gender | hire_date  |
+--------+------------+------------+-----------+--------+------------+
|  10001 | 1953-09-02 | Georgi     | Facello   | M      | 1986-06-26 |
|  10002 | 1964-06-02 | Bezalel    | Simmel    | F      | 1985-11-21 |
|  10003 | 1959-12-03 | Parto      | Bamford   | M      | 1986-08-28 |
|  10004 | 1954-05-01 | Chirstian  | Koblick   | M      | 1986-12-01 |
|  10005 | 1955-01-21 | Kyoichi    | Maliniak  | M      | 1989-09-12 |
|  10006 | 1953-04-20 | Anneke     | Preusig   | F      | 1989-06-02 |
|  10007 | 1957-05-23 | Tzvetan    | Zielinski | F      | 1989-02-10 |
|  10008 | 1958-02-19 | Saniya     | Kalloufi  | M      | 1994-09-15 |
|  10009 | 1952-04-19 | Sumant     | Peac      | F      | 1985-02-18 |
|  10010 | 1963-06-01 | Duangkaew  | Piveteau  | F      | 1989-08-24 |
+--------+------------+------------+-----------+--------+------------+
10 rows in set (0.00 sec)

mysql> SELECT * FROM deidentified_employees LIMIT 10;
+-----------+------------+------------+---------------+--------+------------+
| emp_no    | birth_date | first_name | last_name     | gender | hire_date  |
+-----------+------------+------------+---------------+--------+------------+
| 930277580 | 1953-01-01 | skaidrīte  | molash        | M      | 1986-01-01 |
| 999241458 | 1964-01-01 | grasen     | cessna        | F      | 1985-01-01 |
| 951699030 | 1959-01-01 | imelda     | josephpauline | M      | 1986-01-01 |
| 985905688 | 1954-01-01 | dunc       | burkhardt     | M      | 1986-01-01 |
| 923987335 | 1955-01-01 | karel      | wanamaker     | M      | 1989-01-01 |
| 917751275 | 1953-01-01 | mikrut     | allee         | F      | 1989-01-01 |
| 992344830 | 1957-01-01 | troyvon    | muma          | F      | 1989-01-01 |
| 980277046 | 1958-01-01 | aliziah    | tiwnkal       | M      | 1994-01-01 |
| 964622691 | 1952-01-01 | dominiq    | legnon        | F      | 1985-01-01 |
| 948247243 | 1963-01-01 | sedale     | tunby         | F      | 1989-01-01 |
+-----------+------------+------------+---------------+--------+------------+
10 rows in set (0.01 sec)

The data looks quite different, but remains good enough to apply some analytics and get meaningful results.

Let’s de-identify the table

salaries

mysql> show columns from salaries;
+-----------+---------+------+-----+---------+-------+
| Field     | Type    | Null | Key | Default | Extra |
+-----------+---------+------+-----+---------+-------+
| emp_no    | int(11) | NO   | PRI | NULL    |       |
| salary    | int(11) | NO   |     | NULL    |       |
| from_date | date    | NO   | PRI | NULL    |       |
| to_date   | date    | NO   |     | NULL    |       |
+-----------+---------+------+-----+---------+-------+

We could use something like this:

CREATE VIEW deidentified_salaries
AS
SELECT
gen_range(900000000, 999999999) as emp_no,
gen_range(40000, 80000) as salary,
mask_inner(date_format(from_date, '%Y-%m-%d'), 4, 0) as from_date,
mask_outer(date_format(to_date, '%Y-%m-%d'), 4, 2, '0') as to_date
FROM salaries;

We are using again the function

gen_range

mask_inner

mask_outer

In a real life exercise we would like to have the same values for emp_no across all the tables to keep referential integrity. This is where I think the original MySQL data-masking plugin falls short, as we don’t have deterministic functions using the original value as seed.

mysql> SELECT * FROM salaries LIMIT 10;
+--------+--------+------------+------------+
| emp_no | salary | from_date  | to_date    |
+--------+--------+------------+------------+
|  10001 |  60117 | 1986-06-26 | 1987-06-26 |
|  10001 |  62102 | 1987-06-26 | 1988-06-25 |
|  10001 |  66074 | 1988-06-25 | 1989-06-25 |
|  10001 |  66596 | 1989-06-25 | 1990-06-25 |
|  10001 |  66961 | 1990-06-25 | 1991-06-25 |
|  10001 |  71046 | 1991-06-25 | 1992-06-24 |
|  10001 |  74333 | 1992-06-24 | 1993-06-24 |
|  10001 |  75286 | 1993-06-24 | 1994-06-24 |
|  10001 |  75994 | 1994-06-24 | 1995-06-24 |
|  10001 |  76884 | 1995-06-24 | 1996-06-23 |
+--------+--------+------------+------------+
10 rows in set (0.00 sec)

mysql> SELECT * FROM deidentified_salaries LIMIT 10;
+-----------+--------+------------+------------+
| emp_no    | salary | from_date  | to_date    |
+-----------+--------+------------+------------+
| 929824695 | 61543  | 1986XXXXXX | 0000-06-00 |
| 954275265 | 63138  | 1987XXXXXX | 0000-06-00 |
| 948145700 | 53448  | 1988XXXXXX | 0000-06-00 |
| 937927997 | 54704  | 1989XXXXXX | 0000-06-00 |
| 978459605 | 78179  | 1990XXXXXX | 0000-06-00 |
| 993464164 | 75526  | 1991XXXXXX | 0000-06-00 |
| 946692434 | 51788  | 1992XXXXXX | 0000-06-00 |
| 979870243 | 54807  | 1993XXXXXX | 0000-06-00 |
| 958708118 | 70647  | 1994XXXXXX | 0000-06-00 |
| 945701146 | 76056  | 1995XXXXXX | 0000-06-00 |
+-----------+--------+------------+------------+
10 rows in set (0.00 sec)

Clean-up

Remember that when you’re done, you can free up memory by removing the dictionaries. Restarting the instance will also remove the dictionaries.

mysql> SELECT gen_dictionary_drop('first_names');
+------------------------------------+
| gen_dictionary_drop('first_names') |
+------------------------------------+
| Dictionary removed                 |
+------------------------------------+
1 row in set (0.01 sec)

mysql> SELECT gen_dictionary_drop('last_names');
+------------------------------------+
| gen_dictionary_drop('last_names') |
+------------------------------------+
| Dictionary removed                 |
+------------------------------------+
1 row in set (0.01 sec)

If you use the MySQL data-masking plugin to define different levels of access to the data, remember that you will need to load the dictionaries each time the instance is restarted. With this usage, for example, you could control the data that someone in support has access to, very much like a bargain-basement virtual private database solution. (I’m not proposing this for production systems!)

Other de-identification and masking functions

Percona Server for MySQL Data-Masking includes more functions that the ones we’ve seen here.

We have specialized functions for Primary Account Numbers (PAN), Social Security Numbers (SSN), phone numbers, e-Mail addresses… And also generic functions that will allow us to de-identify types without a specialized method.

Being an open source plugin it should be quite easy to implement any additional methods and contribute it to the broader community.

Next Steps

Using these functions we can de-identify and mask any existing dataset. But if you are populating a lower level environment using production data you would want to store the transformed data only. To achieve this you could choose between various options.

• Small volumes of data: use “de-identified” views to export the data and load into a new database using mysqldump or mysqlpump.
• Medium volumes of data: Clone the original database and de-identify locally the data using updates.
• Large volumes of data option one: using replication, create a master -> slave chain with

  STATEMENT

  binlog format and define triggers de-identifying the data on the slave. Your master can be a slave to the master (using log_slave_updates), so you don’t need to run your primary master in

  STATEMENT

  mode.
• Large volumes of data option two: using multiplexing in ProxySQL, configure ProxySQL to send writes to a clone server where you have defined triggers to de-identify the data.

Future developments

While de-identifying complex schemas we could find that, for example, the name of a person is stored in multiple tables (de-normalized tables). In this case, these functions would generate different names and the resulting data will look broken. You can solve this using a variant of the dictionary functions that will obtain the value based on the original value and passed as parameter:

gen_dictionary_deterministic('Francisco', 'first_names')

This not-yet-available function would always return the same value using that dictionary file, but in such a way that the de-identification cannot be reversed.

Oracle doesn’t currently support this, so we will expand Percona Data-Masking plugin to introduce this as a unique feature. However, that will be in another contribution, so stay tuned for more exciting changes to Percona Server for MySQL Data Masking.

—
Image: Photo by Finan Akbar on Unsplash

The content in this blog is provided in good faith by members of the open source community. Percona has not edited or tested the technical content (although in this case, of course, we have tested the data masking feature incorporated into Percona Server for MySQL 8.0/17, just not the examples in this blog). Views expressed are the authors’ own. When using the advice from this or any other online resource test ideas before applying them to your production systems, and always secure a working back up.

The post Percona Server for MySQL 8.0 – New Data Masking Feature appeared first on Percona Community Blog.