You are here
FromDual TechFeed (en)
Before I forget it and have to search again here a short note about how to rename a MySQL Partition:
My dream:ALTER TABLE history RENAME PARTITION p2015_kw10 INTO p2015_kw09;
In reality: ALTER TABLE history REORGANIZE PARTITION p2015_kw10 INTO ( PARTITION p2015_kw09 VALUES LESS THAN (UNIX_TIMESTAMP('2015-03-02 00:00:00')) );
Caution: REORGANIZE PARTITION causes a full copy of the whole partition!
Hint: I assume it would be very easy for MySQL or MariaDB to make this DDL command an in-place operation...
MySQL Partitioning was introduced in MySQL 5.1.
MySQL Enterprise Backup (MEB) has the capability to make real incremental (differential and cumulative?) backups. The actual releases are quite cool and you should really look at it...
Unfortunately the original MySQL documentation is much too complicated for my simple mind. So I did some testing and simplified it a bit for our customers...
If you want to dive into the original documentation please look here: Making an Incremental Backup .
If you want to use MySQL Enterprise Backup please let us know and we send you a quote...Prepare MySQL Backup infrastructure mkdir /backup/full /backup/incremental1 /backup/incremental2
Full MySQL Backup mysqlbackup --defaults-file=/etc/my.cnf --user=root --backup-dir=/backup/full backup mysqlbackup --defaults-file=/etc/my.cnf --user=root --backup-dir=/backup/full apply-log
First MySQL Incremental Backup mysqlbackup --defaults-file=/etc/my.cnf --user=root --incremental --incremental-base=dir:/backup/full --incremental-backup-dir=/backup/incremental1 backup mysqlbackup --defaults-file=/etc/my.cnf --user=root --backup-dir=/backup/full --incremental-backup-dir=/backup/incremental1 apply-incremental-backup
Second MySQL Incremental Backup mysqlbackup --defaults-file=/etc/my.cnf --user=root --incremental --incremental-base=dir:/backup/full --incremental-backup-dir=/backup/incremental2 backup mysqlbackup --defaults-file=/etc/my.cnf --user=root --backup-dir=/backup/full --incremental-backup-dir=/backup/incremental2 apply-incremental-backup
and so on...MySQL Restore mysqlbackup --defaults-file=/etc/my.cnf --user=root --backup-dir=/backup/full copy-back
Have fun with MySQL Enterprise Backup. If you need any help with your MySQL Backup concept, please let us know.
MySQL Enterprise Monitor (MEM) has by default no Event Handlers created and activated. These Event Handlers you have to define yourself according to your needs.
We would like to be notified by MySQL Enterprise Monitor when the number of connections is near to max_connections.
For this we search first which Advisors are available at all: Configuration -> Advisors -> Availability.
Here we can see that we have an Advisor called Maximum Connection Limit Nearing Or Reached which is scheduled for every 5 minutes and has thresholds at 75, 85, 95 and 100%:
Now we know which Advisor should create and Event. As a next step we have to create and Event which should be triggered: Configuration -> Event Handling -> Create Event Handler.
Here we can create and Event with all its needed configuration: Events -> All -> server.
If we look at the Events we can even see the detailed description and how the values for the Event are collected:
Task: Event Handler for used disk space
For this Event Handler we need the Advisor Filesystem Free Space under Operating System:
In this advisor we can configure the Threshold as well:
In the Event Handler we can define which Assets shall be monitored. For example the mountpoint: /.
Local disks can only be monitored, if a local MySQL Enterprise Monitor Agent is installed. An agent-less MySQL Enterprise Monitor cannot monitor local disk resources...
Have fun using the MySQL Enterprise Monitor. If you need any help in installing or configuring MEM do not hesitate to contact us.
All these functions are also implemented in the FromDual Performance Monitor for MySQL. If you want to relay on Open Source technology only you should consider our Performance Monitor.
FromDual has the pleasure to announce the release of the new version 1.0.0 of its widely used Nagios and Icinga plug-ins for MySQL, Galera Cluster, MariaDB and Percona Server.
All plug-ins are basically renewed and should now work all correctly.
The new Nagios/Icinga plug-ins can be downloaded here.
In the inconceivable case that you find a bug in the Nagios/Icinga plug-ins please report it to our bug tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to firstname.lastname@example.org.Description of the current functionality
Details about the functionality and the usage of each plug-in you get with the option: --help.
The following Nagios/Icinga plug-in for MySQL and MariaDB are currently available:check_db_mysql.pl
This Nagios/Icinga plug-in alerts you if your MySQL database is not up and running.check_errorlog_mysql.pl and errorLogFilterRules.pm
This Nagios/Icinga plug-in alerts you if it finds some suspicious messages in the MySQL error log.
The rules which messages should be ignored can be found in the file errorLogFilterRules.pm. If you want to add your own filter rules please add them in this file as well.
This Nagios/Icinga plug-in alerts you if the actual number of nodes in your a Galera Cluster is not the expected one.check_repl_mysql_cnt_slave_hosts.pl
This Nagios/Icinga plug-in alerts you if your MySQL Slaves have not reported to their Master properly their existence with the report_host variable.check_repl_mysql_heartbeat.pl
This Nagios/Icinga plug-in alerts you if your MySQL Slave is too many heartbeats behind its Master.check_repl_mysql_io_thread.pl
This Nagios/Icinga plug-in alerts you if your MySQL Slaves IO thread is not up an running.check_repl_mysql_read_exec_pos.pl
This Nagios/Icinga plug-in alerts you if your MySQL Slaves read and execution positions differ too much.check_repl_mysql_readonly.pl
This Nagios/Icinga plug-in alerts you if your MySQL Slave is NOT set to readonly.check_repl_mysql_seconds_behind_master.pl
This Nagios/Icinga plug-in alerts you if your MySQL Slave falls too many seconds behind its Master.check_repl_mysql_sql_thread.pl
This Nagios/Icinga plug-in alerts you if your Slaves SQL thread is not up an running.perf_mysql.pl
This Nagios/Icinga plug-in gathers MySQL and MariaDB performance data.Changes in FromDual Nagios/Icinga plug-ins 1.0.0 All plug-ins
- Usage was improved. The usage can be shown with the --help option.
- Usage states now which GRANT privileges are needed for a specific plug-in.
- Examples added how to use each plug-in.
- Default socket location moved from /tmp/mysql.sock to /var/run/mysqld/mysqld.sock.
- New host/socket convention implemented in all scripts similar to MySQL client tools.
- -epn tag added for Icinga.
- Some bugs fixed.
- More filtering rules added.
- Filtering rules separated into own file.
- Entry point finding problem fixed.
- Script name fixed.
- Unknown command problem with Galera Cluster caught.
- mysqladmin ping removed and implemented in Perl.
For subscriptions for commercial use of this software please get in contact with us.
At FromDual, we are currently preparing for our participation in the "Chemnitzer Linux-Tage" in March.
While we don't yet know whether the programme committee accepted our proposed talks, we will have a booth and hope for interesting exchanges with others from the MySQL, database, Linux, ... world. Of course, we will also mention that we are looking for additional colleagues - there are so many tasks that we need more people to handle them all. (In case you got curious, look here: http://www.fromdual.com/mysql-dba-2014-12-de )
While I attended the Chemnitzer Linux-Tage already last year (thank you, Frank Hofmann, for recommending them!), it will be the first time there for FromDual as a company. I enjoyed the informal atmosphere, which made it easy to get into contact with new people, to learn, and to discuss both facts and opinions.
To those of you who didn't yet attend such events, I can only say: You are missing some great experiences! It is good to get stimulated by others, by their problems and their experiences, so that one's thoughts are broadened and start considering new fields. Often, they lead to insights that are helpful for one's own daily work.
In the opposite direction, we at FromDual hope that during the Linux-Tage we can help others to make (better) use of MySQL in all its variants, with its surrounding tools and products, so that everybody will profit.
The Chemnitzer Linux-Tage will take place on March 21 and 22, 2015, in Chemnitz, Germany.
Parts of the programme will probably be delivered in English, so even those of you who don't understand German might enjoy coming there.
If you now consider a visit, you will find further information here:
We will be happy to meet you in Chemnitz - to talk about whatever interests both you and us, maybe even about you joining us?
MySQL provides some great enterprise features beside the MySQL Server. The ones we are asked the most at customers are:
- MySQL Enterprise Backup (MEB)
- MySQL Enterprise Monitor (MEM) and
- MySQL Enterprise Workbench (MWB)
MySQL Enterprise Backup (MEB) is an alternative to the mysqldump backup utility. Its big advantage is its fast backup but even faster restore performance. This is a must for all MySQL users having bigger databases than let's say 10 to 20 Gigabytes and/or having hard requirements for restore times (MTTR).
Last implementation tests we did with a customer for an about 30 Gbyte database were:MEBmysqldumpBackup10 minutes18 minutesRestore12 minutes80 minutes
If you need our help implementing MySQL Enterprise Backup into your backup infrastructure please get in contact with us. MySQL Enterprise Backup also seamlessly integrates into the FromDual Backup Manager for MySQL.MySQL Enterprise Monitor (MEM)
MySQL Enterprise Monitor (MEM) is an Enterprise Monitoring Solution for MySQL which Monitors your business critical MySQL databases. Various predefined advisors rise an alert if something with your precious MySQL database does not work as expected.
Our alternative competitive product is the FromDual Performance Monitor for MySQL.MySQL Enterprise Workbench (MWB)
MySQL Enterprise Workbench (MWB) is the tool modern MySQL Database administrators use to operate their MySQL databases. Old fashion ones still use the CLI... MySQL developers can easily write and test database queries and develop their data model with the ER diagram modeller.Download Enterprise tools
But how can we get now to these precious tools? This is quite easy following the screen shots below:
As a fist step you go to www.mysql.com/downloads:
Here you can find a link to Oracle eDelivery which is the MySQL/Oracle download facility. Then you get to the welcome screen:
Before you get access to the software you have to Sign In with your Oracle/MySQL customer account if you have one. If you do not have an account yet you can Create an Account to get to the software:
Then you have to agree (2 times) to the Terms & Restrictions (this is what Oracle is really good in). Once to the Oracle Trial License Agreement and once to the Export Restrictions:
Then you get to the Media Pack Search. Here you can define what product you are interested in and on which platform you are using it. Unfortunately a sub-product filter cannot be chosen. So you get a long list to pick your final package from:
An last you can download your product of desire:
Unfortunately the packages get some silly names like V59684-01.zip instead of meaningful names. But with the following command you get some information what is included in the package:unzip -l V59684-01.zip Archive: V59684-01.zip Length Date Time Name --------- ---------- ----- ---- 2958631 2014-11-04 13:29 meb-3.11.1-linux-glibc2.5-x86-64bit.tar.gz 185 2014-11-05 08:30 meb-3.11.1-linux-glibc2.5-x86-64bit.tar.gz.asc 77 2014-11-04 13:29 meb-3.11.1-linux-glibc2.5-x86-64bit.tar.gz.md5 2130 2014-11-05 15:06 README.txt --------- ------- 2961023 4 files
Have fun trying the MySQL Enterprise Features. If you need any help installing or integrating them into your infrastructure, do not hesitate to contact FromDual.
Sometimes we face the situation where we have a full MySQL database backup done with mysqldump and then we have to restore and recover just one single table out of our huge mysqldump file.
Further our mysqldump backup was taken hours ago so we want to recover all the changes on that table since our backup was taken up to the end.
In this blog article we cover all the steps needed to achieve this goal for MySQL and MariaDB.
Recommendation: It is recommended to do theses steps on a testing system and then dump and restore your table back to the production system. If you do it directly on your production system you have to know exactly what you are doing...
Further this process should be tested carefully and regularly to get familiar with it and to assure your backup/restore/recovery procedure works properly.
The table we want to recover is called test.test from our backup full_dump.sql.gz. As a first step we have to do the recovery with the following command to our test database:shell> zcat full_dump.sql.gz | extract_table.py --database=test --table=test | mysql -u root
The script extract_table.py is part of the FromDual Recovery Manager to extract one single table from a mysqldump backup.
As a next step we have to extract the binary log file and its position where to start recovery from out of our dump:shell> zcat full_dump.sql.gz | head -n 25 | grep CHANGE CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000216', MASTER_LOG_POS=1300976;
Then we have to find out where we want to stop our Point-in-Time-Recovery. The need for recover is possibly due to a TRUCATE TABLE command or similar operation executed on the wrong system or it is just a time somebody has indicated us to recover to. The position to stop we can find as follows:shell> mysqlbinlog -v mysql-bin.000216 | grep -B5 TRUNCATE --color #150123 19:53:14 server id 35622 end_log_pos 1302950 CRC32 0x24471494 Xid = 3803 COMMIT/*!*/; # at 1302950 #150123 19:53:14 server id 35622 end_log_pos 1303036 CRC32 0xf9ac63a6 Query thread_id=54 exec_time=0 error_code=0 SET TIMESTAMP=1422039194/*!*/; TRUNCATE TABLE test
And as a last step we have to apply all the changes from the binary log to our testing database:shell> mysqlbinlog --disable-log-bin --database=test --start-position=1300976 --stop-position=1302950 mysql-bin.000216 | mysql -u root --force
Now the table test.test is recovered to the wanted point in time and we can dump and restore it to its final location back to the production database.shell> mysqldump --user=root --host=testing test test | mysql --user=root --host=production test
This process has been tested on MySQL 5.1.73, 5.5.38, 5.6.22 and 5.7.5 and also on MariaDB 10.0.10 and 10.1.0.
Often, in such introductions people use the phrase of "the new kid on the block". I won't. If I am to use those words, I will arrange them as "the kid on the new block". The reason is that I don't feel as a new kid in the MySQL village (or is it a city?), let alone in DBMS country.
Ever since I left university (Technical University of Berlin, Germany), I have been involved in SQL DBMS development. After my previous product's team had been dissolved in Berlin and maintenance moved to Riga, Latvia, as a cost-cutting measure, I joined MySQL AB in 2004 as a member of the Build Team. Some of you will remember my name from MySQL release announcements, others may have seen it in bug reports, newsgroup entries, or other MySQL communication. Together with the other MySQL colleagues, I got acquired by Sun in 2008 and then by Oracle in 2010.
In 2012, I decided I wanted a culture change, from a highly (and very strictly) organized corporation to a smaller, more flexible entity. That brought me to an internet site, where I worked as a DBA, concentrating on their (more than 100) MySQL installations. It was very interesting to view MySQL from the DBA side now, I got a big load of new experiences and learned a lot about using, running, and automating MySQL (rather than building and testing it).
That might have continued, but as usual the world is changing, and my DBA job was no exception: Company strategy was to shift those tasks more to the various application teams and to do without a central point for MySQL. It is a sign how far MySQL knowledge has spread, and how little effort may be sufficient to run MySQL, that this worked. OTOH, this invalidated the assumptions on which my job was based, and I did not really fit the alternatives left in that company.
In that situation, my contact to FromDual got renewed, and both the company and me felt we should be a good match. Till now, there is no indication we were wrong: I like the work, I feel I'm productive and make customers happy, the tasks are manifold and interesting (so interesting that writing the first blog gets delayed quite long), and I expect this will continue.
"Ad multos annos!"
We recently run into some troubles with max_allowed_packet size problems during backups with the FromDual Backup/Recovery Manager and thus I investigated a bit more in the symptoms of such problems.
Read more about: max_allowed_packet.
A general rule for max_allowed_packet size to avoid problems is: All clients and the server should have set the same value for max_allowed_packet size!
I prepared some data for the test which looked as follows:mysql> SELECT id, LEFT(data, 30), LENGTH(data), ts FROM test; +----+--------------------------------+--------------+------+ | id | left(data, 30) | length(data) | ts | +----+--------------------------------+--------------+------+ | 1 | Anhang | 6 | NULL | | 2 | Anhang | 6 | NULL | | 3 | Anhangblablablablablablablabla | 2400006 | NULL | | 4 | Anhang | 6 | NULL | +----+--------------------------------+--------------+------+
Max_packet_size was set to a too small value then:mysql> SHOW GLOBAL VARIABLES WHERE variable_name = 'max_allowed_packet'; +--------------------+---------+ | Variable_name | Value | +--------------------+---------+ | max_allowed_packet | 1048576 | +--------------------+---------+
The first test was to retrieve the too big row:mysql> SELECT * FROM test WHERE id = 3; ERROR 2020 (HY000): Got packet bigger than 'max_allowed_packet' bytes mysql> SELECT CURRENT_USER(); ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... Connection id: 6 Current database: test
We got an error message AND we were disconnected from the server. This is indicated with the message MySQL server has gone away which is basically wrong. We were disconnected and not the server has died or similar in this case.
A further symptom is that we get an entry in the MySQL error log about this incident:[Warning] Aborted connection 3 to db: 'test' user: 'root' host: 'localhost' (Got an error writing communication packets)
So watching carefully such error messages in your MySQL error log with the script check_error_log_mysql.pl from our Nagios/Icinga plugins would be a good idea...
The mysqldump utility basically does the same as a SELECT command so I tried this out and got the same error:shell> mysqldump -u root test > /tmp/test_dump.sql mysqldump: Error 2020: Got packet bigger than 'max_allowed_packet' bytes when dumping table `test` at row: 2
And again we get an error message in the error log! This is also a good indicator to see if your backup, made with mysqldump failed in this case.
To get a proper dump we have to configure the mysqldump utility properly:shell> mysqldump --max-allowed-packet=5000000 -u root test > /tmp/test_dump.sql
After the backup we tried to restore the data:shell> mysql -u root test < /tmp/test_dump.sql ERROR 2006 (HY000) at line 40: MySQL server has gone away
Again we got an error on the command line and in the MySQL error log:[Warning] Aborted connection 11 to db: 'test' user: 'root' host: 'localhost' (Got a packet bigger than 'max_allowed_packet' bytes)
and further the data are only partially loaded:mysql> SELECT * FROM test; +----+--------+------+ | id | data | ts | +----+--------+------+ | 1 | Angang | NULL | | 2 | Angang | NULL | +----+--------+------+
Another symptom we can see here is that the MySQL status aborted_clients is increased in all 3 situation:mysql> SHOW GLOBAL STATUS WHERE variable_name = 'aborted_clients'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | Aborted_clients | 10 | +-----------------+-------+
One positive aspect is that with MySQL 5.7.5 the first 2 symptoms do not appear any more...Further information you can find here: Communication Errors and Aborted Connections.
For processing SELECT queries MySQL needs some times the help of temporary tables. These temporary tables can be created either in memory or on disk.
The number of creations of such temporary tables can be found with the following command:mysql> SHOW GLOBAL STATUS LIKE 'created_tmp%tables'; +-------------------------+-------+ | Variable_name | Value | +-------------------------+-------+ | Created_tmp_disk_tables | 4 | | Created_tmp_tables | 36 | +-------------------------+-------+
There are 2 different reasons why MySQL is creating a temporary disk table instead of a temporary memory table:
- The result is bigger than the smaller one of the MySQL variables max_heap_table_size and tmp_table_size.
- The result contains columns of type BLOB or TEXT.
This method can be used if changing the table structure from TEXT to VARCHAR or the use of a RAM disk are not possible solutions.
After properly installing and testing a Galera Cluster we see that the set-up is not finished yet. It needs something in front of the Galera Cluster that balances the load over all nodes.
So we install a load balancer in front of the Galera Cluster. Typically nowadays HAProxy is chosen for this purpose. But then we find, that the whole Galera Cluster is still not high available in case the load balancer fails or dies. So we need a second load balancer for high availability.
But how should we properly fail-over when the HAProxy load balancer dies? For this purpose we put a Virtual IP (VIP) in front of the HAProxy load balancer pair. The Virtual IP is controlled and fail-overed with Keepalived.
First some preparations: For installing socat we need the repoforge repository:shell> cd /tmp shell> wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm shell> yum localinstall rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm shell> yum update shell> yum install socat
Then we can start installing HAProxy and Keepalived:shell> yum install haproxy keepalived shell> chkconfig haproxy on shell> chkconfig keepalived on
We can check the installed HAProxy and Keepalived versions as follows:shell> haproxy -v HA-Proxy version 1.5.2 2014/07/12 shell> keepalived --version Keepalived v1.2.13 (10/15,2014)
Configuration of HAProxy
More details you can find in the HAProxy documentation.shell> cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak shell> cat << _EOF >/etc/haproxy/haproxy.cfg # # /etc/haproxy/haproxy.cfg # #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 1020 # See also: ulimit -n user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats.sock mode 600 level admin stats timeout 2m #--------------------------------------------------------------------- # common defaults that all the 'frontend' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode tcp log global option dontlognull option redispatch retries 3 timeout queue 45s timeout connect 5s timeout client 1m timeout server 1m timeout check 10s maxconn 1020 #--------------------------------------------------------------------- # HAProxy statistics backend #--------------------------------------------------------------------- listen haproxy-monitoring *:80 mode http stats enable stats show-legends stats refresh 5s stats uri / stats realm Haproxy\ Statistics stats auth monitor:AdMiN123 stats admin if TRUE frontend haproxy1 # change on 2nd HAProxy bind *:3306 default_backend galera-cluster backend galera-cluster balance roundrobin server nodeA 192.168.1.61:5201 maxconn 151 check server nodeB 192.168.1.61:5202 maxconn 151 check server nodeC 192.168.1.61:5203 maxconn 151 check _EOF
Starting and testing HAProxy
The HAProxy can be started as follows:shell> service haproxy start
and then be checked either over the socket:shell> socat /var/lib/haproxy/stats.sock readline prompt > show info > show stat > help
or over your favourite web browser entering the username and password (monitor:AdMiN123) specified in the configuration file above:
To check the application over the load balancer we can run the following command:shell> mysql --user=app --password=secret --host=192.168.1.38 --port=3306 --exec="SELECT @@wsrep_node_name;" +-------------------+ | @@wsrep_node_name | +-------------------+ | Node C | +-------------------+ shell> mysql --user=app --password=secret --host=192.168.1.38 --port=3306 --exec="SELECT @@wsrep_node_name;" +-------------------+ | @@wsrep_node_name | +-------------------+ | Node A | +-------------------+ shell> mysql --user=app --password=secret --host=192.168.1.38 --port=3306 --exec="SELECT @@wsrep_node_name;" +-------------------+ | @@wsrep_node_name | +-------------------+ | Node B | +-------------------+
Configuration a Virtual IP (VIP) with Keepalived
Now we have 2 HAProxy load balancers. But what happens if one of them fails. Then we do not want to reconfigure our application to work properly again. The fail-over should happen automatically. For this we need a Virtual IP which should automatically fail-over.
Starting and testing Keepalived
To test the keepalived we can run the following command:shell> keepalived -f /etc/keepalived/keepalived.conf --dont-fork --log-console --log-detail ^C
To finally start it the following command will serve:shell> service keepalived start
To check the Virtual IP the following command will help:shell> ip addr show eth1
And then we can check our application over the VIP:shell> mysql --user=app --password=secret --host=192.168.1.99 --port=3306 --exec="SELECT @@wsrep_node_name;"
- Brian Seitzer: Redundant Load Balancers – HAProxy and Keepalived
We have recently seen a case where the following command was executed on a Galera Cluster node:SQL> GRANT SUPER ON userdb.* TO email@example.com; ERROR 1221 (HY000): Incorrect usage of DB GRANT and GLOBAL PRIVILEGES
2014-12-09 14:53:55 7457 [Warning] Did not write failed 'GRANT SUPER ON userdb.* TO firstname.lastname@example.org' into binary log while granting/revoking privileges in databases. 2014-12-09 14:53:55 7457 [ERROR] Slave SQL: Error 'Incorrect usage of DB GRANT and GLOBAL PRIVILEGES' on query. Default database: ''. Query: 'GRANT SUPER ON userdb.* TO email@example.com', Error_code: 1221 2014-12-09 14:53:55 7457 [Warning] WSREP: RBR event 1 Query apply warning: 1, 17 2014-12-09 14:53:55 7457 [Warning] WSREP: Ignoring error for TO isolated action: source: c5e54ef5-7faa-11e4-97b0-5e5c695f08a5 version: 3 local: 0 state: APPLYING flags: 65 conn_id: 4 trx_id: -1 seqnos (l: 4, g: 17, s: 15, d: 15, ts: 113215863294782)
According to the error message it looks like this command is done in Total Order Isolation (TOI) mode during the Rolling Schema Upgrade (RSU).
Only on the nodes which did NOT receive this wrong command the error log message was written and further they have received a GRA_*.log file.
Analysis of the GRA_*.log (failed transactions) files:hexdump -C GRA_2_16.log 00000000 f3 fe 86 54 02 53 14 00 00 76 00 00 00 76 00 00 |...T.S...v...v..| 00000010 00 00 00 04 00 00 00 00 00 00 00 00 00 00 2a 00 |..............*.| 00000020 00 00 00 00 00 01 00 00 00 40 00 00 00 00 06 03 |.........@......| 00000030 73 74 64 04 21 00 21 00 08 00 0b 04 72 6f 6f 74 |std.!.!.....root| 00000040 09 6c 6f 63 61 6c 68 6f 73 74 00 67 72 61 6e 74 |.localhost.grant| 00000050 20 53 55 50 45 52 20 6f 6e 20 75 73 65 72 64 62 | SUPER on userdb| 00000060 2e 2a 20 74 6f 20 72 6f 6f 74 40 31 32 37 2e 30 |.* to firstname.lastname@example.org| 00000070 2e 30 2e 31 31 31 |.0.111 |
dd if=bin-log.000001 of=binlog.header bs=1 count=120 cat binlog.header GRA_2_17.log > GRA_2_17.binlog_events mysqlbinlog GRA_2_17.binlog_events ... # at 120 #141209 15:04:54 server id 5201 end_log_pos 118 CRC32 0x3432312e Query thread_id=45 exec_time=0 error_code=0 SET TIMESTAMP=1418133894/*!*/; SET @@session.pseudo_thread_id=4/*!*/; SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/; SET @@session.sql_mode=1073741824/*!*/; SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/; /*!\C utf8 *//*!*/; SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=8/*!*/; SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; grant SUPER on userdb.* to email@example.com /*!*/; DELIMITER ; # End of log file ROLLBACK /* added by mysqlbinlog */; /*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/; /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
It further looks like this command was issues by Connection ID number 4: conn_id: 4.
Sometimes, MySQL tablespace file(s) might be deleted by mistake, e.g. delete the shared tablespace (ibdata1) or an individual tablespace (table_name.ibd).
In this post I will show you how to recover those files (on Linux OS) having only one condition, MySQL service should still be running. If MySQL service stopped after deleting that file, this method will not work, so it is extremely important to act as quick as possible to avoid data loss.
The following is a simple table creation (innodb_file_per_table is enabled) and the records count inside that table:SQL> SHOW CREATE TABLE t\G *************************** 1. row *************************** Table: t Create Table: CREATE TABLE `t` ( `id` int(11) NOT NULL AUTO_INCREMENT, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=23 DEFAULT CHARSET=latin1 1 row in set (0.00 sec) SQL> SELECT COUNT(*) FROM t; +----------+ | COUNT(*) | +----------+ | 22 | +----------+ 1 row in set (0.02 sec)
Now, lets delete the individual tablespace for that table:shell> rm -rf /var/lib/mysql/test/t.ibd
At this time, we can still select and modify that table!!SQL> INSERT INTO t VALUES (NULL); Query OK, 1 row affected (0.00 sec) SQL> SELECT COUNT(*) FROM t; +----------+ | COUNT(*) | +----------+ | 23 | +----------+ 1 row in set (0.00 sec)
To be more accurate, rm does not actually delete the file, rather it removes the directory entry pointing to the file's inode. The inode - and in consequence the file - will be removed only if this is the last reference, but as long as the MySQL server process has the file opened, there is another reference which is the open file handle (that's why the "mysqld" server process must still be running).
To list the opened files we can use the Linux command lsof (we filter the output to get only the deleted tablespace information):shell> lsof |grep t.ibd COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME mysqld 11401 mysql 25uW REG 7,0 98304 1010691 /var/lib/mysql/test/t.ibd (deleted)
The file has a tag of (deleted) which means that the directory entry pointing to the file's inode was deleted but there is another reference(s) to that inode, otherwise it won't be listed by the above command.
Now the question is, how can we get the on-disk path to that opened file if the directory entry was removed?
We can use the "/proc" interface to the running processes and their file handles by the following formula:
- File path = /proc/PID/fd/FD-number
According to the above formula and using the output of the "lsof" command, the file we just deleted is located here:shell> ll /proc/11401/fd/25 lrwx------ 1 mysql mysql 64 Oct 28 16:14 /proc/11401/fd/25 -> /var/lib/mysql/test/t.ibd (deleted)
To make sure that this is the on-disk path for the file we deleted, check the reference: it still points to the original path.How can we recover that file??
- First, we should make sure that no other queries are modifying that table: SQL> LOCK TABLE t READ; Query OK, 0 rows affected (0.00 sec)
- Then we copy the data blocks (/proc/11401/fd/25) to a new file (we use the original file path) and change the ownership to the MySQL system user (mysql): shell> cp /proc/11401/fd/25 /var/lib/mysql/test/t.ibd shell> chown mysql:mysql /var/lib/mysql/test/t.ibd
- Restart MySQL service (if we didn't restart MySQL service directly after recovering the tablespace all changes on that table will still be redirected to the open file handle not the just recovered copy and thus will be lost after the restart): shell> service mysql restart ..... SUCCESS! ..... SUCCESS!
- The tablespace is now recovered and we can modify the table normally: SQL> SELECT COUNT(*) FROM t; +----------+ | COUNT(*) | +----------+ | 23 | +----------+ 1 row in set (0.00 sec) SQL> INSERT INTO t VALUES (NULL); Query OK, 1 row affected (0.00 sec) SQL> INSERT INTO t VALUES (NULL); Query OK, 1 row affected (0.00 sec) SQL> select COUNT(*) from t; +----------+ | COUNT(*) | +----------+ | 25 | +----------+ 1 row in set (0.00 sec)
- We can use the same procedures above to recover the shared tablespace (ibdata1) but we should lock all tables before the recovery process by using the SQL command "FLUSH TABLES WITH READ LOCK;"
- If the MySQL server had to deal with more files (.frm, .ibd, .MYI, .MYD, ...) than its "open_file_limit", it might happen that it will close this handle. In that case, the file will also cease to exist, even though the process is still running and that's why it is extremely important to act as quick as possible.
- The same procedure can be used to recover MyISAM files (.MYI and .MYD) but note that the file handle will be released if "FLUSH TABLES;" SQL command was executed.
- The same procedure can be used as well to recover binary logs, general logs, ... etc but note that the file handle will be released if "FLUSH LOGS;" SQL command was executed.
- This method can be used to recover any deleted file on Linux not only MySQL files but if the inode files have other references (lsof).
Real life case:
One of our customers was enabling the general query log on his production system, he noticed that the file was continuously growing and to not to consume the available free disk space on his server he removed that file by "rm /path/to/general_query.log". However, the available free space was still being consumed while he couldn't see the general log file. The customer thought that the file was deleted but in fact, the file handle was still opened by MySQL server process.
To get the problem solved we only issued the SQL command "FLUSH LOGS;" - which the customer should have issued after removing the file - then the file handle was closed, thus the inode was deleted and the consumed disk space freed back to the system.
Global Transaction ID (GTID) is one of the major features that were introduced in MySQL 5.6 which provides a lot of benefits (I have talked about the GTID concept, implementation and possible troubleshooting at Percona Live London 2014, you can download the slides from our presentations repository or from my session at Percona Live.
On the other hand, there are some important things you should consider before deploying GTID in production, I'm going to list them here in this blog post.
- Migration to GTID replication
- Non transactionally safe statement will raise errors now
- MySQL Performance in GTID
- mysql_upgrade script
- Errant transactions!
- Filtration on the slave
It is required to shutdown MySQL service on all servers in the replication setup in order to perform the migration from classic replication (based on binary logs information) to the transaction-based (GTID) replication which means that the migration process requires downtime.
The online migration to GTID replication is not yet available.
Facebook and Booking.com provided some MySQL patches for this, but they are not yet contained in Oracle's binaries.
So, if you can't afford a downtime during the migration process, then you might not be able to make the change.
It is required to enable the system variable (enforce_gtid_consistency) on all servers inside the GTID replication setup which prevent executing the non transactionally safe statements (check GTID restrictions) like:
- CREATE TABLE .. SELECT.
- CREATE TEMPORARY TABLE (inside a transaction).
- Statements that update non-transactional tables inside a transaction.
So, you will have to fix your application first if it contains any of the above statements before using GTID replication.MySQL Performance in GTID
It is required to enable the variables (bin_log and log_slave_updates) on - at least - the slave servers which affects the performance on those slaves negatively.
So, the performance should be tested very well before the production migration to GTID replication.mysql_upgrade script
The mysql_upgrade script's problem when executed on a server having gtid_mode=on has been fixed since MySQL 5.6.7, but it is still not recommended to execute mysql_upgrade when gtid_mode=on as it might change system tables that is using MyISAM, which is non transactional.Errant transactions! Transactions which are executed on a slave apart from the replication transactions (i.e. not executed on the master) are called "Errant transactions", those transactions cause trouble if that slave later is promoted to be a new master in the a fail-over process.
Once the other slaves connect to the new master, they send the value of gtid_executed and the master in turn checks those values, compares it with its own gtid_executed set and sends back all missing transactions (the errant transactions) to the slaves which leads to one of the following problems:
- If those transactions still exist in the new master's binary log files, they will be replicated to the other slave which was not intentional when those were executed only on the slave (new master).
- If those transactions do no longer exist in the new master's binary log file, the replication will break on all slaves.
- Choose some slaves to be possible candidates for promotion in case of fail-over. Thus, stand alone transactions (which are not coming from the master) should NOT be executed there.
- Use one of the MySQL utilities (mysqlfailover or mysqlrpladmin) to find out if there are any errant transactions on the slave before the promotion or not.
In some cases we might need to make filtration to the replication on the slave(s) i.e. not all tables' or databases' changes are propagated to the slave by using the system variables (replicate_ignore_db or replicate_ignore_table). When the slave receives transactions from the master which modify those ignored tables or databases, it simply skips executing them and when the slave restarted it sends the gtid_excuted to the master and the master finds the missing transactions (those for the ignored tables or databases) and sends them back to the slave.
Again, that leads to one of the following two conditions:
- If those transactions still exist in the master's binary log files, then no problem as the slave will skip executing them again!!
- If those transactions are no longer there in the master's binary log files, the replication will break on the slave.
Well, the above problem is supposed to be fixed in MySQL 5.6.18 (Bug #70048) The fix is injecting empty transactions on the slave for those ones modifying ignored tables or databases instead of just skipping them, and when the slave restarted they won't be sent back again from the master.
I listed the above problem although it should be fixed now because I want to mention that having MySQL always updated to the recent release is a good practice to avoid such problems and to get the most bug fixes.Conclusion:
The following are the main things which should be considered before using GTID:
- Migration from classic replication to transaction-based (GTID) replication requires downtime.
- Non-transactionally safe statements will not be executed in GTID replication.
- MySQL performance is a little bit slower in GTID replication, especially, on the slaves.
- mysql_upgrade script might cause troubles on a server having GTID_MODE=ON and it should be tested first.
- Errant transactions might break the replication in the fail-over process, thus, planning slaves for promotion will avoid falling in such cases.
- Some GTID bugs are fixed now (like slave filtration issue), thus MySQL server should be updated to the latest version once there is a new release.
- New bugs are expected to be discovered, so the application should be tested very well with GTID before performing the migration on production.
A few weeks ago, we received an interesting Galera Cluster support case from one of our customers that the application is not working well and they face a lot of troubles in their Galera Cluster setup.
After some investigations, we found a lot of insert queries in state "query end" and lasting for long time without being completed. Also some other queries which were sleeping for long time having the info of "XA COMMIT":SQL> SHOW PROCESSLIST; 27 user host:33214 foodmart Query 14440 sleeping XA COMMIT 0x31302e31312e31332e34372e746d30303336383030303031,0x31302e31312e31332e34372e746d333638,0x SQL> SHOW ENGINE INNODB STATUS; TRANSACTIONS ============ ---TRANSACTION 2DE71D, ACTIVE 14459 sec 9 lock struct(s), heap size 1248, 1 row lock(s), undo log entries 115 MySQL thread id 27, OS thread handle 0x7fc21a42c700, query id 96187 host host-ip foodmart sleeping XA COMMIT 0x31302e31312e31332e34372e746d30303336383030303031,0x31302e31312e31332e34372e746d333638,0x41544f4d ONE PHASE
XA means eXtended Architecture and "XA COMMIT" statement is one of the distributed transactions (XA Transactions) commands which are clearly NOT supported in Galera Cluster and one of its limitations because of possible rollback on commit.
The following command can be used to check if XA Transactions are being used by your application or not:SQL> SHOW GLOBAL STATUS LIKE 'Com_xa%'; +-------------------+---------+ | Variable_name | Value | +-------------------+---------+ | Com_xa_commit | 2828094 | | Com_xa_end | 2828221 | | Com_xa_prepare | 0 | | Com_xa_recover | 2205697 | | Com_xa_rollback | 42 | | Com_xa_start | 2828305 | +-------------------+---------+ 6 rows in set (0.00 sec)
There are only two possible solutions for this problem:
- Get rid of all XA transactions in the application to get the Galera Cluster work.
- Use another HA solution (Active/passive, Master/Slave, ... etc) but not Galera Cluster.
- XA transactions can not be supported in Galera Cluster and that is already stated clearly in the Galera Cluster Limitations.
- Before Moving to Galera Cluster, it is preferred to go through all its limitations and check whether your application can cope with them or not.
Precaution: Before you try this out on your production system do a BACKUP first! FromDual Backup Manager can help you with this.Situation
A MySQL user has delete its InnoDB table files for example like this:shell> rm -f $datadir/test/test.* Analysis
We do some analysis first:mysql> DROP TABLE test; ERROR 1051 (42S02): Unknown table 'test' mysql> CREATE TABLE test (id INT) ENGINE = InnoDB; ERROR 1050 (42S01): Table '`test`.`test`' already exists
The MySQL error log shows us the following information:141022 17:09:04 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. 141022 17:09:04 InnoDB: Error: trying to open a table, but could not InnoDB: open the tablespace file './test/test.ibd'! InnoDB: Have you moved InnoDB .ibd files around without using the InnoDB: commands DISCARD TABLESPACE and IMPORT TABLESPACE? InnoDB: It is also possible that this is a temporary table #sql..., InnoDB: and MySQL removed the .ibd file for this. InnoDB: Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html InnoDB: for how to resolve the issue.
User claims that he does NOT need the table and/or the data any more but wants to get rid of the error messages and/or create a new table with the same name.mysql> CREATE SCHEMA recovery; mysql> use recovery mysql> CREATE TABLE test (id INT) ENGINE = InnoDB; mysql> \! cp $datadir/recovery/test.frm $datadir/test/ mysql> DROP SCHEMA recovery; mysql> use test mysql> DROP TABLE test; Prove
To prove it works we create a new table and fill in some records:mysql> CREATE TABLE test (id int UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY, data VARCHAR(64), ts TIMESTAMP) ENGINE = InnoDB; mysql> INSERT INTO test VALUES (NULL, 'Test data', NULL); Literature
In a few weeks, I'll be speaking at Percona Live London 2014 (November 3-4) about "Transaction-based REPLICATION (GTID) - IMPLEMENTATION AND TROUBLESHOOTING". I'll talk about how to implement GTID replication in different scenarios and how to troubleshoot most of the common issues in GTID replication.
Anyone interested in learning more about GTID replication or planing to go with GTID replication in production should attend this talk.
Face-to-face meetings are very welcome, please let me know (firstname.lastname@example.org) if you are interested so we can schedule a meeting.
Looking forward to seeing you there!!
We have been asked several times by MySQL users about how to install multiple MySQL instances on a single host.
Typically, this is required when testing different MySQL versions or MySQL servers (MySQL server, Percona server and MariaDB server) while no available resources are available.
Sometimes, it is even required to install multiple MySQL instances on a single production server.
In this article, I'll go through the steps needed to install multiple MySQL instances on a single host (using the tar balls binaries) and how our popular tool MyEnv can make such process so easy.Prepare MySQL environment [root@centos-temp ~]# groupadd mysql [root@centos-temp ~]# useradd -g mysql mysql [root@centos-temp ~]# su - mysql [mysql@centos-temp:~]$ mkdir ~/product [mysql@centos-temp:~]$ mkdir ~/data [mysql@centos-temp ~]$ cd /downloads [mysql@centos-temp ~]$ wget http://dev.mysql.com/get/Downloads/MySQL-5.6/mysql-5.6.21-linux-glibc2.5-x86_64.tar.gz [mysql@centos-temp ~]$ cd ~/product [mysql@centos-temp ~]$ tar xf /downloads/mysql-5.6.21-linux-glibc2.5-x86_64.tar.gz [mysql@centos-temp ~]$ ln -s mysql-5.6.21-linux-glibc2.5-x86_64 mysql-5.6.21 Install MyEnv
- Prepare the first instance: [mysql@centos-temp ~]$ mkdir ~/data/master [mysql@centos-temp ~]$ vi ~/data/master/my.cnf [mysqld] port = 3306 basedir = /home/mysql/product/mysql-5.6.21 datadir = /home/mysql/data/master socket = /home/mysql/data/master/master.sock user = mysql
- Launch the myenv installer: [mysql@centos-temp ~]$ ~/product/myenv/bin/installMyEnv.sh PHP is installed on /usr/bin/php Starting MyEnv installer: /home/mysql/product/myenv-1.1.2/bin/installMyEnv.php Configuration file /etc/myenv/myenv.conf does NOT exist. Copy from template or abort (T, a):
- Since this is the first instance, the myenv config file does not exist yet, we take the template (t): Copy from template or abort (T, a): t Copy /home/mysql/product/myenv-1.1.2/etc/myenv.conf.template to /etc/myenv/myenv.conf
- Then MyEnv will detect that no instances are there, we choose the first option (a) to add a new instance: No instance exists yet. An instance is the same as a mysqld process. What do you want to do next? o Add a new instance, o change an existing instance, o delete an existing instance, o save configuration and exit or o quit without saving (A/c/d/s/q)? a
- MyEnv will ask for the new instance name (master): You have to enter a new instance name. An instance name can consist of 0-9, a-z, A-Z, - and _. Examples: test, prod, dev, [mysqld1] mysqld-3306 Enter a new instance name: master
- MyEnv will ask for the MySQL basedir and datadir: Changing instance master: Set basedir. The basedir is the directory where your MySQL binaries are located. Example: /home/mysql/product/myenv /home/mysql/product/myenv-1.1.2 /home/mysql/product/mysql-5.6.21 /home/mysql/product/mysql-5.6.21-linux-glibc2.5-x86_64 /usr/local/mysql /opt/mysql [/usr] basedir = /home/mysql/product/mysql-5.6.21 /home/mysql/product/mysql-5.6.21 seems to exist and has already some files in it . Omitting installation of binary tar ball. Set datadir. The datadir is the directory where your MySQL data are located. Example: /usr/local/mysql/data, /opt/mysql/data or /var/lib/mysql or /home/mysql/product/mysql-5.6.21/data or [/home/mysql/data/master] or /home/mysql/data/master datadir = /home/mysql/data/master WARNING: directory /home/mysql/data/master already exist.
- MyEnv will detect that no mysql database is created yet in the datadir and ask if it should install it for us or not (using the normal mysql_install_db script) Shall I install a mysql database under /home/mysql/data/master (Y/n)? y Installing MySQL instance. Please be patient, this can take a while... /home/mysql/product/mysql-5.6.21/scripts/mysql_install_db --datadir=/home/mysql/data/master --basedir=/home/mysql/product/mysql-5.6.21
- For the new instance, we should specify the port, the socket, and the location of my.cnf (all these must be specific per instance): Set port. Example: 3307 or  port = 3306 Set socket. Example: /tmp/mysql-3306.sock /tmp/mysql-master.sock [/tmp/mysql.sock] /var/run/mysqld/mysql.sock socket = /home/mysql/data/master/master.sock Choose location of my.cnf: Example: /etc/my.cnf /etc/mysql/my.cnf /home/mysql/data/master/my-3306.cnf [/home/mysql/data/master/my.cnf] my.cnf = /home/mysql/data/master/my.cnf
- MyEnv will list the just added instance and ask what should be the next step, we should save and exit (s): The following instances are available: master An instance is the same as a mysqld process. What do you want to do next? o Add a new instance, o change an existing instance, o delete an existing instance, o save configuration and exit or o quit without saving (a/c/d/S/q)? s Writing the configuration file... Rename /etc/myenv/myenv.conf to /etc/myenv/myenv.conf.2014-10-15-05-50-12 Write /etc/myenv/myenv.conf
- MyEnv will propose to add the following lines to the .bash_profile file, we should confirm (y): Do you want to add the following code to your ~/.bash_profile file? . /etc/myenv/MYENV_BASE MYENV_PWD=`pwd` cd $MYENV_BASE/bin . myenv.profile cd $MYENV_BASE; $MYENV_BASE/bin/showMyEnvStatus.php; cd - > /dev/null cd $MYENV_PWD (Y/n)? y Writing /etc/myenv/MYENV_BASE
- We should add the myenv startup script as shown here into the appropriate path (the provided commands should be executed manually from the root/privileged-user after that): Please copy yourself manually the MyEnv start/stop script to the following location: shell> sudo cp /home/mysql/product/myenv-1.1.2/bin/myenv.server /etc/init.d/myenv done (Y)? y and link it to your O/S runlevels: RedHat/CentOS: shell> sudo chkconfig --add myenv; chkconfig myenv on Ubuntu/Debian: shell> sudo update-rc.d myenv defaults SLES/OpenSuSE: shell> sudo chkconfig --add myenv done (Y)? y Now source your profile as follows: . ~/.bash_profile
- Now, we should logout and login so that the .bash_profile code will be executed (we can copy the startup script in the meantime): [mysql@centos-temp ~]$> exit [root@centos-temp ~]# cp /home/mysql/product/myenv-1.1.2/bin/myenv.server /etc/init.d/myenv [root@centos-temp ~]# chkconfig --add myenv [root@centos-temp ~]# su - mysql Up : Down : master (5.6.21) master ( 3306) : test mysql@centos-temp:~ [master, 3306]>
- Now, we can start the just installed instance and check if it is started correctly or not: mysql@centos-temp:~ [master, 3306]> start . SUCCESS! mysql@centos-temp:~ [master, 3306]> up Up : master (5.6.21) Down : master ( 3306) : test mysql@centos-temp:~/data/master [master, 3306]>
Cool! We have got the first instance installed. let's install the second one ...Install the second instance (named slave1)
- Prepare the second instance: mysql@centos-temp:~ [master, 3306]> mkdir ~/data/slave1 mysql@centos-temp:~ [master, 3306]> vi ~/data/slave1/my.cnf [mysqld] port = 3307 basedir = /home/mysql/product/mysql-5.6.21 datadir = /home/mysql/data/slave1 socket = /home/mysql/data/slave1/slave1.sock user = mysql
- Launch the MyEnv installer and add the new instance: mysql@centos-temp:~ [master, 3306]> product/myenv/bin/installMyEnv.sh PHP is installed on /usr/bin/php Starting MyEnv installer: /home/mysql/product/myenv-1.1.2/bin/installMyEnv.php Configuration file /etc/myenv/myenv.conf already exists. Use this, overwrite with template or abort (U,t,a): u The following instances are available: master An instance is the same as a mysqld process. What do you want to do next? o Add a new instance, o change an existing instance, o delete an existing instance, o save configuration and exit or o quit without saving (A/c/d/s/q)? a You have to enter a new instance name. An instance name can consist of 0-9, a-z, A-Z, - and _. Examples: test, prod, dev, [mysqld1] mysqld-3306 Enter a new instance name: slave1 Changing instance slave1: Set basedir. The basedir is the directory where your MySQL binaries are located. Example: /home/mysql/product/myenv /home/mysql/product/myenv-1.1.2 /home/mysql/product/mysql-5.6.21 /home/mysql/product/mysql-5.6.21-linux-glibc2.5-x86_64 /usr/local/mysql /opt/mysql [/usr] basedir = /home/mysql/product/mysql-5.6.21 /home/mysql/product/mysql-5.6.21 seems to exist and has already some files in it. Omitting installation of binary tar ball. Set datadir. The datadir is the directory where your MySQL data are located. Example: /usr/local/mysql/data, /opt/mysql/data or /var/lib/mysql or /home/mysql/product/mysql-5.6.21/data or [/home/mysql/data/slave1] or /home/mysql/data/slave1 datadir = /home/mysql/data/slave1 WARNING: directory /home/mysql/data/slave1 already exist. Shall I install a mysql database under /home/mysql/data/slav1 (Y/n)? y Installing MySQL instance. Please be patient, this can take a while... /home/mysql/product/mysql-5.6.21/scripts/mysql_install_db --datadir=/home/mysql/data/slave1 --basedir=/home/mysql/product/mysql-5.6.21 Set port. Example: 3307 or  port = 3307 Set socket. Example: /tmp/mysql-3307.sock /tmp/mysql-slave1.sock [/tmp/mysql.sock] /var/run/mysqld/mysql.sock socket = /home/mysql/data/slave1/slave1.sock Choose location of my.cnf: Example: /etc/my.cnf /etc/mysql/my.cnf /home/mysql/data/slave1/my-3307.cnf [/home/mysql/data/slave1/my.cnf] my.cnf = /home/mysql/data/slave1/my.cnf The following instances are available: master slave1 An instance is the same as a mysqld process. What do you want to do next? o Add a new instance, o change an existing instance, o delete an existing instance, o save configuration and exit or o quit without saving (a/c/d/S/q)? s Writing the configuration file... Rename /etc/myenv/myenv.conf to /etc/myenv/myenv.conf.2014-10-15-06-44-04 Write /etc/myenv/myenv.conf Now source your profile as follows: . ~/.bash_profile
- Then we logout and login again to load the new configs: mysql@centos-temp:~ [master, 3306]> exit logout [root@centos-temp product]# su - mysql Up : master (5.6.21) Down : slave1 (5.6.21) master ( 3306) : test slave1 ( 3307) : test mysql@centos-temp:~ [master, 3306]>
MyEnv says that we have two instances (master and slave1), master is up and running while slave1 is stopped. Let's start it then ..
- First, all commands we execute are sent to the master instance, we should change to the slave1 instance: mysql@centos-temp:~ [master, 3306]> slave1 mysql@centos-temp:~ [slave1, 3307]> start . SUCCESS! mysql@centos-temp:~ [slave1, 3307]>
- Both instances are now up and running ... mysql@centos-temp:~ [slave1, 3307]> up Up : master (5.6.21) slave1 (5.6.21) Down : master ( 3306) : test slave1 ( 3307) : test mysql@centos-temp:~ [slave1, 3307]>
After we installed the two instances we need to know how can we navigate between them to open MySQL connections.
- Open a MySQL connection to the master instance: mysql@centos-temp:~ [master, 3306]> mysql -u root Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 7 Server version: 5.6.21 MySQL Community Server (GPL) Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. root@localhost master [(none)] SQL>
- While the instance name is already printed in the terminal we need to double check that by a MySQL command (the used port) and exit after that: root@localhost master [(none)] SQL> SHOW GLOBAL VARIABLES LIKE 'port'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | port | 3306 | +---------------+-------+ 1 row in set (0.00 sec) root@localhost master [(none)] SQL> exit Bye mysql@centos-temp:~ [master, 3306]>
- Open a MySQL connection to the slave1 instance and check the used port: mysql@centos-temp:~ [master, 3306]> slave1 mysql@centos-temp:~ [slave1, 3307]> mysql -u root Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 5 Server version: 5.6.21 MySQL Community Server (GPL) Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. root@localhost slave1 [(none)] SQL> SHOW GLOBAL VARIABLES LIKE 'port'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | port | 3307 | +---------------+-------+ 1 row in set (0.00 sec) root@localhost slave1 [(none)] SQL>
- Navigate back to the master instance: mysql@centos-temp:~ [slave1, 3307]> master mysql@centos-temp:~ [master, 3306]>
More information about MyEnv commands can be checked here.Notes:
- Using the same procedures above we can add several instances using either the same binaries or some others.
- We used the smallest configurations in the my.cnf file, you are free to make your own changes.
This week we did some migrations from MariaDB 10.0 to Percona Server 5.6 at the IT department of a big German bank.
We were perfectly aware that since version 10.0 the MariaDB code base started to diverge slightly away from the MySQL and Percona Server code base which are still pretty close to each other.
Because of the Percona Server option enforce_storage_engine we wanted to do this migration.
We stopped the MariaDB 10.0 server replaced the binaries by the Percona Server 5.6 binaries and started the Percona 5.6 server again. After successfully starting the instance we found some error messages in the MySQL error log. By running the mysql_upgrade command some of the problems were fixed but not all of them. Still left problems were:
- The MariaDB binary logs provoked some error messages for the Percona Server: [ERROR] Error in Log_event::read_log_event(): 'Found invalid event in binary log', data_len: 25, event_type: -93 [Warning] Error reading GTIDs from binary log: -1 [ERROR] Incorrect definition of table mysql.db: expected column 'User' at position 2 to have type char(16), found type char(80). [ERROR] Incorrect definition of table mysql.event: expected column 'definer' at position 3 to have type char(77), found type char(141). [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set... A purge of the binary logs solved this issue.
- The tables mysql.event, mysql.innodb_table_stats and mysql.innodb_index_stats where not fixed by mysql_upgrade (a bug to fix for Percona and MySQL/Oracle?). We had to replace those tables manually by copying from an other already working Percona 5.6 Server.
Later in the FromDual technology labs we investigated further and tried the other way from Percona Server 5.6 to MariaDB 10.0. In this direction we found some other errors in the MySQL error log which also where not completely resolved by the mysql_upgrade utility:
- The mysql.innodb_table_stats and mysql.innodb_index_stats tables where recreated manually (here a bug to fix for the MariaDB people?).
- All error messages from tables affected by the following message: InnoDB: in InnoDB data dictionary has unknown flags 40/50/52. could be silenced by a run of the OPTIMIZE TABLE command (which can become quite expensive for very big tables).
Sidegrades from MySQL 5.6 to Percona Server 5.6 and back did not provoke any error message written to the MySQL log files. Sidegrades from MariaDB 10.0 to MySQL 5.6 and vice versa behaved exactly the same as MariaDB 10.0 to Percona Server 5.6 and back.from/to: MySQL 5.6 MariaDB 10.0 Percona Server 5.6 MySQL 5.6 - 2 tables, OPTIMIZE OK MariaDB 10.0 binlog, 3 tables - binlog, 3 tables Percona Server 5.6 OK 2 tables, OPTIMIZE -
During our tests we got rid of the error messages. If they caused any technical harm to the tables or the data we cannot say so far. Further testing and experience from real life is needed. Any feedback is welcome!Observations
It looks like MariaDB 10.0 understands MySQL/Percona Server replication but not the other way around. So replication from MariaDB 10.0 to MySQL 5.6 does probably not work (different implementation of GTID)?Recommendation
To make sure a sigdegrade between these 3 MySQL branches/forks is seamlessly possible the best method seems to be to dump/restore (NOT xtrabackup!) the data. This can be an issue with huge databases (hundreds of Gbyte).Further aid
If you need any help to convert MySQL to MariaDB to Percona Server or the other way do not hesitate to contact the FromDual consultancy team. We will be pleased to assist you as a neutral and vendor independent consulting company.
It is not clear for many MySQL users that Percona XtraDB Cluster (PXC) and MariaDB Galera Cluster depend on the same Galera library i.e used in Galera Cluster for MySQL which is provided by Codership team:
- Galera Cluster: MySQL Server (by Oracle) + Galera library.
- Percona XtraDB Cluster: Percona Server + Galera library.
- MariaDB Galera Cluster: MariaDB Server + Galera library.
But the question is, are there any performance differences between the three of them ?
Let's discover that by doing some simple benchmark to test MySQL write performance in Galera Cluster, PXC and MariaDB Galera Cluster installations.System Information: HW configurations (AWS Servers): Nodes Servers HW configurations:
- CPU: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz (# of cores 8, # of threads 16, HT enabled).
- Memory: 16GB RAM.
- Storage: HDD 120GB/ 5400RPM.
- CPU: Intel(R) Xeon(R) CPU E5-2651 v2 @ 1.80GHz (# of cores 4, # of threads 8, HT enabled).
- Memory: 16GB RAM.
- Storage: HDD 10GB/ 5400RPM.
- CPU: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz (# of cores 16, # of threads 32, HT enabled).
- Memory: 32GB RAM.
- Storage: HDD 10GB/ 5400RPM.
- OS : Red Hat Enterprise Linux Server release 6.5 (Santiago)
- Sysbench : 0.5.3
- GLB : 1.0.0
- Galera Cluster : 5.5.34 and 5.6.16
- Percona XtraDB Cluster : 5.5.37 and 5.6.19
- MariaDB Galera Cluster : 5.5.38 and 10.0.12
- Galera Library : 3.5
- The testing environment consists of 5 AWS servers, three servers for a three-node cluster (each node is installed on a single server), one server for the load balancer and the final server for the load generator in which sysbench is installed to send requests to the load balancer from.
- Sysbench command: sysbench --num-threads=64 --max-requests=1000 --db-driver=mysql --test=/usr/share/doc/sysbench/tests/db/oltp.lua --mysql-table-engine=InnoDB --mysql-user=dev --mysql-password='test' --mysql-host=load_balancer_ip run .
- Table structure which was used by sysbench tests: mysql> show create table sbtest.sbtest\G CREATE TABLE `sbtest` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `k` int(10) unsigned NOT NULL DEFAULT '0', `c` char(120) NOT NULL DEFAULT '', `pad` char(60) NOT NULL DEFAULT '', PRIMARY KEY (`id`), KEY `k` (`k`) ) ENGINE=InnoDB AUTO_INCREMENT=8574 DEFAULT CHARSET=latin1
- The my.cnf used is something like: [mysqld] key_buffer_size = 16M max_allowed_packet = 16M thread_stack = 192K thread_cache_size = 8 innodb_buffer_pool_size = 8G innodb_flush_log_at_trx_commit = 0 expire_logs_days = 10 max_binlog_size = 100M server-id = 1 log-bin = mysql-bin binlog_format = ROW auto_increment_increment = 3 auto_increment_offset = 1 log_slave_updates default_storage_engine = InnoDB # Path to Galera library wsrep_provider = /usr/lib64/galera/libgalera_smm.so # Cluster connection URL contains the IPs of node#1, node#2 and node#3 wsrep_cluster_address = gcomm://nodeB-IP,nodeC-IP innodb_autoinc_lock_mode = 2 # Node #1 address wsrep_node_address = nodeA-IP # Cluster name wsrep_cluster_name = test_cluster # SST method wsrep_sst_method = rsync # Authentication for SST method wsrep_sst_auth = "sst:password"
- The number of threads used in this test is 64 as it generated the highest throughput on all cluster installations.
- Each throughput value for each test case is generated by the average of ten (10) times execution.
The raw results in Transactions / Sec might be useful:
sync_binlog=0innodb_flush_log_ at_trx_commitGalera Cluster 5.5.34PXC 5.5.37MariaDB Galera Cluster 5.5.38Galera Cluster 5.6.16PXC 5.6.15MariaDB Galera Cluster 10.0.120525.119534.022534.249519.575532.19520.7361125.615131.748341.384157.001162.783174.972526.761528.858524.039511.817526.06521.024sync_binlog=10242.201249.622262.516220.313229.807220.97196.82996.759148.815111.995114.8113.0562224.476210.904217.142209.139201.596214.311
According to the above results:
- innodb_flush_log_at_trx_commit = 1 significantly slows down Galera.
- sync_binlog also cuts in half the throughput.
- All other are more or less equal in throughput.