In this blog, we will test a new security feature added in ProxySQL 2.0.9. Since a time ago, we have had the ability to block queries using mysql_query_rules table matching a group of queries using reg exp like a blacklist. Check out a previous blog for how to config “ProxySQL Firewalling” using the mysql_query_rules table.
You can improve a whitelist using the mysql_query_rules table, but it is difficult if you have hundreds of queries.
ProxySQL 2.0.9 introduces two new tables for the firewall whitelist algorithm:
Admin> SELECT name AS tables FROM main.sqlite_master WHERE type='table' AND name IN ('mysql_firewall_whitelist_rules','mysql_firewall_whitelist_users') ORDER BY name;
+--------------------------------+
| tables |
+--------------------------------+
| mysql_firewall_whitelist_rules |
| mysql_firewall_whitelist_users |
+--------------------------------+
2 rows in set (0.00 sec)
How Does It Work?
If you have the param mysql-query_digests enabled, you have all your queries logged in the stats.stats_mysql_query_digest table. The idea is to collect as much traffic as possible to identify normal traffic.
Example
Let’s gets started, so I’ll show you how to configure it using a little example to simulate your traffic:
set mysql-query_digests = 1; set admin-stats_mysql_query_digest_to_disk = 1; LOAD MYSQL VARIABLES TO RUNTIME; SAVE MYSQL VARIABLES TO DISK;
Now we will prepare some databases and tables, and we are going to insert some data to log that traffic into the stats.stats_mysql_query_digest table. Remember, we need to connect through ProxySQL.
mysql -u user1 -p123456 -h 127.0.0.1 -P 6033 create database db_firewall_test1; create database db_firewall_test2; use db_firewall_test1 create table table1 (id int auto_increment, col1 varchar(100), col2 varchar(100), primary key(id)); insert into table1 values (1, 'aa', 'bb'); insert into table1 values (2, 'cc', 'dd'); select * from table1; use db_firewall_test2 create table table2 (id int auto_increment, col1 varchar(100), col2 varchar(100), primary key(id)); insert into table2 values (1, 'ee', 'ff'); insert into table2 values (2, 'gg', 'hh'); select * from table2;
After that, I recommend checking if those queries are in the stats_history.history_mysql_query_digest table and you should see something like this for the MySQL user “user1”:
admin ((none))>select distinct(digest), username, digest_text from stats_history.history_mysql_query_digest WHERE username = 'user1' ORDER BY dump_time; +--------------------+----------+------------------------------------------------------------------------------------------------+ | digest | username | digest_text | +--------------------+----------+------------------------------------------------------------------------------------------------+ | 0x02033E45904D3DF0 | user1 | show databases | | 0x0C8B490E5847BD8A | user1 | create database db_firewall_test2 | | 0xEC489734142A4C69 | user1 | create database db_firewall_test1 | | 0xBA3A41A6FF5DFE4E | user1 | create table table1 (id int auto_increment, col1 varchar(?), col2 varchar(?), primary key(id)) | | 0x99531AEFF718C501 | user1 | show tables | | 0x236BBB7F360888D6 | user1 | insert into table1 values (?, ?, ?) | | 0xA6D96F2525BD0179 | user1 | select * from table1 | | 0x620B328FE9D6D71A | user1 | SELECT DATABASE() | | 0x907932FE528CCC74 | user1 | select * from table2 | | 0xA61313D2974A4080 | user1 | insert into table2 values (?, ?, ?) | | 0xA74FAF80A4A93A29 | user1 | create table table2 (id int auto_increment, col1 varchar(?), col2 varchar(?), primary key(id)) | | 0x92A70B0CCDCE0ACB | user1 | SELECT * FROM table2 WHERE ?=? | | 0x0D15E3987E9E24B1 | user1 | select col1 from table2 | | 0xFF15DBC009550EA3 | user1 | show dataases | | 0xD82EC15EB253E71B | user1 | SELECT * FROM table1 WHERE ?=? | | 0x226CD90D52A2BA0B | user1 | select @@version_comment limit ? | | 0x9DFADD97FB0AF9C7 | user1 | show create table table1 | | 0xFDA30626AB3B972E | user1 | select col2, col1 from table1 | +--------------------+----------+------------------------------------------------------------------------------------------------+
Pay attention to the above list, to the column “digest“, as we will need it to create the whitelist.
After the collection, we can now continue and configure the firewall tables. ProxySQL introduces two new tables for firewall whitelist.
admin ((none))>SELECT name AS tables FROM main.sqlite_master WHERE type='table' AND name IN ('mysql_firewall_whitelist_rules','mysql_firewall_whitelist_users') ORDER BY name;
+--------------------------------+
| tables |
+--------------------------------+
| mysql_firewall_whitelist_rules |
| mysql_firewall_whitelist_users |
+--------------------------------+
2 rows in set (0.00 sec)
The first one “mysql_firewall_whitelist_rules” identifies a specific user to apply the firewall whitelist algorithm and determines the default action for each user. There are three modes for each user (from ProxySQL documentation):
OFF : allow any query DETECTING : allow any query, but queries not explicitly enabled in table mysql_firewall_whitelist_rules generate an error entry in the error log PROTECTING : allows only queries explicitly enabled in mysql_firewall_whitelist_rules , and block any other query
We will enable the firewall on ProxySQL:
SET mysql-firewall_whitelist_enabled = 1; SET mysql-firewall_whitelist_errormsg = 'ProxySQL Firewall blocked this query'; LOAD MYSQL VARIABLES TO RUNTIME; SAVE MYSQL VARIABLES TO DISK;
Note: Take care – if you enabled the firewall, before configuring the above tables, that you will block all users and queries. I’m doing this only for test purposes.
We can continue and enable the MySQL user “user1” to accept all traffic.
INSERT INTO mysql_firewall_whitelist_users (active, username, client_address, mode, comment) VALUES (1, 'user1', '', 'OFF', ''); LOAD MYSQL FIREWALL TO RUNTIME; SAVE MYSQL FIREWALL TO DISK;
Now this “user1” can run any query, existing or not, in the table mysql_firewall_whitelist_rules table.
We will update the rule to mode = DETECTING, and in this case, “user1” can run any query, but if the query doesn’t exist in the table mysql_firewall_whitelist_rules, it will generate an error entry in the proxysql.log file.
UPDATE mysql_firewall_whitelist_users SET mode='DETECTING' WHERE username = 'user1'; LOAD MYSQL FIREWALL TO RUNTIME; SAVE MYSQL FIREWALL TO DISK;
And we can run any query, for example:
mysql -u user1 -p123456 -h 127.0.0.1 -P 6033 mysql> use db_firewall_test1 Database changed mysql> select * from table1; +----+------+------+ | id | col1 | col2 | +----+------+------+ | 1 | aa | bb | | 2 | cc | dd | +----+------+------+ 2 rows in set (0.00 sec) mysql> select col2, col1 from table1; +------+------+ | col2 | col1 | +------+------+ | bb | aa | | dd | cc | +------+------+ 2 rows in set (0.01 sec)
This is the warning we can see in the proxysql.log file. We cannot see the query statement but we can see the digest:
2020-03-28 21:26:01 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall detected unknown query with digest 0x620B328FE9D6D71A from user user1@127.0.0.1 2020-03-28 21:26:10 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall detected unknown query with digest 0xA6D96F2525BD0179 from user user1@127.0.0.1 2020-03-28 21:27:32 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall detected unknown query with digest 0xFDA30626AB3B972E from user user1@127.0.0.1
Anyway, “user1” can run any query and new queries will be logged into the stats_history.history_mysql_query_digest table.
Finally, the last option is mode = PROTECTING, and it will check if the query (in this case, it will check digest) we are running exists in mysql_firewall_whitelist_rules, and we are going to update the mode.
UPDATE mysql_firewall_whitelist_users SET mode='PROTECTING' WHERE username = 'user1'; LOAD MYSQL FIREWALL TO RUNTIME; SAVE MYSQL FIREWALL TO DISK;
And now run any query that exists or not into stats_history.history_mysql_query_digest and check the output. Let’s take a look:
mysql -u user1 -p123456 -h 127.0.0.1 -P 6033 mysql> use db_firewall_test1; Database changed mysql> select * from table1; ERROR 1148 (42000): ProxySQL Firewall blocked this query mysql> show tables; ERROR 1148 (42000): ProxySQL Firewall blocked this query
Perfect! As we can see, those queries are blocked and showing the message “ProxySQL Firewall blocked this query“.
Also, there are warnings in the proxysql.log file. The three first warnings are from this statement “use db_firewall_test1”, as the “use” statement will not be logged into the stats_history.history_mysql_query_digest table.
2020-03-28 21:37:30 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall blocked query with digest 0x620B328FE9D6D71A from user user1@127.0.0.1 2020-03-28 21:37:30 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall blocked query with digest 0x02033E45904D3DF0 from user user1@127.0.0.1 2020-03-28 21:37:30 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall blocked query with digest 0x99531AEFF718C501 from user user1@127.0.0.1 2020-03-28 21:37:44 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall blocked query with digest 0xA6D96F2525BD0179 from user user1@127.0.0.1 2020-03-28 21:37:52 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall blocked query with digest 0x99531AEFF718C501 from user user1@127.0.0.1
Now, we can choose which queries from stats_history.history_mysql_query_digest we want to permit to run from the user “user1”.
The next step is to copy one or all the digests and insert them into mysql_firewall_whitelist_rules table. For this test, I’ll just copy/insert only three digests for test purposes.
From stats_history.history_mysql_query_digest:
| 0x0D15E3987E9E24B1 | user1 | select col1 from table2 | | 0x907932FE528CCC74 | user1 | select * from table2 | | 0xA6D96F2525BD0179 | user1 | select * from table1 |
Insert into mysql_firewall_whitelist_rules:
INSERT INTO mysql_firewall_whitelist_rules VALUES (1, 'user1', '', 'db_firewall_test1', 0, '0x0D15E3987E9E24B1', ''); INSERT INTO mysql_firewall_whitelist_rules VALUES (1, 'user1', '', 'db_firewall_test1', 0, '0x907932FE528CCC74', ''); INSERT INTO mysql_firewall_whitelist_rules VALUES (1, 'user1', '', 'db_firewall_test1', 0, '0xA6D96F2525BD0179', ''); LOAD MYSQL FIREWALL TO RUNTIME; SAVE MYSQL FIREWALL TO DISK;
And test again…
mysql> use db_firewall_test1 Database changed mysql> select * from table1; +----+------+------+ | id | col1 | col2 | +----+------+------+ | 1 | aa | bb | | 2 | cc | dd | +----+------+------+ 2 rows in set (0.00 sec) mysql> use db_firewall_test2 Database changed mysql> select * from table2; +----+------+------+ | id | col1 | col2 | +----+------+------+ | 1 | ee | ff | | 2 | gg | hh | +----+------+------+ 2 rows in set (0.00 sec) mysql> select col1 from table2; +------+ | col1 | +------+ | ee | | gg | +------+ 2 rows in set (0.00 sec) mysql> select id from table2; ERROR 1148 (42000): ProxySQL Firewall blocked this query
It works! My last query was to check if it’s working, because that query was not added in the mysql_firewall_whitelist_rules table.
If you want to add all your users and queries in the whitelist, I recommend the following queries to collect all your traffic for a long time:
INSERT INTO mysql_firewall_whitelist_users (active, username, client_address, mode, comment) SELECT DISTINCT 1, username, '', 'DETECTING', '' FROM mysql_users; INSERT INTO mysql_firewall_whitelist_rules (active, username, client_address, schemaname, flagIN, digest, comment) SELECT DISTINCT 1, username, client_address, schemaname, 0, digest, '' FROM stats_history.history_mysql_query_digest; LOAD MYSQL FIREWALL TO RUNTIME; SAVE MYSQL FIREWALL TO DISK;
Conclusion
This is a solid approach to build a good whitelist, but remember, before/after any deploy from your application with new queries keep in mind you need to add those new queries into the mysql_firewall_whitelist_rules table.
I think it’s good to add permission to one user to run only one or more queries.
Last but not least, take care that the next two queries are the same for you but have a different digest. Check the following example:
mysql> select * from table2; --# 0x907932FE528CCC74 +----+------+------+ | id | col1 | col2 | +----+------+------+ | 1 | ee | ff | | 2 | gg | hh | +----+------+------+ 2 rows in set (0.00 sec) mysql> select * from db_firewall_test2.table2; --# 0x2EDC8B093C50BE48 ERROR 1148 (42000): ProxySQL Firewall blocked this query
I hope this is helpful to you. Thanks so much.
Interested in learning more?
Be sure to get in touch with Percona’s Training Department to schedule a hands-on tutorial session for all things ProxySQL. Our instructors will guide you and your team through the install and setup processes, learn how to manage read-write splitting, chain rules together, implement query firewalling, query caching, high-availability and much more.