Congratulations!

[Valid RSS] This is a valid RSS feed.

Recommendations

This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

Source: http://www.tusacentral.net/joomla/index.php/mysql-blogs?format=feed&type=rss

  1. <?xml version="1.0" encoding="utf-8"?>
  2. <!-- generator="Joomla! - Open Source Content Management" -->
  3. <?xml-stylesheet href="/joomla/plugins/system/jce/css/content.css?badb4208be409b1335b815dde676300e" type="text/css"?>
  4. <rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  5. <channel>
  6. <title>TusaCentral - MySQL Blogs</title>
  7. <description><![CDATA[]]></description>
  8. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs</link>
  9. <lastBuildDate>Wed, 28 Feb 2024 15:23:45 +0000</lastBuildDate>
  10. <generator>Joomla! - Open Source Content Management</generator>
  11. <atom:link rel="self" type="application/rss+xml" href="http://www.tusacentral.net/joomla/index.php/mysql-blogs?format=feed&amp;type=rss"/>
  12. <language>en-gb</language>
  13. <item>
  14. <title>Is MySQL Router 8.2 Any Better?</title>
  15. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/255-is-mysql-router-8-2-any-better</link>
  16. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/255-is-mysql-router-8-2-any-better</guid>
  17. <description><![CDATA[<p>In my previous article, <a href="https://www.percona.com/blog/comparisons-of-proxies-for-mysql/">Comparisons of Proxies for MySQL</a>, I showed how MySQL Router was the lesser performing Proxy in the comparison. From that time to now, we had several MySQL releases and, of course, also some new MySQL Router ones.</p>
  18. <p>Most importantly, we also had MySQL Router going back to being a level 7 proxy capable of redirecting traffic in case of R/W operations (<a href="https://dev.mysql.com/doc/mysql-router/8.2/en/router-read-write-splitting.html" rel="nofollow">see this</a>).</p>
  19. <p>All these bring me hope that we will also have some good improvements in what is a basic functionality in a router: routing.</p>
  20. <p>So with these great expectations, I had to repeat the exact same tests I did in my previous tests, plus I tested for MySQL Router only the cost of encapsulating the select inside a transaction.</p>
  21. <p>Just keep in mind that for all the tests, MySQL Router was configured to use the read/write split option.</p>
  22. <h2>The results</h2>
  23. <p><span style="font-weight: 400;">Given this is the continuation of the previous blog, all the explanations about the tests and commands used are in the <a href="https://www.percona.com/blog/comparisons-of-proxies-for-mysql/">first article</a>. If you did not read that, do it now, or it will be difficult for you to follow what is explained later.</span></p>
  24. <p><span style="font-weight: 400;">As indicated, I was looking to identify when the first proxy would reach a dimension that would not be manageable. The load is all in creating and serving the connections, while the number of operations is capped at 100.</span></p>
  25. <p><a href="https://www.percona.com/blog/wp-content/uploads/2024/01/events_rate_82.png"><img src="https://www.percona.com/blog/wp-content/uploads/2024/01/events_rate_82-1024x608.png" alt="" width="600" height="356" class="aligncenter wp-image-93339 size-large" /></a></p>
  26. <p><span style="font-weight: 400;">As you can see, MySQL Router was reaching the saturation level and was unable to serve traffic at exactly the same time as the previous test.</span></p>
  27. <h2>Test two</h2>
  28. <p><i><span style="font-weight: 400;">When the going gets tough, the tough get going </span></i><span style="font-weight: 400;">reprise ;)&nbsp;</span></p>
  29. <p><span style="font-weight: 400;">Let’s remove the –rate limitation and see what will happen. </span><span style="font-weight: 400;">First, let us compare MySQL router versions only:</span></p>
  30. <p><a href="https://www.percona.com/blog/wp-content/uploads/2024/01/router_82_80_comparison_events.png"><img src="http://www.tusacentral.net/joomla/images/stories/ismysqlrouter82better/router_82_80_comparison_events.png" alt="router 82 80 comparison events" width="600" height="376" class="size-large wp-image-93348 aligncenter" /></a></p>
  31. <p><span style="font-weight: 400;">As we can see the MySQL Router version 8.2 is doing better up to 64 concurrent threads.</span></p>
  32. <p><a href="https://www.percona.com/blog/wp-content/uploads/2024/01/router_82_80_comparison_latency.png"><img src="https://www.percona.com/blog/wp-content/uploads/2024/01/router_82_80_comparison_latency-1024x680.png" alt="" width="600" height="398" class="size-large wp-image-93349 aligncenter" /></a></p>
  33. <p><span style="font-weight: 400;">Latency follows the same trend in old and new cases, and we can see that the new version is acting better, up to 1024 threads.</span></p>
  34. <p><span style="font-weight: 400;">Is this enough to cover the gap with the other proxies? Which, in the end, is what we would like to see.&nbsp;</span></p>
  35. <p><a href="https://www.percona.com/blog/wp-content/uploads/2024/01/events_norate_82.png"><img src="http://www.tusacentral.net/joomla/images/stories/ismysqlrouter82better/events_norate_82.png" alt="events norate 82" width="600" height="326" class="size-large wp-image-93337 aligncenter" /></a></p>
  36. <p><a href="https://www.percona.com/blog/wp-content/uploads/2024/01/latency_norate_82.png"><img src="http://www.tusacentral.net/joomla/images/stories/ismysqlrouter82better/latency_norate_82.png" alt="latency norate 82" width="600" height="364" class="size-large wp-image-93342 aligncenter" /></a></p>
  37. <p><span style="font-weight: 400;">Well, I would say not really; we see a bit of better performance with low concurrent threads, but still not scaling and definitely lower than the other two.</span></p>
  38. <p><span style="font-weight: 400;">Now let us take a look at the CPU saturation:</span></p>
  39. <p><a href="https://www.percona.com/blog/wp-content/uploads/2024/01/cpu_saturation.png"><img src="http://www.tusacentral.net/joomla/images/stories/ismysqlrouter82better/cpu_saturation.png" alt="cpu saturation" width="600" height="300" class="size-full wp-image-93334 aligncenter" /></a></p>
  40. <p><a href="https://www.percona.com/blog/wp-content/uploads/2024/01/cpu.png"><img src="http://www.tusacentral.net/joomla/images/stories/ismysqlrouter82better/cpu.png" alt="cpu" width="600" height="300" class="size-full wp-image-93335 aligncenter" /></a></p>
  41. <p>Here, we can see how MySQL Router hits the top as soon as the rate option is lifted and gets worse with the increase of the running threads.</p>
  42. <h2>Test three</h2>
  43. <p><span style="font-weight: 400;">This simple test was meant to identify the cost of a transaction, or better, what it will cost to include selects inside a transaction.</span></p>
  44. <p><a href="https://www.percona.com/blog/wp-content/uploads/2024/01/read_events_trx.png"><img src="http://www.tusacentral.net/joomla/images/stories/ismysqlrouter82better/read_events_trx.png" alt="read events trx" width="600" height="356" class="size-large wp-image-93344 aligncenter" /></a></p>
  45. <p><a href="https://www.percona.com/blog/wp-content/uploads/2024/01/latency_events_trx.png"><img src="http://www.tusacentral.net/joomla/images/stories/ismysqlrouter82better/latency_events_trx.png" alt="latency events trx" width="600" height="371" class="size-large wp-image-93340 aligncenter" /></a></p>
  46. <p><span style="font-weight: 400;">As we can clearly see, MySQL Router, when handling selects inside a transaction, will drop its performance drastically going back to version 8.0 performance.</span></p>
  47. <h3>Conclusions</h3>
  48. <p>To the initial question — Is MySQL Router 8.2 any better? — we can answer a small (very small) yes.</p>
  49. <p>However, it is still far, far away from being competitive with ProxySQL (same proxy level) or with HAProxy. The fact it is not able to serve efficiently the requests also inside the lower set of concurrent threads, is disappointing.</p>
  50. <p>Even more disappointing because MySQL Router is presented as a critical component in the MySQL InnoDB Cluster solution. How can we use it in the architectures if the product has such limitations?</p>
  51. <p>I know that Oracle suggests to scale out, and I agree with them. When in need to scale with MySQL Router, the only option is to build a forest. However, we must keep in mind that each MySQL Router connects and queries the data nodes constantly and intensively. Given that it requires adding a forest of router nodes to scale, it is not without performance impact, given the increasing noise generated on the data nodes.</p>
  52. <p>Anyhow also, if there is a theoretical option to scale, that is not a good reason to use a poor performing component.</p>
  53. <p>I would prefer to use ProxySQL with Group Replication and add whatever script is needed in mysqlshell to manage it as Oracle is doing for the MySQL InnoDB cluster solution.</p>
  54. <p>What also left me very unhappy is that MySQL InnoDB Cluster is one of the important components of the OCI offer for MySQL. Is Oracle using MySQL Router there as well? I assume so. Can we trust it? I am not feeling like I can.</p>
  55. <p>Finally, what has been done for MySQL Router so far leads me to think that there is no real interest in making it the more robust and performing product that MySQL InnoDB Cluster deserves.</p>
  56. <p>I hope I am wrong and that we will soon see a fully refactored version of MySQL Router. I really hope Oracle will prove me wrong.</p>
  57. <p>Great MySQL to everyone.</p>]]></description>
  58. <category>MySQL</category>
  59. <pubDate>Wed, 10 Jan 2024 18:05:47 +0000</pubDate>
  60. </item>
  61. <item>
  62. <title>Export and import of MySQL passwords using caching_sha2 </title>
  63. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/254-export-and-import-of-mysql-passwords-using-caching-sha2</link>
  64. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/254-export-and-import-of-mysql-passwords-using-caching-sha2</guid>
  65. <description><![CDATA[<p><em><span style="font-weight: 400;">Some fun is coming <a href="https://www.percona.com/blog/wp-content/uploads/2023/09/user-migration.jpg"><img src="https://www.percona.com/blog/wp-content/uploads/2023/09/user-migration-300x136.jpg" alt="" width="380" height="172" class="wp-image-91109 alignright" style="float: right;" /></a></span></em></p>
  66. <p><span style="font-weight: 400;">While I was writing the internal guidelines on how to migrate from MariaDB to Percona Server, I had to export the users accounts in a portable way. This given MariaDB uses&nbsp; some non standard syntax brings me to first test some external tools such as Fred </span><a href="https://github.com/lefred/mysqlshell-plugins/wiki/user#getusersgrants"><span style="font-weight: 400;">https://github.com/lefred/mysqlshell-plugins/wiki/user#getusersgrants</span></a><span style="font-weight: 400;"> and our <a href="https://github.com/percona/percona-toolkit/blob/3.x/bin/pt-show-grants" rel="nofollow">PT-SHOW-GRANTS</a> tool.&nbsp;</span></p>
  67. <p><span style="font-weight: 400;">Useless to say this had open a can worms, given first I had to fix/convert the specifics for MariaDB (not in the scope of this blog), then while testing I discover another nasty issue, that currently prevent us to easily export the new Passwords in MySQL 8 (and PS 8) when caching_sha2 is used.&nbsp;</span></p>
  68. <p>&nbsp;</p>
  69. <p><span style="font-weight: 400;">So what is the problem I am referring to?</span></p>
  70. <p><span style="font-weight: 400;">Well the point is that when you generate passwords with caching_sha2 (default in mysql 8) the password generated can (will) contain characters that are not portable, not even between mysql 8.&nbsp;</span></p>
  71. <p><span style="font-weight: 400;">Let's see a practical example to understand.</span></p>
  72. <p><span style="font-weight: 400;">If I use old </span><i><span style="font-weight: 400;">mysql_native_password</span></i><span style="font-weight: 400;"> and I create a user such as:</span><span style="font-weight: 400;"><br /></span></p>
  73. <pre class="lang:mysql decode:true">create user dba@'192.168.1.%' identified with mysql_native_password by 'dba';
  74. </pre>
  75. <p><span style="font-weight: 400;">My authentication_string will be:&nbsp;</span></p>
  76. <pre class="lang:mysql decode:true">root@localhost) [(none)]&gt;select user,host,authentication_string,plugin from mysql.user where user ='dba' order by 1,2;
  77. +------+-------------+-------------------------------------------+-----------------------+
  78. | user | host        | authentication_string                     | plugin                |
  79. +------+-------------+-------------------------------------------+-----------------------+
  80. | dba  | 192.168.1.% | *381AD08BBFA647B14C82AC1094A29AD4D7E4F51D | mysql_native_password |
  81. +------+-------------+-------------------------------------------+-----------------------+
  82. </pre>
  83. <p><span style="font-weight: 400;">At this point if you want to export the user:</span></p>
  84. <pre class="lang:mysql decode:true">(root@localhost) [(none)]&gt;show create user dba@'192.168.1.%'G
  85. *************************** 1. row ***************************
  86. CREATE USER for dba@192.168.1.%: CREATE USER `dba`@`192.168.1.%` IDENTIFIED WITH 'mysql_native_password' AS '*381AD08BBFA647B14C82AC1094A29AD4D7E4F51D' REQUIRE NONE PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK PASSWORD HISTORY DEFAULT PASSWORD REUSE INTERVAL DEFAULT PASSWORD REQUIRE CURRENT DEFAULT
  87. 1 row in set (0.01 sec)
  88. </pre>
  89. <p><span style="font-weight: 400;">You just need to use the text after the semicolon and all will work fine. Remember that when you want to preserve the already converted password you need to use the </span><i><span style="font-weight: 400;">IDENTIFIED … AS &lt;PW&gt;</span></i><span style="font-weight: 400;"> not the </span><b>BY</b><span style="font-weight: 400;"> or you will re-convert the password ;).</span></p>
  90. <p><span style="font-weight: 400;">&nbsp;Anyhow .. this is simple and what we are all used to.&nbsp;</span></p>
  91. <p><span style="font-weight: 400;">Now if you instead try to use caching_sha2 things will go differently:</span></p>
  92. <pre class="lang:mysql decode:true">root@localhost) [(none)]&gt;create user dba@'192.168.4.%' identified with caching_sha2_password by 'dba';
  93. Query OK, 0 rows affected (0.02 sec)
  94.  
  95. (root@localhost) [(none)]&gt;select user,host,authentication_string,plugin from mysql.user where user ='dba' order by 1,2;
  96. +------+-------------+------------------------------------------------------------------------+-----------------------+
  97. | user | host        | authentication_string                                                  | plugin                |
  98. +------+-------------+------------------------------------------------------------------------+-----------------------+
  99. | dba  | 192.168.1.% | *381AD08BBFA647B14C82AC1094A29AD4D7E4F51D                              | mysql_native_password |
  100. | dba  | 192.168.4.% | $A$005$@&amp;%1H5iNQx|.l{N7T/GosA.Lp4EiO0bxLVQp8Zi0WY2nXLr8TkleQPYjaqVxI7 | caching_sha2_password |
  101. +------+-------------+------------------------------------------------------------------------+-----------------------+
  102. 2 rows in set (0.00 sec)
  103. </pre>
  104. <p><span style="font-weight: 400;">Probably you will not see it here given that while converted on your screen the special characters will be replaced, but the password contains invalid characters.&nbsp;</span></p>
  105. <p><span style="font-weight: 400;">If I try to extract the </span><i><span style="font-weight: 400;">Create USER</span></i><span style="font-weight: 400;"> text I will get:</span></p>
  106. <pre class="lang:mysql decode:true"> (root@localhost) [(none)]&gt;show create user dba@'192.168.4.%'G
  107. *************************** 1. row ***************************
  108. CREATE USER for dba@192.168.4.%: CREATE USER `dba`@`192.168.4.%` IDENTIFIED WITH 'caching_sha2_password' AS '$A$005$@&amp;%1H5iNQx|.l{N7T/GosA.Lp4EiO0bxLVQp8Zi0WY2nXLr8TkleQPYjaqVxI7' REQUIRE NONE PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK PASSWORD HISTORY DEFAULT PASSWORD REUSE INTERVAL DEFAULT PASSWORD REQUIRE CURRENT DEFAULT
  109. 1 row in set (0.00 sec)</pre>
  110. <p><span style="font-weight: 400;">However if I try to use this text to generate the user after I drop it:</span><span style="font-weight: 400;"><br /></span></p>
  111. <pre class="lang:mysql decode:true">(root@localhost) [(none)]&gt;drop user dba@'192.168.4.%';
  112. Query OK, 0 rows affected (0.02 sec)
  113. (root@localhost) [(none)]&gt;create user dba@'192.168.4.%' IDENTIFIED AS 'NQx|.l{N7T/GosA.Lp4EiO0bxLVQp8Zi0WY2nXLr8TkleQPYjaqVxI7';
  114. ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AS 'NQx|.l{N7T/GosA.Lp4EiO0bxLVQp8Zi0WY2nXLr8TkleQPYjaqVxI7'' at line 1
  115. </pre>
  116. <p><span style="font-weight: 400;">Don’t waste time, there is nothing wrong in the query, except the simple fact that you CANNOT use the text coming from the authentication_string when you have </span><i><span style="font-weight: 400;">caching_sha2</span></i><span style="font-weight: 400;">.&nbsp;</span></p>
  117. <p><span style="font-weight: 400;">So? What should we do?&nbsp;</span></p>
  118. <p><span style="font-weight: 400;">The answer is easy, we need to convert the password into binary and use/store that.&nbsp;</span></p>
  119. <p><span style="font-weight: 400;">Let us try.</span></p>
  120. <p><span style="font-weight: 400;">First create the user again:</span></p>
  121. <pre class="lang:mysql decode:true">(root@localhost) [(none)]&gt;select user,host,authentication_string,plugin from mysql.user where user ='dba' order by 1,2;
  122. +------+-------------+------------------------------------------------------------------------+-----------------------+
  123. | user | host        | authentication_string                                                  | plugin                |
  124. +------+-------------+------------------------------------------------------------------------+-----------------------+
  125. | dba  | 192.168.1.% | *381AD08BBFA647B14C82AC1094A29AD4D7E4F51D                              | mysql_native_password |
  126. | dba  | 192.168.4.% | $A$005$X&gt;ztS}WfR"k~aH3Hs0hBbF3WmM2FXubKumr/CId182pl2Lj/gEtxLvV0 | caching_sha2_password |
  127. +------+-------------+------------------------------------------------------------------------+-----------------------+
  128. 2 rows in set (0.00 sec)
  129.  
  130. (root@localhost) [(none)]&gt;exit
  131. Bye
  132. [root@master3 ps80]# ./mysql-3307 -udba -pdba -h192.168.4.57 -P3307
  133. ...
  134. (dba@192.168.4.57) [(none)]&gt;
  135. </pre>
  136. <p><span style="font-weight: 400;">OK as you can see I create the user and can connect, but as we know the PW is not portable.</span></p>
  137. <p><span style="font-weight: 400;">Let us convert it and create the user:</span></p>
  138. <pre class="lang:mysql decode:true">(root@localhost) [(none)]&gt;select user,host,convert(authentication_string using binary),plugin from mysql.user where user ='dba' and host='192.168.4.%' order by 1,2;
  139. +------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
  140. | user | host        | convert(authentication_string using binary)                                                                                                    | plugin                |
  141. +------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
  142. | dba  | 192.168.4.% | 0x2441243030352458193E107A74537D0157055C66527F226B7E5C6148334873306842624633576D4D32465875624B756D722F434964313832706C324C6A2F674574784C765630 | caching_sha2_password |
  143. +------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
  144. </pre>
  145. <p><span style="font-weight: 400;">So the password is: </span></p>
  146. <pre class="lang:mysql decode:true">0x2441243030352458193E107A74537D0157055C66527F226B7E5C6148334873306842624633576D4D32465875624B756D722F434964313832706C324C6A2F674574784C7656
  147. </pre>
  148. <p>Let us use it:</p>
  149. <pre class="lang:mysql decode:true">(root@localhost) [(none)]&gt;drop user dba@'192.168.4.%';
  150. Query OK, 0 rows affected (0.02 sec)
  151.  
  152. (root@localhost) [(none)]&gt;create user dba@'192.168.4.%' IDENTIFIED with 'caching_sha2_password' AS 0x2441243030352458193E107A74537D0157055C66527F226B7E5C6148334873306842624633576D4D32465875624B756D722F434964313832706C324C6A2F674574784C765630;
  153. Query OK, 0 rows affected (0.03 sec)
  154. </pre>
  155. <p><span style="font-weight: 400;">Let us check the user now:</span></p>
  156. <pre class="lang:mysql decode:true">(root@localhost) [(none)]&gt;select user,host, authentication_string,plugin from mysql.user where user ='dba' and host= '192.168.4.%' order by 1,2;
  157. +------+-------------+------------------------------------------------------------------------+-----------------------+
  158. | user | host        | authentication_string                                                  | plugin                |
  159. +------+-------------+------------------------------------------------------------------------+-----------------------+
  160. | dba  | 192.168.4.% | $A$005$X&gt;ztS}WfR"k~aH3Hs0hBbF3WmM2FXubKumr/CId182pl2Lj/gEtxLvV0 | caching_sha2_password |
  161. +------+-------------+------------------------------------------------------------------------+-----------------------+
  162. 1 row in set (0.00 sec)
  163.  
  164. [root@master3 ps80]# ./mysql-3307 -udba -pdba -h192.168.4.57 -P3307
  165.  
  166. (dba@192.168.4.57) [(none)]&gt;select current_user();
  167. +-----------------+
  168. | current_user()  |
  169. +-----------------+
  170. | dba@192.168.4.% |
  171. +-----------------+
  172. 1 row in set (0.00 sec)
  173.  
  174. </pre>
  175. <p><span style="font-weight: 400;">As you can see the user has been created correctly and password is again in encrypted format.&nbsp;</span></p>
  176. <p><span style="font-weight: 400;">In short what you need to do when in need to export users from MySQL/PS 8 is:</span></p>
  177. <ol>
  178. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Read the user information</span></li>
  179. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Convert Password to hex format when plugin is caching_sha2</span></li>
  180. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Push the AS &lt;password&gt; converted to a file or any other way you were used to</span></li>
  181. </ol>
  182. <p>&nbsp;</p>
  183. <p>Another possible solution is to use at session level the parameter <em>print_identified_with_as_hex.</em> If set causes SHOW CREATE USER to display such hash values as hexadecimal strings rather than as regular string literals. Hash values that do not contain unprintable characters still display as regular string literals, even with this variable enabled.</p>
  184. <p><span style="font-weight: 400;">This at the end is exactly what Fred and I have done for our tools:</span></p>
  185. <p><span style="font-weight: 400;">See:</span></p>
  186. <ul>
  187. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Fred: </span><a href="https://github.com/lefred/mysqlshell-plugins/commit/aa5c6bbe9b9aa689bf7266f5a19a35d0091f6568"><span style="font-weight: 400;">https://github.com/lefred/mysqlshell-plugins/commit/aa5c6bbe9b9aa689bf7266f5a19a35d0091f6568</span></a></li>
  188. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Pt-show-grants: <a href="https://github.com/percona/percona-toolkit/blob/4a812d4a79c0973bf176105b0d138ad0a2a46b2f/bin/pt-show-grants#L2058" rel="nofollow">https://github.com/percona/percona-toolkit/blob/4a812d4a79c0973bf176105b0d138ad0a2a46b2f/bin/pt-show-grants#L2058</a></span></li>
  189. </ul>
  190. <h1><span style="font-weight: 400;">Conclusions</span></h1>
  191. <p><span style="font-weight: 400;">MySQL 8 and Percona server comes with a more secure hashing mechanism <i>caching_sha2_password</i> which is also the default. However if you have the need to migrate users and you use your own tools to export and import the passwords, you must update them as indicated. Or use the Percona Toolkit tools that we keep up to date for you.</span></p>
  192. <p>&nbsp;</p>
  193. <p><span style="font-weight: 400;">Have fun with MySQL!!</span></p>]]></description>
  194. <category>MySQL</category>
  195. <pubDate>Sun, 24 Sep 2023 15:39:08 +0000</pubDate>
  196. </item>
  197. <item>
  198. <title>Proof of Concept: Horizontal Write Scaling for MySQL with Kubernetes Operator</title>
  199. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/253-horizontal-write-scaling-for-mysql-with-operator-poc</link>
  200. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/253-horizontal-write-scaling-for-mysql-with-operator-poc</guid>
  201. <description><![CDATA[<p><span style="font-weight: 400;">Historically MySQL is great in horizontal READ scale. The scaling in that case is offered by the different number of Replica nodes, no matter if using standard asynchronous replication or synchronous replication.&nbsp;</span></p>
  202. <p><span style="font-weight: 400;">However those solutions do not offer the same level of scaling for writes operation.&nbsp;</span></p>
  203. <p><span style="font-weight: 400;">Why? Because the solutions still rely on writing in one single node that works as Primary. Also in case of multi-Primary the writes will be distributed by transaction. In both cases, when using virtually-synchronous replication, the process will require certification from each node and local (by node) write, as such the number of writes are NOT distributed across multiple nodes but duplicated.&nbsp;</span></p>
  204. <p><span style="font-weight: 400;">The main reason behind this is that MySQL is a relational database system (RDBMS), and any data that is going to be written in it, must respect the RDBMS rules (</span><a href="https://en.wikipedia.org/wiki/Relational_database"><span style="font-weight: 400;">https://en.wikipedia.org/wiki/Relational_database</span></a><span style="font-weight: 400;">). In short any data that is written must be consistent with the data present. To achieve that the data needs to be checked with the existing through defined relations and constraints. This action is something that can affect very large datasets and be very expensive. Think about updating a table with millions of rows that refer to another table with another million rows.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span></p>
  205. <p><span style="font-weight: 400;">An image may help:</span></p>
  206. <p><a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/schema-db.gif"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/schema-db.gif" alt="schema db" width="478" height="600" class="alignnone size-full wp-image-87678" style="display: block; margin-left: auto; margin-right: auto;" /></a></p>
  207. <p><span style="font-weight: 400;">Every time I will insert an order, I must be sure that all the related elements are in place and consistent.&nbsp;</span></p>
  208. <p><span style="font-weight: 400;">This operation is quite expensive but our database is able to run it in a few milliseconds or less, thanks to several optimizations that allow the node to execute most of them in memory, with none or little access to mass storage.&nbsp;</span></p>
  209. <p><span style="font-weight: 400;">The key factor is that the whole data structure resides in the same location (node) facilitating the operations.</span></p>
  210. <p><span style="font-weight: 400;">Once we have understood that, it will also become clear why we cannot have relational data split in multiple nodes and have to distribute writes by table. If I have a node that manages only the items, another the orders, another the payments, I will need to have my solution able to deal with distributed transactions, each of which needs to certify and verify other nodes data.&nbsp;</span></p>
  211. <p><span style="font-weight: 400;">This level of distribution will seriously affect the efficiency of the operation which will increase the response time significantly. This is it, nothing is impossible however the performances will be so impacted that each operation may take seconds instead milliseconds or fraction of it, unless lifting some of the rules breaking the relational model.</span></p>
  212. <p><span style="font-weight: 400;">MySQL as well as other RDBMS are designed to work respecting the model and cannot scale in any way by fragmenting and distributing a schema, so what can be done to scale?</span></p>
  213. <p>&nbsp;</p>
  214. <p><span style="font-weight: 400;">The alternative is to split a consistent set of data into fragments. What is a consistent set of data? It all depends on the kind of information we are dealing with. Keeping in mind the example above where we have a shop online serving multiple customers, we need to identify which is the most effective way to split the data.</span></p>
  215. <p><span style="font-weight: 400;">For instance if we try to split the data by Product Type (Books, CD/DVD, etc) we will have a huge duplication of data related to customers/orders/shipments and so on, all this data is also quite dynamic given I will have customers constantly ordering things.&nbsp;</span></p>
  216. <p>&nbsp;</p>
  217. <p><span style="font-weight: 400;">Why duplicate the data? Because if I do not duplicate that data I will not know if a customer had already bought or not that specific item, or I will have to ask again about the shipment address and so on. Which means also that any time a customer buys something or puts something in the wish list I have to reconcile the data in all my nodes/clusters.</span></p>
  218. <p>&nbsp;</p>
  219. <p><span style="font-weight: 400;">On the other hand if I choose to split my data by country of customer’s residence the only data I will have to duplicate and keep in sync is the one related to the products, of which the most dynamic one will be the number of items in stock. This of course unless I can organize my products by country as well, which is a bit unusual nowadays, but not impossible.&nbsp;</span></p>
  220. <p>&nbsp;</p>
  221. <p><span style="font-weight: 400;">Another possible case is if I am a health organization and I manage several hospitals. As for the example above, it will be easier to split my data by hospital given most of the data related to patients is bound to the hospital itself as well as treatments and any other element related to hospital management. While it will make no sense to split by patient's country of residence.</span></p>
  222. <p>&nbsp;</p>
  223. <p><span style="font-weight: 400;">This technique of splitting the data into smaller pieces is called </span><b>sharding </b><span style="font-weight: 400;">and at the moment is the only way we have to scale RDBM horizontally.&nbsp;</span></p>
  224. <p>&nbsp;</p>
  225. <p><span style="font-weight: 400;">In the MySQL open source ecosystem we have only two consolidated ways to perform sharding, Vitess and ProxySQL. The first one is a complete solution that takes ownership of your database and manages almost any aspect of its operations in a sharded environment, this includes a lot of specific features for DBA to deal with daily operations like table modifications, backup and more.&nbsp;</span></p>
  226. <p>&nbsp;</p>
  227. <p><span style="font-weight: 400;">While this may look great it also comes with some string attached, including the complexity and proprietary environment. That makes Vitess a good fit for “complex” sharding scenarios where other solutions may not be enough.</span></p>
  228. <p>&nbsp;</p>
  229. <p><span style="font-weight: 400;">ProxySQL does not have a sharding mechanism “per se” but given the way it works and the features it has, allow us to build simple sharding solutions.&nbsp;</span></p>
  230. <p><span style="font-weight: 400;">It is important to note that most of the DBA operations will still be on DBA to be executed with incremented complexity given the sharding environment.&nbsp;</span></p>
  231. <p>&nbsp;</p>
  232. <p><span style="font-weight: 400;">There is a third option which is application aware sharding.&nbsp;</span></p>
  233. <p><span style="font-weight: 400;">This solution sees the application be aware of the need to split the data in smaller fragments and internally point the data to different “connectors” who are connected to multiple data sources.&nbsp;</span></p>
  234. <p><span style="font-weight: 400;">In this case the application is aware of a customer country and will redirect all the operations related to him to the datasource responsible for the specific fragment.</span></p>
  235. <p><span style="font-weight: 400;">Normally this solution requires a full code re-design and could be quite difficult to achieve when it is injected after the initial code architecture definition.&nbsp;&nbsp;</span></p>
  236. <p><span style="font-weight: 400;">On the other hand, if done at design it is probably the best solution, because it will allow the application to define the sharding rules and can also optimize the different data sources using different technologies for different uses.</span></p>
  237. <p><span style="font-weight: 400;">One example could be the use of a RDBMS for most of the </span><b>Online transaction processing (</b><span style="font-weight: 400;">OLTP) data shared by country, and having the products as distributed memory cache with a different technology. At the same time all the data related to orders, payments and customer history can be consolidated in a data warehouse used to generate reporting.&nbsp;&nbsp;&nbsp;&nbsp;</span></p>
  238. <p>&nbsp;</p>
  239. <p><span style="font-weight: 400;">As said the last one is probably the most powerful, scalable and difficult to design and unfortunately it represents probably less than the 5% of the solution currently deployed.&nbsp;</span></p>
  240. <p><span style="font-weight: 400;">As well as very few cases are in the need to have a full system/solution to provide scalability with sharding.&nbsp;</span></p>
  241. <p>&nbsp;</p>
  242. <p><span style="font-weight: 400;">By experience, most of the needs for horizontal scaling fell in the simple scenario, where there is the need to achieve sharding and data separation, very often with sharding-nothing architecture. In shared-nothing, each shard can live in a totally separate logical schema instance / physical database server / data center / continent. There is no ongoing need to retain shared access (from between shards) to the other unpartitioned tables in other shards.</span></p>
  243. <p>&nbsp;</p>
  244. <h2><span style="font-weight: 400;">The POC</span></h2>
  245. <h4><span style="font-weight: 400;">Why this POC?</span></h4>
  246. <p><span style="font-weight: 400;">In the years I have faced a lot of customers that were talking about scaling their database solution and looking at very complex sharding as Vitess as the first and only way to go.&nbsp;</span></p>
  247. <p><span style="font-weight: 400;">This without even considering if their needs were driving them there for real.&nbsp;</span></p>
  248. <p><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/kiss.png" alt="kiss" width="216" height="188" class="wp-image-87680 alignleft" style="margin-right: 10px; float: left;" /></p>
  249. <p><span style="font-weight: 400;">In my experience, and talking with several colleagues, I am not alone, when analyzing the real needs and after discussing with all the parties impacted, only a very small percentage of customers were in the real need of complex solutions. Most of the others were just trying to avoid a project that will implement simple shared-nothing solutions. Why? Because apparently it is simpler to migrate data to a platform that </span><i><span style="font-weight: 400;">does all for you</span></i><span style="font-weight: 400;">, than accept a bit of additional work and challenge at the beginning but keep a simple approach. Also going for the last shining things always has its magic.</span></p>
  250. <p><span style="font-weight: 400;">On top of that, with the rise of Kubernetes and Mysql Operators, a lot of confusion starts to circulate, most of which generate by the total lack of understanding that a Database and a relational database are two separate things. That lack of understanding the difference and the real problems attached to a RDBMS had brought some to talk about horizontal scaling for databases, with a concerning superficiality and without clarifying if they were talking about RDBMS or not. As such some clarification is long due as well as putting back the KISS principle as the main focus.&nbsp;</span></p>
  251. <p>&nbsp;</p>
  252. <p><span style="font-weight: 400;">Given that, I thought that refreshing how ProxySQL could help in building a simple sharding solution may help to clarify the issues, reset the expectations and show how we can do things in a simpler way.&nbsp; (See my old post </span><a href="https://www.percona.com/blog/mysql-sharding-with-proxysql/"><span style="font-weight: 400;">https://www.percona.com/blog/mysql-sharding-with-proxysql/</span></a><span style="font-weight: 400;">).&nbsp;</span></p>
  253. <p>&nbsp;</p>
  254. <p><span style="font-weight: 400;">To do so I had built a simple POC that illustrates how you can use Percona Operator for MySQL (POM) and ProxySQL to build a sharded environment with a good level of automation for some standard operations like backup/restore software upgrade and resource scaling.&nbsp;</span></p>
  255. <p>&nbsp;</p>
  256. <h4><span style="font-weight: 400;">Why Proxysql?</span></h4>
  257. <p><span style="font-weight: 400;">In the following example we mimic a case where we need </span><i><span style="font-weight: 400;">a simple sharding solution</span></i><span style="font-weight: 400;">, which means we just need to redirect the data to different data containers, keeping the database maintenance operations on us. In this common case we do not need to implement a full sharding system such as Vitess.&nbsp;</span></p>
  258. <p><span style="font-weight: 400;"></span></p>
  259. <p><span style="font-weight: 400;">As illustrated above ProxySQL allows us to set up a common entry point for the application and then redirect the traffic on the base of identified sharding keys. It will also allow us to redirect read/write traffic to the primary and read only traffic to all secondaries.&nbsp;</span></p>
  260. <p>&nbsp;</p>
  261. <p><span style="font-weight: 400;">The other interesting thing is that we can have ProxySQL as part of the application pod, or as an independent service. Best practices indicate that having ProxySQL closer to the application will be more efficient especially if we decide to activate the caching feature.&nbsp;&nbsp;</span></p>
  262. <p>&nbsp;</p>
  263. <h4><span style="font-weight: 400;">Why POM</span></h4>
  264. <p><span style="font-weight: 400;">Percona Operator for MySQL comes with three main solutions, Percona Operator for PXC, Percona Operator for MySQL Group Replication and Percona Operator for Percona server. The first two are based on virtually-synchronous replication, and allow the cluster to keep the data state consistent across all pods, which guarantees that the service will always offer consistent data.&nbsp; In K8s context we can see POM as a single service with native horizontal scalability for reads, while for writes we will adopt the mentioned sharding approach.&nbsp;</span></p>
  265. <p>&nbsp;</p>
  266. <p><span style="font-weight: 400;">The other important aspects of using a POM based solution is the automation it comes with. Deploying POM you will be able to set automation for backups, software updates, monitoring (using PMM) and last but not least the possibility to scale UP or DOWN just changing the needed resources.&nbsp;</span></p>
  267. <h3><span style="font-weight: 400;">The elements used</span></h3>
  268. <p><a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/horizontal_scaling.jpg"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/horizontal_scaling.jpg" alt="horizontal scaling" width="450" height="422" class="alignnone wp-image-87688" /></a></p>
  269. <p><span style="font-weight: 400;">In our POC I will use a modified version of sysbench (https://github.com/Tusamarco/sysbench), that has an additional field </span><i><span style="font-weight: 400;">continent</span></i><span style="font-weight: 400;"> and I will use that as a sharding key. At the moment and for the purpose of this simple POC I will only have 2 Shards.</span></p>
  270. <p>&nbsp;</p>
  271. <p><span style="font-weight: 400;">As the diagram above illustrates here we have a simple deployment but good enough to illustrate the sharding approach.</span></p>
  272. <p><span style="font-weight: 400;">We have:</span></p>
  273. <ul>
  274. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">The application(s) node(s), it is really up to you if you want to test with one application node or more, nothing will change, as well as for the ProxySQL nodes, just keep in mind that if you use more proxysql nodes is better to activate the internal cluster support or use consul to synchronize them.&nbsp;</span></li>
  275. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Shard 1 is based on POM with PXC, it has:</span></li>
  276. </ul>
  277. <ul>
  278. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">Load balancer for service entry point</span>
  279. <ul>
  280. <li style="font-weight: 400;" aria-level="3"><span style="font-weight: 400;">Entry point for r/w</span></li>
  281. <li style="font-weight: 400;" aria-level="3"><span style="font-weight: 400;">Entry point for read only</span></li>
  282. </ul>
  283. </li>
  284. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">3 Pods for Haproxy</span>
  285. <ul>
  286. <li style="font-weight: 400;" aria-level="3"><span style="font-weight: 400;">Haproxy container</span></li>
  287. <li style="font-weight: 400;" aria-level="3"><span style="font-weight: 400;">Pmm agent container</span></li>
  288. </ul>
  289. </li>
  290. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">3 Pods with data nodes (PXC)</span>
  291. <ul>
  292. <li style="font-weight: 400;" aria-level="3"><span style="font-weight: 400;">PXC cluster node container</span></li>
  293. <li style="font-weight: 400;" aria-level="3"><span style="font-weight: 400;">Log streaming</span></li>
  294. <li style="font-weight: 400;" aria-level="3"><span style="font-weight: 400;">Pmm container&nbsp;</span></li>
  295. </ul>
  296. </li>
  297. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">Backup/restore service&nbsp;</span></li>
  298. </ul>
  299. <ul>
  300. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Shard 2 is based on POM for Percona server and Group Replication (technical preview)&nbsp;&nbsp;</span>
  301. <ul>
  302. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">Load balancer for service entry point</span>
  303. <ul>
  304. <li style="font-weight: 400;" aria-level="3"><span style="font-weight: 400;">Entry point for r/w</span></li>
  305. <li style="font-weight: 400;" aria-level="3"><span style="font-weight: 400;">Entry point for read only</span></li>
  306. </ul>
  307. </li>
  308. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">3 Pods for MySQL Router (testing)</span>
  309. <ul>
  310. <li style="font-weight: 400;" aria-level="3"><span style="font-weight: 400;">MySQL router container</span></li>
  311. </ul>
  312. </li>
  313. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">3 Pods with data nodes (PS with GR)</span>
  314. <ul>
  315. <li style="font-weight: 400;" aria-level="3"><span style="font-weight: 400;">PS -GR cluster node container</span></li>
  316. <li style="font-weight: 400;" aria-level="3"><span style="font-weight: 400;">Log streaming</span></li>
  317. <li style="font-weight: 400;" aria-level="3"><span style="font-weight: 400;">Pmm container&nbsp;</span></li>
  318. </ul>
  319. </li>
  320. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">Backup/restore on scheduler</span></li>
  321. </ul>
  322. </li>
  323. </ul>
  324. <p>&nbsp;</p>
  325. <p><span style="font-weight: 400;">Now you may have noticed that the representation of the nodes are different in size, this is not a mistake while drawing. It indicates that I have allocated more resources (CPU and Memory) to shard1 than shard2. Why? Because I can and I am simulating a situation where a shard2 gets less traffic, at least temporarily, as such I do not want to give it the same resources as shard1. I will eventually increase them if I see the needs.&nbsp;</span></p>
  326. <h2><span style="font-weight: 400;">The settings</span></h2>
  327. <h3><span style="font-weight: 400;">Data layer</span></h3>
  328. <p><span style="font-weight: 400;">Let us start with the easy one, the Data layer configuration. Configuring correctly the environment is the key, and to do so I am using a tool that I wrote specifically to calculate the needed configuration in K8s POM environment, you can find it here (</span><a href="https://github.com/Tusamarco/mysqloperatorcalculator"><span style="font-weight: 400;">https://github.com/Tusamarco/mysqloperatorcalculator</span></a><span style="font-weight: 400;">).&nbsp;</span></p>
  329. <p><span style="font-weight: 400;">Once you have compiled it and run you can simply ask what “dimensions” are supported, or you can define a custom level of resources, but you will still need to indicate the level of expected load. In any case please refer to the README in the repository which has all the instructions.</span></p>
  330. <p><span style="font-weight: 400;">The full cr.yaml for PXC shard1 is </span><a href="https://github.com/Tusamarco/blogs/blob/master/mysql_horizontal_scaling/cr_pxc.yml"><span style="font-weight: 400;">here</span></a><span style="font-weight: 400;">, while the one for PS-GR </span><a href="https://github.com/Tusamarco/blogs/blob/master/mysql_horizontal_scaling/cr_ps_gr.yml"><span style="font-weight: 400;">here</span></a><span style="font-weight: 400;">.&nbsp;</span></p>
  331. <p><span style="font-weight: 400;">For Shard 1: I asked for resources to cover traffic of type 2 (Light OLTP), configuration type 5 (2XLarge) 1000 connections.</span></p>
  332. <p><span style="font-weight: 400;">For Shard2: I ask for resources to cover traffic of type 2 (Light OLTP), configuration type 2 (Small), 100 connections.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span></p>
  333. <p><span style="font-weight: 400;">Once you have the CRs defined you can follow the official guidelines to set the environment up ( PXC (</span><a href="https://docs.percona.com/percona-operator-for-mysql/pxc/index.html"><span style="font-weight: 400;">https://docs.percona.com/percona-operator-for-mysql/pxc/index.html</span></a><span style="font-weight: 400;">), PS (</span><a href="https://docs.percona.com/percona-operator-for-mysql/ps/index.html"><span style="font-weight: 400;">https://docs.percona.com/percona-operator-for-mysql/ps/index.html</span></a><span style="font-weight: 400;">)</span></p>
  334. <p>&nbsp;</p>
  335. <p><span style="font-weight: 400;">It is time now to see the Proxysql settings.</span></p>
  336. <h3><span style="font-weight: 400;">ProxySQL and Sharding rules</span></h3>
  337. <p><span style="font-weight: 400;">As mentioned before we are going to test the load sharding by continent, and as also mentioned before we know that ProxySQL will not provide additional help to automatically manage the sharded environment.&nbsp;</span></p>
  338. <p><span style="font-weight: 400;">Given that one way to do it is to create a DBA account per shard, or to inject shard information in the commands while executing. </span><span style="font-weight: 400;"><br /></span><span style="font-weight: 400;">I will use the less comfortable one just to prove if it works, the different DBA accounts.&nbsp;</span></p>
  339. <p>&nbsp;</p>
  340. <p><span style="font-weight: 400;">We will have 2 shards, the sharding key is the continent field, and the continents will be grouped as follows:</span></p>
  341. <ul>
  342. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Shard 1:</span>
  343. <ul>
  344. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">Asia</span></li>
  345. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">Africa</span></li>
  346. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">Antarctica</span></li>
  347. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">Europe</span></li>
  348. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">North America</span></li>
  349. </ul>
  350. </li>
  351. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Shard 2:</span>
  352. <ul>
  353. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">Oceania</span></li>
  354. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">South America</span></li>
  355. </ul>
  356. </li>
  357. </ul>
  358. <p>&nbsp;</p>
  359. <p><span style="font-weight: 400;">The DBAs users:</span></p>
  360. <ul>
  361. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">dba_g1</span></li>
  362. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">dba_g2</span></li>
  363. </ul>
  364. <p>&nbsp;</p>
  365. <p><span style="font-weight: 400;">The application user:</span></p>
  366. <ul>
  367. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">app_test</span></li>
  368. </ul>
  369. <p>&nbsp;</p>
  370. <p><span style="font-weight: 400;">The host groups will be:</span></p>
  371. <ul>
  372. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Shard 1</span>
  373. <ul>
  374. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">100 Read and Write</span></li>
  375. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">101 Read only</span></li>
  376. </ul>
  377. </li>
  378. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Shard 2</span>
  379. <ul>
  380. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">200 Read and Write</span></li>
  381. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">201 Read only</span></li>
  382. </ul>
  383. </li>
  384. </ul>
  385. <p>&nbsp;</p>
  386. <p><span style="font-weight: 400;">Once that is defined, we need to identify which query rules will serve us and how.</span></p>
  387. <p><span style="font-weight: 400;">What we want is to redirect all the incoming queries for:</span></p>
  388. <ul>
  389. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Asia, Africa, Antarctica, Europe and North America to shard 1.</span></li>
  390. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Oceania and South America to shard 2</span></li>
  391. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Split the queries in R/W and Read only</span></li>
  392. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Prevent the execution of any query that do not have a shard key</span></li>
  393. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Backup data at regular interval and store it in a safe place</span></li>
  394. </ul>
  395. <p><a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/query_rukes_sharding.png"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/query_rukes_sharding.png" alt="query rukes sharding" width="800" height="280" class="alignnone size-large wp-image-87681" /></a></p>
  396. <p><span style="font-weight: 400;">Given the above we first define the rules for the DBAs accounts:</span></p>
  397. <p><span style="font-weight: 400;">We set the Hostgroup for each DBA and then if the query matches the sharding rule we redirect it to the proper sharding, otherwise the HG will remain as set.</span></p>
  398. <p><span style="font-weight: 400;">This allows us to execute queries like CREATE/DROP table on our shard without problem, but will allow us to send data where needed.&nbsp;</span></p>
  399. <p><span style="font-weight: 400;">For instance the one below is the output of the queries that sysbench will run.</span></p>
  400. <p><b>Prepare:</b></p>
  401. <pre class="lang:mysql decode:true">INSERT INTO windmills_test1 /*  continent=Asia */ (uuid,millid,kwatts_s,date,location,continent,active,strrecordtype) VALUES(UUID(), 79, 3949999,NOW(),'mr18n2L9K88eMlGn7CcctT9RwKSB1FebW397','Asia',0,'quq')
  402. </pre>
  403. <p><span style="font-weight: 400;">In this case I have the application simply injecting a comment in the INSERT SQL declaring the shard key, given I am using the account dba_g1 to create/prepare the schemas, rules&nbsp; 32/32 will be used and give I have sett apply=1, ProxySQL will exit the query rules parsing and send the command to relevant hostgroup.</span></p>
  404. <p><span style="font-weight: 400;">Run:</span></p>
  405. <pre class="lang:mysql decode:true">SELECT id, millid, date,continent,active,kwatts_s FROM windmills_test1 WHERE id BETWEEN ? AND ? AND continent='South America'
  406.  
  407. SELECT SUM(kwatts_s) FROM windmills_test1 WHERE id BETWEEN ? AND ?  and active=1  AND continent='Asia'
  408. SELECT id, millid, date,continent,active,kwatts_s  FROM windmills_test1 WHERE id BETWEEN ? AND ?  AND continent='Oceania' ORDER BY millid
  409.  
  410. SELECT DISTINCT millid,continent,active,kwatts_s   FROM windmills_test1 WHERE id BETWEEN ? AND ? AND active =1  AND continent='Oceania' ORDER BY millid
  411.  
  412. UPDATE windmills_test1 SET active=? WHERE id=?  AND continent='Asia'
  413. UPDATE windmills_test1 SET strrecordtype=? WHERE id=?  AND continent='North America'
  414.  
  415. DELETE FROM windmills_test1 WHERE id=?  AND continent='Antarctica'
  416.  
  417. INSERT INTO windmills_test1 /* continent=Antarctica */ (id,uuid,millid,kwatts_s,date,location,continent,active,strrecordtype) VALUES (?, UUID(), ?, ?, NOW(), ?, ?, ?,?) ON DUPLICATE KEY UPDATE kwatts_s=kwatts_s+1
  418. </pre>
  419. <p><span style="font-weight: 400;">The above are executed during the tests.&nbsp;</span></p>
  420. <p><span style="font-weight: 400;">In all of them the sharding key is present, either in the WHERE clause OR as comment.&nbsp;</span></p>
  421. <p><span style="font-weight: 400;">Of course if I execute one of them without the sharding key,&nbsp; the firewall rule will stop the query execution, ie:</span><span style="font-weight: 400;"><br /></span></p>
  422. <pre class="lang:mysql decode:true">mysql&gt; SELECT id, millid, date,continent,active,kwatts_s FROM windmills_test1 WHERE id BETWEEN ? AND ?;
  423. ERROR 1148 (42000): It is impossible to redirect this command to a defined shard. Please be sure you Have the Continent definition in your query, or that you use a defined DBA account (dba_g{1/2})
  424. </pre>
  425. <p><span style="font-weight: 400;">Check </span><a href="https://github.com/Tusamarco/blogs/blob/master/mysql_horizontal_scaling/operator_sharding_with_proxysql_public.txt"><span style="font-weight: 400;">here </span></a><span style="font-weight: 400;">for full command list</span></p>
  426. <h3><span style="font-weight: 400;">Setting up the dataset</span></h3>
  427. <p><span style="font-weight: 400;">Once we have the rules set it is time to setup the schemas and the data using sysbench (</span><a href="https://github.com/Tusamarco/sysbench"><span style="font-weight: 400;">https://github.com/Tusamarco/sysbench</span></a><span style="font-weight: 400;">), remember to use windmills_sharding tests.&nbsp;&nbsp;</span></p>
  428. <p><span style="font-weight: 400;">The first operation is to build the schema on SHARD2 without filling it with data. This is a DBA action as such we will execute it using the dba_g2 account:</span></p>
  429. <pre class="lang:sh decode:true">sysbench ./src/lua/windmills_sharding/oltp_read.lua  --mysql-host=10.0.1.96  --mysql-port=6033 --mysql-user=dba_g2 --mysql-password=xxx --mysql-db=windmills_large --mysql_storage_engine=innodb --db-driver=mysql --tables=4 --table_size=0 --table_name=windmills --mysql-ignore-errors=all --threads=1  prepare
  430. </pre>
  431. <p><span style="font-weight: 400;">Setting table_size and pointing to the ProxySQL IP/port will do, and I will have:</span><span style="font-weight: 400;"><br /></span></p>
  432. <pre class="lang:mysql decode:true">mysql&gt; select current_user(), @@hostname;
  433. +----------------+-------------------+
  434. | current_user() | @@hostname        |
  435. +----------------+-------------------+
  436. | dba_g2@%       | ps-mysql1-mysql-0 |
  437. +----------------+-------------------+
  438. 1 row in set (0.01 sec)
  439.  
  440. mysql&gt; use windmills_large;
  441. Database changed
  442. mysql&gt; show tables;
  443. +---------------------------+
  444. | Tables_in_windmills_large |
  445. +---------------------------+
  446. | windmills1                |
  447. | windmills2                |
  448. | windmills3                |
  449. | windmills4                |
  450. +---------------------------+
  451. 4 rows in set (0.01 sec)
  452.  
  453. mysql&gt; select count(*) from windmills1;
  454. +----------+
  455. | count(*) |
  456. +----------+
  457. |        0 |
  458. +----------+
  459. 1 row in set (0.09 sec)
  460. </pre>
  461. <p><span style="font-weight: 400;">All set but empty.</span></p>
  462. <p><span style="font-weight: 400;">Now let us do the same but with the other DBA user:</span></p>
  463. <pre class="lang:sh decode:true">sysbench ./src/lua/windmills_sharding/oltp_read.lua  --mysql-host=10.0.1.96  --mysql-port=6033 --mysql-user=dba_g1 --mysql-password=xxx --mysql-db=windmills_large --mysql_storage_engine=innodb --db-driver=mysql --tables=4 --table_size=400 --table_name=windmills --mysql-ignore-errors=all --threads=1  prepare
  464. </pre>
  465. <p><span style="font-weight: 400;">If I do now the select above with user dba_g2:</span></p>
  466. <pre class="lang:mysql decode:true">mysql&gt; select current_user(), @@hostname;select count(*) from windmills1;
  467. +----------------+-------------------+
  468. | current_user() | @@hostname        |
  469. +----------------+-------------------+
  470. | dba_g2@%       | ps-mysql1-mysql-0 |
  471. +----------------+-------------------+
  472. 1 row in set (0.00 sec)
  473.  
  474. +----------+
  475. | count(*) |
  476. +----------+
  477. |      113 |
  478. +----------+
  479. 1 row in set (0.00 sec)
  480. </pre>
  481. <p><span style="font-weight: 400;">While If I reconnect and use dba_g1:</span></p>
  482. <pre class="lang:mysql decode:true">mysql&gt; select current_user(), @@hostname;select count(*) from windmills1;
  483. +----------------+--------------------+
  484. | current_user() | @@hostname         |
  485. +----------------+--------------------+
  486. | dba_g1@%       | mt-cluster-1-pxc-0 |
  487. +----------------+--------------------+
  488. 1 row in set (0.00 sec)
  489.  
  490. +----------+
  491. | count(*) |
  492. +----------+
  493. |      287 |
  494. +----------+
  495. 1 row in set (0.01 sec)
  496. </pre>
  497. <p><span style="font-weight: 400;">I can also check on ProxySQL to see which rules were utilized:</span></p>
  498. <pre class="lang:mysql decode:true">select active,hits,destination_hostgroup, mysql_query_rules.rule_id, match_digest, match_pattern, replace_pattern, cache_ttl, apply,flagIn,flagOUT FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_id;
  499.  
  500. +------+-----------------------+---------+---------------------+----------------------------------------------------------------------------+-------+--------+---------+
  501. | hits | destination_hostgroup | rule_id | match_digest        | match_pattern                                                              | apply | flagIN | flagOUT |
  502. +------+-----------------------+---------+---------------------+----------------------------------------------------------------------------+-------+--------+---------+
  503. | 3261 | 100                   | 20      | NULL                | NULL                                                                       | 0     | 0      | 500     |
  504. | 51   | 200                   | 21      | NULL                | NULL                                                                       | 0     | 0      | 600     |
  505. | 2320 | 100                   | 31      | NULL                | scontinents*(=|like)s*'*(Asia|Africa|Antarctica|Europe|North America)'* | 1     | 500    | 0       |
  506. | 880  | 200                   | 32      | NULL                | scontinents*(=|like)s*'*(Oceania|South America)'*                       | 1     | 500    | 0       |
  507. | 0    | 100                   | 34      | NULL                | scontinents*(=|like)s*'*(Asia|Africa|Antarctica|Europe|North America)'* | 1     | 600    | 0       |
  508. | 0    | 200                   | 35      | NULL                | scontinents*(=|like)s*'*(Oceania|South America)'*                       | 1     | 600    | 0       |
  509. | 2    | 100                   | 51      | NULL                | scontinents*(=|like)s*'*(Asia|Africa|Antarctica|Europe|North America)'* | 0     | 0      | 1001    |
  510. | 0    | 200                   | 54      | NULL                | scontinents*(=|like)s*'*(Oceania|South America)'*                       | 0     | 0      | 1002    |
  511. | 0    | 100                   | 60      | NULL                | NULL                                                                       | 0     | 50     | 1001    |
  512. | 0    | 200                   | 62      | NULL                | NULL                                                                       | 0     | 60     | 1002    |
  513. | 7    | NULL                  | 2000    | .                   | NULL                                                                       | 1     | 0      | NULL    |
  514. | 0    | 100                   | 2040    | ^SELECT.*FOR UPDATE | NULL                                                                       | 1     | 1001   | NULL    |
  515. | 2    | 101                   | 2041    | ^SELECT.*$          | NULL                                                                       | 1     | 1001   | NULL    |
  516. | 0    | 200                   | 2050    | ^SELECT.*FOR UPDATE | NULL                                                                       | 1     | 1002   | NULL    |
  517. | 0    | 201                   | 2051    | ^SELECT.*$          | NULL                                                                       | 1     | 1002   | NULL    |
  518. +------+-----------------------+---------+---------------------+----------------------------------------------------------------------------+-------+--------+---------+
  519. </pre>
  520. <h3><span style="font-weight: 400;">Running the application</span></h3>
  521. <p><span style="font-weight: 400;">Now that the data load test was successful, let us do the real load following the indication as above but use 80 Tables and just a bit more records like 20000, nothing huge.&nbsp;</span></p>
  522. <p>&nbsp;</p>
  523. <p><span style="font-weight: 400;">Once the data is loaded we will have the 2 shards with different numbers of records, if all went well the shard2 should have ¼ of the total and shard1 ¾ .</span></p>
  524. <p>&nbsp;</p>
  525. <p><span style="font-weight: 400;">When load is over I have as expected:</span></p>
  526. <pre class="lang:mysql decode:true">mysql&gt; select current_user(), @@hostname;select count(*) as shard1 from windmills_large.windmills80;select /* continent=shard2 */ count(*) as shard2 from windmills_large.windmills80;
  527. +----------------+--------------------+
  528. | current_user() | @@hostname         |
  529. +----------------+--------------------+
  530. | dba_g1@%       | mt-cluster-1-pxc-0 |
  531. +----------------+--------------------+
  532. 1 row in set (0.00 sec)
  533.  
  534. +--------+
  535. | shard1 |
  536. +--------+
  537. |  14272 | ← Table windmills80 in SHARD1
  538. +--------+
  539. +--------+
  540. | shard2 |
  541. +--------+
  542. |   5728 | ← Table windmills80 in SHARD2
  543. +--------+
  544. </pre>
  545. <p><span style="font-weight: 400;">As you may have already noticed, I used a trick to query the other shard using the dba_g1 user, I just passed in the query&nbsp; the shard2 definition as a comment. That is all we need.</span></p>
  546. <p><span style="font-weight: 400;">Let us execute the </span><i><span style="font-weight: 400;">run</span></i><span style="font-weight: 400;"> command for writes in sysbench and see what happens.</span></p>
  547. <p><span style="font-weight: 400;">The first thing we can notice while doing writes is the query distribution:</span><span style="font-weight: 400;"><br /></span></p>
  548. <pre class="lang:mysql decode:true">+--------+-----------+----------------------------------------------------------------------------+----------+--------+----------+----------+--------+---------+-------------+---------+
  549. | weight | hostgroup | srv_host                                                                   | srv_port | status | ConnUsed | ConnFree | ConnOK | ConnERR | MaxConnUsed | Queries |
  550. +--------+-----------+----------------------------------------------------------------------------+----------+--------+----------+----------+--------+---------+-------------+---------+
  551. | 10000  | 100       | ac966f7d46c04400fb92a3603f0e2634-193113472.eu-central-1.elb.amazonaws.com  | 3306     | ONLINE | 24     | 0        | 138    | 66      | 25          | 1309353 |
  552. | 100    | 101       | a5c8836b7c05b41928ca84f2beb48aee-1936458168.eu-central-1.elb.amazonaws.com | 3306     | ONLINE | 0     | 0        | 0      | 0       | 0           |       0 |
  553. | 10000  | 200       | a039ab70e9f564f5e879d5e1374d9ffa-1769267689.eu-central-1.elb.amazonaws.com | 3306     | ONLINE | 24     | 1        | 129    | 66      | 25          |  516407 |
  554. | 10000  | 201       | a039ab70e9f564f5e879d5e1374d9ffa-1769267689.eu-central-1.elb.amazonaws.com | 6447     | ONLINE | 0     | 0        | 0      | 0       | 0           |       0 |
  555. +--------+-----------+----------------------------------------------------------------------------+----------+--------+----------+----------+--------+---------+-------------+---------+
  556. </pre>
  557. <p><span style="font-weight: 400;">Where we can notice that the load in connection is evenly distributed, while the load is mainly going to shard1 as we expected given we have an unbalanced sharding by design.</span></p>
  558. <p>&nbsp;</p>
  559. <p><span style="font-weight: 400;">At MySQL level we had:</span></p>
  560. <p>&nbsp;</p>
  561. <p><span style="font-weight: 400;">Questions</span></p>
  562. <p><a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/pxc_ps_write_questions.png"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/pxc_ps_write_questions.png" alt="pxc ps write questions" width="1024" height="187" class="alignnone size-large wp-image-87684" /></a></p>
  563. <p><span style="font-weight: 400;">Com Type</span></p>
  564. <p><a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/pxc_ps_write_com.png"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/pxc_ps_write_com.png" alt="pxc ps write com" width="1024" height="179" class="alignnone size-large wp-image-87683" /></a></p>
  565. <p><a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/number_operation_writes.png"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/number_operation_writes.png" alt="number operation writes" width="473" height="133" class="alignnone size-full wp-image-87687" /></a></p>
  566. <p><span style="font-weight: 400;">The final point is, what is the gain using this sharding approach?</span></p>
  567. <p><span style="font-weight: 400;">Well we still need to consider the fact we are testing on a very small set of data, however if we can already identify some benefit here, that will be an interesting result.&nbsp;</span></p>
  568. <p>&nbsp;</p>
  569. <p><span style="font-weight: 400;">Let see the write operations with 24 and 64 threads:</span></p>
  570. <p><a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/writes_write.png"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/writes_write.png" alt="writes write" width="450" height="395" class="alignnone wp-image-87693" /></a> <a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/latency_write.png"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/latency_write.png" alt="latency write" width="450" height="395" class="alignnone wp-image-87686" /></a></p>
  571. <p><span style="font-weight: 400;">We get a gain of ~33% just using sharding, while for latency we do not have a cost on the contrary also with small load increase we can see how the sharded solution performs better. Of course we are still talking about low number of rows and running threads but gain is there.&nbsp;</span></p>
  572. <p>&nbsp;</p>
  573. <h3><span style="font-weight: 400;">Backup&nbsp;</span></h3>
  574. <p><span style="font-weight: 400;">The backup and restore operation when using POM is completely managed by the operator (see instructions in the POM documentation </span><a href="https://docs.percona.com/percona-operator-for-mysql/pxc/backups.html"><span style="font-weight: 400;">https://docs.percona.com/percona-operator-for-mysql/pxc/backups.html</span></a><span style="font-weight: 400;"> and </span><a href="https://docs.percona.com/percona-operator-for-mysql/ps/backups.html"><span style="font-weight: 400;">https://docs.percona.com/percona-operator-for-mysql/ps/backups.html</span></a><span style="font-weight: 400;"> ).&nbsp;</span></p>
  575. <p><span style="font-weight: 400;">The interesting part is that we can have multiple kind of backup solution, like:</span></p>
  576. <ul>
  577. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">On demand</span></li>
  578. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Scheduled&nbsp;</span></li>
  579. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Full Point in time recovery with log streaming</span></li>
  580. </ul>
  581. <p><span style="font-weight: 400;">Automation will allow us to set schedule as simple as this:</span><span style="font-weight: 400;"><br /></span></p>
  582. <pre class="lang:yaml decode:true">    schedule:
  583.     - name: "sat-night-backup"
  584.        schedule: "0 0 * * 6"
  585.        keep: 3
  586.        storageName: s3-eu-west
  587.      - name: "daily-backup"
  588.        schedule: "0 3 * * *"
  589.        keep: 7
  590.        storageName: s3-eu-west
  591. </pre>
  592. <p><span style="font-weight: 400;">Or if you want to run the on demand:</span></p>
  593. <pre class="lang:sh decode:true">kubectl apply -f backup.yaml</pre>
  594. <p><span style="font-weight: 400;">Where the backup.yaml file has very simple informations:</span></p>
  595. <pre class="lang:yaml decode:true">apiVersion: ps.percona.com/v1alpha1
  596. kind: PerconaServerMySQLBackup
  597. metadata:
  598.  name: ps-gr-sharding-test-2nd-of-may
  599. #  finalizers:
  600. #    - delete-backup
  601. spec:
  602.  clusterName: ps-mysql1
  603.  storageName: s3-ondemand
  604. </pre>
  605. <p><span style="font-weight: 400;">Using both methods we will be able to soon have a good set of backups like:</span></p>
  606. <p><span style="font-weight: 400;">POM (PXC)</span></p>
  607. <pre class="lang:sh decode:true">cron-mt-cluster-1-s3-eu-west-20234293010-3vsve   mt-cluster-1   s3-eu-west    s3://mt-bucket-backup-tl/scheduled/mt-cluster-1-2023-04-29-03:00:10-full   Succeeded   3d9h        3d9h
  608. cron-mt-cluster-1-s3-eu-west-20234303010-3vsve   mt-cluster-1   s3-eu-west    s3://mt-bucket-backup-tl/scheduled/mt-cluster-1-2023-04-30-03:00:10-full   Succeeded   2d9h        2d9h
  609. cron-mt-cluster-1-s3-eu-west-2023513010-3vsve    mt-cluster-1   s3-eu-west    s3://mt-bucket-backup-tl/scheduled/mt-cluster-1-2023-05-01-03:00:10-full   Succeeded   33h         33h
  610. cron-mt-cluster-1-s3-eu-west-2023523010-3vsve    mt-cluster-1   s3-eu-west    s3://mt-bucket-backup-tl/scheduled/mt-cluster-1-2023-05-02-03:00:10-full   Succeeded   9h          9h
  611. </pre>
  612. <p><span style="font-weight: 400;">POM (PS) *</span></p>
  613. <pre class="lang:sh decode:true">NAME                             STORAGE       DESTINATION                                                                     STATE       COMPLETED   AGE
  614. ps-gr-sharding-test              s3-ondemand   s3://mt-bucket-backup-tl/ondemand/ondemand/ps-mysql1-2023-05-01-15:10:04-full   Succeeded   21h         21h
  615. ps-gr-sharding-test-2nd-of-may   s3-ondemand   s3://mt-bucket-backup-tl/ondemand/ondemand/ps-mysql1-2023-05-02-12:22:24-full   Succeeded   27m         27m
  616. </pre>
  617. <p><span style="font-weight: 400;">To note that as DBA, we still need to validate the backups with a restore procedure, that part is not automated (yet).&nbsp;</span></p>
  618. <p><i><span style="font-weight: 400;">*Note that Backup for POM PS is available only on demand given the solution is still in technical preview</span></i></p>
  619. <h3><span style="font-weight: 400;">When will this solution fit in?</span></h3>
  620. <p><span style="font-weight: 400;">As mentioned multiple times, this solution can cover simple cases of sharding, better if you have shared-nothing.&nbsp;</span></p>
  621. <p><span style="font-weight: 400;">It also requires work from the DBA side in case of DDL operations or resharding.&nbsp;</span></p>
  622. <p><span style="font-weight: 400;">You also need to be able to change some SQL code in order to be sure to have present the sharding key/information, in any SQL executed.</span></p>
  623. <p>&nbsp;</p>
  624. <h3><span style="font-weight: 400;">When will this solution not fit in?</span></h3>
  625. <p><span style="font-weight: 400;">There are several things that could prevent you to use this solution, the most common ones are:</span></p>
  626. <ul>
  627. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">You need to query multiple shards at the same time. This is not possible with ProxySQL</span></li>
  628. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">You do not have a DBA to perform administrative work and need to rely on automated system</span></li>
  629. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Distributed transaction cross shard</span></li>
  630. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">No access to SQL code.</span></li>
  631. </ul>
  632. <h2><span style="font-weight: 400;">Conclusions</span></h2>
  633. <p><span style="font-weight: 400;">We do not have the Amletic dilemma about sharding or not sharding.&nbsp;</span></p>
  634. <p><span style="font-weight: 400;">When using a RDBMS like MySQL, if you need horizontal scalability, you need to shard.&nbsp;</span></p>
  635. <p><span style="font-weight: 400;">The point is there is no magic wand or solution, moving to sharding is an expensive and impacting operation. If you choose it at the beginning, before doing any application development, the effort can be significantly less.&nbsp;</span></p>
  636. <p><span style="font-weight: 400;">Doing sooner will also allow you to test proper solutions, where </span><i><span style="font-weight: 400;">proper</span></i><span style="font-weight: 400;"> is a KISS solution, always go for the less complex things, in 2 years you will be super happy about your decision.&nbsp;&nbsp;</span></p>
  637. <p><span style="font-weight: 400;">If instead you must convert a current solution, then prepare for a bloodshed, or at least for a long journey.&nbsp;</span></p>
  638. <p><span style="font-weight: 400;">In any case we need to keep in mind few key points:</span></p>
  639. <ul>
  640. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Do not believe most of the articles on the internet that promise you infinite scalability for your database. If there is no distinction in the article about a simple database and a RDBMS, run away.&nbsp;</span></li>
  641. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Do not go for the last shiny things just because they shine. Test them and evaluate IF it makes sense for you. Better to spend a quarter testing now a few solutions, than fight for years with something that you do not fully comprehend.&nbsp;&nbsp;</span></li>
  642. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Using containers/operators/kubernetes do not scale </span><i><span style="font-weight: 400;">per se</span></i><span style="font-weight: 400;">, you must find a mechanism to have the solution scaling, there is absolutely NO difference with premises. What you may get is a good level of automation, however that will come with a good level of complexity, it is up to you to evaluate if it makes sense or not.&nbsp;&nbsp;</span></li>
  643. </ul>
  644. <p>&nbsp;</p>
  645. <p><span style="font-weight: 400;">As said at the beginning for MYSQL the choice is limited. Vitess is the full complete solution, with a lot of coding to provide you a complete platform to deal with your scaling needs.</span></p>
  646. <p><span style="font-weight: 400;">However do not be so fast to exclude ProxySQL as possible solutions. There are out there already many using it also for sharding.&nbsp;</span></p>
  647. <p><span style="font-weight: 400;">This small POC used a synthetic case, but it also shows that with just 4 rules you can achieve a decent solution. A real scenario could be a bit more complex … or not.&nbsp;</span></p>
  648. <p>&nbsp;</p>
  649. <h2><span style="font-weight: 400;">References</span></h2>
  650. <p><span style="font-weight: 400;">Vitess (</span><a href="https://vitess.io/docs/"><span style="font-weight: 400;">https://vitess.io/docs/</span></a><span style="font-weight: 400;">)</span></p>
  651. <p><span style="font-weight: 400;">ProxySQL (</span><a href="https://proxysql.com/documentation/"><span style="font-weight: 400;">https://proxysql.com/documentation/</span></a><span style="font-weight: 400;">)</span></p>
  652. <p><span style="font-weight: 400;">Firewalling with ProxySQL (</span><a href="https://www.tusacentral.com/joomla/index.php/mysql-blogs/197-proxysql-firewalling"><span style="font-weight: 400;">https://www.tusacentral.com/joomla/index.php/mysql-blogs/197-proxysql-firewalling</span></a><span style="font-weight: 400;">)</span></p>
  653. <p><span style="font-weight: 400;">Sharding:</span></p>
  654. <ul>
  655. <li style="font-weight: 400;" aria-level="1"><a href="https://www.percona.com/blog/mysql-sharding-with-proxysql/"><span style="font-weight: 400;">https://www.percona.com/blog/mysql-sharding-with-proxysql/</span></a></li>
  656. <li style="font-weight: 400;" aria-level="1"><a href="https://www.percona.com/blog/horizontal-scaling-in-mysql-sharding-followup/"><span style="font-weight: 400;">https://https://medium.com/pinterest-engineering/sharding-pinterest-how-we-scaled-our-mysql-fleet-3f341e96ca6fwww.percona.com/blog/horizontal-scaling-in-mysql-sharding-followup/</span></a></li>
  657. <li style="font-weight: 400;" aria-level="1"><a href="https://quoraengineering.quora.com/MySQL-sharding-at-Quora"><span style="font-weight: 400;">https://quoraengineering.quora.com/MySQL-sharding-at-Quora</span></a></li>
  658. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">https://skywalking.apache.org/blog/skywalkings-new-storage-feature-based-on-shardingsphere-proxy-mysql-sharding/</span></li>
  659. </ul>
  660. <p>&nbsp;</p>]]></description>
  661. <category>MySQL</category>
  662. <pubDate>Thu, 11 May 2023 12:20:20 +0000</pubDate>
  663. </item>
  664. <item>
  665. <title>Which is the best Proxy for Percona MySQL Operator?</title>
  666. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/252-comparisons-of-proxies-for-mysql</link>
  667. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/252-comparisons-of-proxies-for-mysql</guid>
  668. <description><![CDATA[<h2>Overview</h2>
  669. <p>HAProxy, ProxySQL, MySQL Router (AKA MySQL Proxy), in the last few years I had to answer multiple times on what proxy to use and in what scenario. When designing an architecture there are many components that need to be considered before deciding what is the best solution.</p>
  670. <p>When deciding what to pick, there are many things to consider like where the proxy needs to be, if it “just” needs to redirect the connections or if more features need to be in, like caching, filtering, or if it needs to be integrated with some MySQL embedded automation.</p>
  671. <p>Given that, there never was a single straight answer, instead an analysis needs to be done. Only after a better understanding of the environment, the needs and the evolution that the platform needs to achieve is it possible to decide what will be the better choice.</p>
  672. <p>However recently we have seen an increase in usage of MySQL on Kubernetes, especially with the adoption of Percona Operator for MySQL.<br />In this case we have a quite well define scenario that can resemble the image below:</p>
  673. <p><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-default.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/proxys_gr_comparison-default.png" alt="proxys gr comparison default" width="536" height="614" class="aligncenter wp-image-86219 size-full" style="display: block; margin-left: auto; margin-right: auto;" /></a></p>
  674. <p><span style="font-weight: 400;">In this scenario the proxies need to sit inside Pods balancing the incoming traffic from the Service LoadBalancer connecting with the active data nodes.</span></p>
  675. <p><span style="font-weight: 400;">Their role is merely to be sure that any incoming connection is redirected to nodes that are able to serve them, which include having a separation between Read/Write and Read Only traffic, separation that can be achieved, at service level, with automatic recognition or with two separate entry points.&nbsp;</span></p>
  676. <p><span style="font-weight: 400;">In this scenario it is also crucial to be efficient in resource utilization and scaling with frugality. In this context features like filtering, firewalling or caching are redundant and may consume resources that could be allocated to scaling. Those are also features that will work better outside the K8s/Operator cluster, given the closer to the application they are located, the better they will serve.&nbsp;</span></p>
  677. <p><span style="font-weight: 400;">About that we must always remember the concept that each K8s/Operator cluster needs to be seen as a single service, not as a real cluster. In short each cluster is in reality a single database with High Availability and other functionalities built in.&nbsp;</span></p>
  678. <p><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-default.png"></a></p>
  679. <p><span style="font-weight: 400;">Anyhow, we are here to talk about Proxies. Once we have defined that we have one clear mandate in mind, we need to identify which product allow our K8s/Operator solution to:</span></p>
  680. <ul>
  681. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Scale at the maximum the number of incoming connections</span></li>
  682. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Serve the request with the higher efficiency</span></li>
  683. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Consume less resources as possible</span></li>
  684. </ul>
  685. <p><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-default.png"></a></p>
  686. <h2><span style="font-weight: 400;">The Environment</span></h2>
  687. <p><span style="font-weight: 400;">To identify the above points I have simulated a possible K8s/Operator environment, creating:</span></p>
  688. <ul>
  689. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">One powerful application node, where I run sysbench read only tests, scaling from 2 to 4096 threads. (Type c5.4xlarge)</span></li>
  690. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">3 Mid data nodes with several gigabytes of data in with MySQL and Group Replication (Type m5.xlarge)</span></li>
  691. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">1 Proxy node running on a resource limited box (Type t2.micro)</span></li>
  692. </ul>
  693. <p><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-default.png"></a></p>
  694. <h2><span style="font-weight: 400;">The Tests&nbsp;</span></h2>
  695. <p><span style="font-weight: 400;">We will have very simple test cases. The first one has the scope to define the baseline, identifying the moment when we will have the first level of saturation due to the number of connections. In this case we will increase the number of connections and keep a low number of operations.&nbsp;</span></p>
  696. <p><span style="font-weight: 400;">The second test will define how well the increasing load is served inside the range we had previously identified.&nbsp;</span></p>
  697. <p><span style="font-weight: 400;">For documentation the sysbench commands are:</span></p>
  698. <p><span style="font-weight: 400;">Test1</span></p>
  699. <pre class="lang:sh decode:true">sysbench ./src/lua/windmills/oltp_read.lua  --db-driver=mysql --tables=200 --table_size=1000000
  700. --rand-type=zipfian --rand-zipfian-exp=0 --skip_trx=true  --report-interval=1 --mysql-ignore-errors=all
  701. --mysql_storage_engine=innodb --auto_inc=off --histogram  --stats_format=csv --db-ps-mode=disable --point-selects=50
  702. --reconnect=10 --range-selects=true –rate=100 --threads=&lt;#Threads from 2 to 4096&gt; --time=1200 run
  703. </pre>
  704. <p><span style="font-weight: 400;">Test2</span></p>
  705. <pre class="lang:sh decode:true">sysbench ./src/lua/windmills/oltp_read.lua  --mysql-host=&lt;host&gt; --mysql-port=&lt;port&gt; --mysql-user=&lt;user&gt;
  706. --mysql-password=&lt;pw&gt; --mysql-db=&lt;schema&gt; --db-driver=mysql --tables=200 --table_size=1000000  --rand-type=zipfian
  707. --rand-zipfian-exp=0 --skip_trx=true  --report-interval=1 --mysql-ignore-errors=all --mysql_storage_engine=innodb
  708. --auto_inc=off --histogram --table_name=&lt;tablename&gt;  --stats_format=csv --db-ps-mode=disable --point-selects=50
  709. --reconnect=10 --range-selects=true --threads=&lt;#Threads from 2 to 4096&gt; --time=1200 run
  710. </pre>
  711. <h2><span style="font-weight: 400;">Results</span></h2>
  712. <h3><span style="font-weight: 400;">Test 1</span></h3>
  713. <p><span style="font-weight: 400;">As indicated here I was looking to identify when the first Proxy will reach a dimension that would not be manageable. The load is all in creating and serving the connections, while the number of operations is capped to 100.&nbsp;</span></p>
  714. <p><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/events_rate2.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/events_rate2.png" alt="events rate2" width="600" height="401" class="alignnone size-large wp-image-86207" style="display: block; margin-left: auto; margin-right: auto;" /></a></p>
  715. <p><span style="font-weight: 400;">As you can see and as I was expecting, the three Proxies were behaving more or less the same, serving the same number of operations (they were capped so why not), until not.</span></p>
  716. <p><span style="font-weight: 400;">MySQL router after 2048 connection was not able to serve anything more.</span></p>
  717. <p><b>NOTE: MySQL Router was actually stopped working at 1024 threads, but using version 8.0.32 I enabled the feature: </b><b><i>connection_sharing</i></b><b>. That allows it to go a bit further</b><span style="font-weight: 400;">.&nbsp;&nbsp;</span></p>
  718. <p><span style="font-weight: 400;">Let us take a look also the the latency:</span></p>
  719. <p><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/latency95_rate.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/latency95_rate.png" alt="latency95 rate" width="600" height="424" class="alignnone size-large wp-image-86209" style="display: block; margin-left: auto; margin-right: auto;" /></a></p>
  720. <p><span style="font-weight: 400;">Here the situation starts to be a little bit more complicated. MySQL Router is the one that has the higher latency no matter what. However HAProxy and ProxySQL have an interesting behavior. HAProxy is performing better with a low number of connections, while ProxySQL is doing better when a high number of connections is in place.&nbsp;&nbsp;</span></p>
  721. <p><span style="font-weight: 400;">This is due to the multiplexing and the very efficient way ProxySQL uses to deal with high load.</span></p>
  722. <p><span style="font-weight: 400;">Everything has a cost:</span></p>
  723. <p><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/node-summary-cpu.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/node-summary-cpu.png" alt="node summary cpu" width="600" height="241" class="alignnone size-large wp-image-86215" style="display: block; margin-left: auto; margin-right: auto;" /></a></p>
  724. <p><span style="font-weight: 400;">HAProxy is definitely using less user’s CPU resources than ProxySQL or MySQL Router …</span></p>
  725. <p><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/node-summary-cpu-saturation.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/node-summary-cpu-saturation.png" alt="node summary cpu saturation" width="600" height="238" class="alignnone size-large wp-image-86214" style="display: block; margin-left: auto; margin-right: auto;" /></a></p>
  726. <p><span style="font-weight: 400;">.. we can also notice that HAProxy barely reaches on average the 1.5 CPU load while ProxySQL is at 2.50 and MySQL Router around 2.&nbsp;</span></p>
  727. <p><span style="font-weight: 400;">To be honest I was expecting something like this, given ProxySQL's need to handle the connections and the other basic routing. What was instead a surprise was MySQL Router, why does it have a higher load?</span></p>
  728. <h4><span style="font-weight: 400;">Brief summary</span></h4>
  729. <p><span style="font-weight: 400;">This test highlights that HAProxy and ProxySQL are able to reach a level of connection higher than the slowest runner in the game (MySQL Router). It is also clear that traffic is better served under a high number of connections by ProxySQL but it requires more resources.&nbsp;</span></p>
  730. <h2><span style="font-weight: 400;">Test 2</span></h2>
  731. <p><i><span style="font-weight: 400;">When the going gets tough, the tough gets going</span></i></p>
  732. <p><span style="font-weight: 400;">Well let us remove the </span><i><span style="font-weight: 400;">–rate</span></i><span style="font-weight: 400;"> limitation and see what will happen.&nbsp;</span></p>
  733. <p><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/events_perf.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/events_perf.png" alt="events perf" width="800" height="475" class="alignnone size-large wp-image-86205" style="display: block; margin-left: auto; margin-right: auto;" /></a></p>
  734. <p><span style="font-weight: 400;">The scenario with load changes drastically. We can see how HAProxy is able to serve the connection and allow the execution of a higher number of operations for the whole test. ProxySQL is immediately after it and is behaving quite well up to 128 threads, then it just collapses.&nbsp;</span></p>
  735. <p><span style="font-weight: 400;">MySQL Router never takes off; it always stays below the 1k reads/second while HAProxy was able to serve 8.2k and&nbsp; ProxySQL 6.6k.</span></p>
  736. <p><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/latency95_perf.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/latency95_perf.png" alt="latency95 perf" width="800" height="531" class="alignnone size-large wp-image-86208" style="display: block; margin-left: auto; margin-right: auto;" /></a></p>
  737. <p><span style="font-weight: 400;">Taking a look at the Latency, we can see that HAProxy had a gradual increase as expected, while ProxySQL and MySQL Router just went up from the 256 threads on.&nbsp;</span></p>
  738. <p><span style="font-weight: 400;">To observe that both ProxySQL and MySQL Router were not able to complete the tests with 4096 threads.</span></p>
  739. <p><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/node-summary-cpu-perf.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/node-summary-cpu-perf.png" alt="node summary cpu perf" width="800" height="318" class="alignnone size-large wp-image-86212" style="display: block; margin-left: auto; margin-right: auto;" /></a></p>
  740. <p><span style="font-weight: 400;">Why? HAProxy stays always below 50% cpu, no matter the increasing number of threads/connections, scaling the load very efficiently. MySQL router was almost immediately reaching the saturation point, being affected not only by the number of threads/connections but also by the number of operations, and that was not expected given we do not have a level 7 capability in MySQL Router.</span></p>
  741. <p><span style="font-weight: 400;">Finally ProxySQL, which was working fine up to a certain limit, then it reached saturation point and was not able to serve the load. I am saying load because ProxySQL is a level 7 proxy and is aware of the content of the load. Given that on top of multiplexing, additional resource consumption was expected.&nbsp;&nbsp;&nbsp;</span></p>
  742. <p><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/node-summary-cpu-saturation-perf.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/node-summary-cpu-saturation-perf.png" alt="node summary cpu saturation perf" width="800" height="319" class="alignnone size-large wp-image-86213" style="display: block; margin-left: auto; margin-right: auto;" /></a></p>
  743. <p><span style="font-weight: 400;">Here we just have a clear confirmation of what already said above, with 100% cpu utilization reached by MySQL Router with just 16 threads, and ProxySQL way after at 256 threads.</span></p>
  744. <h2><span style="font-weight: 400;">Brief Summary</span></h2>
  745. <p><span style="font-weight: 400;">HAProxy comes up as the champion in this test, there is no doubt that it was able to scale the increasing load in connection without being affected significantly by the load generated by the requests. The lower consumption in resources also indicates the possible space for even more scaling.</span></p>
  746. <p><span style="font-weight: 400;">ProxySQL was penalized by the limited resources, but this was the game, we have to get the most out of the few available. This test indicates that it is not optimal to use ProxySQL inside the Operator, actually it is a wrong choice if low resource and scalability is a must.&nbsp;&nbsp;&nbsp;&nbsp;</span></p>
  747. <p><span style="font-weight: 400;">MySQL Router was never in the game. Unless a serious refactoring, MySQL Router is designed for very limited scalability, as such the only way to adopt it is to have many of them at application node level. Utilizing it close to the data nodes in a centralized position is a mistake.&nbsp;&nbsp;</span></p>
  748. <h2><span style="font-weight: 400;">Conclusions</span></h2>
  749. <p><span style="font-weight: 400;">I started showing an image on how the MySQL service is organized and want to close showing the variation that for me is the one to be considered the default approach:</span></p>
  750. <p><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-haproxy.jpg"><img src="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-haproxy.jpg" alt="" width="536" height="651" class="size-full wp-image-86220 aligncenter" style="display: block; margin-left: auto; margin-right: auto;" /></a></p>
  751. <p><span style="font-weight: 400;">This to highlight that we always need to choose the right tool for the job.&nbsp;</span></p>
  752. <p><span style="font-weight: 400;">The Proxy in architectures involving MySQL/PS/PXC is a crucial element for the scalability of the cluster, no matter if using K8s or not. It is important to choose the one that serves us better, which in some cases can be ProxySQL over HAProxy.&nbsp;</span><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-haproxy.jpg"><br /></a></p>
  753. <p><span style="font-weight: 400;">However when talking of K8s and Operators we must recognize the need to optimize the resources usage for the specific service. In that context there is no discussion about it, HAProxy is the best solution and the one we should go to.&nbsp;</span><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-haproxy.jpg"><br /></a></p>
  754. <p><span style="font-weight: 400;">My final observation is about MySQL Router (aka MySQL Proxy).&nbsp;</span></p>
  755. <p><span style="font-weight: 400;">Unless a significant refactoring of the product at the moment it is not even close to what the other two can do. From the tests done so far, it requires a complete reshaping starting to identify why it is so subject to the load coming from the query, more than the load coming from the connections.&nbsp;&nbsp;&nbsp;</span></p>
  756. <p><a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-haproxy.jpg"></a></p>
  757. <p><span style="font-weight: 400;">Great MySQL to everyone.&nbsp;</span></p>
  758. <h2>References</h2>
  759. <p>https://www.percona.com/blog/boosting-percona-distribution-for-mysql-operator-efficiency/</p>
  760. <p>https://www.slideshare.net/marcotusa/my-sql-on-kubernetes-demystified</p>
  761. <p>https://docs.haproxy.org/2.7/configuration.html</p>
  762. <p>https://proxysql.com/documentation/</p>
  763. <p>https://dev.mysql.com/doc/mysql-router/8.0/en/</p>
  764. <p>&nbsp;</p>
  765. <p>&nbsp;</p>]]></description>
  766. <category>MySQL</category>
  767. <pubDate>Sun, 19 Mar 2023 17:57:59 +0000</pubDate>
  768. </item>
  769. <item>
  770. <title>Help! I am out of disk space!</title>
  771. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/251-help-i-am-out-of-disk-space</link>
  772. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/251-help-i-am-out-of-disk-space</guid>
  773. <description><![CDATA[<p><em><span style="font-weight: 400;">How we could&nbsp; fix a nasty out of space issue leveraging the flexibility of Percona MySQL operator (PMO)&nbsp;&nbsp;<img src="http://www.tusacentral.net/joomla/images/stories/DiskSpaceFull.png" alt="DiskSpaceFull" width="350" height="208" style="float: right;" /></span></em></p>
  774. <p><span style="font-weight: 400;">When planning a database deployment, one of the most challenging factors to consider is the amount of space we need to dedicate for Data on disk.</span></p>
  775. <p><span style="font-weight: 400;">This is even more cumbersome when working on bare metal. Given it is definitely more difficult to add space when using this kind of solution in respect to the cloud.&nbsp;</span></p>
  776. <p><span style="font-weight: 400;">This is it, when using cloud storage like EBS or similar it is normally easy(er) to extend volumes, which gives us the luxury to plan the space to allocate for data with a good grade of relaxation.&nbsp;</span></p>
  777. <p><span style="font-weight: 400;">Is this also true when using a solution based on Kubernetes like Percona Operator for MySQL? Well it depends on where you run it, however if the platform you choose supports the option to extend volumes K8s per se is giving you the possibility to do so as well.</span></p>
  778. <p><span style="font-weight: 400;">However, if it can go wrong it will, and ending up with a fully filled device with MySQL is not a fun experience.&nbsp;</span></p>
  779. <p><span style="font-weight: 400;">As you know, on normal deployments, when mysql has no space left on the device, it simply stops working, ergo it will cause a production down event, which of course is an unfortunate event that we want to avoid at any cost.&nbsp;&nbsp;</span></p>
  780. <p><span style="font-weight: 400;">This blog is the story of what happened, what was supposed to happen and why.&nbsp;</span></p>
  781. <h2><span style="font-weight: 400;">The story&nbsp;</span></h2>
  782. <p><span style="font-weight: 400;">Case was on AWS using EKS.</span></p>
  783. <p><span style="font-weight: 400;">Given all the above, I was quite surprised when we had a case in which a deployed solution based on PMO went out of space. However we start to dig and review what was going on and why.</span></p>
  784. <p><span style="font-weight: 400;">The first thing we did was to quickly investigate what was really taking space, that could have been an easy win if most of the space was taken by some log, but unfortunately this was not the case, data was really taking all the available space.&nbsp;</span></p>
  785. <p><span style="font-weight: 400;">The next step was to check what storage class was used for the PVC</span></p>
  786. <pre class="lang:sh decode:true">k get pvc
  787. NAME                         VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS
  788. datadir-mt-cluster-1-pxc-0   pvc-&lt;snip&gt;   233Gi      RWO            io1
  789. datadir-mt-cluster-1-pxc-1   pvc-&lt;snip&gt;   233Gi      RWO            io1
  790. datadir-mt-cluster-1-pxc-2   pvc-&lt;snip&gt;   233Gi      RWO            io1
  791. </pre>
  792. <p><span style="font-weight: 400;">Ok we use the io1 SC, it is now time to check if the SC is supporting volume expansion:</span></p>
  793. <pre class="lang:sh decode:true">kubectl describe sc io1
  794. Name:            io1
  795. IsDefaultClass:  No
  796. Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"},"name":"io1"},"parameters":{"fsType":"ext4","iopsPerGB":"12","type":"io1"},"provisioner":"kubernetes.io/aws-ebs"}
  797. ,storageclass.kubernetes.io/is-default-class=false
  798. Provisioner:           kubernetes.io/aws-ebs
  799. Parameters:            fsType=ext4,iopsPerGB=12,type=io1
  800. AllowVolumeExpansion:  &lt;unset&gt; &lt;------------
  801. MountOptions:          &lt;none&gt;
  802. ReclaimPolicy:         Delete
  803. VolumeBindingMode:     Immediate
  804. Events:                &lt;none&gt;
  805. </pre>
  806. <p><span style="font-weight: 400;">And no is not enabled, in this case we cannot just go and expand the volume, must change the storage class settings first. </span><span style="font-weight: 400;"><br /></span><span style="font-weight: 400;">To enable volume expansion, you need to delete the storage class and enable it again.&nbsp;</span></p>
  807. <p><span style="font-weight: 400;">Unfortunately we were unsuccessful in doing that operation, because the storage class kept staying as unset for&nbsp; ALLOWVOLUMEEXPANSION.&nbsp;</span></p>
  808. <p><span style="font-weight: 400;">As said this is a production down event, so we cannot invest too much time in digging why it was not correctly changing the mode, we had to act quickly.&nbsp;</span></p>
  809. <p><span style="font-weight: 400;">The only option we had to fix it was:</span></p>
  810. <ul>
  811. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Expand the io1 volumes from AWS console (or aws client)</span></li>
  812. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Resize the file system&nbsp;</span></li>
  813. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Patch any K8 file to allow K8 to correctly see the new volumes dimension&nbsp;&nbsp;</span></li>
  814. </ul>
  815. <p><span style="font-weight: 400;">Expanding EBS volumes from the console is trivial, just go to Volumes, select the volume you want to modify, choose modify and change the size of it with the one desired, done.&nbsp;</span></p>
  816. <p><span style="font-weight: 400;">Once that is done connect to the Node hosting the pod which has the volume mounted like:</span></p>
  817. <pre class="lang:sh decode:true"> k get pods -o wide|grep mysql-0
  818. NAME                                        READY     STATUS    RESTARTS   AGE    IP            NODE            
  819. cluster-1-pxc-0                               2/2     Running   1          11d    10.1.76.189     &lt;mynode&gt;.eu-central-1.compute.internal</pre>
  820. <p><span style="font-weight: 400;">Then we need to get the id of the pvc to identify it on the node</span></p>
  821. <pre class="lang:sh decode:true">k get pvc
  822. NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
  823. datadir-cluster-1-pxc-0   Bound    pvc-1678c7ee-3e50-4329-a5d8-25cd188dc0df   233Gi      RWO            io1
  824. </pre>
  825. <p><span style="font-weight: 400;">One note, when doing this kind of recovery with a PXC based solution, always recover node-0 first, then the others.&nbsp;&nbsp;</span></p>
  826. <p><span style="font-weight: 400;">So we connect to &lt;mynode&gt; and identify the volume:&nbsp;</span></p>
  827. <pre class="lang:sh decode:true">lslbk |grep pvc-1678c7ee-3e50-4329-a5d8-25cd188dc0df
  828. nvme1n1      259:4    0  350G  0 disk /var/lib/kubelet/pods/9724a0f6-fb79-4e6b-be8d-b797062bf716/volumes/kubernetes.io~aws-ebs/pvc-1678c7ee-3e50-4329-a5d8-25cd188dc0df &lt;-----
  829. </pre>
  830. <p><span style="font-weight: 400;">At this point we can resize it:</span></p>
  831. <pre class="lang:sh decode:true">root@ip-&lt;snip&gt;:/# resize2fs  /dev/nvme1n1
  832. resize2fs 1.45.5 (07-Jan-2020)
  833. Filesystem at /dev/nvme1n1 is mounted on /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-central-1a/vol-0ab0db8ecf0293b2f; on-line resizing required
  834. old_desc_blocks = 30, new_desc_blocks = 44
  835. The filesystem on /dev/nvme1n1 is now 91750400 (4k) blocks long.
  836. </pre>
  837. <p><span style="font-weight: 400;">The good thing is that as soon as you do that the MySQL daemon see the space and will restart, however it will happen only on the current pod and K8 will still see the old dimension:</span></p>
  838. <pre class="lang:sh decode:true">k get pv
  839. NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                            STORAGECLASS   REASON
  840. pvc-1678c7ee-3e50-4329-a5d8-25cd188dc0df   333Gi      RWO            Delete           Bound    pxc/datadir-cluster-1-pxc-0   io1
  841. </pre>
  842. <p><span style="font-weight: 400;">To allow k8 to be align with the real dimension we must patch the information stored, the command is the following:</span></p>
  843. <pre class="lang:sh decode:true">kubectl patch pvc &lt;pvc-name&gt;  -n &lt;pvc-namespace&gt; -p '{ "spec": { "resources": { "requests": { "storage": "NEW STORAGE VALUE" }}}}'
  844. Ie:
  845. kubectl patch pvc datadir-cluster-1-pxc-0 -n pxc -p '{ "spec": { "resources": { "requests": { "storage": "350" }}}}'
  846. </pre>
  847. <p><span style="font-weight: 400;">Remember to use as pvc-name the NAME coming from </span></p>
  848. <pre><span style="font-weight: 400;">kubectl get pvc</span><span style="font-weight: 400;">.</span></pre>
  849. <p><span style="font-weight: 400;">Once this is done k8 will see the new volume dimension correctly.</span></p>
  850. <p><span style="font-weight: 400;">Just repeat the process for Node-1 and Node-2 and …done the cluster is up again.</span></p>
  851. <p><span style="font-weight: 400;">Finally do not forget to modify your custom resources file (cr.yaml) to match the new volume size. IE:</span></p>
  852. <pre class="lang:sh decode:true">    volumeSpec:
  853.      persistentVolumeClaim:
  854.        storageClassName: "io1"
  855.        resources:
  856.          requests:
  857.            storage: 350G
  858. </pre>
  859. <p><span style="font-weight: 400;">The whole process took just a few minutes, it was time now to investigate why the incident happened and why the storage class was not allowing extension in the first place.&nbsp;&nbsp;</span></p>
  860. <p>&nbsp;</p>
  861. <h2><span style="font-weight: 400;">Why it happened</span></h2>
  862. <p><span style="font-weight: 400;">Well first and foremost the platform was not correctly monitored. As such there was lack of visibility about the space utilization, and no alert about disk space.&nbsp;</span></p>
  863. <p><span style="font-weight: 400;">This was easy to solve just enabling the PMM feature in the cluster cr and set the alert in PMM once the nodes join it (see </span><a href="https://docs.percona.com/percona-monitoring-and-management/get-started/alerting.html"><span style="font-weight: 400;">https://docs.percona.com/percona-monitoring-and-management/get-started/alerting.html</span></a><span style="font-weight: 400;"> for details on how to).</span></p>
  864. <p><span style="font-weight: 400;">The second issue was the problem with the storage class. Once we had the time to carefully review the configuration files, we identified that there was an additional tab in the SC class, which was causing k8 to ignore the directive.&nbsp;</span></p>
  865. <p><span style="font-weight: 400;">Was suppose to be:</span></p>
  866. <pre class="lang:sh decode:true">kind: StorageClass
  867. apiVersion: storage.k8s.io/v1
  868. metadata:
  869.  name: io1
  870.  annotations:
  871.    storageclass.kubernetes.io/is-default-class: "false"
  872. provisioner: kubernetes.io/aws-ebs
  873. parameters:
  874.  type: io1
  875.  iopsPerGB: "12"
  876.  fsType: ext4
  877. allowVolumeExpansion: true &lt;----------
  878.  
  879. It was:
  880. kind: StorageClass
  881. apiVersion: storage.k8s.io/v1
  882. metadata:
  883.  name: io1
  884.  annotations:
  885.    storageclass.kubernetes.io/is-default-class: "false"
  886. provisioner: kubernetes.io/aws-ebs
  887. parameters:
  888.  type: io1
  889.  iopsPerGB: "12"
  890.  fsType: ext4
  891.  allowVolumeExpansion: true. &lt;---------</pre>
  892. <p><span style="font-weight: 400;">What was concerning was the lack of error returned by the Kubernetes API, so in theory the configuration was accepted but not really validated.&nbsp;</span></p>
  893. <p><span style="font-weight: 400;">In any case once we had fix the typo and recreated the SC, the setting for volume expansion was correctly accepted:</span></p>
  894. <pre class="lang:sh decode:true">kubectl describe sc io1
  895. Name:            io1
  896. IsDefaultClass:  No
  897. Annotations:     kubectl.kubernetes.io/last-applied-configuration={"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"},"name":"io1"},"parameters":{"fsType":"ext4","iopsPerGB":"12","type":"io1"},"provisioner":"kubernetes.io/aws-ebs"}
  898. ,storageclass.kubernetes.io/is-default-class=false
  899. Provisioner:           kubernetes.io/aws-ebs
  900. Parameters:            fsType=ext4,iopsPerGB=12,type=io1
  901. AllowVolumeExpansion:  True    
  902. MountOptions:          &lt;none&gt;
  903. ReclaimPolicy:         Delete
  904. VolumeBindingMode:     Immediate
  905. Events:                &lt;none&gt;
  906. </pre>
  907. <h2><span style="font-weight: 400;">What should have happened instead?</span></h2>
  908. <p><span style="font-weight: 400;">If a proper monitoring and alerting was in place, the administrators would have the time to act and extend the volumes without downtime.&nbsp;</span></p>
  909. <p><span style="font-weight: 400;">However, the procedure for extending volumes on K8 is not complex but also not as straightforward as you may think. My colleague Natalia Marukovich wrote this blog post (</span><a href="https://www.percona.com/blog/percona-operator-volume-expansion-without-downtime/"><span style="font-weight: 400;">https://www.percona.com/blog/percona-operator-volume-expansion-without-downtime/</span></a><span style="font-weight: 400;">)&nbsp; that gives you the step by step instructions on how to extend the volumes without downtime.&nbsp;</span></p>
  910. <h2><span style="font-weight: 400;">Conclusions</span></h2>
  911. <p><span style="font-weight: 400;">Using the cloud, containers, automation or more complex orchestrators like Kubernetes, do not solve all, do not prevent mistakes from happening, and more importantly do not make the right decisions for you.&nbsp;</span></p>
  912. <p><span style="font-weight: 400;">You must set up a proper architecture that includes backup, monitoring and alerting. You must set the right alerts and act on them in time.&nbsp;</span></p>
  913. <p><span style="font-weight: 400;">Finally automation is cool, however the devil is in the details and typos are his day to day joy. Be careful and check what you put online, do not rush it. Validate, validate validate…&nbsp;</span></p>
  914. <p><span style="font-weight: 400;">Great stateful MySQL to all.&nbsp;</span></p>]]></description>
  915. <category>MySQL</category>
  916. <pubDate>Thu, 19 Jan 2023 14:11:24 +0000</pubDate>
  917. </item>
  918. <item>
  919. <title>MySQL Dual password how to manage them programmatically</title>
  920. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/250-mysql-dual-password-how-to-manage-them-programmatically</link>
  921. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/250-mysql-dual-password-how-to-manage-them-programmatically</guid>
  922. <description><![CDATA[<p><span style="font-weight: 400;">What is dual password in MYSQL and how it works was already covered by my colleague Brian Sumpter here (</span><a href="https://www.percona.com/blog/using-mysql-8-dual-passwords/"><span style="font-weight: 400;">https://www.percona.com/blog/using-mysql-8-dual-passwords/</span></a><span style="font-weight: 400;">).&nbsp;<img src="http://www.tusacentral.net/joomla/images/stories/rpa-cognitive-blog-img.jpg" alt="rpa cognitive blog img" width="400" height="215" style="float: right;" /></span></p>
  923. <p><span style="font-weight: 400;">However let me do a brief recap here about it.</span></p>
  924. <p><span style="font-weight: 400;">Dual password is the MySQL mechanism that allows you to keep two passwords active at the same time. This feature is part of a more extended set of Password management features implemented in MySQL 8 to enforce better security and secrets management, like:</span></p>
  925. <ul>
  926. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Internal Versus External Credentials Storage</span></li>
  927. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Password Expiration Policy</span></li>
  928. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Password Reuse Policy</span></li>
  929. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Password Verification-Required Policy</span></li>
  930. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Dual Password Support</span></li>
  931. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Random Password Generation</span></li>
  932. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Failed-Login Tracking and Temporary Account Locking</span></li>
  933. </ul>
  934. <p><span style="font-weight: 400;">The most important and requested features are the password expiration and verification policy. The problem in implementing them is the complexity of replacing passwords for accounts on very large platforms, like with thousands of applications and hundreds of MySQL servers.&nbsp;</span></p>
  935. <p><span style="font-weight: 400;">In fact, while for a single user it is not so complex to change his own password when requested at login, for an application using thousands of sub-services it may require some time. The problem in performing the password change is that while executing the modification some services will have the updated password while others will still use the old one. Without Dual Password a segment of nodes will receive error messages in connecting creating service disruption.&nbsp;</span></p>
  936. <p><span style="font-weight: 400;"><span style="font-weight: 400;">Now let us cover this blog topic. </span></span></p>
  937. <p><span style="font-weight: 400;">With Dual Password it is instead possible to declare a new password keeping the old still active until the whole upgrade has been completed.&nbsp;</span></p>
  938. <p><span style="font-weight: 400;">This highlight two very important aspects:</span></p>
  939. <ul>
  940. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">When automating the password update, it is better to not use a password expiration policy, but base the expiration on the completion of the new password deployment.</span></li>
  941. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">We need to be sure the account we are changing the password to keeps the password active until we need it, and that is correctly removed when done.&nbsp;</span></li>
  942. </ul>
  943. <p><span style="font-weight: 400;">As you see I am focusing on the cases when we have automation and not the single interactive user update.&nbsp;</span></p>
  944. <p><span style="font-weight: 400;">How dual password works</span></p>
  945. <p><span style="font-weight: 400;">Let us assume we have create a user like:</span></p>
  946. <pre class="lang:mysql decode:true">create user dualtest@'192.168.4.%' identified by 'password1';
  947. grant all on test.* to dualtest@'192.168.4.%';
  948. </pre>
  949. <p><span style="font-weight: 400;">This will generate an entry in MySQL mysql.user table as:</span></p>
  950. <pre class="lang:mysql decode:true">(root@localhost) [(none)]&gt;select user,host, plugin, authentication_string, password_last_changed,User_attributes from mysql.user where user = 'dualtest'\G
  951. *************************** 1. row ***************************
  952.                 user: dualtest
  953.                 host: 192.168.4.%
  954.               plugin: mysql_native_password
  955. authentication_string: *668425423DB5193AF921380129F465A6425216D0
  956. password_last_changed: 2022-11-17 08:31:37
  957.      User_attributes: NULL
  958. 1 row in set (0.00 sec)
  959. </pre>
  960. <p><span style="font-weight: 400;">At this point our user will be able to connect from any application located in the correct network and act on the </span><i><span style="font-weight: 400;">test</span></i><span style="font-weight: 400;"> schema.&nbsp;</span></p>
  961. <p><span style="font-weight: 400;">After some time, you as application owner,will be notified by your DBA team that the user </span><i><span style="font-weight: 400;">dualtest</span></i><span style="font-weight: 400;"> is required to change the password in order to respect the security constraints.</span></p>
  962. <p><span style="font-weight: 400;">At this point there are two options:</span></p>
  963. <ol>
  964. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">You have privileges&nbsp; to use Dual Password (the required dynamic privilege to use dual Password is </span><b>APPLICATION PASSWORD ADMIN</b><span style="font-weight: 400;">).</span></li>
  965. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">You do not have the right privileges.</span></li>
  966. </ol>
  967. <p><span style="font-weight: 400;">In case 2 your DBA team must perform the change for you, and then they will let you know the new password.</span></p>
  968. <p><span style="font-weight: 400;">In case 1 you can do the operation yourself.&nbsp;</span></p>
  969. <p><span style="font-weight: 400;">In the last case what you will do is:</span><span style="font-weight: 400;"><br /></span></p>
  970. <pre class="lang:mysql decode:true crayon-selected">ALTER USER 'dualtest'@'192.168.4.%' IDENTIFIED BY 'password2' RETAIN CURRENT PASSWORD;
  971. </pre>
  972. <p><span style="font-weight: 400;">Then check it is done properly:</span></p>
  973. <pre class="lang:mysql decode:true">select user,host, plugin, authentication_string, password_last_changed,User_attributes from mysql.user where user ='dualtest' order by 1,2\G
  974. *************************** 1. row ***************************
  975.                 user: dualtest
  976.                 host: 192.168.4.%
  977.               plugin: mysql_native_password
  978. authentication_string: *DC52755F3C09F5923046BD42AFA76BD1D80DF2E9
  979. password_last_changed: 2022-11-17 08:46:28
  980.      User_attributes: {"additional_password": "*668425423DB5193AF921380129F465A6425216D0"}
  981. 1 row in set (0.00 sec)
  982. </pre>
  983. <p><span style="font-weight: 400;">As you can see here the OLD password has been moved to the </span><i><span style="font-weight: 400;">User_attributes</span></i><span style="font-weight: 400;"> JSON field that is used in MYSQL8 to store several values.&nbsp;</span></p>
  984. <p><span style="font-weight: 400;">At this point you can rollout safely the password change and that change can take an hour or a week, no production impact given the applications will be able to use either of them.&nbsp;</span></p>
  985. <p><span style="font-weight: 400;">Once the process is complete, you can ask your DBA team to remove OLD password, or do:</span></p>
  986. <pre class="lang:mysql decode:true">ALTER USER 'dualtest'@'192.168.4.%' DISCARD OLD PASSWORD;</pre>
  987. <p><span style="font-weight: 400;">Then check if the password has being removed properly:</span></p>
  988. <pre class="lang:mysql decode:true">(root@localhost) [(none)]&gt;select user,host, plugin, authentication_string, password_last_changed,User_attributes from mysql.user where user ='dualtest' order by 1,2\G
  989. *************************** 1. row ***************************
  990.                 user: dualtest
  991.                 host: 192.168.4.%
  992.               plugin: mysql_native_password
  993. authentication_string: *DC52755F3C09F5923046BD42AFA76BD1D80DF2E9
  994. password_last_changed: 2022-11-17 08:46:28
  995.      User_attributes: NULL
  996. 1 row in set (0.00 sec)
  997. </pre>
  998. <p><span style="font-weight: 400;">If all is clean the process can be considered complete.&nbsp;</span></p>
  999. <p><span style="font-weight: 400;">Of course all this should be automated and executed by code not by hand high level it should be more or less like this:</span></p>
  1000. <pre class="lang:sh decode:true">{}Input new password  
  1001. - Check for additional_password in User_attributes in mysql.user
  1002. &lt;&gt; If no value you can proceed otherwise exit (another change is in place)
  1003. - Read and store authentication_string for the user you need to change password
  1004. - Change current password with: Alter user ... RETAIN CURRENT PASSWORD
  1005. - Check for additional_password in User_attributes in mysql.user
  1006. &lt;&gt; If value is present and match the password stored then you can proceed otherwise exit given there is an error in Dual Password or the passwords are different
  1007. - Run update on all application nodes, and verify new password on each application node
  1008. &lt;&gt; At regular interval check the number of completed changes and check the additional_password in User_attributes in mysql.user to be sure it is still there
  1009. [] When all application nodes are up to date
  1010. &lt;&gt; If verification is successful 100%
  1011. - Remove OLD password with: ALTER USER ... DISCARD OLD PASSWORD;
  1012. - Check for additional_password in User_attributes in mysql.user
  1013. &lt;&gt; If no value is present close with OK otherwise report Error for password not removed
  1014. () complete
  1015. </pre>
  1016. <h2><span style="font-weight: 400;">Conclusion</span></h2>
  1017. <p><span style="font-weight: 400;">As also Brian mentioned, those are the small things that could make the difference when in large deployments and enterprise environments. Security is a topic that very often is underestimated in small companies or start-ups, but that is wrong, security operations like password rotation are crucial for your safety.&nbsp;</span></p>
  1018. <p><span style="font-weight: 400;">It is nice to see that MySQL is finally adopting simple but effective steps to help DBAs to implement proper procedures without causing production impact and without the need to become too creative.&nbsp;</span></p>
  1019. <h2>&nbsp;</h2>
  1020. <h2><span style="font-weight: 400;">References&nbsp;</span></h2>
  1021. <p><a href="https://www.percona.com/blog/using-mysql-8-dual-passwords/"><span style="font-weight: 400;">https://www.percona.com/blog/using-mysql-8-dual-passwords/</span></a></p>
  1022. <p><a href="https://dev.mysql.com/doc/refman/8.0/en/password-management.html#dual-passwords"><span style="font-weight: 400;">https://dev.mysql.com/doc/refman/8.0/en/password-management.html#dual-passwords</span></a></p>]]></description>
  1023. <category>MySQL</category>
  1024. <pubDate>Thu, 17 Nov 2022 16:26:45 +0000</pubDate>
  1025. </item>
  1026. <item>
  1027. <title>ProxySQL support for MySQL caching_sha2_password</title>
  1028. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/249-proxysql-support-for-mysql-caching-sha2-password</link>
  1029. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/249-proxysql-support-for-mysql-caching-sha2-password</guid>
  1030. <description><![CDATA[<p><span style="font-weight: 400;">In our time, every day we use dozens if not hundreds of applications connecting to some kind of data repository. This simple step is normally executed over the network and given&nbsp;so, it is subject to possible sniffing with all the possible related consequences.<img src="http://www.tusacentral.net/joomla/images/stories/proxysql/brokenlock.png" alt="brokenlock" style="float: right;" /><br style="clear: left;" /></span></p>
  1031. <p><span style="font-weight: 400;">Given that it is normally better to protect your connection using data encryption like SSL, or at the minimum, make the information you pass to connect less easy to be intercepted.&nbsp;</span></p>
  1032. <p><span style="font-weight: 400;">At the same time it is best practice to not store connection credential in clear text, not even inside a table in your database. Doing that is the equivalent of writing your password over a sticky note on your desk. Not a good idea.</span></p>
  1033. <p><span style="font-weight: 400;">The main options are instead in either transforming the passwords to be less identifiable like hashing or to store the information in an external centralized vault.&nbsp;</span></p>
  1034. <p><span style="font-weight: 400;">In MySQL the passwords are transformed in order to not be clear text, and several different plugins are used to authenticate the user. From version 8 MySQL uses </span><i><span style="font-weight: 400;">caching_sha2_password</span></i><span style="font-weight: 400;"> as default authentication plugin. The </span><i><span style="font-weight: 400;">caching_sha2_password</span></i><span style="font-weight: 400;"> and </span><i><span style="font-weight: 400;">sha256_password</span></i><span style="font-weight: 400;"> authentication plugins provide more secure password encryption than the </span><i><span style="font-weight: 400;">mysql_native_password</span></i><span style="font-weight: 400;"> plugin, and </span><i><span style="font-weight: 400;">caching_sha2_password</span></i><span style="font-weight: 400;"> provides better performance than </span><i><span style="font-weight: 400;">sha256_password</span></i><span style="font-weight: 400;">. Due to these superior security and performance characteristics of </span><i><span style="font-weight: 400;">caching_sha2_password</span></i><span style="font-weight: 400;">, it is as of MySQL 8.0 the preferred authentication plugin, and is also the default authentication plugin rather than mysql_native_password.</span></p>
  1035. <p><span style="font-weight: 400;">In this regard recently I got the same question again “Can we use ProxySQL with MySQL 8 authorization mechanism?”, and I decided it was time to write this short blog post.</span></p>
  1036. <p><span style="font-weight: 400;">The short answer is “Yes you can”, however do not expect to have full </span><i><span style="font-weight: 400;">caching_sha2_password</span></i><span style="font-weight: 400;"> support.</span></p>
  1037. <p><span style="font-weight: 400;">This is because ProxySQL does not fully support the </span><i><span style="font-weight: 400;">caching_sha2_password </span></i><span style="font-weight: 400;">mechanism internally and given that a “trick” must be used.&nbsp;</span></p>
  1038. <p><span style="font-weight: 400;">So, what should we do when using MySQL 8 and ProxySQL?&nbsp;</span></p>
  1039. <p><span style="font-weight: 400;">In the text below we will see what can be done to continue to use ProxySQL with MySQL and Percona server 8.&nbsp;</span></p>
  1040. <p><span style="font-weight: 400;">Note that I have used the Percona </span><i><span style="font-weight: 400;">proxysql_admin</span></i><span style="font-weight: 400;"> tool to manage the users except in the last case. <br />Percona </span><i><span style="font-weight: 400;">proxysql_admin</span></i><span style="font-weight: 400;"> tool is a nice tool that helps you to manage ProxySQL and in regard to user it also manage and synchronize users from your Percona or MySQL&nbsp;&nbsp;</span></p>
  1041. <p><span style="font-weight: 400;">In the following examples:</span></p>
  1042. <p><span style="font-weight: 400;">Proxysql is on 192.168.4.191</span></p>
  1043. <p><span style="font-weight: 400;">User name/password is msandbox/msandbox</span></p>
  1044. <h2><span style="font-weight: 400;">Using hashing.</span></h2>
  1045. <p><span style="font-weight: 400;">By default MySQL comes with </span><i><span style="font-weight: 400;">caching_sha2_password</span></i><span style="font-weight: 400;"> as such if I create a user names </span><i><span style="font-weight: 400;">msandbox</span></i><span style="font-weight: 400;"> I will have:</span><span style="font-weight: 400;"><br /></span></p>
  1046. <pre class="lang:mysql decode:true">DC1-1(root@localhost) [(none)]&gt;select user,host, authentication_string,plugin from mysql.user order by 1,2;
  1047. +----------------------------+--------------------+------------------------------------------------------------------------+-----------------------+
  1048. | user                       | host               | authentication_string                                                  | plugin                |
  1049. +----------------------------+--------------------+------------------------------------------------------------------------+-----------------------+
  1050. | msandbox                   | %                  | $A$005$Z[z@l'O%[Q5t^ EKJDgxjWXJjDpDEUv91oL7Hoh/0NydTeCzpV.aI06C9.      | caching_sha2_password |      &lt;---- this user    
  1051. +----------------------------+--------------------+------------------------------------------------------------------------+-----------------------+
  1052. </pre>
  1053. <p><span style="font-weight: 400;">Then I use percona_scheduler_admin to sync the users:</span></p>
  1054. <pre class="lang:mysql decode:true">./percona-scheduler-admin --config-file=config.toml --syncusers&nbsp;
  1055. Syncing user accounts from PXC(192.168.4.205:3306) to ProxySQL
  1056. Removing existing user from ProxySQL: msandbox
  1057. Adding user to ProxySQL: msandbox
  1058.  
  1059. Synced PXC users to the ProxySQL database!
  1060.  
  1061. mysql&gt; select * from mysql_users ;
  1062. +------------+------------------------------------------------------------------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+-----------------------------+
  1063. | username &nbsp; | password &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | active | use_ssl | default_hostgroup | default_schema | schema_locked | transaction_persistent | fast_forward | backend | frontend | max_connections | attributes | comment &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; |
  1064. +------------+------------------------------------------------------------------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+-----------------------------+
  1065. | msandbox &nbsp; | $A$005$Z[z@l'O%[Q5t^ EKJDgxjWXJjDpDEUv91oL7Hoh/0NydTeCzpV.aI06C9       | 1&nbsp; &nbsp; &nbsp; | 0 &nbsp; &nbsp; &nbsp; | 100 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | NULL &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | 0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | 1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | 0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | 1 &nbsp; &nbsp; &nbsp; | 1&nbsp; &nbsp; &nbsp; &nbsp; | 10000 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; |&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; |
  1066. +------------+------------------------------------------------------------------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+-----------------------------+</pre>
  1067. <p><span style="font-weight: 400;">And set the query rules:</span></p>
  1068. <pre class="lang:mysql decode:true">insert into mysql_query_rules (rule_id,proxy_port,username,destination_hostgroup,active,retries,match_digest,apply) values(1048,6033,'msandbox',100,1,3,'^SELECT.*FOR UPDATE',1);
  1069. insert into mysql_query_rules (rule_id,proxy_port,username,destination_hostgroup,active,retries,match_digest,apply) values(1050,6033,'msandbox',101,1,3,'^SELECT.*$',1);
  1070.  
  1071. load mysql query rules to run;save mysql query rules to disk;
  1072. </pre>
  1073. <p><span style="font-weight: 400;">Now I try to connect passing by ProxySQL:</span></p>
  1074. <pre class="lang:sh decode:true"># mysql -h 192.168.4.191 -P6033  -umsandbox -pmsandbox
  1075. ERROR 1045 (28000): ProxySQL Error: Access denied for user 'msandbox'@'192.168.4.191' (using password: YES)
  1076. </pre>
  1077. <p><span style="font-weight: 400;">My account will fail to connect given failed authentication.</span></p>
  1078. <p><span style="font-weight: 400;">To fix this I need to drop the user and recreate it with a different authentication plugin in my MySQL server:</span></p>
  1079. <pre class="lang:mysql decode:true">drop user msandbox@'%';
  1080. create user 'msandbox'@'%' identified with mysql_native_password  BY 'msandbox';
  1081. grant select on *.* to 'msandbox'@'%';
  1082.  
  1083. select user,host, authentication_string,plugin from mysql.user order by 1,2;
  1084. +----------+--------------------+-------------------------------------------+-----------------------+
  1085. | user     | host               | authentication_string                     | plugin                |
  1086. +----------+--------------------+-------------------------------------------+-----------------------+
  1087. | msandbox | %                  | *6C387FC3893DBA1E3BA155E74754DA6682D04747 | mysql_native_password |
  1088. +----------+--------------------+-------------------------------------------+-----------------------+
  1089. </pre>
  1090. <p><span style="font-weight: 400;">At this point I can re-run </span></p>
  1091. <pre class="lang:sh decode:true">./percona-scheduler-admin --config-file=config.toml --syncusers</pre>
  1092. <p><span style="font-weight: 400;">if I try to connect again:</span><span style="font-weight: 400;"><br /></span></p>
  1093. <pre class="lang:mysql decode:true"># mysql -h 192.168.4.191 -P6033&nbsp; -umsandbox -pmsandbox
  1094. mysql: [Warning] Using a password on the command line interface can be insecure.
  1095. Welcome to the MySQL monitor.&nbsp; Commands end with ; or \g.
  1096. Your MySQL connection id is 6708563
  1097. Server version: 8.0.28 (ProxySQL). &lt;---------------------------- Connecting to proxysql
  1098.  
  1099. Copyright (c) 2000, 2022, Oracle and/or its affiliates.
  1100.  
  1101. Oracle is a registered trademark of Oracle Corporation and/or its
  1102. affiliates. Other names may be trademarks of their respective
  1103. owners.
  1104.  
  1105. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
  1106.  
  1107. mysql&gt; show global variables like 'version%';
  1108. +-------------------------+------------------------------------------------------------------------------------+
  1109. | Variable_name &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | Value&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; |
  1110. +-------------------------+------------------------------------------------------------------------------------+
  1111. | version &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | 8.0.25-15.1 &nbsp; &nbsp; &nbsp; &nbsp; &lt;--- Percona/MySQL version &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; |
  1112. | version_comment &nbsp; &nbsp; &nbsp; &nbsp; | Percona XtraDB Cluster binary (GPL) 8.0.25, Revision 8638bb0, WSREP version 26.4.3 |
  1113. | version_compile_machine | x86_64 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; |
  1114. | version_compile_os&nbsp; &nbsp; &nbsp; | Linux&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; |
  1115. | version_compile_zlib&nbsp; &nbsp; | 1.2.11 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; |
  1116. | version_suffix&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | .1 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; |
  1117. +-------------------------+------------------------------------------------------------------------------------+
  1118. 6 rows in set (0.02 sec)</pre>
  1119. <p><span style="font-weight: 400;">This is the only way to keep the password hashed in MySQL and in ProxySQL.</span></p>
  1120. <h2><span style="font-weight: 400;">Not using Hashing</span></h2>
  1121. <p><span style="font-weight: 400;">What if you cannot use mysql_native_password for the password in your MySQL server?</span></p>
  1122. <p><span style="font-weight: 400;">There is a way to still connect, however I do not recommend it given for me is highly insecure, but for completeness I am going to illustrate it.</span></p>
  1123. <p><span style="font-weight: 400;">First of all disable password hashing in Proxysql:</span></p>
  1124. <pre class="lang:mysql decode:true">update global_variables set Variable_Value='false' where Variable_name='admin-hash_passwords'; </pre>
  1125. <p><span style="font-weight: 400;">At this point instead sync the user you can locally create the user like:</span></p>
  1126. <pre class="lang:mysql decode:true">insert into mysql_users (username,password,active,default_hostgroup,default_schema,transaction_persistent,comment) values ('msandbox','msandbox',1,100,'mysql',1,'generic test for security');&nbsp;
  1127. mysql&gt; select * from runtime_mysql_users where username ='msandbox';&nbsp;
  1128. +----------+----------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+---------------------------+
  1129. | username | password | active | use_ssl | default_hostgroup | default_schema | schema_locked | transaction_persistent | fast_forward | backend | frontend | max_connections | attributes | comment &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; |
  1130. +----------+----------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+---------------------------+
  1131. | msandbox | msandbox | 1&nbsp; &nbsp; &nbsp; | 0 &nbsp; &nbsp; &nbsp; | 100 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | mysql&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | 0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | 1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | 0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | 1 &nbsp; &nbsp; &nbsp; | 0&nbsp; &nbsp; &nbsp; &nbsp; | 10000 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; |&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | generic test for security |
  1132. | msandbox | msandbox | 1&nbsp; &nbsp; &nbsp; | 0 &nbsp; &nbsp; &nbsp; | 100 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | mysql&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | 0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | 1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | 0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | 0 &nbsp; &nbsp; &nbsp; | 1&nbsp; &nbsp; &nbsp; &nbsp; | 10000 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; |&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | generic test for security |
  1133. +----------+----------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+---------------------------+</pre>
  1134. <p><span style="font-weight: 400;">As you can see doing that will prevent the password to be hashed and instead it will be clear text.</span><span style="font-weight: 400;"></span></p>
  1135. <p><span style="font-weight: 400;">At this point you will be able to connect to MySQL 8 using the caching_sha2_password, but the password is visible in ProxySQL.</span></p>
  1136. <p><span style="font-weight: 400;">Let me repeat, I DO NOT recommend using it this way, because for me it is highly insecure.&nbsp;</span></p>
  1137. <p>&nbsp;</p>
  1138. <h1><span style="font-weight: 400;">Conclusion</span></h1>
  1139. <p><span style="font-weight: 400;">While it is still possible to configure your user in MySQL to connect using ProxySQL, it is obvious that we have a gap in the way ProxySQL supports security.&nbsp;</span></p>
  1140. <p><span style="font-weight: 400;">The hope is that this gap will be filled soon by the ProxySQL development team, also if looking to the past issues this seems pending from years now.&nbsp;</span></p>
  1141. <h1><span style="font-weight: 400;">References</span></h1>
  1142. <p class="p1"><span class="s1"><a href="https://proxysql.com/documentation/mysql-8-0/">https://proxysql.com/documentation/mysql-8-0/</a></span></p>
  1143. <p class="p1"><span class="s1"><a href="https://github.com/sysown/proxysql/issues/2580">https://github.com/sysown/proxysql/issues/2580</a></span></p>
  1144. <p class="p2"><span class="s2">https://www.percona.com/blog/upgrade-your-libraries-authentication-plugin-caching_sha2_password-cannot-be-loaded/</span></p>]]></description>
  1145. <category>MySQL</category>
  1146. <pubDate>Thu, 03 Nov 2022 13:47:19 +0000</pubDate>
  1147. </item>
  1148. <item>
  1149. <title>Zero impact on index creation with Aurora 3</title>
  1150. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/248-zero-impact-on-index-creation-with-aurora-3</link>
  1151. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/248-zero-impact-on-index-creation-with-aurora-3</guid>
  1152. <description><![CDATA[<p><span style="font-weight: 400;"><span style="font-weight: 400;"><img src="http://www.tusacentral.net/joomla/images/stories/ddl/aurora_ddl_notes.jpeg" alt="aurora ddl notes" style="float: right;" /></span>Last quarter of 2021 AWS released Aurora version 3. This new version aligns Aurora with the latest MySQL 8 version porting many of the advantages MySQL 8 has over previous versions.</span></p>
  1153. <p><span style="font-weight: 400;">While this brings a lot of new interesting features for Aurora, what we are going to cover here is to see how DDLs behave when using the ONLINE option. With a quick comparison with what happens in MySQL 8 standard and with Group Replication.&nbsp;</span></p>
  1154. <h2><span style="font-weight: 400;">Tests</span></h2>
  1155. <p><span style="font-weight: 400;">All tests were run on an Aurora instance r6g.large with secondary availability zone.<br /></span><span style="font-weight: 400;">The test was composed by:</span></p>
  1156. <p><span style="font-weight: 400;">&nbsp; &nbsp; &nbsp; &nbsp; 4 connections</span></p>
  1157. <ul>
  1158. <li style="list-style-type: none;">
  1159. <ul>
  1160. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">#1 to perform ddl</span></li>
  1161. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">#2 to perform insert data in the table I am altering</span></li>
  1162. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">#3 to perform insert data on a different table&nbsp;</span></li>
  1163. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">#4 checking the other node operations</span></li>
  1164. </ul>
  1165. </li>
  1166. </ul>
  1167. <p><span style="font-weight: 400;">In the Aurora instance, a sysbench schema with 10 tables and 5 million rows was created, just to get a bit of traffic. While the test table with 5ml rows as well was:</span><span style="font-weight: 400;"><br /></span></p>
  1168. <pre class="lang:mysql decode:true">CREATE TABLE `windmills_test` (
  1169.  `id` bigint NOT NULL AUTO_INCREMENT,
  1170.  `uuid` char(36) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  1171.  `millid` smallint NOT NULL,
  1172.  `kwatts_s` int NOT NULL,
  1173.  `date` date NOT NULL,
  1174.  `location` varchar(50) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  1175.  `active` tinyint NOT NULL DEFAULT '1',
  1176.  `time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  1177.  `strrecordtype` char(3) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  1178.  PRIMARY KEY (`id`),
  1179.  KEY `IDX_millid` (`millid`,`active`),
  1180.  KEY `IDX_active` (`id`,`active`),
  1181.  KEY `kuuid_x` (`uuid`),
  1182.  KEY `millid_x` (`millid`),
  1183.  KEY `active_x` (`active`),
  1184.  KEY `idx_1` (`uuid`,`active`)
  1185. ) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8mb3 COLLATE=utf8_bin ROW_FORMAT=DYNAMIC
  1186. </pre>
  1187. <p><span style="font-weight: 400;">The executed commands:</span></p>
  1188. <pre class="lang:sh decode:true">Connection 1:
  1189.    ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE, LOCK=NONE;
  1190.    ALTER TABLE windmills_test drop INDEX idx_1, ALGORITHM=INPLACE;
  1191.    
  1192. Connection 2:
  1193. while [ 1 = 1 ];do da=$(date +'%s.%3N');mysql --defaults-file=./my.cnf -D windmills_large -e "insert into windmills_test  select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmills4 limit 1;" -e "select count(*) from windmills_large.windmills_test;" &gt; /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done
  1194.  
  1195. Connection 3:
  1196. while [ 1 = 1 ];do da=$(date +'%s.%3N');mysql --defaults-file=./my.cnf -D windmills_large -e "insert into windmills3  select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmills4 limit 1;" -e "select count(*) from windmills_large.windmills_test;" &gt; /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done
  1197.  
  1198. Connections 4:
  1199.     while [ 1 = 1 ];do echo "$(date +'%T.%3N')";mysql --defaults-file=./my.cnf -h &lt;secondary aurora instance&gt; -D windmills_large -e "show full processlist;"|egrep -i -e "(windmills_test|windmills_large)"|grep -i -v localhost;sleep 1;done
  1200.    
  1201. </pre>
  1202. <p>Operations:<br />1) start inserts from connections<br />2) start commands in connections 4 - 5 on the other nodes<br />3) execute: <span style="background-color: #f4f4f4; font-family: 'Courier 10 Pitch', Courier, monospace; font-size: 12.8px;">DC1-1(root@localhost) [windmills_large]&gt;ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE, LOCK=NONE;</span></p>
  1203. <p><span style="font-weight: 400;">With this, what I was looking to capture is the operation impact in doing a common action as creating an Index. My desired expectation is to have no impact when doing operations that are declared “ONLINE” such as creating an index, as well as data consistency between nodes.&nbsp;</span></p>
  1204. <p><span style="font-weight: 400;">Let us see what happened…</span></p>
  1205. <h2><span style="font-weight: 400;">Results</span></h2>
  1206. <p><span style="font-weight: 400;">While running the insert in the same table, performing the alter:</span></p>
  1207. <pre class="lang:mysql decode:true">mysql&gt;  ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE, LOCK=NONE;
  1208. Query OK, 0 rows affected (16.51 sec)
  1209. Records: 0  Duplicates: 0  Warnings: 0
  1210. </pre>
  1211. <p><span style="font-weight: 400;">Is NOT stopping the operation in the same table or any other table in the Aurora instance.</span></p>
  1212. <p><span style="font-weight: 400;">We can only identify a minimal performance impact:</span></p>
  1213. <pre class="lang:sh decode:true">[root@ip-10-0-0-11 tmp]# while [ 1 = 1 ];do da=$(date +'%s.%3N');mysql --defaults-file=./my.cnf -D windmills_large -e "insert into windmills_test  select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmills4 limit 1;" -e "select count(*) from windmills_large.windmills_test;" &gt; /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done
  1214. .347
  1215. .283
  1216. .278
  1217. .297
  1218. .291
  1219. .317
  1220. .686  ← start
  1221. &lt;Snip&gt;
  1222. .512  ← end
  1223. .278
  1224. .284
  1225. .279
  1226. </pre>
  1227. <p><span style="font-weight: 400;">The secondary node is not affected at all, and this is because Aurora managed at storage level the data replication. Given that there is no such thing as Apply from Relaylog, as we have in standard MySQL asynchronous or data replicated with Group Replication.&nbsp;&nbsp;</span></p>
  1228. <p><span style="font-weight: 400;">The result is that in Aurora 3, we can have zero impact index (or any other ONLINE/INSTANT) operation, with this I include the data replicated in the other instances for High Availability.&nbsp;</span></p>
  1229. <p><span style="font-weight: 400;">If we compare this with Group replication (see <a href="https://www.percona.com/blog/online-ddl-with-group-replication-in-mysql-8-0-27/">blog</a>):</span></p>
  1230. <pre class="lang:sh decode:true">                                GR         Aurora 3
  1231. Time on hold for insert for altering table   ~0.217 sec   ~0.523 sec
  1232. Time on hold for insert for another table   ~0.211 sec   ~0.205 sec
  1233. </pre>
  1234. <p><span style="font-weight: 400;">However, keep in mind that MySQL with Group Replication will still need to apply the data on the Secondaries. This means that if your alter was taking 10 hours to build the index, the Secondary nodes will be misaligned with the Source for approximately another 10 hours.&nbsp;</span></p>
  1235. <p><span style="font-weight: 400;">With Aurora 3 or with PXC, changes will be there when Source has completed the operation.&nbsp;&nbsp;&nbsp;&nbsp;</span></p>
  1236. <p><span style="font-weight: 400;">What about Percona XtraDB Cluster (PXC)? Well, with PXC we have a different scenario:</span></p>
  1237. <pre class="lang:sh decode:true">                               PXC(NBO)     Aurora 3
  1238. Time on hold for insert for altering table   ~120 sec      ~0.523 sec
  1239. Time on hold for insert for another table   ~25  sec      ~0.205 sec</pre>
  1240. <p><span style="font-weight: 400;">We will have a higher impact while doing the Alter operation, but the data will be on all nodes at the same time maintaining a high level of consistency in the cluster.&nbsp;</span></p>
  1241. <h2><span style="font-weight: 400;">Conclusion</span></h2>
  1242. <p><span style="font-weight: 400;">Aurora is not for all use, and not for all budgets, however it has some very good aspects like the one we have just seen. The Difference between standard MySQL and Aurora is not in the time of holding/locking (aka operation impact), but on the HA aspects. If I have my data/structure on all my Secondary at the same time of the Source, I will feel much more comfortable, than having to wait an additional T time.&nbsp;</span></p>
  1243. <p><span style="font-weight: 400;">This is why PXC in that case is a better alternative if you can afford locking time, if not, well Aurora 3 is your solution, just do your math properly and be conservative with the instance resources.&nbsp;</span></p>
  1244. <p>&nbsp;</p>]]></description>
  1245. <category>MySQL</category>
  1246. <pubDate>Wed, 20 Apr 2022 12:05:56 +0000</pubDate>
  1247. </item>
  1248. <item>
  1249. <title>A face to face with semi-synchronous replication</title>
  1250. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/247-a-face-to-face-with-semi-synchronous-replication</link>
  1251. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/247-a-face-to-face-with-semi-synchronous-replication</guid>
  1252. <description><![CDATA[<p><a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/face_to_face.jpeg"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/face_to_face.jpeg" alt="" width="300" height="212" class="size-medium wp-image-80671 alignright" style="float: right;" /></a></p>
  1253. <p><span style="font-weight: 400;">Last month I performed a review of the Percona Operator for MySQL Server (<a href="https://www.percona.com/doc/kubernetes-operator-for-mysql/ps/index.html">https://www.percona.com/doc/kubernetes-operator-for-mysql/ps/index.html</a>) which is still Alpha.&nbsp; That operator is based on Percona Server and uses standard asynchronous replication, with the option to activate semi-synchronous&nbsp; replication to gain higher levels of data consistency between nodes.&nbsp;</span></p>
  1254. <p><span style="font-weight: 400;">The whole solution is composed as:</span></p>
  1255. <p><a href="https://www.percona.com/blog/wp-content/uploads/2022/04/operator.svg"><img src="https://www.percona.com/blog/wp-content/uploads/2022/04/operator.svg" alt="" width="499" height="409" class="alignnone wp-image-80663" /></a></p>
  1256. <p><span style="font-weight: 400;">Additionally, Orchestrator (</span><a href="https://github.com/openark/orchestrator"><span style="font-weight: 400;">https://github.com/openark/orchestrator</span></a><span style="font-weight: 400;">) is used to manage the topology and the settings to enable on the replica nodes, the semi-synchronous flag if required.<br /></span><span style="font-weight: 400;">While we have not too much to say when using standard Asynchronous replication, I want to spend two words on the needs and expectations on the semi-synchronous (semi-sync) solution.&nbsp;</span></p>
  1257. <h2><span style="font-weight: 400;">A look into semi-synchronous</span></h2>
  1258. <p><span style="font-weight: 400;">Difference between Async and Semi-sync.<br /></span>Asynchronous:</p>
  1259. <p><a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/async-replication-diagram.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/async-replication-diagram.png" alt="" width="725" height="304" class="alignnone size-full wp-image-80662" /></a></p>
  1260. <p><span style="font-weight: 400;">The above diagram represents the standard asynchronous replication. This method is expected by design, to have transactions committed on the Source that are not present on the Replicas. The Replica is supposed to </span><i><span style="font-weight: 400;">catch-up</span></i><span style="font-weight: 400;"> when possible.&nbsp;&nbsp;&nbsp;</span></p>
  1261. <p><span style="font-weight: 400;">It is also important to understand that there are two steps in replication:</span></p>
  1262. <ul>
  1263. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Data copy, which is normally very fast. The Data is copied from the binlog of the Source to the relay log on the Replica (IO_Thread).</span></li>
  1264. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Data apply, where the data is read from the relay log on the Replica node and written inside the database itself (SQL_Thread). This step is normally the bottleneck and while there are some parameters to tune, the efficiency to apply transactions depends on many factors including schema design.&nbsp;</span></li>
  1265. </ul>
  1266. <p><span style="font-weight: 400;">Production deployments that utilize the Asynchronous solution are typically designed to manage the possible inconsistent scenario given data on Source is not supposed to be on Replica at commit. At the same time the level of High Availability assigned to this solution is lower than the one we normally obtain with (virtually-)synchronous replication, given we may need to wait for the Replicare to catch-up the gap accumulated in the relay-logs before performing the fail-over.</span></p>
  1267. <p>Semi-sync:</p>
  1268. <p><a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/semisync-replication-diagram.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/semisync-replication-diagram.png" alt="" width="725" height="396" class="alignnone size-full wp-image-80665" /></a></p>
  1269. <p><span style="font-weight: 400;">The above diagram represents the Semi-sync replication method.The introduction of semi-sync adds a checking step on the Source before it returns the acknowledgement to the client.<br /></span><span style="font-weight: 400;">This step happens at the moment of the data-copy, so when the data is copied from the Binary-log on Source to the Relay-log on Replica.&nbsp;</span></p>
  1270. <p><span style="font-weight: 400;">This is important, there is NO mechanism to ensure a more resilient or efficient data replication, there is only an additional step, that tells the Source to wait a given amount of time for an answer from N replicas, and then return the acknowledgement or timeout and return to the client no matter what.&nbsp;</span></p>
  1271. <p><span style="font-weight: 400;">This mechanism is introducing a </span><i><span style="font-weight: 400;">possible</span></i><span style="font-weight: 400;"> significant delay in the service, without giving the 100% guarantee of data consistency.&nbsp;</span></p>
  1272. <p><span style="font-weight: 400;">In terms of </span><i><span style="font-weight: 400;">availability</span></i><span style="font-weight: 400;"> of the service, when in presence of high load, this method may lead the Source to stop serving the request while waiting for acknowledgements, significantly reducing the availability of the service itself.&nbsp;&nbsp;&nbsp;</span></p>
  1273. <p><span style="font-weight: 400;">At the same time only acceptable settings for rpl_semi_sync_source_wait_point is AFTER_SYNC (default) because:&nbsp; </span><i><span style="font-weight: 400;">In the event of source failure, all transactions committed on the source have been replicated to the replica (saved to its relay log). An unexpected exit of the source server and failover to the replica is lossless because the replica is up to date.</span></i></p>
  1274. <p><span style="font-weight: 400;">All clear? No? Let me simplify the thing.&nbsp;</span></p>
  1275. <ul>
  1276. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">In standard replication you have two moments (I am simplifying)</span>
  1277. <ul>
  1278. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">Copy data from Source to Replica</span></li>
  1279. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">Apply data in the Replica node</span></li>
  1280. </ul>
  1281. </li>
  1282. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">There is no certification on the data applied about its consistency with the Source</span></li>
  1283. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">With asynchronous the Source task is to write data in the binlog and forget</span></li>
  1284. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">With semi-sync the Source writes the data on binlog and waits </span><b><i>T</i></b><span style="font-weight: 400;"> seconds to receive </span><i><span style="font-weight: 400;">acknowledgement</span></i><span style="font-weight: 400;"> from </span><b><i>N</i></b><span style="font-weight: 400;"> servers about them having received the data.</span></li>
  1285. </ul>
  1286. <p><span style="font-weight: 400;">To enable semi-sync you follow these steps:&nbsp; </span><a href="https://dev.mysql.com/doc/refman/8.0/en/replication-semisync-installation.html"><span style="font-weight: 400;">https://dev.mysql.com/doc/refman/8.0/en/replication-semisync-installation.html</span></a><span style="font-weight: 400;"></span></p>
  1287. <p><span style="font-weight: 400;">In short:</span></p>
  1288. <ul>
  1289. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Register the plugins</span></li>
  1290. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Enable Source rpl_semi_sync_source_enabled=1</span></li>
  1291. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Enable Replica rpl_semi_sync_replica_enabled = 1</span>
  1292. <ul>
  1293. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">If replication is already running STOP/START REPLICA IO_THREAD</span></li>
  1294. </ul>
  1295. </li>
  1296. </ul>
  1297. <p><span style="font-weight: 400;">And here starts the fun, be ready for many “</span><i><span style="font-weight: 400;">wait whaaat?”</span></i><span style="font-weight: 400;">.&nbsp;</span></p>
  1298. <p><span style="font-weight: 400;">What is the T and N I have just mentioned above?</span></p>
  1299. <p><span style="font-weight: 400;">Well the T is a timeout that you can set to avoid having the source wait forever for the Replica acknowledgement. The default is 10 seconds. What happens if the Source waits for more than the timeout?&nbsp; <i><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/wait_whaat.jpeg" alt="" width="115" height="79" class="wp-image-80666 alignright" style="float: right;" /></i><br /></span><b><i>rpl_semi_sync_source_timeout</i></b><i><span style="font-weight: 400;"> controls how long the source waits on a commit for acknowledgment from a replica before timing out and </span></i><b><i>reverting to asynchronous replication</i></b><i><span style="font-weight: 400;">.</span></i></p>
  1300. <p><span style="font-weight: 400;">Careful of the wording here! The manual says </span><b>SOURCE</b><span style="font-weight: 400;">, so it is not that MySQL revert to asynchronous, by transaction or connection, </span><span style="font-weight: 400;">it is for the <strong>whole server</strong>.</span></p>
  1301. <p><span style="font-weight: 400;">Now analyzing the work-log (see </span><a href="https://dev.mysql.com/worklog/task/?id=1720"><span style="font-weight: 400;">https://dev.mysql.com/worklog/task/?id=1720</span></a><span style="font-weight: 400;"> and more in the references) the Source should revert to semi-synchronous as soon as all involved replicas are aligned again.&nbsp;</span></p>
  1302. <p><span style="font-weight: 400;">However, checking the code (see&nbsp; </span><a href="https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/plugin/semisync/semisync_source.cc#L844"><span style="font-weight: 400;">https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/plugin/semisync/semisync_source.cc#L844</span></a><span style="font-weight: 400;"> and following), we can see that we do not have a 100% guarantee that the Source will be able to switch back.&nbsp;</span></p>
  1303. <p><span style="font-weight: 400;">Also in the code:</span><span style="font-weight: 400;"><br /></span><i><span style="font-weight: 400;">But, it is not that easy to detect that the replica has caught up.&nbsp; This is caused by the fact that MySQL's replication protocol is&nbsp; asynchronous, meaning that if thesource does not use the semi-sync protocol, the replica would not send anything to thesource.</span></i></p>
  1304. <p><span style="font-weight: 400;">In all the runned tests the Source was not able to switch back. In short Source was moving out from semi-sync and that was </span><span style="font-weight: 400;">forever</span><span style="font-weight: 400;">, no rollback. Keep in mind that while we go ahead.</span></p>
  1305. <p><span style="font-weight: 400;">What is the N I mentioned above? It represents the number of Replicas that must provide the acknowledgement back.&nbsp;</span></p>
  1306. <p><span style="font-weight: 400;">If you have a cluster of 10 nodes you may need to have only 2 of them involved in the semi-sync, no need to include them all. But if you have a cluster of 3 nodes where 1 is the Source, relying on 1 Replica only, is not really secure. What I mean here is that if you choose to be semi-synchronous to ensure the data replicates, having it enabled for one single node is not enough, if that node crashes or whatever, you are doomed, as such you need at least 2 nodes with semi-sync.</span></p>
  1307. <p><span style="font-weight: 400;">Anyhow, the point is that if one of the Replica takes more than T to reply, the whole mechanism stops working, probably forever.&nbsp;</span></p>
  1308. <p><span style="font-weight: 400;">As we have seen above, to enable semi-sync on Source we manipulate the value of the GLOBAL variable </span><i><span style="font-weight: 400;">rpl_semi_sync_source_enabled</span></i><span style="font-weight: 400;">.</span></p>
  1309. <p><span style="font-weight: 400;">However if I check the value of </span><i><span style="font-weight: 400;">rpl_semi_sync_source_enabled</span></i><span style="font-weight: 400;"> when the Source shift to simple Asynchronous replication because timeout:</span></p>
  1310. <p><span style="font-weight: 400;">select @@rpl_semi_sync_source_enabled;</span></p>
  1311. <pre class="lang:mysql decode:true">select @@rpl_semi_sync_source_enabled;
  1312. +--------------------------------+
  1313. | @@rpl_semi_sync_source_enabled |
  1314. +--------------------------------+
  1315. |                              1 |
  1316. +--------------------------------+
  1317. </pre>
  1318. <p><span style="font-weight: 400;">As you can see the Global variable reports a value of 1, meaning that semi-sync is active also if not.</span></p>
  1319. <p><span style="font-weight: 400;">In the documentation it is reported that to monitor the semi-sync activity we should check for Rpl_semi_sync_source_status. Which means that you can have <a href="https://www.percona.com/blog/wp-content/uploads/2022/04/wait_whaat.jpeg"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/wait_whaat.jpeg" alt="" width="113" height="77" class="wp-image-80666 alignright" style="float: right;" /></a> Rpl_semi_sync_source_status = 0 and rpl_semi_sync_source_enabled =1 at the same time.</span></p>
  1320. <p><span style="font-weight: 400;">Is this a bug? Well according to documentation:</span><span style="font-weight: 400;"><br /></span><i><span style="font-weight: 400;">When the source switches between asynchronous or semisynchronous replication due to commit-blocking timeout or a replica catching up, it sets the value of the Rpl_semi_sync_source_status or Rpl_semi_sync_source_status status variable appropriately. Automatic fallback from semisynchronous to asynchronous replication on the source means that it is possible for the rpl_semi_sync_source_enabled or rpl_semi_sync_source_enabled system variable to have a value of 1 on the source side even when semisynchronous replication is in fact not operational at the moment. You can monitor the Rpl_semi_sync_source_status or Rpl_semi_sync_source_status status variable to determine whether the source currently is using asynchronous or semisynchronous replication.</span></i></p>
  1321. <p><b>It is not a bug</b><span style="font-weight: 400;">. However, because you documented it, it doesn’t change the fact this is a weird/unfriendly/counterintuitive way of doing, that opens the door to many, many possible issues. Especially given you know the Source may fail to switch semi-synch back.&nbsp;&nbsp;</span></p>
  1322. <p><span style="font-weight: 400;">Just to close this part, we can summarize as follows:</span></p>
  1323. <ul>
  1324. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">You activate semi-sync setting a global variable</span></li>
  1325. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Server/Source can disable it (silently) without changing that variable&nbsp;</span></li>
  1326. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Server will never restore semi-sync automatically</span></li>
  1327. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">The way to check if semi-sync works is to use the Status variable</span></li>
  1328. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">When Rpl_semi_sync_source_status = 0 and rpl_semi_sync_source_enabled =1 you had a Timeout and Source is now working in asynchronous replication</span></li>
  1329. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">The way to reactivate semi-sync is to set rpl_semi_sync_source_enabled&nbsp; to OFF first then rpl_semi_sync_source_enabled = ON.&nbsp;</span></li>
  1330. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Replicas can be set with semi-sync ON/OFF but unless you do not STOP/START the replica_IO_THREAD the state of the variable can be inconsistent with the state of the Server.</span></li>
  1331. </ul>
  1332. <p><span style="font-weight: 400;">What can go wrong?</span></p>
  1333. <h2><span style="font-weight: 400;">Semi-synchronous is not seriously affecting the performance</span></h2>
  1334. <p><span style="font-weight: 400;">Others had already discussed semi-sync performance in better details. However I want to add some color given the recent experience with our operator testing.<br /></span><span style="font-weight: 400;">In the next graphs I will show you the behavior of writes/reads using Asynchronous replication and the same load with Semi-synchronous.<br /></span><span style="font-weight: 400;">For the record the test was a simple Sysbench-tpcc test using 20 tables, 20 warehouses, 256 threads for 600 seconds.&nbsp;&nbsp;&nbsp;</span></p>
  1335. <p><a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/2.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/2.png" alt="" width="900" height="401" class="alignnone wp-image-80652 size-large" /></a> <a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/3.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/3.png" alt="" width="900" height="421" class="alignnone size-large wp-image-80653" /></a></p>
  1336. <p><span style="font-weight: 400;">The one above indicates a nice and consistent set of load in r/w with minimal fluctuations. This is what we like to have.&nbsp;</span></p>
  1337. <p><span style="font-weight: 400;">The graphs below, represent the exact same load on the exact same environment but with semi-sync activated and no timeout.&nbsp;</span></p>
  1338. <p><a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/9.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/9.png" alt="" width="900" height="421" class="alignnone wp-image-80657 size-large" /></a> <a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/10.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/10.png" alt="" width="900" height="420" class="alignnone wp-image-80658 size-large" /></a></p>
  1339. <p><span style="font-weight: 400;">Aside from the performance loss (we went from Transaction 10k/s to 3k/s), the constant stop/go imposed by the semi-sync mechanism has a very bad effect on the application behavior when you have many concurrent threads and high loads. I challenge any serious production system to work in this way.&nbsp;&nbsp;&nbsp;</span></p>
  1340. <p><span style="font-weight: 400;">Of course results are inline with this yoyo game:</span></p>
  1341. <p><a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/1c.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/1c.png" alt="" width="900" height="356" class="alignnone size-large wp-image-80651" /></a></p>
  1342. <p><span style="font-weight: 400;">In the best case, when all was working as expected, and no crazy stuff happening I had something around the 60% loss. I am not oriented to see this as&nbsp; a minor performance drop.&nbsp;</span></p>
  1343. <h2><span style="font-weight: 400;">But at least your data is safe</span></h2>
  1344. <p><span style="font-weight: 400;">As already stated at the beginning the scope of semi-synchronous replication is to guarantee that the data in server A reaches server B before returning the OK to the application.&nbsp;</span></p>
  1345. <p><span style="font-weight: 400;">In short, given a period of 1 second we should have minimal transactions in flight and limited transactions in the apply queue. While for standard replication (asynchronous), we may have … thousands.&nbsp;</span></p>
  1346. <p><span style="font-weight: 400;">In the graphs below we can see two lines:</span></p>
  1347. <ul>
  1348. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">The yellow line represents the number of GTIDs “in flight” from Source to destination, Y2 axes.&nbsp; In case of Source crash, those transactions are lost and we will have data loss.</span></li>
  1349. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">The blue line represents the number of GTIDs already copied over from Source to Replica but not applied in the database Y1 axes. In case of Source crash we must wait for the Replica to process these entries, before making the node Write active, or we will have data inconsistency.</span></li>
  1350. </ul>
  1351. <p><span style="font-weight: 400;">Asynchronous replication:</span></p>
  1352. <p><a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/4.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/4.png" alt="" width="900" height="526" class="alignnone wp-image-80654 size-large" /></a></p>
  1353. <p><span style="font-weight: 400;">As expected we can see a huge queue in applying the transactions from relay-log, and some spike of transactions in flight.&nbsp;</span></p>
  1354. <p><span style="font-weight: 400;">Using Semi-synchronous replication:</span></p>
  1355. <p><a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/8.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/8.png" alt="" width="900" height="421" class="alignnone size-large wp-image-80656" /></a></p>
  1356. <p><span style="font-weight: 400;">Yes, apparently we have reduced the queue and no spikes so no data loss.</span></p>
  1357. <p><span style="font-weight: 400;">But this happens&nbsp; when all goes as expected, and we know in production this is not the normal.<br /></span><span style="font-weight: 400;">What if we need to enforce the semi-sync but at the same time we cannot set the Timeout to ridiculous values like 1 week?&nbsp;</span></p>
  1358. <p><span style="font-weight: 400;">Simple, we need to have a check that puts back the semi-sync as soon as it is silently disabled (for any reason).<br /></span><span style="font-weight: 400;">However doing this without waiting for the Replicas to cover the replication gap, cause the following interesting effects:</span></p>
  1359. <p><a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/5.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/5.png" alt="" width="900" height="525" class="alignnone size-large wp-image-80655" /></a></p>
  1360. <p><span style="font-weight: 400;">Thousands of transactions queued and shipped with the result of having a significant increase of the possible data loss and still a huge number of data to apply from the relay-log.&nbsp;</span></p>
  1361. <p><span style="font-weight: 400;">So the only possible alternative is to set the Timeout to a crazy value, However this can cause a full production stop in the case a Replica hangs or for any reason it disables the semi-sync locally.&nbsp;</span></p>
  1362. <p>&nbsp;</p>
  1363. <h2><span style="font-weight: 400;">Conclusion</span></h2>
  1364. <p><span style="font-weight: 400;">First of all I want to say that the tests on our operator using Asynchronous replication, shows a consistent behavior with the standard deployments in the cloud or premises.&nbsp; It has the same benefits, like better performance and same possible issues as longer time to failover when it needs to wait a Replica to apply the relay-log queue.&nbsp;</span></p>
  1365. <p><span style="font-weight: 400;">The semi-synchronous flag in the operator is disabled, and the tests I have done bring me to say “keep it like that!”. At least unless you know very well what you are doing and are able to deal with a semi-sync timeout of days.</span></p>
  1366. <p>I was happy to have the chance to perform these tests, because they gives me a way/time/need to investigate more on the semi-synchronous feature.<br /><span style="font-weight: 400;">Personally, I was not convinced about the semi-synchronous replication when it came out, and I am not now. I never saw a less consistent and less trustable feature in MySQL as semi-sync.&nbsp;&nbsp;</span></p>
  1367. <p><span style="font-weight: 400;">If you need to have a higher level of synchronicity in your database just go for Group Replication, or Percona XtraDB Cluster and stay away from semi-sync.&nbsp;</span></p>
  1368. <p><span style="font-weight: 400;">Otherwise, stay on Asynchronous replication, which is not perfect but it is predictable.&nbsp;&nbsp;</span></p>
  1369. <h2><span style="font-weight: 400;">References</span></h2>
  1370. <p><a href="https://www.percona.com/blog/2012/01/19/how-does-semisynchronous-mysql-replication-work/"><span style="font-weight: 400;">https://www.percona.com/blog/2012/01/19/how-does-semisynchronous-mysql-replication-work/</span></a></p>
  1371. <p><a href="https://www.percona.com/blog/percona-monitoring-and-management-mysql-semi-sync-summary-dashboard/"><span style="font-weight: 400;">https://www.percona.com/blog/percona-monitoring-and-management-mysql-semi-sync-summary-dashboard/</span></a></p>
  1372. <p><a href="https://www.percona.com/blog/2012/06/14/comparing-percona-xtradb-cluster-with-semi-sync-replication-cross-wan/">https://www.percona.com/blog/2012/06/14/comparing-percona-xtradb-cluster-with-semi-sync-replication-cross-wan/</a></p>
  1373. <p><a href="https://datto.engineering/post/lossless-mysql-semi-sync-replication-and-automated-failover"><span style="font-weight: 400;">https://datto.engineering/post/lossless-mysql-semi-sync-replication-and-automated-failover</span></a></p>
  1374. <p><a href="https://planetscale.com/blog/mysql-semi-sync-replication-durability-consistency-and-split-brains"><span style="font-weight: 400;">https://planetscale.com/blog/mysql-semi-sync-replication-durability-consistency-and-split-brains</span></a></p>
  1375. <p><a href="https://percona.community/blog/2018/08/23/question-about-semi-synchronous-replication-answer-with-all-the-details/"><span style="font-weight: 400;">https://percona.community/blog/2018/08/23/question-about-semi-synchronous-replication-answer-with-all-the-details/</span></a></p>
  1376. <p><a href="https://dev.mysql.com/doc/refman/8.0/en/replication-semisync-installation.html"><span style="font-weight: 400;">https://dev.mysql.com/doc/refman/8.0/en/replication-semisync-installation.html</span></a></p>
  1377. <p><a href="https://dev.mysql.com/worklog/task/?id=1720"><span style="font-weight: 400;">https://dev.mysql.com/worklog/task/?id=1720</span></a></p>
  1378. <p><a href="https://dev.mysql.com/worklog/task/?id=6630"><span style="font-weight: 400;">https://dev.mysql.com/worklog/task/?id=6630</span></a></p>
  1379. <p><a href="https://dev.mysql.com/worklog/task/?id=4398"><span style="font-weight: 400;">https://dev.mysql.com/worklog/task/?id=4398</span></a></p>
  1380. <p><a href="https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/plugin/semisync/semisync_source.cc#L844"><span style="font-weight: 400;">https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/plugin/semisync/semisync_source.cc#L844</span></a></p>
  1381. <p><a href="https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/plugin/semisync/semisync_source.cc#L881"><span style="font-weight: 400;">https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/plugin/semisync/semisync_source.cc#L881</span></a></p>]]></description>
  1382. <category>MySQL</category>
  1383. <pubDate>Tue, 12 Apr 2022 10:00:01 +0000</pubDate>
  1384. </item>
  1385. <item>
  1386. <title>Online DDL with Group Replication In MySQL 8.0.27</title>
  1387. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/246-online-ddl-with-group-replication-in-mysql-8-0-27</link>
  1388. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/246-online-ddl-with-group-replication-in-mysql-8-0-27</guid>
  1389. <description><![CDATA[<p><span style="font-weight: 400;">Last April 2021 I wrote an <a href="https://www.percona.com/blog/2021/04/15/online-ddl-with-group-replication-in-percona-server-for-mysql-8-0-22/">article</a> about Online DDL and Group Replication. At that time we were dealing with MySQL 8.0.23 </span><span style="font-weight: 400;">and also opened a <a href="https://bugs.mysql.com/bug.php?id=103421">bug</a> report </span><span style="font-weight: 400;">which did not have the right answer to the case presented.&nbsp;</span></p>
  1390. <p><span style="font-weight: 400;">Anyhow, in that article I have shown how an online DDL was de facto locking the whole cluster for a very long time even when using the consistency level set to EVENTUAL.</span></p>
  1391. <p><span style="font-weight: 400;">This article is to give justice to the work done by the MySQL/Oracle engineers to correct that annoying inconvenience.&nbsp;</span></p>
  1392. <p><span style="font-weight: 400;">Before going ahead, let us remember how an Online DDL was propagated in a group replication cluster, and identify the differences with what happens now, all with the consistency level set to EVENTUAL (<a href="https://dev.mysql.com/doc/refman/8.0/en/group-replication-configuring-consistency-guarantees.html">see</a></span><span style="font-weight: 400;">).</span></p>
  1393. <p><span style="font-weight: 400;">In MySQL 8.0.23 we were having:</span></p>
  1394. <table>
  1395. <tbody>
  1396. <tr>
  1397. <td><a href="https://www.percona.com/blog/wp-content/uploads/2022/01/1-gr-ddl.png"><img src="http://www.tusacentral.net/joomla/images/stories/gr_ddl_8_0_27/1-gr-ddl.png" alt="1 gr ddl" width="201" height="222" class="alignnone size-full wp-image-79683" /></a></td>
  1398. <td><a href="https://www.percona.com/blog/wp-content/uploads/2022/01/gr-ddl-2-old.png"><img src="http://www.tusacentral.net/joomla/images/stories/gr_ddl_8_0_27/gr-ddl-2-old.png" alt="gr ddl 2 old" width="201" height="222" class="alignnone size-full wp-image-79689" /></a></td>
  1399. <td><a href="https://www.percona.com/blog/wp-content/uploads/2022/01/gr-ddl-3-old.png"><img src="http://www.tusacentral.net/joomla/images/stories/gr_ddl_8_0_27/gr-ddl-3-old.png" alt="gr ddl 3 old" width="211" height="242" class="alignnone size-full wp-image-79684" /></a></td>
  1400. </tr>
  1401. </tbody>
  1402. </table>
  1403. <p><span style="font-weight: 400;">While in MySQL 8.0.27 we have:</span></p>
  1404. <table>
  1405. <tbody>
  1406. <tr>
  1407. <td><a href="https://www.percona.com/blog/wp-content/uploads/2022/01/1-gr-ddl.png"><img src="http://www.tusacentral.net/joomla/images/stories/gr_ddl_8_0_27/1-gr-ddl-new.png" alt="1 gr ddl new" width="201" height="222" class="alignnone size-full wp-image-79683" /></a></td>
  1408. <td><a href="https://www.percona.com/blog/wp-content/uploads/2022/01/gr-ddl-2-new.png"><img src="http://www.tusacentral.net/joomla/images/stories/gr_ddl_8_0_27/gr-ddl-2-new.png" alt="gr ddl 2 new" width="231" height="257" class="alignnone size-full wp-image-79686" /></a></td>
  1409. <td><a href="https://www.percona.com/blog/wp-content/uploads/2022/01/gr-ddl-3-new.png"><img src="http://www.tusacentral.net/joomla/images/stories/gr_ddl_8_0_27/gr-ddl-3-new.png" alt="gr ddl 3 new" width="201" height="212" class="alignnone size-full wp-image-79687" /></a></td>
  1410. </tr>
  1411. </tbody>
  1412. </table>
  1413. <p>&nbsp;</p>
  1414. <p><span style="font-weight: 400;">As you can see from the images we have 3 different phases. Phase 1 is the same between version 8.0.23 and version 8.0.27.&nbsp;</span></p>
  1415. <p><span style="font-weight: 400;">Phase 2 and 3 instead are quite different. In MySQL 8.0.23 after the DDL is applied on the Primary it is propagated to the other nodes, but a metalock was also acquired and the control was NOT returned. The result was that not only the session executing the DDL was kept on hold, but also all the other sessions performing modifications.&nbsp;</span></p>
  1416. <p><span style="font-weight: 400;">Only when the operation was over on all secondaries, the DDL was pushed to Binlog and disseminated for Asynchronous replication, lock raised and operation can restart.</span></p>
  1417. <p><span style="font-weight: 400;">Instead, in MySQL 8.0.27,&nbsp; once the operation is over on the primary the DDL is pushed to binlog, disseminated to the secondaries and control returned. The result is that the write operations on primary have no interruption whatsoever and the DDL is distributed to secondary and Asynchronous replication at the same time.&nbsp;</span></p>
  1418. <p><span style="font-weight: 400;">This is a fantastic improvement, available only with consistency level EVENTUAL, but still, fantastic.</span></p>
  1419. <h3><span style="font-weight: 400;">Let's see some numbers.</span></h3>
  1420. <p><span style="font-weight: 400;">To test the operation, I have used the same approach used in the previous tests in the article mentioned above.</span></p>
  1421. <pre class="lang:sh decode:true">Connection 1:
  1422.    ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE, LOCK=NONE;
  1423.    ALTER TABLE windmills_test drop INDEX idx_1, ALGORITHM=INPLACE;
  1424.    
  1425. Connection 2:
  1426. while [ 1 = 1 ];do da=$(date +'%s.%3N');/opt/mysql_templates/mysql-8P/bin/mysql --defaults-file=./my.cnf -uroot -D windmills_large -e "insert into windmills_test  select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmill7 limit 1;" -e "select count(*) from windmills_large.windmills_test;" &gt; /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done
  1427.  
  1428. Connection 3:
  1429. while [ 1 = 1 ];do da=$(date +'%s.%3N');/opt/mysql_templates/mysql-8P/bin/mysql --defaults-file=./my.cnf -uroot -D windmills_large -e "insert into windmill8  select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmill7 limit 1;" -e "select count(*) from windmills_large.windmills_test;" &gt; /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done
  1430.  
  1431. Connections 4-5:
  1432.     while [ 1 = 1 ];do echo "$(date +'%T.%3N')";/opt/mysql_templates/mysql-8P/bin/mysql --defaults-file=./my.cnf -uroot -D windmills_large -e "show full processlist;"|egrep -i -e "(windmills_test|windmills_large)"|grep -i -v localhost;sleep 1;done
  1433. </pre>
  1434. <p><span style="font-weight: 400;">Modifying a table with ~5 million rows:</span></p>
  1435. <pre class="lang:mysql decode:true">node1-DC1 (root@localhost) [windmills_large]&gt;select count(*) from  windmills_test;
  1436. +----------+
  1437. | count(*) |
  1438. +----------+
  1439. |  5002909 |
  1440. +----------+
  1441. </pre>
  1442. <p><span style="font-weight: 400;">The numbers below represent the time second.milliseconds taken by the operation to complete. While I was also catching the state of the ALTER on the other node I am not reporting it here given it is not relevant.&nbsp;</span></p>
  1443. <pre class="lang:sh decode:true">EVENTUAL (on the primary only)
  1444. -------------------
  1445. Node 1 same table:
  1446. .184
  1447. .186 &lt;--- no locking during alter on the same node
  1448. .184
  1449. &lt;snip&gt;
  1450. .184
  1451. .217 &lt;--- moment of commit
  1452. .186
  1453. .186
  1454. .186
  1455. .185
  1456.  
  1457. Node 1 another table :
  1458. .189
  1459. .198 &lt;--- no locking during alter on the same node
  1460. .188
  1461. &lt;snip&gt;
  1462. .191
  1463. .211  &lt;--- moment of commit
  1464. .194
  1465. </pre>
  1466. <p><span style="font-weight: 400;">As you can see there is just a very small delay at the moment of commit but other impact.</span></p>
  1467. <p><span style="font-weight: 400;">Now if we compare this with the recent tests I have done for PXC Non Blocking operation (see <a href="https://www.percona.com/blog/percona-xtradb-cluster-non-blocking-operation-for-online-schema-upgrade/">here</a></span><span style="font-weight: 400;">) with same number of rows and same kind of table/data:</span></p>
  1468. <table border="1">
  1469. <thead>
  1470. <tr><th>Action</th><th align="right">Group Replication</th><th align="right">PXC (NBO)</th></tr>
  1471. </thead>
  1472. <tbody>
  1473. <tr>
  1474. <td>Time on hold for insert in altering table</td>
  1475. <td align="right">~ 0.217 sec</td>
  1476. <td align="right">~ 120 sec</td>
  1477. </tr>
  1478. <tr>
  1479. <td>Time on hold for insert in another table</td>
  1480. <td align="right">~ 0.211 sec</td>
  1481. <td align="right">~ 25 sec</td>
  1482. </tr>
  1483. </tbody>
  1484. </table>
  1485. <p>&nbsp;</p>
  1486. <p><span style="font-weight: 400;"><strong>However</strong>, yes there is a <strong>however</strong>, PXC was maintaining consistency between the different nodes during the DDL execution, while MySQL 8.0.27 with Group Replication was postponing consistency on the secondaries, thus Primary and Secondary were not in sync until full DDL finalization on the secondaries.</span></p>
  1487. <h1><span style="font-weight: 400;">Conclusions</span></h1>
  1488. <p><span style="font-weight: 400;">MySQL 8.0.27 comes with this nice fix that significantly reduces the impact of an online DDL operation on a busy server. But we can still observe a significant misalignment of the data between the nodes when a DDL is executing.&nbsp;</span></p>
  1489. <p><span style="font-weight: 400;">On the other hand PXC with NBO is a bit more “expensive” in time, but nodes remain aligned all the time.</span></p>
  1490. <p><span style="font-weight: 400;">At the end is what is more important for you to choose one or the other solution, consistency vs. operational impact.</span></p>
  1491. <p><span style="font-weight: 400;">Great MySQL to all.</span></p>]]></description>
  1492. <category>MySQL</category>
  1493. <pubDate>Tue, 11 Jan 2022 13:04:00 +0000</pubDate>
  1494. </item>
  1495. <item>
  1496. <title>A look into Percona XtraDB Cluster Non Blocking Operation for Online Schema Upgrade</title>
  1497. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/245-a-look-into-percona-xtradb-cluster-non-blocking-operation-for-online-schema-upgrade</link>
  1498. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/245-a-look-into-percona-xtradb-cluster-non-blocking-operation-for-online-schema-upgrade</guid>
  1499. <description><![CDATA[<p><span style="font-weight: 400;">Percona XtraDB Cluster 8.0.25 has introduced a new option to perform online schema modifications: NBO (<a href="https://www.percona.com/doc/percona-xtradb-cluster/LATEST/features/nbo.html#nbo">Non Blocking Operation</a>).</span></p>
  1500. <p><span style="font-weight: 400;">When using PXC the cluster relies on </span><b>wsrep_OSU_method</b><span style="font-weight: 400;"> parameter to define the Online Schema Upgrade (OSU) method the node uses to replicate DDL statements.&nbsp;&nbsp;<img src="http://www.tusacentral.net/joomla/images/stories/NBO/breaking_bariers.jpg" alt="breaking bariers" width="400" height="239" style="float: right;" /></span></p>
  1501. <p><span style="font-weight: 400;">Until now we normally have 3 options:</span></p>
  1502. <ul>
  1503. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Use Total Isolation Order (TOI, the default)</span></li>
  1504. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Use Rolling Schema Upgrade (RSU)</span></li>
  1505. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Use Percona’s online schema change tool (TOI + <a href="https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html">PTOSC</a>)</span></li>
  1506. </ul>
  1507. <p><span style="font-weight: 400;">Each method has some positive and negative aspects. TOI will lock the whole cluster from being able to accept data modifications for the entire time it takes to perform the DDL operation. RSU will misalign the schema definition between the nodes, and in any case the node performing the DDL operation is still locked. Finally TOI+PTOSC will rely on creating triggers and copying data, so in some cases this can be very impactful.&nbsp;</span></p>
  1508. <p><span style="font-weight: 400;">The new Non Blocking Operation (NBO) method is to help to reduce the impact on the cluster and make it easier to perform some DDL operations.</span></p>
  1509. <p><span style="font-weight: 400;">At the moment we only support a limited set of operations with NBO like:</span></p>
  1510. <ul>
  1511. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">ALTER INDEX</span></li>
  1512. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">CREATE INDEX</span></li>
  1513. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">DROP INDEX</span></li>
  1514. </ul>
  1515. <p><span style="font-weight: 400;">Any other command will result in an error message ER_NOT_SUPPORTED_YET.</span></p>
  1516. <p><span style="font-weight: 400;">But let us see how it works and what is the impact while we will also compare it with the default method TOI.</span></p>
  1517. <p><span style="font-weight: 400;">What we will do is working with 4 connections:</span></p>
  1518. <p style="padding-left: 40px;"><span style="font-weight: 400;">1 to perform ddl<br /></span><span style="font-weight: 400;">2 to perform insert data in the table being altered<br /></span><span style="font-weight: 400;">3 to perform insert data on a different table&nbsp;<br /></span><span style="font-weight: 400;">4-5 checking the other two nodes operations</span></p>
  1519. <p><span style="font-weight: 400;">PXC must be at least Ver 8.0.25-15.1.</span></p>
  1520. <p><span style="font-weight: 400;">The table we will modify is :</span></p>
  1521. <pre class="lang:mysql decode:true">DC1-1(root@localhost) [windmills_s]&gt;show create table windmills_test\G
  1522. *************************** 1. row ***************************
  1523.       Table: windmills_test
  1524. Create Table: CREATE TABLE `windmills_test` (
  1525.  `id` bigint NOT NULL AUTO_INCREMENT,
  1526.  `uuid` char(36) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  1527.  `millid` smallint NOT NULL,
  1528.  `kwatts_s` int NOT NULL,
  1529.  `date` date NOT NULL,
  1530.  `location` varchar(50) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  1531.  `active` tinyint NOT NULL DEFAULT '1',
  1532.  `time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  1533.  `strrecordtype` char(3) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  1534.  PRIMARY KEY (`id`),
  1535.  KEY `IDX_millid` (`millid`,`active`),
  1536.  KEY `IDX_active` (`id`,`active`),
  1537.  KEY `kuuid_x` (`uuid`),
  1538.  KEY `millid_x` (`millid`),
  1539.  KEY `active_x` (`active`)
  1540. ) ENGINE=InnoDB AUTO_INCREMENT=8199260 DEFAULT CHARSET=utf8mb3 COLLATE=utf8_bin ROW_FORMAT=DYNAMIC
  1541. 1 row in set (0.00 sec)
  1542. </pre>
  1543. <p><span style="font-weight: 400;">And contains ~5 million rows.</span></p>
  1544. <pre class="lang:sh decode:true">DC1-1(root@localhost) [windmills_s]&gt;select count(*) from windmills_test;
  1545. +----------+
  1546. | count(*) |
  1547. +----------+
  1548. |  5002909 |
  1549. +----------+
  1550. 1 row in set (0.44 sec)
  1551. </pre>
  1552. <p><span style="font-weight: 400;">The commands.<br /></span><span style="font-weight: 400;">Connection 1:</span></p>
  1553. <pre class="lang:sh decode:true">  ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE;
  1554.  ALTER TABLE windmills_test drop INDEX idx_1, ALGORITHM=INPLACE;
  1555. </pre>
  1556. <p>&nbsp;</p>
  1557. <p><span style="font-weight: 400;">Connection 2:</span></p>
  1558. <pre class="lang:sh decode:true">while [ 1 = 1 ];do da=$(date +'%s.%3N');/opt/mysql_templates/PXC8P/bin/mysql --defaults-file=./my.cnf -uroot -D windmills_s -e "insert into windmills_test  select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmills7 limit 1;" -e "select count(*) from windmills_s.windmills_test;" &gt; /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done</pre>
  1559. <p>&nbsp;</p>
  1560. <p><span style="font-weight: 400;">Connection 3:</span></p>
  1561. <pre class="lang:sh decode:true"> while [ 1 = 1 ];do da=$(date +'%s.%3N');/opt/mysql_templates/PXC8P/bin/mysql --defaults-file=./my.cnf -uroot -D windmills_s -e "insert into windmills8  select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmills7 limit 1;" -e "select count(*) from windmills_s.windmills_test;" &gt; /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done</pre>
  1562. <p>&nbsp;</p>
  1563. <p><span style="font-weight: 400;">Connections 4-5:</span></p>
  1564. <pre class="lang:sh decode:true">while [ 1 = 1 ];do echo "$(date +'%T.%3N')";/opt/mysql_templates/PXC8P/bin/mysql --defaults-file=./my.cnf -uroot -D windmills_s -e "show full processlist;"|egrep -i -e "(windmills_test|windmills_s)"|grep -i -v localhost;sleep 1;done</pre>
  1565. <p><span style="font-weight: 400;">Operations:</span></p>
  1566. <ul>
  1567. <li><span style="font-weight: 400;">start inserts from connections</span></li>
  1568. <li><span style="font-weight: 400;">start commands in connections 4 - 5 on the other nodes</span></li>
  1569. <li><span style="font-weight: 400;">execute:&nbsp;</span>
  1570. <ul>
  1571. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">for TOI&nbsp;</span>
  1572. <ul>
  1573. <li aria-level="2">
  1574. <pre class="">DC1-1(root@localhost) [windmills_s]&gt;SET SESSION wsrep_OSU_method=TOI;</pre>
  1575. </li>
  1576. </ul>
  1577. </li>
  1578. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">for NBO</span>
  1579. <ul>
  1580. <li aria-level="2">
  1581. <pre class="">DC1-1(root@localhost) [windmills_s]&gt;SET SESSION wsrep_OSU_method=NBO;</pre>
  1582. </li>
  1583. </ul>
  1584. </li>
  1585. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">For both</span>
  1586. <ul>
  1587. <li>
  1588. <pre class="">DC1-1(root@localhost) [windmills_s]&gt;ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE, LOCK=shared;</pre>
  1589. </li>
  1590. </ul>
  1591. </li>
  1592. </ul>
  1593. </li>
  1594. </ul>
  1595. <h1>&nbsp;</h1>
  1596. <h1 style="padding-left: 40px;"><span style="font-weight: 400;">Let us run it</span></h1>
  1597. <h2><span style="font-weight: 400;">Altering a table with TOI.</span></h2>
  1598. <pre class="lang:sh decode:true">DC1-1(root@localhost) [windmills_s]&gt;ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE;
  1599. Query OK, 0 rows affected (1 min 4.74 sec)
  1600. Records: 0  Duplicates: 0  Warnings: 0
  1601. </pre>
  1602. <p>&nbsp;</p>
  1603. <p><span style="font-weight: 400;">Inserts in the altering table (connection 2):</span></p>
  1604. <pre class="lang:sh decode:true">.450
  1605. .492
  1606. 64.993 &lt;--- Alter blocks all inserts on the table we are altering
  1607. .788
  1608. .609
  1609. </pre>
  1610. <p>&nbsp;</p>
  1611. <p><span style="font-weight: 400;">Inserts on the other table (connection 3):</span></p>
  1612. <pre class="lang:sh decode:true">.455
  1613. .461
  1614. 64.161 &lt;--- Alter blocks all inserts on all the other tables as well
  1615. .641
  1616. .483
  1617. </pre>
  1618. <p>&nbsp;</p>
  1619. <p><span style="font-weight: 400;">On the other nodes at the same time of the ALTER we can see:</span></p>
  1620. <pre class="lang:sh decode:true">Id  User             db         Command Time  State             Info                                                                            Time_ms Rows_sent Rows_examined
  1621. 15 system user windmills_s Query 102  altering table ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE 102238 0       0    &lt;--- time from start
  1622. </pre>
  1623. <p><span style="font-weight: 400;">So in short we have the whole cluster locked for ~64 seconds. During this period of time, all the operations to modify data or structure were on hold.&nbsp;</span></p>
  1624. <h2>&nbsp;</h2>
  1625. <h2><span style="font-weight: 400;">Let us now try with NBO</span></h2>
  1626. <p><span style="font-weight: 400;">Inserts in the altering table:</span></p>
  1627. <pre class="lang:sh decode:true">.437
  1628. .487
  1629. 120.758 &lt;---- Execution time increase
  1630. .617
  1631. .510
  1632. </pre>
  1633. <p>&nbsp;</p>
  1634. <p><span style="font-weight: 400;">Inserts on the other table:</span></p>
  1635. <pre class="lang:sh decode:true">.468
  1636. .485
  1637. 25.061 &lt;---- still a metalock, but not locking the other tables for the whole duration
  1638. .494
  1639. .471
  1640. </pre>
  1641. <p>&nbsp;</p>
  1642. <p><span style="font-weight: 400;">On the other nodes at the same time of the ALTER we can see:</span></p>
  1643. <pre class="lang:sh decode:true">Id      User         db             Command Time  State             Info                                                                            Time_ms Rows_sent Rows_examined
  1644. 110068 system user windmills_s Connect 86  altering table ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE 120420 0          0
  1645. </pre>
  1646. <p>&nbsp;</p>
  1647. <p><span style="font-weight: 400;">In this case what is also interesting to note is that:</span></p>
  1648. <ol>
  1649. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">We have a moment of metalock:</span><ol>
  1650. <li aria-level="2">
  1651. <pre class="">110174 pmm 127.0.0.1:42728 NULL Query 2 Waiting for table metadata lock  SELECT x FROM information_schema.tables&nbsp; WHERE TABLE_SCHEMA = 'windmills_s' 1486 10  &nbsp; 0</pre>
  1652. </li>
  1653. <li aria-level="2">
  1654. <pre class="">110068  system user connecting host windmills_s Connect 111 closing tables ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE 111000 0 0</pre>
  1655. </li>
  1656. </ol></li>
  1657. <li><span style="font-weight: 400;">The execution time is longer </span></li>
  1658. </ol>
  1659. <p><span style="font-weight: 400;">Summarizing:</span></p>
  1660. <pre class="lang:sh decode:true">                                      TOI            NBO
  1661. Time on hold for insert for altering table   ~64 sec     ~120 sec
  1662. Time on hold for insert for another table   ~64 sec      ~25 sec
  1663. metalock                       whole time   only at the end
  1664. </pre>
  1665. <h2>&nbsp;</h2>
  1666. <h2><span style="font-weight: 400;">What is happening, what are the differences and why takes longer with NBO?</span></h2>
  1667. <p><span style="font-weight: 400;">Let see at very high level how the two works:</span></p>
  1668. <ul>
  1669. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">TOI: when you issue a DDL like ADD INDEX a metadata lock is taken on the table and it will be released only at the end of the operation. During this time, you cannot:&nbsp;</span>
  1670. <ul>
  1671. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">Perform DMLs on any cluster node</span></li>
  1672. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">Alter another table in the cluster</span></li>
  1673. </ul>
  1674. </li>
  1675. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">NBO: the metadata lock is taken at the start and at the end for a very brief period of time. The ADD INDEX operation will then work on each node independently. The lock taken at the end is to have all the nodes agree on the operation and commit or roll back (using cluster error voting). This final phase costs a bit more in time and is what adds a few seconds to the operation execution. But during the operation:</span>
  1676. <ul>
  1677. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">You can alter another table (using NBO)</span></li>
  1678. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">You can continue to insert data, except in the table(s) you are altering.</span></li>
  1679. <li style="font-weight: 400;" aria-level="2"><span style="font-weight: 400;">On node crash the operation will continue on the other nodes, and if successful it will persist.&nbsp;&nbsp;</span></li>
  1680. </ul>
  1681. </li>
  1682. </ul>
  1683. <p><span style="font-weight: 400;">In short the cluster server behavior changes significantly when using NBO, offering significant flexibility compared to TOI. The cost in time should not linearly increase with the dimension of the table, but more in relation to the single node efficiency in performing the ALTER operation.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span></p>
  1684. <h1><span style="font-weight: 400;">Conclusion</span></h1>
  1685. <p><span style="font-weight: 400;">NBO can be significantly helpful to reduce the impact of DDL on the cluster, for now limited to the widely used creation/modification/drop of an index. But in the future … we may expand it.&nbsp;</span></p>
  1686. <p><span style="font-weight: 400;">The feature is still a technology preview, so do not trust in production, but test it and let us know what you think.&nbsp;</span></p>
  1687. <p><span style="font-weight: 400;">Final comment. Another distribution has introduced NBO, but only if you buy the enterprise version.</span></p>
  1688. <p><span style="font-weight: 400;">Percona, which is truly open source with facts not just words, has implemented NBO in standard PXC, and the code is fully open source. This is not the first one, but just another of the many features Percona is offering for free, while others ask you to buy the enterprise version.</span></p>
  1689. <p><span style="font-weight: 400;">Enjoy the product and let us have your feedback!</span></p>
  1690. <p><span style="font-weight: 400;">Great MySQL to all!&nbsp;</span></p>
  1691. <p>&nbsp;</p>]]></description>
  1692. <category>MySQL</category>
  1693. <pubDate>Fri, 10 Dec 2021 10:00:29 +0000</pubDate>
  1694. </item>
  1695. <item>
  1696. <title>What if … MySQL’s repeatable reads cause you to lose money?</title>
  1697. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/244-what-if-mysql-s-repeatable-reads-cause-you-to-lose-money</link>
  1698. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/244-what-if-mysql-s-repeatable-reads-cause-you-to-lose-money</guid>
  1699. <description><![CDATA[<p><span style="font-weight: 400;">Well, let me say if that happens because there is a logic mistake in your application. But you need to know and understand what happens in MySQL to be able to avoid the problem.&nbsp;</span></p>
  1700. <p><span style="font-weight: 400;">In short the WHY of this article is to inform you about possible pitfalls and how to prevent that to cause you damage.&nbsp;<img src="http://www.tusacentral.net/joomla/images/stories/repeatable_read/pitfalls1.jpg" alt="pitfalls1" style="float: right;" /></span></p>
  1701. <p><span style="font-weight: 400;">Let us start by having a short introduction to what Repeatable reads are about. Given I am extremely lazy, I am going to use (a lot) existing documentation from <a href="https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html">MySQL documentation.</a></span></p>
  1702. <p><i><span style="font-weight: 400;">Transaction isolation is one of the foundations of database processing. Isolation is the I in the acronym ACID; the isolation level is the setting that fine-tunes the balance between performance and reliability, consistency, and reproducibility of results when multiple transactions are making changes and performing queries at the same time.</span></i></p>
  1703. <p><i><span style="font-weight: 400;">InnoDB offers all four transaction isolation levels described by the SQL:1992 standard: READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ, and SERIALIZABLE. The default isolation level for InnoDB is REPEATABLE READ.</span></i></p>
  1704. <ul>
  1705. <li style="font-weight: 400;" aria-level="1"><i><span style="font-weight: 400;">REPEATABLE READ</span></i><i><span style="font-weight: 400;"><br /></span></i><i><span style="font-weight: 400;">This is the default isolation level for InnoDB. Consistent reads within the same transaction read the snapshot established by the first read. This means that if you issue several plain (nonlocking) SELECT statements within the same transaction, these SELECT statements are consistent also with respect to each other.</span></i></li>
  1706. </ul>
  1707. <ul>
  1708. <li style="font-weight: 400;" aria-level="1"><i><span style="font-weight: 400;">READ COMMITTED</span></i><i><span style="font-weight: 400;"><br /></span></i><i><span style="font-weight: 400;">Each consistent read, even within the same transaction, sets and reads its own fresh snapshot.</span></i></li>
  1709. </ul>
  1710. <p><span style="font-weight: 400;">And about Consistent Non blocking reads:</span><span style="font-weight: 400;"><br /></span><i><span style="font-weight: 400;">A consistent read means that InnoDB uses multi-versioning to present to a query a snapshot of the database at a point in time. The query sees the changes made by transactions that committed before that point in time, and no changes made by later or uncommitted transactions. The exception to this rule is that the query sees the changes made by earlier statements within the same transaction. This exception causes the following anomaly: If you update some rows in a table, a SELECT sees the latest version of the updated rows, but it might also see older versions of any rows. If other sessions simultaneously update the same table, the anomaly means that you might see the table in a state that never existed in the database.</span></i></p>
  1711. <p><span style="font-weight: 400;">Ok, but what does all this mean in practice?</span></p>
  1712. <p><span style="font-weight: 400;">To understand, let us simulate this scenario:</span></p>
  1713. <p><span style="font-weight: 400;">I have a shop and I decide to grant a bonus discount to a selected number of customers that:</span></p>
  1714. <ul>
  1715. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Have an active account to my shop</span></li>
  1716. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Match my personal criteria to access the bonus</span></li>
  1717. </ul>
  1718. <p><span style="font-weight: 400;">My application is set to perform batch operations in a moment of less traffic and unattended.&nbsp;</span></p>
  1719. <p><span style="font-weight: 400;">This includes reactivating dormant accounts that customers ask to reactivate.&nbsp;&nbsp;</span></p>
  1720. <p><span style="font-weight: 400;">What we will do is to see what happens, by default, then see what we can do to avoid pitfalls.</span></p>
  1721. <h1><span style="font-weight: 400;">The scenario</span></h1>
  1722. <p><span style="font-weight: 400;">I will use 3 different connections to connect to the same MySQL 8.0.27 instance. The only relevant setting I have modified is the Innodb_timeout that I set to 50 seconds.&nbsp;</span></p>
  1723. <ul>
  1724. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Session 1 will simulate a process that should activate the bonus feature for the selected customers.</span></li>
  1725. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Session 2 is an independent process that reactivate the given list of customers</span></li>
  1726. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Session 3 is used to collect lock information.</span></li>
  1727. </ul>
  1728. <p><span style="font-weight: 400;">For this simple test I will use the customer table in the sakila schema modified as below:</span></p>
  1729. <pre class="lang:mysql decode:true">CREATE TABLE `customer` (
  1730.  `customer_id` smallint unsigned NOT NULL AUTO_INCREMENT,
  1731.  `store_id` tinyint unsigned NOT NULL,
  1732.  `first_name` varchar(45) NOT NULL,
  1733.  `last_name` varchar(45) NOT NULL,
  1734.  `email` varchar(50) DEFAULT NULL,
  1735.  `address_id` smallint unsigned NOT NULL,
  1736.  `active` tinyint(1) NOT NULL DEFAULT '1',
  1737.  `create_date` datetime NOT NULL,
  1738.  `last_update` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  1739.  `bonus` int NOT NULL DEFAULT '0',
  1740.  `activate_bonus` varchar(45) NOT NULL DEFAULT '0',
  1741.  PRIMARY KEY (`customer_id`),
  1742.  KEY `idx_fk_store_id` (`store_id`),
  1743.  KEY `idx_fk_address_id` (`address_id`),
  1744.  KEY `idx_last_name` (`last_name`),
  1745.  KEY `idx_bonus` (`bonus`),
  1746.  CONSTRAINT `fk_customer_address` FOREIGN KEY (`address_id`) REFERENCES `address` (`address_id`) ON DELETE RESTRICT ON UPDATE CASCADE,
  1747.  CONSTRAINT `fk_customer_store` FOREIGN KEY (`store_id`) REFERENCES `store` (`store_id`) ON DELETE RESTRICT ON UPDATE CASCADE
  1748. ) ENGINE=InnoDB AUTO_INCREMENT=600 DEFAULT CHARSET=utf8mb4 COLLATE=ut
  1749. </pre>
  1750. <p><span style="font-weight: 400;">As you can see I have added the </span><i><span style="font-weight: 400;">bonus</span></i><span style="font-weight: 400;"> and </span><i><span style="font-weight: 400;">activate_bonus</span></i><span style="font-weight: 400;"> fields plus the </span><i><span style="font-weight: 400;">idx_bonus</span></i><span style="font-weight: 400;"> index.</span></p>
  1751. <p><span style="font-weight: 400;">To be able to trace the locks these are the threads ids by session:</span></p>
  1752. <pre class="lang:sh decode:true">session 1 17439
  1753. session 2 17430
  1754. session 3 17443
  1755. </pre>
  1756. <p><span style="font-weight: 400;">To collect the lock information:</span></p>
  1757. <pre class="lang:mysql decode:true">SELECT
  1758.    index_name, lock_type, lock_mode, lock_data, thread_id
  1759. FROM
  1760.    performance_schema.data_locks
  1761. WHERE
  1762.    object_schema = 'sakila'
  1763.        AND object_name = 'customer'
  1764.        AND lock_type = 'RECORD'
  1765.        AND thread_id IN (17439 , 17430)
  1766. ORDER BY index_name , lock_data DESC;
  1767. </pre>
  1768. <p><span style="font-weight: 400;">ok , ready? Let us start!</span></p>
  1769. <h1><span style="font-weight: 400;">The run…</span></h1>
  1770. <p><span style="font-weight: 400;">While the following steps can be done in a more </span><i><span style="font-weight: 400;">compressed</span></i><span style="font-weight: 400;"> way, I prefer to do it in a more verbose way, to make it more human readable.</span></p>
  1771. <p><span style="font-weight: 400;">First let us set the environment:</span></p>
  1772. <pre class="lang:mysql decode:true">session1 &gt;set transaction_isolation = 'REPEATABLE-READ';
  1773. Query OK, 0 rows affected (0.07 sec)
  1774. session1 &gt;Start Transaction;
  1775. Query OK, 0 rows affected (0.07 sec)
  1776. </pre>
  1777. <p><span style="font-weight: 400;">Then let see the list of the customers we will modify:</span></p>
  1778. <pre class="lang:mysql decode:true">session1 &gt;SELECT * FROM sakila.customer where bonus = 1 and active =1 order by last_name;
  1779. +-------------+----------+------------+-----------+-------+----------------+
  1780. | customer_id | store_id | first_name | last_name | bonus | activate_bonus |
  1781. +-------------+----------+------------+-----------+-------+----------------+
  1782. |         383 |        1 | MARTIN     | BALES     |     1 | 0              |
  1783. |         539 |        1 | MATHEW     | BOLIN     |     1 | 0              |
  1784. |         441 |        1 | MARIO      | CHEATHAM  |     1 | 0              |
  1785. |         482 |        1 | MAURICE    | CRAWLEY   |     1 | 0              |
  1786. |         293 |        2 | MAE        | FLETCHER  |     1 | 0              |
  1787. |          38 |        1 | MARTHA     | GONZALEZ  |     1 | 0              |
  1788. |         444 |        2 | MARCUS     | HIDALGO   |     1 | 0              |
  1789. |         252 |        2 | MATTIE     | HOFFMAN   |     1 | 0              |
  1790. |         256 |        2 | MABEL      | HOLLAND   |     1 | 0              |
  1791. |         226 |        2 | MAUREEN    | LITTLE    |     1 | 0              |
  1792. |         588 |        1 | MARION     | OCAMPO    |     1 | 0              |
  1793. |         499 |        2 | MARC       | OUTLAW    |     1 | 0              |
  1794. |         553 |        1 | MAX        | PITT      |     1 | 0              |
  1795. |         312 |        2 | MARK       | RINEHART  |     1 | 0              |
  1796. |          80 |        1 | MARILYN    | ROSS      |     1 | 0              |
  1797. |         583 |        1 | MARSHALL   | THORN     |     1 | 0              |
  1798. |         128 |        1 | MARJORIE   | TUCKER    |     1 | 0              |
  1799. |          44 |        1 | MARIE      | TURNER    |     1 | 0              |
  1800. |         267 |        1 | MARGIE     | WADE      |     1 | 0              |
  1801. |         240 |        1 | MARLENE    | WELCH     |     1 | 0              |
  1802. |         413 |        2 | MARVIN     | YEE       |     1 | 0              |
  1803. +-------------+----------+------------+-----------+-------+----------------+
  1804. 21 rows in set (0.08 sec)
  1805. </pre>
  1806. <p><span style="font-weight: 400;">As you can see we have 21 customers fitting our criteria.<br /></span><span style="font-weight: 400;">How much money is involved in this exercise?</span></p>
  1807. <pre class="lang:mysql decode:true">session1 &gt;SELECT
  1808.    -&gt;     SUM(amount) income,
  1809.    -&gt;     SUM(amount) * 0.90 income_with_bonus,
  1810.    -&gt;    (SUM(amount) - (SUM(amount) * 0.90)) loss_because_bonus
  1811.    -&gt; FROM
  1812.    -&gt;     sakila.customer AS c
  1813.    -&gt;         JOIN
  1814.    -&gt;     sakila.payment AS p ON c.customer_id = p.customer_id
  1815.    -&gt;     where active = 1 and bonus =1 ;
  1816. +---------+-------------------+--------------------+
  1817. | income  | income_with_bonus | loss_because_bonus |
  1818. +---------+-------------------+--------------------+
  1819. | 2416.23 |         2174.6070 |           241.6230 |
  1820. +---------+-------------------+--------------------+
  1821. </pre>
  1822. <p><span style="font-weight: 400;">This exercise is going to cost me </span><b>~242</b><span style="font-weight: 400;"> dollars. Keep this number in mind.<br /></span><span style="font-weight: 400;">What locks do I have at this point?</span></p>
  1823. <pre class="lang:mysql decode:true">session3 &gt;select index_name, lock_type, lock_mode,lock_data from performance_schema.data_locks where  object_schema = 'sakila' and object_name = 'customer' and lock_type = 'RECORD'  and
  1824. thread_id in (17439,17430) order by index_name, lock_data desc;
  1825. Empty set (0.00 sec)
  1826. </pre>
  1827. <p><span style="font-weight: 400;">Answer is none.&nbsp;</span></p>
  1828. <p><span style="font-weight: 400;">Meanwhile we have the other process that needs to reactivate the customers:</span></p>
  1829. <pre class="lang:mysql decode:true">session2 &gt;set transaction_isolation = 'REPEATABLE-READ';
  1830. Query OK, 0 rows affected (0.00 sec)
  1831.  
  1832. session2 &gt;Start Transaction;
  1833. Query OK, 0 rows affected (0.00 sec)
  1834.  
  1835. session2 &gt;SELECT * FROM sakila.customer where bonus = 1 and active =0 ;
  1836. +-------------+----------+------------+-----------+-------+----------------+
  1837. | customer_id | store_id | first_name | last_name | bonus | activate_bonus |
  1838. +-------------+----------+------------+-----------+-------+----------------+
  1839. |           1 |        1 | MARY       | SMITH     |     1 | 0              |
  1840. |           7 |        1 | MARIA      | MILLER    |     1 | 0              |
  1841. |           9 |        2 | MARGARET   | MOORE     |     1 | 0              |
  1842. |         178 |        2 | MARION     | SNYDER    |     1 | 0              |
  1843. |         236 |        1 | MARCIA     | DEAN      |     1 | 0              |
  1844. |         246 |        1 | MARIAN     | MENDOZA   |     1 | 0              |
  1845. |         254 |        2 | MAXINE     | SILVA     |     1 | 0              |
  1846. |         257 |        2 | MARSHA     | DOUGLAS   |     1 | 0              |
  1847. |         323 |        2 | MATTHEW    | MAHAN     |     1 | 0              |
  1848. |         408 |        1 | MANUEL     | MURRELL   |     1 | 0              |
  1849. +-------------+----------+------------+-----------+-------+----------------+
  1850. 10 rows in set (0.00 sec)
  1851. </pre>
  1852. <p><span style="font-weight: 400;">In this case the process needs to reactivate 10 users.</span></p>
  1853. <pre class="lang:mysql decode:true">session2 &gt;update sakila.customer set active = 1 where bonus = 1 and active =0 ;
  1854. Query OK, 10 rows affected (0.00 sec)
  1855. Rows matched: 10  Changed: 10  Warnings: 0
  1856. session2 &gt;commit;
  1857. Query OK, 0 rows affected (0.00 sec)
  1858. </pre>
  1859. <p><span style="font-weight: 400;">All good, right? But before going ahead let us double check, session 1:</span></p>
  1860. <pre class="lang:mysql decode:true">session1 &gt;SELECT * FROM sakila.customer where bonus = 1 and active =1 order by last_name;
  1861. +-------------+----------+------------+-----------+-------+----------------+
  1862. | customer_id | store_id | first_name | last_name | bonus | activate_bonus |
  1863. +-------------+----------+------------+-----------+-------+----------------+
  1864. |         383 |        1 | MARTIN     | BALES     |     1 | 0              |
  1865. |         539 |        1 | MATHEW     | BOLIN     |     1 | 0              |
  1866. |         441 |        1 | MARIO      | CHEATHAM  |     1 | 0              |
  1867. |         482 |        1 | MAURICE    | CRAWLEY   |     1 | 0              |
  1868. |         293 |        2 | MAE        | FLETCHER  |     1 | 0              |
  1869. |          38 |        1 | MARTHA     | GONZALEZ  |     1 | 0              |
  1870. |         444 |        2 | MARCUS     | HIDALGO   |     1 | 0              |
  1871. |         252 |        2 | MATTIE     | HOFFMAN   |     1 | 0              |
  1872. |         256 |        2 | MABEL      | HOLLAND   |     1 | 0              |
  1873. |         226 |        2 | MAUREEN    | LITTLE    |     1 | 0              |
  1874. |         588 |        1 | MARION     | OCAMPO    |     1 | 0              |
  1875. |         499 |        2 | MARC       | OUTLAW    |     1 | 0              |
  1876. |         553 |        1 | MAX        | PITT      |     1 | 0              |
  1877. |         312 |        2 | MARK       | RINEHART  |     1 | 0              |
  1878. |          80 |        1 | MARILYN    | ROSS      |     1 | 0              |
  1879. |         583 |        1 | MARSHALL   | THORN     |     1 | 0              |
  1880. |         128 |        1 | MARJORIE   | TUCKER    |     1 | 0              |
  1881. |          44 |        1 | MARIE      | TURNER    |     1 | 0              |
  1882. |         267 |        1 | MARGIE     | WADE      |     1 | 0              |
  1883. |         240 |        1 | MARLENE    | WELCH     |     1 | 0              |
  1884. |         413 |        2 | MARVIN     | YEE       |     1 | 0              |
  1885. +-------------+----------+------------+-----------+-------+----------------+
  1886. 21 rows in set (0.08 sec)
  1887. </pre>
  1888. <p><span style="font-weight: 400;">Perfect! My Repeatable Read still sees the same snapshot. Let me apply the changes:</span></p>
  1889. <pre class="lang:mysql decode:true">session1 &gt;update sakila.customer set activate_bonus=1 where bonus = 1 and active =1 ;
  1890. <strong>Query OK, 31 rows affected (0.06 sec)</strong>
  1891. Rows matched: 31  Changed: 31  Warnings: 0
  1892. </pre>
  1893. <p><span style="font-weight: 400;">Wait, what? My list reports 21 entries, but I have modified 31! And if I check the cost:</span></p>
  1894. <pre class="lang:mysql decode:true">session1 &gt;SELECT
  1895.    -&gt;     SUM(amount) income,
  1896.    -&gt;     SUM(amount) * 0.90 income_with_bonus,
  1897.    -&gt;    (SUM(amount) - (SUM(amount) * 0.90)) loss_because_bonus
  1898.    -&gt; FROM
  1899.    -&gt;     sakila.customer AS c
  1900.    -&gt;         JOIN
  1901.    -&gt;     sakila.payment AS p ON c.customer_id = p.customer_id
  1902.    -&gt;     where active = 1 and bonus =1 ;
  1903. +---------+-------------------+--------------------+
  1904. | income  | income_with_bonus | loss_because_bonus |
  1905. +---------+-------------------+--------------------+
  1906. | 3754.01 |         3378.6090 |           375.4010 |
  1907. +---------+-------------------+--------------------+
  1908. </pre>
  1909. <p><span style="font-weight: 400;">Well now the cost of this operation is</span><b> 375</b><span style="font-weight: 400;"> dollars not </span><b>242</b><span style="font-weight: 400;">. In this case we are talking about peanuts, but guess what could be if we do something similar on thousands of users.&nbsp;</span></p>
  1910. <p><span style="font-weight: 400;">Anyhow let us first:</span></p>
  1911. <pre class="lang:mysql decode:true">session1 &gt;rollback;
  1912. Query OK, 0 rows affected (0.08 sec)
  1913. </pre>
  1914. <p><span style="font-weight: 400;">And cancel the operation.</span></p>
  1915. <p><span style="font-weight: 400;">So what happened, is this a bug?</span></p>
  1916. <p><span style="font-weight: 400;">No it is not! It is how repeatable reads work in MySQL. The snapshot is relevant to the read operation, But if another session is able to write given also the absence of lock at the moment of reads, the next update operation will touch any value in the table matching the conditions.&nbsp;</span></p>
  1917. <p><span style="font-weight: 400;">As shown above this can be very dangerous. But only if you don’t do the right things in the code.&nbsp;</span></p>
  1918. <h2><span style="font-weight: 400;">How can I prevent this from happening?</span></h2>
  1919. <p><span style="font-weight: 400;">When coding, hope for the best, plan for the worse, always! Especially when dealing with databases. That approach may save you from spending nights trying to fix the impossible.&nbsp;</span></p>
  1920. <p><span style="font-weight: 400;">So how can this be prevented? You have two ways, both simple, but both with positive and negative consequences.</span></p>
  1921. <ol>
  1922. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Add this simple clausole to the select statement: </span><i><span style="font-weight: 400;">for share</span></i></li>
  1923. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Add the other simple clausole to the select statement: </span><i><span style="font-weight: 400;">for update</span></i></li>
  1924. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Use Read-committed</span></li>
  1925. </ol>
  1926. <p><span style="font-weight: 400;">Solution 1 is easy and clean, no other change in the code. BUT create locks, and if your application is lock sensitive this may be an issue for you.</span></p>
  1927. <p><span style="font-weight: 400;">Solution 2 is also easy, but a bit more encapsulated and locking.&nbsp;</span></p>
  1928. <p><span style="font-weight: 400;">On the other hand Solution 3 does not add locks, but requires modifications in the code and it still leaves some space for problems.</span></p>
  1929. <p><span style="font-weight: 400;">Let us see them in detail.</span></p>
  1930. <h3><span style="font-weight: 400;">Solution 1</span></h3>
  1931. <p><span style="font-weight: 400;">Let us repeat the same steps</span></p>
  1932. <pre class="lang:mysql decode:true">session1 &gt;set transaction_isolation = 'REPEATABLE-READ';
  1933. session1 &gt;Start Transaction;
  1934. session1 &gt;SELECT * FROM sakila.customer where bonus = 1 and active =1 order by last_name for share;
  1935. </pre>
  1936. <p><span style="font-weight: 400;">If now we check the locks for brevity I have cut some entries and kept a few as a sample:</span></p>
  1937. <pre class="lang:mysql decode:true">session3 &gt;select index_name, lock_type, lock_mode,lock_data,thread_id from performance_schema.data_locks where
  1938.    -&gt; object_schema = 'sakila'
  1939.    -&gt; and object_name = 'customer'
  1940.    -&gt; and lock_type = 'RECORD'
  1941.    -&gt; and thread_id = 17439
  1942.    -&gt; order by index_name, lock_data desc;
  1943. +------------+-----------+---------------+------------------------+-----------+
  1944. | index_name | lock_type | lock_mode     | lock_data              | thread_id |
  1945. +------------+-----------+---------------+------------------------+-----------+
  1946. | idx_bonus  | RECORD    | S             | supremum pseudo-record |     17439 |
  1947. | idx_bonus  | RECORD    | S             | 1, 9                   |     17439 |
  1948. &lt;snip&gt;
  1949. | idx_bonus  | RECORD    | S             | 1, 128                 |     17439 |
  1950. | idx_bonus  | RECORD    | S             | 1, 1                   |     17439 |
  1951. | PRIMARY    | RECORD    | S,REC_NOT_GAP | 9                      |     17439 |
  1952. &lt;snip&gt;
  1953. | PRIMARY    | RECORD    | S,REC_NOT_GAP | 1                      |     17439 |
  1954. +------------+-----------+---------------+------------------------+-----------+
  1955. 63 rows in set (0.00 sec)
  1956. </pre>
  1957. <p><span style="font-weight: 400;">This time we can see that the select has raised a few locks all in S (shared) mode.&nbsp;</span></p>
  1958. <p><span style="font-weight: 400;">For brevity I am skipping to report exactly the same results as in the first exercise.</span></p>
  1959. <p><span style="font-weight: 400;">If we go ahead and try with session 2:</span></p>
  1960. <pre class="lang:mysql decode:true">session2 &gt;set transaction_isolation = 'READ-COMMITTED';
  1961. session2 &gt;Start Transaction;
  1962. session2 &gt;SELECT * FROM sakila.customer where bonus = 1 and active =0 ;
  1963. session2 &gt;update sakila.customer set active = 1 where bonus = 1 and active =0 ;
  1964. ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
  1965. </pre>
  1966. <p><span style="font-weight: 400;">Here we go! Attempting to change the values in the customer table is now on hold waiting to acquire the lock. If the duration exceeds the Innodb_wait_timeout, then the execution is interrupted and the application must have a </span><i><span style="font-weight: 400;">try-catch</span></i><span style="font-weight: 400;"> mechanism to retry the operation. This last one is a best practice that you should already have in place in your code. If not, well add it now!</span></p>
  1967. <p><span style="font-weight: 400;">At this point session 1 can proceed and complete the operations. After that and before the final </span><i><span style="font-weight: 400;">commit</span></i><span style="font-weight: 400;">, we will be able to observe:</span></p>
  1968. <pre class="lang:mysql decode:true">+------------+-----------+---------------+------------------------+-----------+
  1969. | index_name | lock_type | lock_mode     | lock_data              | thread_id |
  1970. +------------+-----------+---------------+------------------------+-----------+
  1971. | idx_bonus  | RECORD    | S             | supremum pseudo-record |     17439 |
  1972. | idx_bonus  | RECORD    | X             | supremum pseudo-record |     17439 |
  1973. | idx_bonus  | RECORD    | S             | 1, 9                   |     17439 |
  1974. | idx_bonus  | RECORD    | X             | 1, 9                   |     17439 |
  1975. &lt;snip&gt;
  1976. | idx_bonus  | RECORD    | X             | 1, 1                   |     17439 |
  1977. | idx_bonus  | RECORD    | S             | 1, 1                   |     17439 |
  1978. | PRIMARY    | RECORD    | X,REC_NOT_GAP | 9                      |     17439 |
  1979. | PRIMARY    | RECORD    | S,REC_NOT_GAP | 9                      |     17439 |
  1980. &lt;snip&gt;|
  1981. | PRIMARY    | RECORD    | X,REC_NOT_GAP | 1                      |     17439 |
  1982. | PRIMARY    | RECORD    | S,REC_NOT_GAP | 1                      |     17439 |
  1983. +------------+-----------+---------------+------------------------+-----------+
  1984. 126 rows in set (0.00 sec)
  1985. </pre>
  1986. <p><span style="font-weight: 400;">As you can see we now have two different lock types, the Shared one from the select, and the exclusive lock (X) from the update.</span></p>
  1987. <h3><span style="font-weight: 400;">Solution 2</span></h3>
  1988. <p><span style="font-weight: 400;">In this case we just change the kind of lock we set on the select. In solution 1 we opt for a shared lock, which allows other sessions to eventually acquire another shared lock. Here we go for an exclusive lock that will put all the other operations on hold. </span></p>
  1989. <pre class="lang:mysql decode:true">session1 &gt;set transaction_isolation = 'REPEATABLE-READ';
  1990. session1 &gt;Start Transaction;
  1991. session1 &gt;SELECT * FROM sakila.customer where bonus = 1 and active =1 order by last_name for UPDATE;
  1992. </pre>
  1993. <p><span style="font-weight: 400;">If now we check the locks for brevity I have cut some entries and kept few as a sample:</span></p>
  1994. <pre class="lang:mysql decode:true">session3 &gt;select index_name, lock_type, lock_mode,lock_data,thread_id from performance_schema.data_locks where
  1995.    -&gt; object_schema = 'sakila'
  1996.    -&gt; and object_name = 'customer'
  1997.    -&gt; and lock_type = 'RECORD'
  1998.    -&gt; and thread_id = 17439
  1999.    -&gt; order by index_name, lock_data desc;
  2000. +------------+-----------+---------------+------------------------+-----------+
  2001. | index_name | lock_type | lock_mode     | lock_data              | thread_id |
  2002. +------------+-----------+---------------+------------------------+-----------+
  2003. | idx_bonus  | RECORD    | X             | supremum pseudo-record |     17439 |
  2004. | idx_bonus  | RECORD    | X             | 1, 9                   |     17439 |
  2005. &lt;snip&gt;
  2006. | idx_bonus  | RECORD    | X             | 1, 128                 |     17439 |
  2007. | idx_bonus  | RECORD    | X             | 1, 1                   |     17439 |
  2008. | PRIMARY    | RECORD    | X,REC_NOT_GAP | 9                      |     17439 |
  2009. | PRIMARY    | RECORD    | X,REC_NOT_GAP | 80                     |     17439 |
  2010. &lt;snip&gt;
  2011. | PRIMARY    | RECORD    | X,REC_NOT_GAP | 128                    |     17439 |
  2012. | PRIMARY    | RECORD    | X,REC_NOT_GAP | 1                      |     17439 |
  2013. +------------+-----------+---------------+------------------------+-----------+
  2014. 63 rows in set (0.09 sec)
  2015. </pre>
  2016. <p><span style="font-weight: 400;">In this case the kind of lock is X as exclusive, so the other operations when requesting a lock must wait for this transaction to complete.</span></p>
  2017. <pre class="lang:mysql decode:true">session2 &gt;#set transaction_isolation = 'READ-COMMITTED';
  2018. session2 &gt;
  2019. session2 &gt;Start Transaction;
  2020. Query OK, 0 rows affected (0.06 sec)
  2021.  
  2022. session2 &gt;SELECT * FROM sakila.customer where bonus = 1 and active =0 for update;
  2023. </pre>
  2024. <p><span style="font-weight: 400;">And it will wait for N time where N is either the Innodb_lock_wait_timeout or the commit time from season 1.</span></p>
  2025. <p><span style="font-weight: 400;">Note the lock request on hold:</span></p>
  2026. <pre class="lang:mysql decode:true crayon-selected">+------------+-----------+---------------+------------------------+-----------+
  2027. | index_name | lock_type | lock_mode     | lock_data              | thread_id |
  2028. +------------+-----------+---------------+------------------------+-----------+
  2029. | idx_bonus  | RECORD    | X             | supremum pseudo-record |     17439 |
  2030. &lt;snip&gt;
  2031. | idx_bonus  | RECORD    | X             | 1, 128                 |     17439 |
  2032. | idx_bonus  | RECORD    | X             | 1, 1                   |     17430 |&lt;--
  2033. | idx_bonus  | RECORD    | X             | 1, 1                   |     17439 |
  2034. | PRIMARY    | RECORD    | X,REC_NOT_GAP | 9                      |     17439 |
  2035. &lt;snip&gt;
  2036. | PRIMARY    | RECORD    | X,REC_NOT_GAP | 1                      |     17439 |
  2037. +------------+-----------+---------------+------------------------+-----------+
  2038. 64 rows in set (0.09 sec)
  2039. </pre>
  2040. <p><span style="font-weight: 400;">Session 2 is trying to lock exclusively the idx_bonus but cannot proceed.</span></p>
  2041. <p><span style="font-weight: 400;">Once session 1 goes ahead we have:</span></p>
  2042. <pre class="lang:mysql decode:true">session1 &gt;update sakila.customer set activate_bonus=1 where bonus = 1 and active =1 ;
  2043. Query OK, 0 rows affected (0.06 sec)
  2044. Rows matched: 21  Changed: 0  Warnings: 0
  2045.  
  2046. session1 &gt;
  2047. session1 &gt;SELECT * FROM sakila.customer where bonus = 1 and active =1 ;
  2048. +-------------+----------+------------+-----------+-------+----------------+
  2049. | customer_id | store_id | first_name | last_name | bonus | activate_bonus |
  2050. +-------------+----------+------------+-----------+-------+----------------+
  2051. |          38 |        1 | MARTHA     | GONZALEZ  |     1 | 1              |
  2052. |          44 |        1 | MARIE      | TURNER    |     1 | 1              |
  2053. |          80 |        1 | MARILYN    | ROSS      |     1 | 1              |
  2054. |         128 |        1 | MARJORIE   | TUCKER    |     1 | 1              |
  2055. |         226 |        2 | MAUREEN    | LITTLE    |     1 | 1              |
  2056. |         240 |        1 | MARLENE    | WELCH     |     1 | 1              |
  2057. |         252 |        2 | MATTIE     | HOFFMAN   |     1 | 1              |
  2058. |         256 |        2 | MABEL      | HOLLAND   |     1 | 1              |
  2059. |         267 |        1 | MARGIE     | WADE      |     1 | 1              |
  2060. |         293 |        2 | MAE        | FLETCHER  |     1 | 1              |
  2061. |         312 |        2 | MARK       | RINEHART  |     1 | 1              |
  2062. |         383 |        1 | MARTIN     | BALES     |     1 | 1              |
  2063. |         413 |        2 | MARVIN     | YEE       |     1 | 1              |
  2064. |         441 |        1 | MARIO      | CHEATHAM  |     1 | 1              |
  2065. |         444 |        2 | MARCUS     | HIDALGO   |     1 | 1              |
  2066. |         482 |        1 | MAURICE    | CRAWLEY   |     1 | 1              |
  2067. |         499 |        2 | MARC       | OUTLAW    |     1 | 1              |
  2068. |         539 |        1 | MATHEW     | BOLIN     |     1 | 1              |
  2069. |         553 |        1 | MAX        | PITT      |     1 | 1              |
  2070. |         583 |        1 | MARSHALL   | THORN     |     1 | 1              |
  2071. |         588 |        1 | MARION     | OCAMPO    |     1 | 1              |
  2072. +-------------+----------+------------+-----------+-------+----------------+
  2073. 21 rows in set (0.08 sec)
  2074.  
  2075. session1 &gt;SELECT      SUM(amount) income,     SUM(amount) * 0.90 income_with_bonus,    (SUM(amount) - (SUM(amount) * 0.90)) loss_because_bonus FROM     sakila.customer AS c         JOIN     sakila.payment AS p ON c.customer_id = p.customer_id     where active = 1 and bonus =1;
  2076. +---------+-------------------+--------------------+
  2077. | income  | income_with_bonus | loss_because_bonus |
  2078. +---------+-------------------+--------------------+
  2079. | 2416.23 |         2174.6070 |           241.6230 |
  2080. +---------+-------------------+--------------------+
  2081. 1 row in set (0.06 sec)
  2082. </pre>
  2083. <p><span style="font-weight: 400;">Now my update matches the expectations and the other operations are on hold.</span></p>
  2084. <p><span style="font-weight: 400;">After session 1 commits.</span></p>
  2085. <pre class="lang:mysql decode:true">session2 &gt;SELECT * FROM sakila.customer where bonus = 1 and active =0 for update;
  2086. +-------------+----------+------------+-----------+-------+----------------+
  2087. | customer_id | store_id | first_name | last_name | bonus | activate_bonus |
  2088. +-------------+----------+------------+-----------+-------+----------------+
  2089. |           1 |        1 | MARY       | SMITH     |     1 | 1              |
  2090. |           7 |        1 | MARIA      | MILLER    |     1 | 1              |
  2091. |           9 |        2 | MARGARET   | MOORE     |     1 | 1              |
  2092. |         178 |        2 | MARION     | SNYDER    |     1 | 1              |
  2093. |         236 |        1 | MARCIA     | DEAN      |     1 | 1              |
  2094. |         246 |        1 | MARIAN     | MENDOZA   |     1 | 1              |
  2095. |         254 |        2 | MAXINE     | SILVA     |     1 | 1              |
  2096. |         257 |        2 | MARSHA     | DOUGLAS   |     1 | 1              |
  2097. |         323 |        2 | MATTHEW    | MAHAN     |     1 | 1              |
  2098. |         408 |        1 | MANUEL     | MURRELL   |     1 | 1              |
  2099. +-------------+----------+------------+-----------+-------+----------------+
  2100. <span style="text-decoration: underline;"><strong>10 rows in set (52.72 sec) &lt;-- Note the time!</strong></span>
  2101.  
  2102. session2 &gt;update sakila.customer set active = 1 where bonus = 1 and active =0 ;
  2103. Query OK, 10 rows affected (0.06 sec)
  2104. Rows matched: 10  Changed: 10  Warnings: 0
  2105. </pre>
  2106. <p><span style="font-weight: 400;">Session 2 is able to complete BUT it was on hold for 52 seconds waiting for session 1.<br /></span><span style="font-weight: 400;">As said this solution is a good one if you can afford locks and waiting time.</span></p>
  2107. <h3><span style="font-weight: 400;">Solution 3</span></h3>
  2108. <p><span style="font-weight: 400;">In this case we will use a different isolation model that will allow session 1 to </span><i><span style="font-weight: 400;">see</span></i><span style="font-weight: 400;"> what session 2 has modified.</span></p>
  2109. <pre class="lang:mysql decode:true">session1 &gt;set transaction_isolation = 'READ-COMMITTED';
  2110. session1 &gt;Start Transaction;
  2111. session1 &gt;SELECT * FROM sakila.customer where bonus = 1 and active =1 order by last_name;
  2112. +-------------+----------+------------+-----------+-------+----------------+
  2113. | customer_id | store_id | first_name | last_name | bonus | activate_bonus |
  2114. +-------------+----------+------------+-----------+-------+----------------+
  2115. |         383 |        1 | MARTIN     | BALES     |     1 | 0              |
  2116. |         539 |        1 | MATHEW     | BOLIN     |     1 | 0              |
  2117. |         441 |        1 | MARIO      | CHEATHAM  |     1 | 0              |
  2118. |         482 |        1 | MAURICE    | CRAWLEY   |     1 | 0              |
  2119. |         293 |        2 | MAE        | FLETCHER  |     1 | 0              |
  2120. |          38 |        1 | MARTHA     | GONZALEZ  |     1 | 0              |
  2121. |         444 |        2 | MARCUS     | HIDALGO   |     1 | 0              |
  2122. |         252 |        2 | MATTIE     | HOFFMAN   |     1 | 0              |
  2123. |         256 |        2 | MABEL      | HOLLAND   |     1 | 0              |
  2124. |         226 |        2 | MAUREEN    | LITTLE    |     1 | 0              |
  2125. |         588 |        1 | MARION     | OCAMPO    |     1 | 0              |
  2126. |         499 |        2 | MARC       | OUTLAW    |     1 | 0              |
  2127. |         553 |        1 | MAX        | PITT      |     1 | 0              |
  2128. |         312 |        2 | MARK       | RINEHART  |     1 | 0              |
  2129. |          80 |        1 | MARILYN    | ROSS      |     1 | 0              |
  2130. |         583 |        1 | MARSHALL   | THORN     |     1 | 0              |
  2131. |         128 |        1 | MARJORIE   | TUCKER    |     1 | 0              |
  2132. |          44 |        1 | MARIE      | TURNER    |     1 | 0              |
  2133. |         267 |        1 | MARGIE     | WADE      |     1 | 0              |
  2134. |         240 |        1 | MARLENE    | WELCH     |     1 | 0              |
  2135. |         413 |        2 | MARVIN     | YEE       |     1 | 0              |
  2136. +-------------+----------+------------+-----------+-------+----------------+
  2137. 21 rows in set (0.08 sec)
  2138. </pre>
  2139. <p><span style="font-weight: 400;">Same result as before. Now let us execute commands in session2:</span></p>
  2140. <pre class="lang:mysql decode:true">session2 &gt;set transaction_isolation = 'READ-COMMITTED';
  2141. Query OK, 0 rows affected (0.00 sec)
  2142.  
  2143. session2 &gt;
  2144. session2 &gt;Start Transaction;
  2145. Query OK, 0 rows affected (0.00 sec)
  2146.  
  2147. session2 &gt;SELECT * FROM sakila.customer where bonus = 1 and active =0 ;
  2148. +-------------+----------+------------+-----------+-------+----------------+
  2149. | customer_id | store_id | first_name | last_name | bonus | activate_bonus |
  2150. +-------------+----------+------------+-----------+-------+----------------+
  2151. |           1 |        1 | MARY       | SMITH     |     1 | 0              |
  2152. |           7 |        1 | MARIA      | MILLER    |     1 | 0              |
  2153. |           9 |        2 | MARGARET   | MOORE     |     1 | 0              |
  2154. |         178 |        2 | MARION     | SNYDER    |     1 | 0              |
  2155. |         236 |        1 | MARCIA     | DEAN      |     1 | 0              |
  2156. |         246 |        1 | MARIAN     | MENDOZA   |     1 | 0              |
  2157. |         254 |        2 | MAXINE     | SILVA     |     1 | 0              |
  2158. |         257 |        2 | MARSHA     | DOUGLAS   |     1 | 0              |
  2159. |         323 |        2 | MATTHEW    | MAHAN     |     1 | 0              |
  2160. |         408 |        1 | MANUEL     | MURRELL   |     1 | 0              |
  2161. +-------------+----------+------------+-----------+-------+----------------+
  2162. 10 rows in set (0.00 sec)
  2163. session2 &gt;update sakila.customer set active = 1 where bonus = 1 and active =0 ;
  2164. Query OK, 10 rows affected (0.00 sec)
  2165. Rows matched: 10  Changed: 10  Warnings: 0
  2166.  
  2167. session2 &gt;commit;
  2168. Query OK, 0 rows affected (0.00 sec)
  2169. </pre>
  2170. <p><span style="font-weight: 400;">All seems the same, but if we check again in session 1:</span></p>
  2171. <pre class="lang:mysql decode:true">session1 &gt;SELECT * FROM sakila.customer where bonus = 1 and active =1 order by last_name;
  2172. +-------------+----------+------------+-----------+------+----------------+
  2173. | customer_id | store_id | first_name | last_name |bonus | activate_bonus |
  2174. +-------------+----------+------------+-----------+------+----------------+
  2175. |         383 |        1 | MARTIN     | BALES     |    1 | 1              |
  2176. |         539 |        1 | MATHEW     | BOLIN     |    1 | 1              |
  2177. |         441 |        1 | MARIO      | CHEATHAM  |    1 | 1              |
  2178. |         482 |        1 | MAURICE    | CRAWLEY   |    1 | 1              |
  2179. |         236 |        1 | MARCIA     | DEAN      |    1 | 0              |
  2180. |         257 |        2 | MARSHA     | DOUGLAS   |    1 | 0              |
  2181. |         293 |        2 | MAE        | FLETCHER  |    1 | 1              |
  2182. |          38 |        1 | MARTHA     | GONZALEZ  |    1 | 1              |
  2183. |         444 |        2 | MARCUS     | HIDALGO   |    1 | 1              |
  2184. |         252 |        2 | MATTIE     | HOFFMAN   |    1 | 1              |
  2185. |         256 |        2 | MABEL      | HOLLAND   |    1 | 1              |
  2186. |         226 |        2 | MAUREEN    | LITTLE    |    1 | 1              |
  2187. |         323 |        2 | MATTHEW    | MAHAN     |    1 | 0              |
  2188. |         246 |        1 | MARIAN     | MENDOZA   |    1 | 0              |
  2189. |           7 |        1 | MARIA      | MILLER    |    1 | 0              |
  2190. |           9 |        2 | MARGARET   | MOORE     |    1 | 0              |
  2191. |         408 |        1 | MANUEL     | MURRELL   |    1 | 0              |
  2192. |         588 |        1 | MARION     | OCAMPO    |    1 | 1              |
  2193. |         499 |        2 | MARC       | OUTLAW    |    1 | 1              |
  2194. |         553 |        1 | MAX        | PITT      |    1 | 1              |
  2195. |         312 |        2 | MARK       | RINEHART  |    1 | 1              |
  2196. |          80 |        1 | MARILYN    | ROSS      |    1 | 1              |
  2197. |         254 |        2 | MAXINE     | SILVA     |    1 | 0              |
  2198. |           1 |        1 | MARY       | SMITH     |    1 | 0              |
  2199. |         178 |        2 | MARION     | SNYDER    |    1 | 0              |
  2200. |         583 |        1 | MARSHALL   | THORN     |    1 | 1              |
  2201. |         128 |        1 | MARJORIE   | TUCKER    |    1 | 1              |
  2202. |          44 |        1 | MARIE      | TURNER    |    1 | 1              |
  2203. |         267 |        1 | MARGIE     | WADE      |    1 | 1              |
  2204. |         240 |        1 | MARLENE    | WELCH     |    1 | 1              |
  2205. |         413 |        2 | MARVIN     | YEE       |    1 | 1              |
  2206. +-------------+----------+------------+-----------+------+----------------+
  2207. 31 rows in set (0.09 sec)
  2208. </pre>
  2209. <p><span style="font-weight: 400;">We now can see all the 31 customers and also the value calculation:</span></p>
  2210. <pre class="lang:mysql decode:true">+---------+-------------------+--------------------+
  2211. | income  | income_with_bonus | loss_because_bonus |
  2212. +---------+-------------------+--------------------+
  2213. | 3754.01 |         3378.6090 |           375.4010 |
  2214. +---------+-------------------+--------------------+
  2215. </pre>
  2216. <p><span style="font-weight: 400;">Is reporting the right values.</span></p>
  2217. <p><span style="font-weight: 400;">At this point we can program some logic in the process to check the possible tolerance, and have the process either complete performing the update operations or to stop the process and exit.</span></p>
  2218. <p><span style="font-weight: 400;">This as mentioned requires additional coding and more logic.</span></p>
  2219. <p><span style="font-weight: 400;">To be fully honest, this solution still leaves space to some possible interference, but at least allow the application to be informed about what is happening. Still it should be used only if you cannot afford to have a higher level of locking, but this is another story/article.</span></p>
  2220. <h1><span style="font-weight: 400;">Conclusions</span></h1>
  2221. <p><span style="font-weight: 400;">When writing applications that interact with a RDBMS, you must be very careful in what you do and HOW you do it. While using data facilitation layers like Object Relational Mapping (ORM), seems to make your life easier, in reality you may lose control on a few crucial aspects of the application’s interaction. So be very careful when opting for how to access your data.</span></p>
  2222. <p><span style="font-weight: 400;">About the case reported above we can summarize a few pitfalls:</span></p>
  2223. <ol>
  2224. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">First and most important, never ever have processes that may interfere with the one running at the same time. Be very careful when you design them and even more careful when planning their executions.&nbsp;&nbsp;</span></li>
  2225. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Use options such as </span><i><span style="font-weight: 400;">for share</span></i><span style="font-weight: 400;"> or </span><i><span style="font-weight: 400;">for update </span></i><span style="font-weight: 400;">in your select statements. Use them carefully to avoid unuseful locks, but use them.&nbsp;</span></li>
  2226. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">In case you have a long process that requires modifying data, and you need to be able to check if other processes have altered the status of the data, but at the same time, cannot have a long wait time for locks. Then use a mix, setting READ-COMMITTED as isolation, to allow your application to check, but also add things like </span><i><span style="font-weight: 400;">for share </span></i><span style="font-weight: 400;">or </span><i><span style="font-weight: 400;">for update</span></i><span style="font-weight: 400;"> in the select statement, immediately before the DML and the commit. That will allow you to prevent writes while you are finalizing, and also to significantly reduce the locking time.&nbsp;</span></li>
  2227. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Keep in mind </span><b>that long running processes and long running transactions </b><span style="font-weight: 400;">can be</span><b> the source of a lot of pain</b><span style="font-weight: 400;">, especially when using REPEATABLE-READ. Try whenever you can to split the operations in small chunks.&nbsp;</span></li>
  2228. </ol>
  2229. <p><span style="font-weight: 400;">Finally when developing, remember that DBAs are friends, and are there to help you to do the things at the best. It may seem they are giving you a hard time, but that is because their point of view is different, focusing on data consistency, availability and durability. But they can help you to save a lot of time after the code is released, especially when you try to identify why something has gone wrong.&nbsp;</span></p>
  2230. <p><span style="font-weight: 400;">So involve them soon in the design process, use them and let them be part of the process. </span></p>
  2231. <h1><span style="font-weight: 400;">References</span></h1>
  2232. <p><a href="https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html"><span style="font-weight: 400;">https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html</span></a></p>
  2233. <p><a href="https://dev.mysql.com/doc/refman/8.0/en/innodb-consistent-read.html"><span style="font-weight: 400;">https://dev.mysql.com/doc/refman/8.0/en/innodb-consistent-read.html</span></a></p>
  2234. <p><a href="https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-model.html"><span style="font-weight: 400;">https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-model.html</span></a></p>]]></description>
  2235. <category>MySQL</category>
  2236. <pubDate>Tue, 09 Nov 2021 10:11:31 +0000</pubDate>
  2237. </item>
  2238. <item>
  2239. <title>MySQL on Kubernetes demystified</title>
  2240. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/243-mysql-on-kubernetes-demystified</link>
  2241. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/243-mysql-on-kubernetes-demystified</guid>
  2242. <description><![CDATA[<h1>Why<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/stop-befool.jpg" alt="stop befool" style="float: right;" /></h1>
  2243. <p>Marco, why did you write this long article?</p>
  2244. <p>Yes, it is long, and I know most of the people will not read it in full, but my hope is that at least someone will, and I count on them to make the wave of sanity.&nbsp;</p>
  2245. <p>Why I wrote it is simple. We write articles to share something we discover, or to share new approaches or as in this case to try to demystify and put in the right perspective the “last shining thing” that will save the world.&nbsp;&nbsp;</p>
  2246. <p>The “last shining thing” is the use of containers for relational databases management systems (RDBMS) and all the attached solutions like Kubernetes or similar.&nbsp;</p>
  2247. <p>Why is this a problem? The use of containers for RDBMS is not really a problem per se, but it had become a problem because it was not correctly contextualized and even more important, the model that should be used to properly design the solutions, was not reviewed and modified in respect to the classic one.</p>
  2248. <p>One example for all is this image:</p>
  2249. <p><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/db_on_kube_1.png" alt="db on kube 1" width="671" height="386" /></p>
  2250. <p>Source (<a href="https://www.cockroachlabs.com/blog/kubernetes-trends/"></a><a href="https://www.cockroachlabs.com/blog/kubernetes-trends/">https://www.cockroachlabs.com/blog/kubernetes-trends/</a>)&nbsp;</p>
  2251. <p>In this report we find the use of the term Database multiple times, and reference how easy it is to adopt and scale a Database on Kubernetes. But the problem is … what Database? Not all the Databases are the same, not all can be adopted so easily, some use more restrictive design to be real RDBMS, others less.&nbsp;</p>
  2252. <p>Some are designed to scale horizontally, others no. The other missed part is that, to get something you need to give something. If I design a system to scale horizontally, I probably have to pay a price on something else. It could be the lack of referential integrity or the Isolation level, it doesn’t matter, but you do not get anything for free.&nbsp;</p>
  2253. <p>Generalizing without mentioning the difference is misleading.</p>
  2254. <p>What happens instead is that we have a lot of people, presenting solutions that are so generic that are unusable. The most hilarious thing is that they present that as an inevitable evolution, the step into the future that will solve every problem. But when doing this, they do not clarify what is the “Database” in use, what you get, what you lose. This may lead to misunderstanding and future frustration.&nbsp;</p>
  2255. <p>For instance we constantly see presentations that illustrate how easy it will be to manage hundreds, thousands of pods containing RDBMS, without even understanding the concept of RDBMS, the data they may host and its dimension.&nbsp;</p>
  2256. <p>The more we go ahead the more dangerous this disinformation is becoming, because we can say that on 100 companies currently using RDBMS to manage their crucial data, only 10 have a good team of experts that understand how to really use containers, and probably only 1 or 2 of them have real data experts that are able to redesign the data layer to be correctly utilized in containers or able to see to what proper Database to migrate to achieve the expected results.</p>
  2257. <p>Another common expectation is to move to container/kubernetes to reduce the costs, no matter if related to the iron (physical servers) or of the management of them.&nbsp;</p>
  2258. <p>Indeed you can optimise that part, but you need to understand the limitations of your future solution. You must take into account that you may not have the same level of service in a way or another. Honestly I haven't seen that addressed in current blogging or reports.&nbsp;</p>
  2259. <p>This is why this article. What I want to do is to open a door to a discussion, that will lead us to review the original model used to design data inside RDBMS, that will allow anyone to safely approach a different model to use for modern database design.</p>
  2260. <h1>When</h1>
  2261. <p class="btn-inverse">(if you know how we get here you can skip this)</p>
  2262. <p>To fully understand what we are talking about we need to do a jump back in time, because without history we are nothing, without memory we are lost. Just note that I am going to touch on things at a very high level and for what we are concerned, otherwise this will be a book not an article.</p>
  2263. <p>A long time ago in a world far far away where the internet was not, there was: client server approach.</p>
  2264. <p><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/ClientServerArchitecture1.png" alt="ClientServerArchitecture1" width="340" height="223" />&nbsp;</p>
  2265. <p>At that time we had many clients connecting to a server to provide access to whatever. Most of the clients were performing information rendering and local data validation, then they sent the information back to the application server who processed them and store… where? Well there was a high variety of containers. Some were just proprietary files with custom format, others connected to an external database.&nbsp; What is relevant here is that we had a very limited number of clients (often less than 1k) and at the end the amount of data in transit and then store was very small (if a database had more than 1GB data in total&nbsp; it was considered a monster).&nbsp;</p>
  2266. <p>&nbsp;</p>
  2267. <p>Then came the internet … and many things started to change.&nbsp;</p>
  2268. <p><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/clientserver-web.gif" alt="clientserver web" width="431" height="205" /></p>
  2269. <p>We still have a client on our PC, but now is called a web Browser, the information rendering is now standardize using SGML standards (HTML tagging), the server is not anymore a simple Application Server, but a web server who is connected to one or many application(s) using common gateway interface (CGI). Each application was then handling the data in/out independently, some using databases some not.&nbsp;</p>
  2270. <p>This new approach add to the previous model many challenges like:</p>
  2271. <ul>
  2272. <li aria-level="1">Anyone from anywhere can access the Web Server and any application hosted</li>
  2273. <li aria-level="1">The number of requests and clients connected jump from well predictable numbers to something impossible to define.</li>
  2274. <li aria-level="1">The connection to a web server follows a different approach based on a request issued by a browser and a set of information that will be sent as an answer from the server. (later we will have more interactive/active protocols but let us stay high ok?)&nbsp;</li>
  2275. <li aria-level="1">Many application were duplicating the same functions but with different approach</li>
  2276. <li aria-level="1">Data received and sent needs to be consistent not only locally on the server, but also between requests.&nbsp;</li>
  2277. </ul>
  2278. <p>So on top of the problem generated by the number of possible clients connected and the amount of requests per second a web server was in need to process, the initial model was wasting resources in development and when running. There was the need to optimise the interactions between applications and their functions, to cover that the Service Oriented Architecture (SOA) model was largely adopted.</p>
  2279. <p><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/1920px-SOA_Metamodelsvg.png" alt="1920px SOA Metamodelsvg" width="500" height="413" /></p>
  2280. <p>The data problem was partially solved identifying RDBMS as the best tool to use to guarantee the level of data consistency and at the same time it was possible to organize data in containers (tables) with validated interactions (Foreign keys and constraints) solving the online transaction processing (OLTP) problem. While for business analysis and reporting the Online analytical processing (OLAP) model was chosen. The OLAP model is based on facts and dimensions, which are elements of the data hypercube which is the base model that is utilized to first transform and then access the data.&nbsp;&nbsp;</p>
  2281. <p>During this time, the data as volume starts to increase significantly, most of the systems, when well designed, start to implement the concept of archiving, partitioning, and sharding.&nbsp;</p>
  2282. <p>We are around 2005-2008.</p>
  2283. <p>&nbsp;</p>
  2284. <p>Then social media and online shopping explode...&nbsp;</p>
  2285. <p>Well we know that, right? We are still in this huge big bang of nonsense. What changed was not only that now anyone can access whatever is on a web server as certified information, but we can chat with anyone, we can buy almost anything, we can sell anything, we can make photos and share them, videos and ask anyone from anywhere to comment and we can say whatever and that crap will remain forever around.&nbsp;</p>
  2286. <p>And guess what? To do that you need a very, very, very powerful scalable platform and it should be resilient, but for real not as a joke. When we look for something today, we open our phones, we expect to have that now and no matter what. If we do not get it, we may look to a different place, like another shop, or another chat system or whatever.&nbsp; To achieve this you do not only need to have some specialized bloks, but to split the load in many small parts, then be able to have redundancy for each one of them, and be able to have any functional block able to connect to another functional block. Moreover, if one of the blocks goes down I must be able to retry the operation using another functional block of the same kind, and be sure I will have the same results.&nbsp;&nbsp;</p>
  2287. <p>To achieve this the SOA concept was evolved into the Microservice concept. But microservice also brings new challenges. One for all is how to manage these thousands of different functional bloks, and how to efficiently deploy/remove/modify them.&nbsp;</p>
  2288. <p>Interestingly the term DevOps also started to get more relevance, and automation became a crucial part of ANY environment.&nbsp;</p>
  2289. <p>In short the modern application architectures moves from Monolithic or SOA to microservice as shown here:</p>
  2290. <p><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/microservice-draft-3-03-db.png" alt="microservice draft 3 03 db" width="500" height="263" /></p>
  2291. <p>Of course reality sees each microservice block be redundant and scalable. Given that for each microservice kind we can have 1 to infinite number of instances.&nbsp;</p>
  2292. <p>It is in this context that the use of Kubernetes comes in help. Kubernetes is designed to help manage the resources as microservices, each element (pod) is seen in function of the service provided, the service will scale up and down&nbsp; adding/removing the pod, keeping the service resilient.&nbsp;</p>
  2293. <p>Expecting to deal with a structure like this without proper tooling/automation is just not possible:</p>
  2294. <p><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/amazon-death-star.png" alt="amazon death star" width="500" height="332" /></p>
  2295. <p>This picture represent the Amazon microservice infrastructure in 2008 <a href="https://twitter.com/Werner/status/741673514567143424">https://twitter.com/Werner/status/741673514567143424</a></p>
  2296. <h2>In conclusion</h2>
  2297. <p>In short, we can say that the complex topologies we have to deal with today are the results of a needed evolution from monolithic application approach to a distributed one in services/microservices. The proliferation of which had made absolutely necessary the creation and usage of a tool to facilitate the management, reducing complexity, time to perform the operations and costs.&nbsp;</p>
  2298. <h1>Cool eh? What about the database?&nbsp;</h1>
  2299. <p>The first thing to say is that RDBMS as Oracle, Postgres and MySQL did not evolve in the same direction. They remain centralized structures. We have clusters but mostly to answer to High Availability needs and not scaling.&nbsp;</p>
  2300. <p>This is the part of the story, the current evangelists of the “last shining thing” are not telling you. There is no coverage for the RDBMS like Oracle, Postgres or MySQL in the microservice approach. None, zero, nich, nada!</p>
  2301. <p>To bypass that problem, many microservices developers implemented complex mechanisms of caching or use noSQL approaches. <br />Using caches to speedup the processing is absolutely ok, trying to use them instead of an ACID solution, is not. Same as the use of noSQL, if it makes sense because the service requires then ok otherwise no.</p>
  2302. <p>But what is the problem in seeing a RDBMS as a microservice?<br />Let us start with a positive thing. If you design it as a microservice and reset the expectations, then it is ok.</p>
  2303. <h2>What benefits (pros)?&nbsp;</h2>
  2304. <p>The first thought is why I should move to kubernetes/containers. What am I expecting to get more that I do not already have?</p>
  2305. <p>Well let see the few basic points a microservice solution should help me with in case of application:</p>
  2306. <ul>
  2307. <li aria-level="1">Modularity:&nbsp;
  2308. <ul>
  2309. <li aria-level="2">This makes the application easier to understand, develop, test, and become more resilient to architecture erosion.</li>
  2310. </ul>
  2311. </li>
  2312. <li aria-level="1">Scalability:&nbsp;
  2313. <ul>
  2314. <li aria-level="2">Since microservices are implemented and deployed independently of each other, i.e. they run within independent processes, they can be monitored and scaled independently.</li>
  2315. </ul>
  2316. </li>
  2317. <li aria-level="1">Better Fault Isolation for More Resilient Applications.
  2318. <ul>
  2319. <li aria-level="2">With a microservices architecture, the failure of one service is less likely to negatively impact other parts of the application because each microservice runs autonomously from the others.&nbsp;</li>
  2320. <li aria-level="2">Nevertheless, large distributed microservices architectures tend to have many dependencies, so developers need to protect the application from a dependency failure related shut down.</li>
  2321. </ul>
  2322. </li>
  2323. <li aria-level="1">Integration of heterogeneous and legacy systems:&nbsp;
  2324. <ul>
  2325. <li aria-level="2">microservices is considered as a viable means for modernizing existing monolithic software applications.</li>
  2326. <li aria-level="2">There are experience reports of several companies who have successfully replaced (parts of) their existing software with microservices, or are in the process of doing so.</li>
  2327. <li aria-level="2">The process for Software modernization of legacy applications is done using an incremental approach.</li>
  2328. </ul>
  2329. </li>
  2330. <li aria-level="1">Faster Time to Market and “Future-Proofing”&nbsp;
  2331. <ul>
  2332. <li aria-level="2">The pluggability of a microservices application architecture allows for easier and faster application development and upgrades. Developers can quickly build or change a microservice, then plug it into the architecture with less risk of coding conflicts and service outages. Moreover, due to the independence of each microservice, teams don’t have to worry about coding conflicts, and they don’t have to wait for slower-moving projects before launching their part of the application.</li>
  2333. <li aria-level="2">Microservices also support the CI/CD/CD (Continuous Integration, Continuous Delivery, Continuous Deployment) development philosophy. This means that you quickly deploy the core microservices of an application – as a minimum viable product (MVP) – and continuously update the product over time as you complete additional microservices. It also means that you can respond to security threats and add new technologies as they appear. The result for the user is a “future proof” product that’s constantly evolving with the times.</li>
  2334. </ul>
  2335. </li>
  2336. </ul>
  2337. <p>Of course this is for applications let us try to map similar things for a database if it makes any sense.&nbsp;</p>
  2338. <ul>
  2339. <li aria-level="1">Modularity. We can have two different level of modularity:
  2340. <ul>
  2341. <li aria-level="2">By service. When using by service we need to have data modules that serve the microservices who use that segment of data. IE login needs data about user login only, not information about customer orders.</li>
  2342. <li aria-level="2">By context. When by context, we need to group as much as possible the data relevant to a specific topic. In a multitenant database we will need to separate the data such that all microservices will be able to scale accessing the data relevant to only one tenant, not all, then filtering. IE in our example above, we will have the whole schema in our service but only for one single tenant&nbsp;&nbsp;</li>
  2343. </ul>
  2344. </li>
  2345. <li aria-level="1">Scalability. To gain infinite scalability as we do for applications, we should be able to:
  2346. <ul>
  2347. <li aria-level="2">Either the database service is able to perform non impacting distributed locking. What I mean here is that independently by the type of modularity used, if we need to be able to scale the number of requests/sec duplicating the service who answer the requests, we need to be able to avoid two different pods to overwrite eachother without or with minimal impact on the performance. As for today in the MySQL area we have two solutions that could cover that, PXC with galera and MySQL/PS with group replication. But both solutions have performance impact when using multiple writers. Not only, in these solutions data is not distributed by partitions, but duplicated in full for each pod. This means that unless you have a very good archiving strategy your space utilization will always be S x N (where s is the total space in one pod and N the number of pods).<br />Because of this, if we choose this approach, we need to keep our dataset to the minimum to reduce any locking impact and data utilization.&nbsp;&nbsp;&nbsp;&nbsp;</li>
  2348. <li aria-level="2">Or sharding the data by pods, which means we need to have a controller able to distribute the data. Of course we need to have High Availability so we need to consider at least a copy of each node or distributed partitioning. As far as I know MySQL or Postgres don’t have this functionality, Mongo not sure.</li>
  2349. </ul>
  2350. </li>
  2351. </ul>
  2352. <p>In the end, a RDBMS microservice will not be able to scale in the same way an application microservice does.&nbsp;</p>
  2353. <ul>
  2354. <li aria-level="1">Better Fault Isolation for More Resilient Applications. <br />Well it comes without saying. If the model used is by context, the fact we will isolate the data service in smaller entities will help the resilience of the whole system.&nbsp; But the relationship that exists inside a schema can make the by service model more fragile, given that when 1 segment is down it will not be possible to validate the reference .&nbsp; &nbsp;</li>
  2355. <li aria-level="1">Integration of heterogeneous and legacy systems. To be honest I can see this is possible only if modularity by service is used. Otherwise we will still have to deal with the whole structure and multiple services. In the last case I don’t see any benefit and we will have to perform operations like modifying a table definition, that can be as fast as renaming a field, or very impactful.</li>
  2356. <li aria-level="1">Faster Time to Market and “Future-Proofing”<br />Indeed the divide and conquer approach especially if using the by service model will help in making each service more flexible to changes, but if by context is used, then we may have a negative effect because an application microservice may read data from N data microservices. If we change one data microservice (ie adding an index to a table), this change must be applied to all of the N microservice by context or the behaviour of the microservices will be different from the one with the change.</li>
  2357. <li aria-level="1">Distributed development. Also in this case we need to consider it easier to achieve if modularity by service is used. Otherwise I don’t really see how it will be possible to perform any of the mentioned points that are valid for applications.&nbsp;</li>
  2358. </ul>
  2359. <h2>What concerns (cons)?</h2>
  2360. <p>Now let us discuss the possible concerns.</p>
  2361. <ul>
  2362. <li aria-level="1">Services form information barriers. Well it comes without saying.&nbsp;</li>
  2363. <li aria-level="1">Inter-service calls over a network have a higher cost in terms of network latency and message processing time than in-process calls within a monolithic service process. Just think about some standard maintenance work that needs to be done to consolidate the data from each service to a single location for business intelligence purposes. And keep in mind that all must be encrypted for security reasons.&nbsp;&nbsp;</li>
  2364. <li aria-level="1">Testing and deployment are more complicated. While acting on a single module will be easier, the deployment of the full solution and its testing will become extremely complicated.&nbsp;&nbsp;</li>
  2365. <li aria-level="1">Viewing the size of services as the primary structuring mechanism can lead to too many services when the alternative of internal modularization may lead to a simpler design. This is true when using the by service modularity, we may end up having too many fragments of data, with serious difficulties in consolidating it. And of course, we still need to know and keep an eye on what is happening on the rest of the schema used by other services.</li>
  2366. <li aria-level="1">Two-phased commits are regarded as an anti-pattern in microservices-based architectures as this results in a tighter coupling of all the participants within the transaction. However, lack of this technology causes awkward dances which have to be implemented by all the transaction participants in order to maintain data consistency.</li>
  2367. <li aria-level="1">Development and support of many services is more challenging if they are built with different tools and technologies. It is not uncommon to have complex applications use different data sources, like RDBMS and a document store. While the use of microservice can help in simplifying how the data is collected and handled from these different sources, the additional fragmentation leads to a significant increase of the complexity to handle the platform. To be honest, solutions as Kubernetes can help a lot here, but still an issue.&nbsp;&nbsp;</li>
  2368. <li aria-level="1">The protocol typically used with microservices (HTTP) was designed for public-facing services, and as such is unsuitable for working internal microservices that often must be impeccably reliable. This issue applies to databases in a different way. Because we will have 2 different aspects. The first one is the application microservice connecting and reading/writing to the data microservice. These operations must use the proprietary client protocol of the RDBMS. Then the second part is when the application microservice is sharing/serving the data to others. Say I am posting a payment and a network glitch happens between application microservices. I must be able to:
  2369. <ul>
  2370. <li aria-level="2">Retry the data sharing</li>
  2371. <li aria-level="2">Rollback the operation on the database if failing</li>
  2372. </ul>
  2373. </li>
  2374. </ul>
  2375. <p>The HTTP protocol is not the best/most performant way to that given the post/get approach used.&nbsp;&nbsp;</p>
  2376. <h2>Let me give an example to clarify this.&nbsp;</h2>
  2377. <p>We are an organization who does multiple activities. We have shops all over the globe and a huge segment of my sales is online. We also organize travels and host web-stores. To deal with the traffic and load I have moved my large part of the application architecture to microservice. To provide better services to my customers I also have to differentiate my catalog by geographic regions, splitting my market into North America, Europe, Middle-East, Asia and pacific. I have data centers in each geo-zone. Each geo-zone also hosts a local data repository that is focused on the specifics of the area. Finally my sales are consolidated in two steps, first locally by geo-area, then centrally in the North America HQ. We have hundreds of database services for each segment of business, and thousands of pods of the application.</p>
  2378. <p>One of mine online shop that use microservices at high level looks like this for each geo-zone:</p>
  2379. <p><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/db-as-service1.png" alt="db as service1" width="349" height="221" /></p>
  2380. <p>Where Service-1 can be the login of a user and its authentication, Service-2 the show/retrieval of the items in special-sales, Service-3 dealing with existing order information … and so on.&nbsp;</p>
  2381. <p>So I have specialized microservices that may serve multiple applications, like sign on, but each microservice at the moment needs to retrieve and send data from/to the same repository.&nbsp;</p>
  2382. <p>Each database contains data relevant to a specific business activity. For instance our main store data is separate from the travel data and so on. Because this the “My DB” above just represents one of many databases we have. The detailed design of it may resemble this:</p>
  2383. <p><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/schema-db.gif" alt="schema db" width="693" height="870" /></p>
  2384. <p>All credits for this diagram go to Barry Williams</p>
  2385. <h2>Let us play the game, what do I need to do to convert this to a microservice?</h2>
  2386. <p>Let us now try to apply what we have discussed so far about the expectations (pros/cons) and see if we will be able to make this fit. Given I am a MySQL expert I am going to cover the next step in the MySQL context.</p>
  2387. <h3>Case one Modular by context</h3>
  2388. <p>The first step is to choose the kind of modularity we want/can use. Let us start with the one which requires less changes, the module by context.&nbsp;</p>
  2389. <p>When using that approach our schema remains the same, what we do is in case of multi-tenant to make it single-tenant, then we need to see how to cover the other aspects.&nbsp;</p>
  2390. <p>&nbsp;</p>
  2391. <p>The next step is scalability, and here we start to have problems. As already mentioned above there is no way to achieve the same level of scalability we have with the application layer. So what can we really do?&nbsp;</p>
  2392. <p>As mentioned in the MySQL ecosystem we have two major solutions, one is Percona XtraDB Cluster (PXC) using galera virtually synchronous replication, the other is MySQL or Percona Server with group replication (GR). We have another player here which may actually fit much better, but at the time of writing there is no container solution for it. I am talking about the MySQL NDB cluster.</p>
  2393. <p>Anyhow both PXC and GR allow to have Single primary (one writer) or multiple Primary (multiple writers). But the use of multi Primary is discouraged because of the loss of performance caused by the write conflict resolution. What this means is that at each commit the node receiving the writes must certify the commit against the whole cluster. The more nodes write at the same time, the more chances of conflicts will rise. Each conflict will end up with a conflict resolution and a possible rollback of the operation. All these instead adding scalability is actually impacting it and significantly reducing the performance.</p>
  2394. <p>On the other hand, reads can easily scale using multiple nodes. But keep in mind that the two solutions cannot really scale in number of nodes to the infinite. GR has a hard limit of nine nodes, while PXC has no hard limit, but more nodes, more latency at each write commit and more cost in maintaining the cluster, in short when you reach a cluster with 7 nodes you already start to experience significant slow down and latency.&nbsp;&nbsp;&nbsp;&nbsp;</p>
  2395. <p>So what level of scalability do we have here? The answer is easy, none!&nbsp;</p>
  2396. <p>Yes none, as it is you can have and will have only 1 node (pod) writing at a given time, so your level of scalability in writing is the capacity of 1 node, period.&nbsp;</p>
  2397. <p>&nbsp;</p>
  2398. <p>Unless you adopt sharding. In that case you may have multiple microservice each covering one shard with a fixed number of nodes (pods). So the scaling is not internal to the microservice but in the duplication of it.&nbsp;</p>
  2399. <p>Application at this point has three ways to use this approach:</p>
  2400. <ul>
  2401. <li aria-level="1">Be shard aware and have a list of microservice to use when in the need (a shard catalog)</li>
  2402. <li aria-level="1">Be shard aware but refer to a generalized catalog service to access the data.</li>
  2403. <li aria-level="1">Not shard aware but pass information to the generalized catalog service to identify the shard.</li>
  2404. </ul>
  2405. <p>&nbsp;The first option is not really optimal because anytime we add a shard we need to “inform/modify” the catalog in all relevant application microservice.&nbsp;</p>
  2406. <p>The second and the third assume the presence of another service dedicated to resolving the sharding.&nbsp;&nbsp;</p>
  2407. <p>I am not going into the details here but this can be achieved by adding a ProxySQL microservice. Such that the scenario will be:</p>
  2408. <p><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/db-as-service-proxy-microservice-sharding.png" alt="db as service proxy microservice sharding" width="439" height="402" /></p>
  2409. <p>Where each data service shard is a Percona Distribution for MySQL Operator (PDMO).</p>
  2410. <p>Scalability is then guaranteed by adding a PDMO shard and just an entry in ProxySQL in the mysql servers and query rules.&nbsp;</p>
  2411. <p>&nbsp;</p>
  2412. <p>Better Fault Isolation for More Resilient Applications. Adopting the above model will allow us to be not only more resilient in case we migrate from a multi-tenant approach, but also in case we do not. No matter what if the application is correctly written if a shard will go down only that segment will be affected.&nbsp;</p>
  2413. <p>&nbsp;</p>
  2414. <p>Integration and changes will become more cumbersome given the scale of the services. This is in the model and will represent a risk in case of mistakes or improper data definition modifications. But it will be easier to roll back by shard than against the monolithic data service.&nbsp;</p>
  2415. <p>Distributed development. There will not be any advantage here, as developer or DBA I still need to consider the whole data definition scenario and not just a small segment. So while a developer can focus only on one aspect (i.e. login), a DBA still needs to take care of the whole schema and the impact of the operations (like locking) on the single table (i.e. users) for that service.&nbsp;</p>
  2416. <p>&nbsp;</p>
  2417. <p>Dimension of the dataset. But here we have another important concern, at least for now. To be so agile each single service (or microservice) needs to be able to be rebuilt quickly. This means that when a pod inside a data microservice crashes the cluster needs to rebuild it as fast as possible, or the QOS (quality of service) will be impacted. As for now both PXC and GR if in need to rebuild a node must actually copy the data over the new pod. This can be very fast in case of small datasets, or take days in case of very large. Given that when we plan to use a data microservice we also need to plan/design a very efficient archiving mechanism that allows us to stay in our microservices only the data that is really needed at the moment. All the data that can be considered not actual should be moved to an OLAP system and have dedicated service to retrieve it.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</p>
  2418. <p>&nbsp;</p>
  2419. <p><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/db-as-service-etl.png" alt="db as service etl" width="594" height="632" /></p>
  2420. <p>&nbsp;</p>
  2421. <h3>Case two modularity by service.&nbsp;</h3>
  2422. <p>What modularity by service means? It means I can split the data schema definition into small segments that correspond to the services. So my data segment will be functional to the use done by the application microservices. As an example I may have microservice dealing with: Login; shipments; invoicing … and so on.&nbsp;</p>
  2423. <p>This means that in case of the invoice I should fragment the schema as follow:</p>
  2424. <p><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/schema-invoice-order.png" alt="schema invoice order" width="500" height="357" /></p>
  2425. <p>But these are the same tables I also need to serve shipments, so I must use the same data microservice, for both. This is ok … but … the question is can we? Can we split in small segments a schema that is built on relationships? Orders are linked to Customers and to products. How do we manage these referential integrity links if we start to split the schema?&nbsp;&nbsp;</p>
  2426. <p>Again a simple answer, we cannot. If you are using a RDBMS the R is for relational and you cannot break the model and hope to have some data integrity.&nbsp;</p>
  2427. <p>If you need that approach, then please use something else like noSQL Cassandra or similar. And then consolidate the data in the OLAP system. But do not use a RDBMS.</p>
  2428. <p>Given the above the by service approach, which was the one closer to give us what we need in the microsystem area, is not utilizable when talking of RDBMS. Period.&nbsp;</p>
  2429. <h2>I want to move to containers/Kubernetes to rationalize the resource utilizations and lower management cost</h2>
  2430. <p>So far we have talked about moving to containers/kubernetes as an answer to application microservices, but there is another possible need that may lead you to see containers/kubernetes as “the solution”, the need to optimize resources.</p>
  2431. <p>&nbsp;</p>
  2432. <p>It goes without saying that if you have hundreds or more small instances of a RDBMS, sparse n many different physical servers, or also some virtualization, the move may help you. The current operators like PDMO offer you a lot of automation, to perform not only installation of the main environment, but you will be able to easily automate and manage:</p>
  2433. <ul>
  2434. <li aria-level="1">Installation</li>
  2435. <li aria-level="1">Geographic distribution</li>
  2436. <li aria-level="1">Backup/restore</li>
  2437. <li aria-level="1">Point in time recovery</li>
  2438. </ul>
  2439. <p>All this with simple commands through kubernetes.&nbsp;</p>
  2440. <p>&nbsp;</p>
  2441. <p>As also indicated in my blog <a href="http://www.tusacentral.net/joomla/index.php/mysql-blogs/242-compare-percona-distribution-for-mysql-operator-vs-aws-aurora-and-standard-rds">here</a>, you will also be able to rationalize the resource utilization reducing the costs.&nbsp;</p>
  2442. <p>&nbsp;</p>
  2443. <p>So yes, of course this can be helpful and probably you should seriously look into migrating your environment to it. But it is not for all.&nbsp;</p>
  2444. <p>Again keep in mind that Kubernetes/containers are mainly designed to work with distributed services, and when talking about resiliency, the resilient factor for kubernetes is the service, not the single pod.</p>
  2445. <p>What it means is that it could happen that a pod has issues and that it will be destroyed and rebuilt. If you have a pod containing stateless application, this takes seconds and has no impact on the cluster.&nbsp;</p>
  2446. <p>But if you are talking about the node of a cluster with several terabytes of data, that will impact the cluster and the service.&nbsp;</p>
  2447. <p>&nbsp;</p>
  2448. <p>This is it, when dealing with large clusters and kubernetes, you may need to keep into account factors that may trigger actions, like dropping a pod, that instead require more flexibility.&nbsp;</p>
  2449. <p>A good example is when a pod starts to have temporary network glitches. In the case of PDMO with PXC, the cluster will try to heal itself at PXC level. That operation may temporarily exclude a node from the cluster, and then let it join again. This operation has a cost in performance and efficiency of the cluster itself, but normally it resolves in a few seconds.</p>
  2450. <p>The same network glitches can be seen from kubernetes as a pod failure, and action may be taken directly on the pod, removing it from the cluster and rebuilding a new node. That operation not only may take hours or days, but will impact on the cluster performance in a significant way until all the data is copied over from a data node to another.</p>
  2451. <p>The addition of kubernetes also adds the need to have additional understanding on how effectively tuning the service to prevent wrong management actions like the one described. Tuning that needs to be done in respect to the traffic served and the dimension of the dataset.&nbsp;&nbsp;&nbsp;&nbsp;</p>
  2452. <p>&nbsp;</p>
  2453. <p>The advice here is to start small, adopt the help of experts in setting up the environment and tune it properly, gain good understanding and then start to increase the challenge. And never trust the defaults settings in the configuration files.</p>
  2454. <h1>Wrapping up …&nbsp;</h1>
  2455. <p>Given all the above.</p>
  2456. <p>If we are looking to containers/Kubernetes for our databases as an addition to an existing or new environment based on microservices, then the question is: <em>Are we following evolution and really providing a benefit adopting the kubernetes/microservice approach to RDBMS?</em></p>
  2457. <p>As we have seen the move to microservice was an answer to provide better flexibility, scalability and many other *bility to the application layer.&nbsp;</p>
  2458. <p>An RDBMS is, on the other hand, a system designed to answer to the ACID paradigm, which is the core of a RDBM and the model should never be broken.&nbsp;&nbsp;</p>
  2459. <p>As we have seen, it is possible to create fragments and scale, but this needs to happen in getting consistent sets of data. <br />Sharding is one option, but it requires a lot of investment in re-design the data layer and adapting the application. It will also open a lot of new problems, including data archiving and possible/probable data redundancy. All problems that require a deep analysis and proper design before moving. Not impossible but not something you should do without preparation, care and investment in time and money.&nbsp;</p>
  2460. <p>If instead the scope is to reduce the overhead in management and resource optimization, the task is easier. The scaling factor is not there, and you are not bounding your data layer to a microservice concept. But, you still need to be careful and properly analyze the move, considering the side effects not only today, but in the long run. Questions like, how big my dataset will be in 1 year, are pivotal.&nbsp;&nbsp;&nbsp;&nbsp;</p>
  2461. <p>Blindly moving your database stack to containers/kubernetes or similar, without raising the right questions will only be your source of pain and despair for the next few years. It will not be your saviour but your executioner.&nbsp;</p>
  2462. <p>At the same time there are cases where it may fit, as shown previously, but is not a blanket solution.&nbsp;</p>
  2463. <p>Keep in mind this brief list:</p>
  2464. <ul>
  2465. <li aria-level="1">An RDBMS like MySQL, does not scale writes inside a single service (at the moment of writing)</li>
  2466. <li aria-level="1">Bigger is the dataset longer it will be the possible downtime you will suffer (consider review your RTO/RPO)</li>
  2467. <li aria-level="1">You cannot have a relation 1:1 with an application microservice, unless your RDBMS contains only non relational schemas.</li>
  2468. <li aria-level="1">In case you go for microservice (and shard). You MUST have an OLAP system to serve consolidated data.</li>
  2469. <li aria-level="1">In case you are looking to reduce management cost and gain resource optimization. Start small, and keep going like that.&nbsp;&nbsp;&nbsp;&nbsp;</li>
  2470. </ul>
  2471. <p>Said that, if you instead of moving foolishly, stop and ask DBA and DATA experts for help, and design a proper system, then the use of RDBMS on containers/Kubernetes could become a nice surprise.&nbsp;</p>
  2472. <p>Last thing, you may discover that you do not need a tight RDBMS, you may discover that something else could serve your needs better like Cassandra or CockroachDB, and then you need to migrate to it as a real solution. In that case your need for data experts is even more urgent and the need to have them working closely with your developers is a mandatory step to success.&nbsp;</p>
  2473. <p>&nbsp;</p>
  2474. <h1>Disclaimer&nbsp;</h1>
  2475. <p>My blog, my opinions.&nbsp;</p>
  2476. <p>I am not, in any way or aspects, presenting here the official line of thoughts of the company I am working for.</p>
  2477. <h1>Comments</h1>
  2478. <p>Comments are welcome but given the topic and the fact this is going to be more probable (hopefully) an open discussion, I advice you to comment on <a href="https://www.linkedin.com/posts/marcotusa_mysql-on-kubernetes-demystified-activity-6854020804426915840-45mW">this linkedin post</a>.&nbsp;</p>]]></description>
  2479. <category>MySQL</category>
  2480. <pubDate>Wed, 13 Oct 2021 07:00:09 +0000</pubDate>
  2481. </item>
  2482. <item>
  2483. <title>Compare Percona Distribution for MySQL Operator VS AWS Aurora and standard RDS</title>
  2484. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/242-compare-percona-distribution-for-mysql-operator-vs-aws-aurora-and-standard-rds</link>
  2485. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/242-compare-percona-distribution-for-mysql-operator-vs-aws-aurora-and-standard-rds</guid>
  2486. <description><![CDATA[<p>I decided to write this article after a discussion with other colleagues.&nbsp; We were wondering how and if the Percona Distribution for MySQL Operator(PDMO) could help a starting bu<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/performance-best-practices.png" alt="best performance rubber stamp grunge design dust scratches effects can be easily removed clean crisp look color 88158699" width="280" height="300" style="float: right;" />siness or a new project inside an enterprise.&nbsp; We were also wondering how it behaves in relation to already well established solutions, like Amazon RDS or Google SQL.&nbsp;</p>
  2487. <p>In fact, we often see analysis and benchmarking covering huge datasets and instances with very high levels of resources . But we tend to forget that any application or solution started as something small, and it is important to identify a database platform that allows us to scale from zero to a decent amount of load, while keeping the cost as low as possible.&nbsp;&nbsp;</p>
  2488. <p>This also happened while I was trying to get a bit more performance and stability out of Percona Operator for MySQL, and identify the possible minimal entrypoint and its usage.&nbsp;</p>
  2489. <p>Given all the above I decided to perform an investigation comparing the Percona Distribution for MySQL Operatorin the two most used cloud services, AWS and Google cloud. I am working on the Azure AKS and will publish results when available.</p>
  2490. <p>The investigation was taking in consideration three possible scenarios:</p>
  2491. <ul>
  2492. <li aria-level="1">1S -&nbsp;I have a new website and my editorial system is X, my content is mainly static (Mainly Read Only)</li>
  2493. <li aria-level="1">2S - I have a new social platform, what should I use (OLTP style)</li>
  2494. <li aria-level="1">3S - I have a small application dealing with orders/payments, highly transactional&nbsp; (more TPC-C</li>
  2495. </ul>
  2496. <p>And try to identify which is the best solution to use.</p>
  2497. <p>Of course, I do not pretend to cover all cases or even at 100% the one mentioned. But I think we can say a good 70% of cases nowaday match one of the three above</p>
  2498. <h1>Tests</h1>
  2499. <p>I have split the tests in three segment one for each major provider. This to be able to better compare the results, and following the assumption that if you are already registered on a platform you are more interested in comparing data locally than cross providers.</p>
  2500. <p>As usual for the tests I have used sysbench adapted version from here (<a href="https://github.com/Tusamarco/sysbench"></a><a href="https://github.com/Tusamarco/sysbench">https://github.com/Tusamarco/sysbench</a>) and sysbench-tpcc style from here (<a href="https://github.com/Tusamarco/sysbench-tpcc"></a><a href="https://github.com/Tusamarco/sysbench-tpcc">https://github.com/Tusamarco/sysbench-tpcc</a>). The customization for standard sysbench is in the table definition, and for tpcc like because I have improved the report layout.&nbsp;</p>
  2501. <p>Given we are talking about an entry point for a new product, I was considering two different scenarios:</p>
  2502. <ul>
  2503. <li aria-level="1">When dataset fits in memory</li>
  2504. <li aria-level="1">When dataset doesn’t fits in memory&nbsp;&nbsp;</li>
  2505. </ul>
  2506. <p>Data sets dimensions:</p>
  2507. <pre class="language-bash language-bash bash"><code>+--------------------+--------+-----------+----------+-----------+-----------+<br />| TABLE_SCHEMA       | TABLES | ROWS      | DATA (M) | INDEX (M) | TOTAL(M)  |<br />+--------------------+--------+-----------+----------+-----------+-----------+<br />| tpcc               |    180 | 293113897 | 44762.00 |   9033.07 |  53795.07 |<br />| tpcc_small         |     90 |  48899475 |  7611.15 |   1542.35 |   9153.51 |<br />| windmills_large    |     80 | 317752934 | 35360.00 |  38463.79 |  73823.79 |<br />| windmills_small    |     80 |  39731506 |  4447.50 |   6075.82 |  10523.32 |<br />+--------------------+--------+-----------+----------+-----------+-----------+<br /><br /></code></pre>
  2508. <p>All tables are using InnoDB. Keep in mind that scenario 1S and scenario 2S use the same schemas (windmills_xx).</p>
  2509. <p>This is because normally when you start your new adventure, you do not have Terabytes of data in your database, but .. you plan to have them in the future. So most of the time you start with something very small and want to have a solution able to scale at least a bit.</p>
  2510. <p>I was also considering whether they need to be able to scale (a bit) so I was running tests using 64, 92,128, 256, 512 threads.&nbsp;</p>
  2511. <p>Finally each test was executed for 100 minutes, multiple times in different moments of the day and of the week. When multiple crashes happen the result will be zero.&nbsp;</p>
  2512. <p>Summarizing we have:</p>
  2513. <ul>
  2514. <li aria-level="1">3 different scenarios
  2515. <ul>
  2516. <li aria-level="2">Read Only (when write are so scarce that do not represent a variable)</li>
  2517. <li aria-level="2">OLTP Read and write some are inside transaction some not (especially reads)</li>
  2518. <li aria-level="2">TPC-C like load almost all inside transactions</li>
  2519. </ul>
  2520. </li>
  2521. <li aria-level="1">2 Dimensions of dataset&nbsp; for each environment dimension&nbsp;
  2522. <ul>
  2523. <li aria-level="2">Fits in memory (small)&nbsp;</li>
  2524. <li aria-level="2">Large enough to cause more flushing and disk activity (large)</li>
  2525. </ul>
  2526. </li>
  2527. <li aria-level="1">5 increasing number of threads&nbsp;</li>
  2528. </ul>
  2529. <h2>Assumptions</h2>
  2530. <p>Assumptions and clarification before starting the review:</p>
  2531. <p>Q: Why are you using only one RDS instance instead of also spinning a Replica, we can also have the PDMO using only one node and reducing the cost impact.</p>
  2532. <p>A: &nbsp;We are testing minimal production solutions. Using only a node for PDMO is delivering a solution that has no High Availability. RDS with multi AZ is providing High Availability, with lower level of nines but still functional. See: <a href="https://aws.amazon.com/blogs/database/amazon-rds-under-the-hood-multi-az/">https://aws.amazon.com/blogs/database/amazon-rds-under-the-hood-multi-az/</a> &nbsp; <a href="https://aws.amazon.com/blogs/database/amazon-rds-under-the-hood-single-az-instance-recovery/">https://aws.amazon.com/blogs/database/amazon-rds-under-the-hood-single-az-instance-recovery/</a>&nbsp;</p>
  2533. <p>Q: Did you tune any of the default settings coming with the service?&nbsp;</p>
  2534. <p>A: No. The assumption was that the provider is already providing optimised setup in relation to the solution offered.&nbsp;</p>
  2535. <p>Q: Did you modify the PDMO settings?</p>
  2536. <p>A: Yes. The basic config coming with the PDMO is set for testing on a very minimal/test setup. It is not production ready and should not be used in production as well. The settings for PDMO had been changed to match the resources available and the expected load. Nothing more than what we should expect from the service providers.</p>
  2537. <h2>The layout</h2>
  2538. <h3>AWS</h3>
  2539. <p>The following is a simplified layout of the tested solutions.</p>
  2540. <p>In general terms each solution spans across multiple subnets and AZ, each solution has a dedicated Application for testing and application connected directly to the closest endpoint provided. In case of EKS the nodes are distributed over 3 different subnets.</p>
  2541. <p><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/operator_layout_aws.png" alt="operator layout aws" style="display: block; margin-left: auto; margin-right: auto;" /></p>
  2542. <p>I am not reporting the whole internal details about how the Percona Distribution for MySQL Operator (PDMO) works, for that please refer to the documentation <a href="https://www.percona.com/doc/kubernetes-operator-for-pxc/index.html">https://www.percona.com/doc/kubernetes-operator-for-pxc/index.html</a></p>
  2543. <p>The table below is reporting the high level description of the solutions tested:</p>
  2544. <p><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/machines.PNG" alt="machines" width="700" height="160" /></p>
  2545. <p>*For PDMO we had to use machines with more resources to host additional services like HAProxy and the operator itself. But the resources allocated for the database services were reflecting the ones used AWS for Aurora and standard RDS. Said that, while for this exercise we have used machines with a bit more resources, it is common practice to have one powerful host node with multiple database clusters to optimise cost and resource utilization, I will cover this better later in the cost section.&nbsp;&nbsp;&nbsp;</p>
  2546. <h1>Down into the rabbit hole&nbsp;</h1>
  2547. <p>A lot of graphs can be found <a href="https://github.com/Tusamarco/blogs/tree/master/comparing_rds_operator/files/aws-files">here</a>:&nbsp;</p>
  2548. <p>The name includes the test and the data set dimension.</p>
  2549. <p>In this section we will review the results of the tests by provider.</p>
  2550. <h2>AWS</h2>
  2551. <p>AWS is the most utilized Cloud provider and is also the one with more solutions when discussing Database services. I have tested and compared PDMO vs Aurora and standard RDS.</p>
  2552. <p>For clarity I am reporting two different sets of graphs for details. One is related to CPU, memory etc, the other is more MySQL specific.&nbsp;</p>
  2553. <p>In this context we have the following:</p>
  2554. <ul>
  2555. <li aria-level="1">aurora-eks-test1-instance-1; Primary node of the Aurora service</li>
  2556. <li aria-level="1">aurora-eks-test1-instance-1-eu-central-1a; Secondary node of the Aurora service</li>
  2557. <li aria-level="1">Mt-rds-kube-test; RDS instance&nbsp;</li>
  2558. <li aria-level="1">pxc-cluster1-pxc-0 ; Pod serving the MySQL traffic for PDMO&nbsp;</li>
  2559. </ul>
  2560. <p>As you may notice from the graphs, the Aurora service automatically swap/failover from a node to another more than once.&nbsp;&nbsp;</p>
  2561. <h3>Checking Read Only traffic (scenario 1S)</h3>
  2562. <p>Let us start with the easiest case, a simple new website with some editorial. As already mentioned this can be considered a Read Only case given the writes should be counted in few by minutes or even less.&nbsp;</p>
  2563. <h4>Small Dataset</h4>
  2564. <p>As previously mentioned, this case is the first baby step, when your DB is really at an infant stage.</p>
  2565. <p>Y axis reports operations/sec</p>
  2566. <p>&nbsp;<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Reads-sec-RO-small-_10.png" alt="chart 2021 09 28 Reads sec RO small 10" width="600" height="407" /></p>
  2567. <p>In case no matter if the dataset fits in memory or not, we can notice how the Percona MySQL operator is able to serve a higher amount of traffic in respect to the other AWS solutions. If we check what seems to be the limitation factor, we can see that:</p>
  2568. <p>&nbsp;2 CPU</p>
  2569. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_small_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_small_OS_cpu.png" alt="2 cpu ro small OS cpu" width="700" height="116" /></a></p>
  2570. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_small_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_small_threads.png" alt="2 cpu ro small threads" width="700" height="157" /></a></p>
  2571. <p>4 CPU</p>
  2572. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_small_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_small_OS_cpu.png" alt="4 cpu ro small OS cpu" width="700" height="133" /></a></p>
  2573. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_small_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_small_threads.png" alt="4 cpu ro small threads" width="700" height="176" /></a></p>
  2574. <p>We can see how in both cases the AWs solutions reach 100% of CPU utilization, limiting the number of operations they can execute in memory. <br />Also notice how the load was automatically moved from one Aurora instance to another during the tests (CPUs graph).</p>
  2575. <h4>Large Dataset</h4>
  2576. <p>Now let's see what happens if I am successful and my website starts to increase in content, such that my dataset does not fit in memory, and as such I will have MySQL evicting old pages from memory</p>
  2577. <p><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Ops-sec-RO-Large_1.png" alt="chart 2021 09 28 Ops sec RO Large 1" width="700" height="474" /></p>
  2578. <p>In case the dataset exceeds the dimension of the memory, PDMO is still performing significantly better than the other two solutions when using 4 CPUs. But it has a significant reduction of performance when using 2 CPUs.</p>
  2579. <p>2 CPUs</p>
  2580. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_large_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_large_OS_cpu.png" alt="2 cpu ro large OS cpu" width="700" height="116" /></a></p>
  2581. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_large_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_large_threads.png" alt="2 cpu ro large threads" width="700" height="159" /></a></p>
  2582. <p>Disk Latency</p>
  2583. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_large_OS_disk_latency.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_large_OS_disk_latency.png" alt="2 cpu ro large OS disk latency" width="700" height="114" /></a></p>
  2584. <p>The other relevant fact to notice is how the disk latency affected the reads for the PDMO when using a large dataset and 2 CPUs. <br />This negative effect doesn't show up in case of 4 CPUs&nbsp;</p>
  2585. <p>4 CPUs&nbsp;</p>
  2586. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_large_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_large_OS_cpu.png" alt="4 cpu ro large OS cpu" width="700" height="133" /></a></p>
  2587. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_large_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_large_threads.png" alt="4 cpu ro large threads" width="700" height="183" /></a></p>
  2588. <p>Also in the case of 4 CPUs we see the same behavior, with AWS solutions not able to scale and failing to serve traffic.&nbsp;</p>
  2589. <p>Disk Latency</p>
  2590. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_large_OS_disk_latency.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_large_OS_disk_latency.png" alt="4 cpu ro large OS disk latency" width="700" height="132" /></a></p>
  2591. <h3>Checking OLTP (scenario 2S)</h3>
  2592. <p>This case is obviously more complicated given the writes and transactions. The level of reads is still high, around 93% but all operations are enclosed inside transactions.&nbsp;</p>
  2593. <p>In this case we MUST review in parallel the SMALL and LARGE dataset to better understand what is happening.</p>
  2594. <h4>Operation/sec</h4>
  2595. <table>
  2596. <tbody>
  2597. <tr>
  2598. <td>&nbsp;<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Reads-sec-OLTP-small-_11.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Reads-sec-OLTP-small-_11.png" alt="chart 2021 09 28 Reads sec OLTP small 11" width="400" height="271" /></a></td>
  2599. <td><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Ops-sec-OLTP-large_2.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Ops-sec-OLTP-large_2.png" alt="chart 2021 09 28 Ops sec OLTP large 2" width="400" height="272" /></a>&nbsp;</td>
  2600. </tr>
  2601. </tbody>
  2602. </table>
  2603. <p>&nbsp;</p>
  2604. <p>As you can see the solution using the Operator is overperforming in the case of small and large datasets. While AWS solutions are still suffering from the same CPU limitation we have seen in the read only load.&nbsp;</p>
  2605. <p>2 CPUs small</p>
  2606. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_small_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_small_OS_cpu.png" alt="2 cpu rw small OS cpu" width="700" height="118" /></a></p>
  2607. <p>&nbsp;<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_small_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_small_threads.png" alt="2 cpu rw small threads" width="700" height="156" /></a></p>
  2608. <p>2 CPUs Large</p>
  2609. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_large_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_large_OS_cpu.png" alt="2 cpu rw large OS cpu" width="700" height="117" /></a></p>
  2610. <p>We can observe how CPUs resources in the case of AWS solutions are always at 100%, while PDMO can count on resource scaling.</p>
  2611. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_large_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_large_threads.png" alt="2 cpu rw large threads" width="700" height="157" /></a></p>
  2612. <p>2 CPUs large Disk Latency</p>
  2613. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_large_OS_disk_latency.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_large_OS_disk_latency.png" alt="2 cpu rw large OS disk latency" width="700" height="116" /></a></p>
  2614. <p>4 CPUs small</p>
  2615. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_small_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_small_OS_cpu.png" alt="4 cpu rw small OS cpu" width="700" height="132" /></a></p>
  2616. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_small_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_small_threads.png" alt="4 cpu rw small threads" width="700" height="178" /></a>&nbsp;</p>
  2617. <p>4 CPUs Small Disk Latency</p>
  2618. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_small_OS_disk_latency.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_small_OS_disk_latency.png" alt="4 cpu rw small OS disk latency" width="700" height="130" /></a></p>
  2619. <p>4 CPUs large</p>
  2620. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_large_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_large_OS_cpu.png" alt="4 cpu rw large OS cpu" width="700" height="134" /></a></p>
  2621. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_large_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_large_threads.png" alt="4 cpu rw large threads" width="700" height="183" /></a></p>
  2622. <p>4 CPUs Large Disk Latency</p>
  2623. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_large_OS_disk_latency.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_large_OS_disk_latency.png" alt="4 cpu rw large OS disk latency" width="700" height="131" /></a></p>
  2624. <p>With 4 CPUs available the Aurora solution can count in a bit more “space” to scale, but still not enough to result as efficient as PDMO.</p>
  2625. <p>As for the read only, we can see the disk latency significantly increase in the case of Large dataset, especially for the PDMO solution.&nbsp;&nbsp;</p>
  2626. <p>It is interesting to see how Aurora is penalized not only by the CPU utilization but also by a very high disk latency.&nbsp;</p>
  2627. <p>&nbsp;</p>
  2628. <h4>Latency 95pct (lower is better)</h4>
  2629. <table>
  2630. <tbody>
  2631. <tr>
  2632. <td><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-OLTP-small-_20.png" target="_blank"> <img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-OLTP-small-_20.png" alt="chart 2021 09 28 Latency 95 Pct (y axis in ms) OLTP small 20" width="400" height="272" /></a></td>
  2633. <td><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-OLTP-large_19.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-OLTP-large_19.png" alt="chart 2021 09 28 Latency 95 Pct (y axis in ms) OLTP large 19" width="400" height="272" /></a>&nbsp;</td>
  2634. </tr>
  2635. </tbody>
  2636. </table>
  2637. <p>&nbsp;</p>
  2638. <p>Latency metrics reflect the operation trend, highlighting how the low CPU-resources/Disk-latency affects all solutions, especially in case of large datasets.&nbsp;</p>
  2639. <h3>Checking TPC-c (inspired) load (scenario 3S)</h3>
  2640. <p>Also in this case we have that all the queries are encapsulated in a transaction and in this kind of test the read and write operations are close to being split at the 50% mark, so we expect to see a significant load on disk write operation and for the data replication mechanism.&nbsp;&nbsp;</p>
  2641. <h4>Operation/sec</h4>
  2642. <table>
  2643. <tbody>
  2644. <tr>
  2645. <td><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Reads-sec-TPCC-small-_12.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Reads-sec-TPCC-small-_12.png" alt="chart 2021 09 28 Reads sec TPCC small 12" width="400" height="272" /></a></td>
  2646. <td><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Writes-sec-TPCC-large_14.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Writes-sec-TPCC-large_14.png" alt="chart 2021 09 28 Writes sec TPCC large 14" width="400" height="271" /></a></td>
  2647. </tr>
  2648. </tbody>
  2649. </table>
  2650. <p>&nbsp;</p>
  2651. <p>In this scenario PDMO results to be more penalized when the dataset is larger than the available memory. In some way this is expected, given the level of writing is such that the solution pays the cost of the virtual-synchronous replication mechanism used in PXC.</p>
  2652. <p>If we check the details we can see:</p>
  2653. <p>4 CPUs Large</p>
  2654. <p>&nbsp;<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_OS_cpu.png" alt="4 cpu tpcc large OS cpu" width="700" height="135" /></a></p>
  2655. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_threads.png" alt="4 cpu tpcc large threads" width="700" height="181" /></a></p>
  2656. <p>4 CPUs Disck Latency</p>
  2657. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_OS_disk_latency.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_OS_disk_latency.png" alt="4 cpu tpcc large OS disk latency" width="700" height="131" /></a>&nbsp;</p>
  2658. <p>As we can see the PDMO node receiving the direct load is not crazy loaded, while the disk latency is higher than RDS with an average of 4ms write and 8ms for read operations. <br />But that is not enough to justify the delta we have in performance loss. What is preventing the solution from being as efficient as the other two?</p>
  2659. <p>PXC flow control</p>
  2660. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_pxc_flow_control_new.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_pxc_flow_control_new.png" alt="4 cpu tpcc large pxc flow control new" width="700" height="218" /></a></p>
  2661. <p>PDMO is using PXC as technology to offer service high availability and full data consistency. This means that the cluster must certify and apply all writes in any node at each commit, with the increase of the write operations this solution also increases the pressure on the replication layer and if any node has a delay in applying the write, that delay will affect the efficiency of the whole cluster.&nbsp;</p>
  2662. <p>In our case we can see that nodes pxc-0 and pxc-2 are on hold for pxc-1 to perform the writes.&nbsp;</p>
  2663. <p>We also have as side effect the increase of the sending queue on the primary node (pxc-0)</p>
  2664. <p><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_pxc_send_queue.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_pxc_send_queue.png" alt="4 cpu tpcc large pxc send queue" width="700" height="204" /></a></p>
  2665. <h4>&nbsp;</h4>
  2666. <h4>Latency</h4>
  2667. <table>
  2668. <tbody>
  2669. <tr>
  2670. <td><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-TPCC-small-_22.png" target="_blank"> <img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-TPCC-small-_22.png" alt="chart 2021 09 28 Latency 95 Pct (y axis in ms) TPCC small 22" width="400" height="272" /></a></td>
  2671. <td><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-TPCC-large_21.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-TPCC-large_21.png" alt="chart 2021 09 28 Latency 95 Pct (y axis in ms) TPCC large 21" width="400" height="272" /> </a></td>
  2672. </tr>
  2673. </tbody>
  2674. </table>
  2675. <p>Latency is just confirming us the picture we had from the operations.&nbsp;&nbsp;</p>
  2676. <h2>What should I pick?&nbsp;</h2>
  2677. <p>Before indicating what should be a good solution for what, let us talk about costs.</p>
  2678. <h3>Costs</h3>
  2679. <p>To discuss the cost, we need to make some assumptions.&nbsp;</p>
  2680. <p>The first assumption is that we need to calculate the cost by solutions and not by node. This is because we cannot consider a solution with a single node for PDMO or Aurora. To be effective both require their production minimum requirements, which for Aurora is two nodes and for PDMO given is based on PXC 3 nodes. Given that calculating by node is a fictitious artifact, while calculating by solution gives you the real cost.&nbsp;&nbsp;&nbsp;</p>
  2681. <p>The second assumption is that we can have more PDMO solutions running on the same Node. Which means we will have a better relation cost/benefit in utilizing larger nodes and use them to run multiple PDMO solutions. In that case, the only cost that will need to be added is the cost of the attached EBS volume per delivered POD.&nbsp;</p>
  2682. <p>&nbsp;</p>
  2683. <p>Let me provide an example of PDMO calculation.&nbsp;</p>
  2684. <p>If I consider the minimal Node resource requirements, as we did for these tests, I will need to have:</p>
  2685. <table border="1">
  2686. <thead>
  2687. <tr style="background-color: #bbbbbb;">
  2688. <td>
  2689. <p>Solution</p>
  2690. </td>
  2691. <td>
  2692. <p>Node Type</p>
  2693. </td>
  2694. <td>
  2695. <p>Monthly Cost</p>
  2696. </td>
  2697. </tr>
  2698. </thead>
  2699. <tbody>
  2700. <tr>
  2701. <td>
  2702. <p>PDMO 2 CPU</p>
  2703. </td>
  2704. <td>
  2705. <p>R5.xlarge</p>
  2706. </td>
  2707. <td>
  2708. <p>1,609 USD</p>
  2709. </td>
  2710. </tr>
  2711. <tr>
  2712. <td>
  2713. <p>PDMO 4 CPU</p>
  2714. </td>
  2715. <td>
  2716. <p>R5.2xlarge</p>
  2717. </td>
  2718. <td>
  2719. <p>2,274 USD</p>
  2720. </td>
  2721. </tr>
  2722. </tbody>
  2723. </table>
  2724. <p>&nbsp;</p>
  2725. <p>But if we use a larger node that can host multiple PDMO like a r5.4xlarge (16 CPU 128GB) that can host multiple PDMO :</p>
  2726. <table border="1">
  2727. <tbody>
  2728. <tr style="background-color: #bbbbbb;">
  2729. <td>
  2730. <p>Solution</p>
  2731. </td>
  2732. <td>
  2733. <p>PDMO instances per node</p>
  2734. </td>
  2735. <td>
  2736. <p>Monthly Cost - full solution for PDMO</p>
  2737. </td>
  2738. </tr>
  2739. <tr>
  2740. <td>
  2741. <p>PDMO 2 CPU</p>
  2742. </td>
  2743. <td>
  2744. <p>3</p>
  2745. </td>
  2746. <td>
  2747. <p>1,202 USD</p>
  2748. </td>
  2749. </tr>
  2750. <tr>
  2751. <td>
  2752. <p>PDMO 4 CPU</p>
  2753. </td>
  2754. <td>
  2755. <p>2</p>
  2756. </td>
  2757. <td>
  2758. <p>1,803 USD</p>
  2759. </td>
  2760. </tr>
  2761. </tbody>
  2762. </table>
  2763. <p>&nbsp;</p>
  2764. <p>As you can see also if a solution based on r5.4xlarge has higher absolute cost per node, the fact you can run multiple PDMO in a node allows you to reduce the cost in a significant way by solution.&nbsp;&nbsp;&nbsp;</p>
  2765. <p>&nbsp;</p>
  2766. <p>Said that let us summarize the cost of the solutions:</p>
  2767. <table border="1">
  2768. <tbody>
  2769. <tr style="background-color: #bbbbbb;">
  2770. <td>
  2771. <p>Solution</p>
  2772. </td>
  2773. <td>
  2774. <p>Node Type</p>
  2775. </td>
  2776. <td>
  2777. <p>Monthly Cost</p>
  2778. </td>
  2779. </tr>
  2780. <tr>
  2781. <td>
  2782. <p>Aurora* 2 CPU</p>
  2783. </td>
  2784. <td>
  2785. <p>db.r5.large</p>
  2786. </td>
  2787. <td>
  2788. <p>$6,323.30&nbsp;</p>
  2789. </td>
  2790. </tr>
  2791. <tr>
  2792. <td>
  2793. <p>Aurora* 4 CPU</p>
  2794. </td>
  2795. <td>
  2796. <p>db.r5.xlarge</p>
  2797. </td>
  2798. <td>
  2799. <p>$6,834.30&nbsp;</p>
  2800. </td>
  2801. </tr>
  2802. <tr>
  2803. <td>
  2804. <p>RDS 2 CPU</p>
  2805. </td>
  2806. <td>
  2807. <p>db.r5.large</p>
  2808. </td>
  2809. <td>
  2810. <p>$1,329.80&nbsp;</p>
  2811. </td>
  2812. </tr>
  2813. <tr>
  2814. <td>
  2815. <p>RDS 4 CPU</p>
  2816. </td>
  2817. <td>
  2818. <p>db.r5.xlarge</p>
  2819. </td>
  2820. <td>
  2821. <p>$1,753.20&nbsp;</p>
  2822. </td>
  2823. </tr>
  2824. <tr>
  2825. <td>
  2826. <p>PDMO 2 CPU (single instance per delivered node)</p>
  2827. </td>
  2828. <td>
  2829. <p>r5.xlarge</p>
  2830. </td>
  2831. <td>
  2832. <p>$1,609.20&nbsp;</p>
  2833. </td>
  2834. </tr>
  2835. <tr>
  2836. <td>
  2837. <p>PDMO 4 CPU (single instance per delivered node)</p>
  2838. </td>
  2839. <td>
  2840. <p>r5.2xlarge</p>
  2841. </td>
  2842. <td>
  2843. <p>$2,274.96&nbsp;</p>
  2844. </td>
  2845. </tr>
  2846. <tr>
  2847. <td>
  2848. <p>PDMO 2 CPU (multiple instance per delivered node)</p>
  2849. </td>
  2850. <td>
  2851. <p>r5.4xlarge</p>
  2852. </td>
  2853. <td>
  2854. <p>$1,202</p>
  2855. </td>
  2856. </tr>
  2857. <tr>
  2858. <td>
  2859. <p>PDMO 4 CPU (single instance per delivered node)</p>
  2860. </td>
  2861. <td>
  2862. <p>r5.4xlarge</p>
  2863. </td>
  2864. <td>
  2865. <p>$1,803</p>
  2866. </td>
  2867. </tr>
  2868. </tbody>
  2869. </table>
  2870. <p>* For Aurora we had to include an average number of operations based on the test executed. Please refer to: <a href="https://calculator.aws/#/createCalculator/AuroraMySQL"></a><a href="https://calculator.aws/#/createCalculator/AuroraMySQL">https://calculator.aws/#/createCalculator/AuroraMySQL</a></p>
  2871. <p>&nbsp;</p>
  2872. <p>As you can see PDMO is just a little bit more expensive than RDS when deployed in single instance mode. But it also provides better performance in most cases and always has a higher level of High Availability, given PXC is used in the background.&nbsp;</p>
  2873. <p>There is no discussion if you can use multiple PDMO on larger nodes, in that case the cost goes lower than RDS.</p>
  2874. <h3>So what to pick</h3>
  2875. <h4>Read-only</h4>
  2876. <p>Choosing is not only based on performance, but also how efficient the solution we pick is to perform the maintenance of our platform. In this regard both operator and Aurora offer a good level of automation. But PDMO was always performing better than the other two solutions. On top of that Aurora based solutions are more expensive than the others, so in the end, for me, choosing PDMO is the logical thing to do, being ready to eventually do a small scale-up in case I need more resources. This is another advantage of PDMO, I can eventually decide to add fragments of the CPU cycle, without the need to move to a higher and more expensive system.&nbsp;&nbsp;</p>
  2877. <h4>OLTP</h4>
  2878. <p>In this case there is no doubt that PDMO is the solution to go to. Aurora can be seen as a possible replacement given that the RDS solution has a lower level of HA in respect to the other two. But as we know the Aurora costs are higher than any other compared solutions here.</p>
  2879. <h4>TPCC</h4>
  2880. <p>RDS is the solution to go to serve the load and save the money, but you will have a lower level of HA. If you are willing to cover the cost Aurora is the solution offering you good performance and high level of High Availability.&nbsp;</p>
  2881. <p>Finally we can say that PDMO was penalized by the replication/certification mechanism in use in PXC but, the significant saving in terms of cost can justify the adoption, especially considering that with a limited scaling up in resources we can gain more traffic served. Scaling will never be as expensive as if we adopt Aurora as a solution.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</p>
  2882. <h1>Conclusions</h1>
  2883. <p>Trying to identify an entry point solution for a starting activity is harder than choosing a platform for something already well defined. There is a lot of uncertainty, about the traffic, the kind of operations, the frequency of them, the growth (if any).&nbsp;</p>
  2884. <p>In my opinion there are already so many variables and things that can possibly go wrong, that we should try to adopt, if not the safest solution, the one that gives us a good level of confidence in what we will be able to achieve.&nbsp;</p>
  2885. <p>I was not surprised to see products such as Aurora shine when we have Tpcc like load, but I was pleased to see Percona solution with the operator performing in an admirable way.&nbsp;</p>
  2886. <p>Of course I was not using the settings coming from the vanilla installation, given they are for testing. So if you install PDMO in production you MUST change them&nbsp; (see <a href="https://github.com/Tusamarco/blogs/tree/master/comparing_rds_operator/files/crs">here</a> for my cr.yaml) and I was counting on version 1.9.0 at least, without which PDMO is not even in the game.&nbsp;</p>
  2887. <p>But it is obvious that Percona is on the right path, with a long road ahead, but we it is moving in the right direction.</p>
  2888. <p>Percona Distribution for MySQL Operatoris still a baby, but is growing fast, and the more we have people using, testing and providing feedback, the fastest Percona will be able to provide software that not only early adopters implement, but also the more conservative ones can use for their production.</p>
  2889. <h2>References</h2>
  2890. <p><a href="https://calculator.aws/#/createCalculator/RDSMySQL"></a><a href="https://calculator.aws/">https://calculator.aws/</a></p>
  2891. <h1>Disclaimer&nbsp;</h1>
  2892. <p>My blog, my opinions.&nbsp;</p>
  2893. <p>I am not, in any way or aspects, presenting here the official line of the company I am working for.</p>
  2894. <h1>Comments</h1>
  2895. <p>If you have comments or suggestions please add them to the <a href="https://www.linkedin.com/posts/marcotusa_compare-percona-distribution-for-mysql-operator-activity-6852136882537545728-jfTe">linkedin post</a></p>]]></description>
  2896. <category>MySQL</category>
  2897. <pubDate>Sat, 09 Oct 2021 07:00:45 +0000</pubDate>
  2898. </item>
  2899. <item>
  2900. <title>Boosting Percona MySQL Operator efficiency</title>
  2901. <link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/241-boosting-percona-mysql-operator-efficiency</link>
  2902. <guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/241-boosting-percona-mysql-operator-efficiency</guid>
  2903. <description><![CDATA[<h1><span style="font-weight: 400;">Preface</span></h1>
  2904. <p><span style="font-weight: 400;">Percona is well known for its offer of several outstanding fully open source, free to <a href="https://www.percona.com/software">download</a> software packages. And while Percona started as a MySQL focus company, nowadays it covers different technologies such as MySQL, Postgres and Mongodb. <a href="https://www.percona.com/blog/wp-content/uploads/2018/11/lightspeed.jpg"><img src="https://www.percona.com/blog/wp-content/uploads/2018/11/lightspeed-300x245.jpg" alt="" width="300" height="245" class="size-medium wp-image-53798 alignright" style="float: right;" /></a></span></p>
  2905. <p><span style="font-weight: 400;">In its constant effort to make life easier for our utilizer Percona had moved from providing single software packages, to Percona Distributions for MySQL, MongoDB and Postgres. Percona Distributions are a set of software packages that Percona has tested and certifies working together. That it easier to deploy architectural solutions requiring the use of multiple components, such as proxy, topology manager, backup software and more.&nbsp;</span></p>
  2906. <p><span style="font-weight: 400;">But we are going even further, and with the release of Percona Distribution for MySQL/MongoDB <a href="https://www.percona.com/doc/kubernetes-operator-for-pxc/index.html">Operator</a> </span><span style="font-weight: 400;">we are providing a high level of automation to roll out and manage solutions based on Percona Distributions.&nbsp;</span></p>
  2907. <p><span style="font-weight: 400;">One of my tasks, as MySQL technical leader, is to identify optimal architectures to serve several common cases. Such as Percona Distribution for MySQL: <a href="https://www.percona.com/blog/2021/04/14/percona-distribution-for-mysql-high-availability-with-group-replication-solution/">High Availability with Group Replication Solutio</a>n</span><span style="font-weight: 400;">. Or in the case of&nbsp; Percona Distribution for MySQL Operator, identify the different dimensions (low/mid/high utilization) and suggest a Vanilla setup with the scope to get the most out of the solution deployed.&nbsp;&nbsp;</span></p>
  2908. <p><span style="font-weight: 400;">This is a long exercise, which started with a lot of internal discussions to identify what can make sense as traffic, then testing, identifying the saturation points, testing again and so on.&nbsp;</span></p>
  2909. <p><span style="font-weight: 400;">It is during this process that I found a small problem (<a href="https://jira.percona.com/browse/K8SPXC-749">Feature Request</a></span><span style="font-weight: 400;">). This small issue is preventing us from easily and dynamically modifying some parameters in the checks the Operator&nbsp; uses. Given that we had to put the testing on hold until the above FR is implemented. As you can see it is a small thing but it will give us better control over the Operator behavior and will help to have a well tuned platform.&nbsp;</span></p>
  2910. <p><span style="font-weight: 400;">This article is to show the level of improvement you can have with small but targeted tuning.To do so I used the smallest solution we have identified. The solution is dimensioned to serve a small website or a simple application with low level of traffic.&nbsp;</span></p>
  2911. <h1><span style="font-weight: 400;">The environment</span></h1>
  2912. <p><span style="font-weight: 400;">To help identify a balanced setup we were using sysbench and sysbench-tpcc. The whole stack on GCP was composed of Application nodes with sysbench, 2 ProxySQL nodes for R/W split only, 3 VMS 8 CPU 32GB RAM, with the Percona operator managing the MySQL service.</span></p>
  2913. <p><a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Untitled-Diagram.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Untitled_Diagram.png" alt="Untitled Diagram" width="370" height="396" class="wp-image-76889 aligncenter" style="display: block; margin-left: auto; margin-right: auto;" /></a></p>
  2914. <h2><span style="font-weight: 400;">Tests</span></h2>
  2915. <p><span style="font-weight: 400;">For this specific test we were running 68 - 96 -128 - 256 threads:</span></p>
  2916. <ul>
  2917. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Sysbench read only</span></li>
  2918. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Sysbench read/write</span></li>
  2919. <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Sysbench Tpc-c like&nbsp;</span></li>
  2920. </ul>
  2921. <p><span style="font-weight: 400;">The tests were run multiple times and the data considered is the consolidation of the multiple runs.&nbsp;</span></p>
  2922. <p><span style="font-weight: 400;">We always run first on basic environments for baseline. Meaning no tuning for the MySQL or Operator, just dimension correctly disk space and BufferPool (and related).</span></p>
  2923. <p><span style="font-weight: 400;">Then we apply some tuning and run the tests multiple times eventually refining when/where needed.&nbsp;</span></p>
  2924. <p>The code can be found <a href="https://github.com/Tusamarco/sysbench">here</a> and <a href="https://github.com/Tusamarco/sysbench-tpcc">here</a></p>
  2925. <h1><span style="font-weight: 400;">And now the results…</span></h1>
  2926. <h2><span style="font-weight: 400;">Sysbench r/w tests</span></h2>
  2927. <p><span style="font-weight: 400;">I am not going to describe in detail the images that I think are clear enough. Just keep in mind on the left we have the results from our baseline, on the right the same tests on the same platform with the optimization applied.</span></p>
  2928. <h3><span style="font-weight: 400;">Operations</span></h3>
  2929. <p><a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture6.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture6.png" alt="Picture6" width="800" height="392" class="alignnone size-large wp-image-76891" /></a></p>
  2930. <p><span style="font-weight: 400;">Worth to mention that without tuning, the platform was not able to consistently scale up to 256 threads. While with a bit of adjustment not only it was able to serve 256 threads but we could have gone a bit further.</span></p>
  2931. <h3><span style="font-weight: 400;">Reads</span></h3>
  2932. <p><a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture7.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture7.png" alt="Picture7" width="800" height="392" class="alignnone size-large wp-image-76892" /></a></p>
  2933. <h3><span style="font-weight: 400;">Writes</span></h3>
  2934. <p><a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture8.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture8.png" alt="Picture8" width="800" height="392" class="alignnone size-large wp-image-76893" /></a></p>
  2935. <h3><span style="font-weight: 400;">Comments</span></h3>
  2936. <p><span style="font-weight: 400;">As you can see the sysbench tests clearly indicate that the platform with minor adjustment was acting better, and that it was able to serve more and with constant scaling. Let me add that almost all the tests runned on the “basic” platform had incidents, meaning as soon as the traffic was increasing, Sysbench was reporting connection interruptions or errors.</span></p>
  2937. <h2><span style="font-weight: 400;">TPC-C</span></h2>
  2938. <h3><span style="font-weight: 400;">Operations</span></h3>
  2939. <h3><a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture10.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture10.png" alt="Picture10" width="800" height="395" class="alignnone size-large wp-image-76895" /></a></h3>
  2940. <h3><span style="font-weight: 400;">Reads</span></h3>
  2941. <p><a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture11.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture11.png" alt="Picture11" width="800" height="395" class="alignnone size-large wp-image-76896" /></a></p>
  2942. <h3><span style="font-weight: 400;">Writes</span></h3>
  2943. <h3><a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture12.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture12.png" alt="Picture12" width="800" height="395" class="alignnone size-large wp-image-76897" /></a></h3>
  2944. <h3><span style="font-weight: 400;">Comments</span></h3>
  2945. <p><span style="font-weight: 400;">Also for Tpc-c like tests we have exactly the same trend. With our “optimised” solution able to serve up to 1516 qps while the “basic” one was able to reach only 322. In this case Also the “optimised” solution was not able to scale up to 256 threads, but that makes sense, given the more intense write workload present in this test and the small dimension of the platform.</span></p>
  2946. <p>&nbsp;</p>
  2947. <h1><span style="font-weight: 400;">Wooha what have you changed?</span></h1>
  2948. <p><span style="font-weight: 400;">You may think we have done crazy things to get this difference, but we did not.</span></p>
  2949. <p><span style="font-weight: 400;">Let us jump back. As indicated at the beginning I had opened a FR (</span><a href="https://jira.percona.com/browse/K8SPXC-749"><span style="font-weight: 400;">https://jira.percona.com/browse/K8SPXC-749</span></a><span style="font-weight: 400;">) to be able to tune some/most of the timeouts existing in the operator. <a href="https://www.percona.com/blog/wp-content/uploads/2021/06/8e4.jpg"><img src="https://www.percona.com/blog/wp-content/uploads/2021/06/8e4.jpg" alt="" width="287" height="198" class="wp-image-76885 alignright" style="float: right;" /></a></span></p>
  2950. <p><span style="font-weight: 400;">Why? Think about this, when you install a cluster on iron, you do not set it to be able to work only when the load is low, and all the components of the server are able to answer in nanoseconds. What you do instead is tune the whole system to accommodate the increasing load, and you will give to some elements more space for “flexibility” eventually expecting to have delays in answer. When doing so you also need to correctly align all the parameters that will be affected on cascade. For instance if you know your data nodes will be very busy serving queries, they may also slow down in answering internal health checks, but if you relax the cluster health checks and not the checks used for testing the cluster from operator point of view, the platform will be unbalanced and will not work correctly.</span></p>
  2951. <p><span style="font-weight: 400;">At the same time, if you do not tune the solution at all, you may end up with a platform that is theoretically able to serve the load, but that is crashing for artificial limitations.&nbsp;</span></p>
  2952. <p><span style="font-weight: 400;">The last one is exactly what was happening with our “basic” solution. As it is the operator comes with parameters that allow it to work well, but that are not designed to scale. Is like having a server where your CPUs are always at 20% and if the applications ask more, a controller will chop them in fear of having too much load. But the fact is that you want to have the CPUs at 80% or the server will be underutilized.&nbsp;</span></p>
  2953. <p><span style="font-weight: 400;">Anyhow what we have changed was some Innodb parameters, to allow internal operations to work better. Then we force consistent reads in PXC, which actually SLOW down the operations, and finally we tune the PXC cluster to be more flexible in its internal checks, avoiding having it to expel nodes unless really needed to.&nbsp;</span></p>
  2954. <p><span style="font-weight: 400;">All the above were done using the Operator configuration, but then we had to work </span><b>manually changing</b><span style="font-weight: 400;"> all the timeouts parameters used by the operator checks to be aligned with what we had defined in the cluster.&nbsp;</span></p>
  2955. <p><span style="font-weight: 400;">In particular what we have changed was:</span></p>
  2956. <pre class="lang:sh decode:true">script                      line    value
  2957. /usr/bin/clustercheckcron   33      TIMEOUT=10
  2958. liveness-check.sh           23      TIMEOUT=5
  2959. readiness-check.sh          21      TIMEOUT=10
  2960. /usr/local/bin/check_pxc.sh 15      TIMEOUT=${CUSTOM_TIMEOUT:-10}</pre>
  2961. <h2><span style="font-weight: 400;">Wait .. why ProxySQL?</span></h2>
  2962. <p><span style="font-weight: 400;">Ok this is another long discussion and I will cover it better in another article. For now just consider that HAProxy does not allow r/w splitting or other nice functionalities like firewalling etcetera etcetera. So the idea is simple, let us use the operator with what fits it better, and then decouple the special needs, eventually adding proxysql in a separate deployment.&nbsp;</span></p>
  2963. <p><span style="font-weight: 400;">If you are scared of the cost of adding an additional block to the architecture:</span></p>
  2964. <p><a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture1-a.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture1-a.png" alt="Picture1 a" width="800" height="401" class="alignnone wp-image-76883" /></a></p>
  2965. <p><a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture1-b.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture1-b.png" alt="Picture1 b" width="800" height="264" class="alignnone wp-image-76884" /></a></p>
  2966. <p><span style="font-weight: 400;">Where:<br /></span><span style="font-weight: 400;">ProxySQL means: Application → ProxySQL → HAProxy.<br /></span><span style="font-weight: 400;">HAProxy means: Application → HAProxy.<br /></span></p>
  2967. <p><span style="font-weight: 400;">Hope this puts your worries at rest, of course this is using the “optimised” solution.</span></p>
  2968. <h1><span style="font-weight: 400;">Conclusions</span></h1>
  2969. <p><span style="font-weight: 400;">Percona Distribution for MySQL Operator, is a constantly growing/improving solution. It also has a lot of interesting features, like being able to manage your backup/restore, point in time recovery, and more. But its adoption is still limited and it is normal to have some drawbacks like this one. It is on us who play with real production environments, or as in this case playing to define certified solutions, to give feedback to improve how the operator works, in order to make it a stronger product able to serve you better day by day.</span></p>
  2970. <p><span style="font-weight: 400;">Now we are going to wait for the FR to be implemented, and then we will recover our dimensioning work.&nbsp;</span></p>]]></description>
  2971. <category>MySQL</category>
  2972. <pubDate>Thu, 24 Jun 2021 08:09:19 +0000</pubDate>
  2973. </item>
  2974. </channel>
  2975. </rss>
  2976.  

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

  1. Download the "valid RSS" banner.

  2. Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)

  3. Add this HTML to your page (change the image src attribute if necessary):

If you would like to create a text link instead, here is the URL you can use:

http://www.feedvalidator.org/check.cgi?url=http%3A//www.tusacentral.net/joomla/index.php/mysql-blogs%3Fformat%3Dfeed%26type%3Drss

Copyright © 2002-9 Sam Ruby, Mark Pilgrim, Joseph Walton, and Phil Ringnalda