Comments on: Progress on MySQL Proxy Partitioning http://pero.blogs.aprilmayjune.org/2008/03/29/progress-on-mysql-proxy-partitioning/ Thu, 28 Aug 2008 23:05:06 +0000 http://wordpress.org/?v=2.3.3 By: Gavin Towey http://pero.blogs.aprilmayjune.org/2008/03/29/progress-on-mysql-proxy-partitioning/#comment-1300 Gavin Towey Mon, 05 May 2008 21:49:19 +0000 http://pero.blogs.aprilmayjune.org/2008/03/29/progress-on-mysql-proxy-partitioning/#comment-1300 I was playing around with creating a similar transparent partitioning scheme, although not quite as sophisticated as what you've done here. I was using a combination of modulus and a directory service, so keys are hashed out, but into something like 1k-4k buckets and the location of each bucket is looked up in a directory. This would be a compromise for having the overhead of looking up each key in a directory, and the inflexibility of a fixed hash lookup. Buckets can be moved between partitions at will, and with a bit of planning each should contain a manageable chunk of records. The problem I ran into was that the connect phase happens before the query is sent (obviously) so by the time you parse the query, you can't decide then which server among a pool of nodes to send the query to, or even send to multiple servers. Or can you? Of course that's probably a question for Jan. I was playing around with creating a similar transparent partitioning scheme, although not quite as sophisticated as what you’ve done here. I was using a combination of modulus and a directory service, so keys are hashed out, but into something like 1k-4k buckets and the location of each bucket is looked up in a directory. This would be a compromise for having the overhead of looking up each key in a directory, and the inflexibility of a fixed hash lookup. Buckets can be moved between partitions at will, and with a bit of planning each should contain a manageable chunk of records.

The problem I ran into was that the connect phase happens before the query is sent (obviously) so by the time you parse the query, you can’t decide then which server among a pool of nodes to send the query to, or even send to multiple servers. Or can you? Of course that’s probably a question for Jan.

]]> By: pero on anything » Blog Archive » MySQL Proxy LUA scripts cannot be written as a module? http://pero.blogs.aprilmayjune.org/2008/03/29/progress-on-mysql-proxy-partitioning/#comment-327 pero on anything » Blog Archive » MySQL Proxy LUA scripts cannot be written as a module? Sat, 05 Apr 2008 00:28:16 +0000 http://pero.blogs.aprilmayjune.org/2008/03/29/progress-on-mysql-proxy-partitioning/#comment-327 [...] I went through hell while testing the MySQL Proxy partition LUA scripts I am working on in a high concurrency environment. I am sending multiple queries to the server and [...] […] I went through hell while testing the MySQL Proxy partition LUA scripts I am working on in a high concurrency environment. I am sending multiple queries to the server and […]

]]> By: Log Buffer #91: a Carnival of the Vanities for DBAs http://pero.blogs.aprilmayjune.org/2008/03/29/progress-on-mysql-proxy-partitioning/#comment-324 Log Buffer #91: a Carnival of the Vanities for DBAs Fri, 04 Apr 2008 16:44:50 +0000 http://pero.blogs.aprilmayjune.org/2008/03/29/progress-on-mysql-proxy-partitioning/#comment-324 [...] Romianowski of pero on anything writes that he has made progress using MySQL proxy to implement database [...] […] Romianowski of pero on anything writes that he has made progress using MySQL proxy to implement database […]

]]> By: pero http://pero.blogs.aprilmayjune.org/2008/03/29/progress-on-mysql-proxy-partitioning/#comment-321 pero Sat, 29 Mar 2008 22:58:43 +0000 http://pero.blogs.aprilmayjune.org/2008/03/29/progress-on-mysql-proxy-partitioning/#comment-321 Xaprb, I agree, using a simple modulo or any other sort of hashing method is not sufficient. Actually I left out the partitioning function (I took the modulo function as a simple example). Anything could be implemented here. In my previous post (http://pero.blogs.aprilmayjune.org/2008/03/26/mysql-partitioning-on-application-side/) I described something like a directory / lookup service. This is definitely the type of partitioning function we will use in our application. But the MySQL Proxy partition module we are working on lets the user choose the way of partitioning by simply providing a partitioning function (via LUA). Cheers, Peter Xaprb,

I agree, using a simple modulo or any other sort of hashing method is not sufficient.

Actually I left out the partitioning function (I took the modulo function as a simple example). Anything could be implemented here. In my previous post (http://pero.blogs.aprilmayjune.org/2008/03/26/mysql-partitioning-on-application-side/) I described something like a directory / lookup service. This is definitely the type of partitioning function we will use in our application. But the MySQL Proxy partition module we are working on lets the user choose the way of partitioning by simply providing a partitioning function (via LUA).

Cheers,

Peter

]]> By: Xaprb http://pero.blogs.aprilmayjune.org/2008/03/29/progress-on-mysql-proxy-partitioning/#comment-320 Xaprb Sat, 29 Mar 2008 20:52:23 +0000 http://pero.blogs.aprilmayjune.org/2008/03/29/progress-on-mysql-proxy-partitioning/#comment-320 Sounds like you are thinking of using modulus as the partitioning function. Many people have tried this: it's usually jumping from the frying pan into the fire. I'd use what Jeremy Cole calls a "directory service" instead. Sounds like you are thinking of using modulus as the partitioning function. Many people have tried this: it’s usually jumping from the frying pan into the fire. I’d use what Jeremy Cole calls a “directory service” instead.

]]>