Planet Drizzle

Jay Pipes: Recent Work on Improving Drizzle’s Storage Engine API

Sat, 13 Mar 2010 07:07:59 +0000

Over the past six weeks or so, I have been working on cleaning up the pluggable storage engine API in Drizzle. I’d like to describe some of this work and talk a bit about the next steps I’m taking in the coming months as we roll towards implementing Log Shipping in Drizzle.

First, how did it come about that I started working on the storage engine API?

From Commands to Transactions

Well, it really goes back to my work on Drizzle’s replication system. I had implemented a simple, fast, and extensible log which stored records of the data changes made to a server. Originally, the log was called the Command Log, because the Google Protobuffer messages it contained were called message::Commands. The API for implementing replication plugins was very simple and within a month or so of debuting the API, quite a few replication plugins had been built, including one replicating to Memcached, a prototype one replicating to Gearman, and a filtering replicator plugin.

In addition, Marcus Eriksson had created the RabbitReplication project which could replicate from Drizzle to other data stores, including Cassandra and Project Valdemort. However, Marcus did not actually implement any C/C++ plugins using the Drizzle replication API. Instead, RabbitReplication simply read the new Command Log, which due to it simply being a file full of Google Protobuffer messages, was quick and easy to read into memory using a variety of different programming languages. RabbitReplication is written in Java, and it was great to see other programming languages be able to read Drizzle’s replication log so easily. Marcus later coded up a C++ TransactionApplier plugin which replaces the Drizzle replication log and instead replicates the GPB messages directly to RabbitMQ.

And there, you’ll note that one of the plugins involved in Drizzle’s replication system is called TransactionApplier. It used to be called CommandApplier. That was because the GPB Command messages were individual row change events for the most part. However, I made a series of changes to the replication API and now the GPB messages sent through the APIs are of class message::Transaction. message::Transaction objects contain a transaction context, with information about the transaction’s start and end time, it’s transaction identifer, along with a series of message::Statement objects, each of which representing a part of the data changes that the SQL transaction made.

Thus, the Command Log now turned into the Transaction Log, and everywhere the term Command was used now was replaced with the terms Transaction and Statement (depending on whether you were talking about the entire Transaction or a piece of it). Log entries were now written at COMMIT to the Transaction Log and were not written if no COMMIT occurred¹.

After finishing this work to make the transaction log write Transaction messages at commit time, I was keen to begin coding up the publisher and subscriber plugins which represent a node in the replication environment. However, Brian had asked me to delay working on other replication features and ensure that the replication API could support fully distributed transactions via the X/Open XA distributed transaction protocol. XA support had been removed from Drizzle when the MySQL binlog and original replication system was ripped out and needed some TLC. Fair enough, I said. So, off I went to work on XA.

If Only It Were Simple…

As anyone who has worked on the MySQL source code or developed storage engines for MySQL knows, working with the MySQL pluggable storage engine API is sometimes not the easiest or most straightforward thing. I think the biggest problem with the MySQL storage engine API is that, due to understandable historical reasons, it’s an API that was designed with the MyISAM and HEAP storage engines in mind. Much of the transactional pieces of the API seem to be a bolted-on afterthought and can be very confusing to work with.

As an example, Paul McCullagh, developer of the transactional storage engine PBXT, recently emailed the mysql internals mailing list asking how the storage engine could tell when a SQL statement started and ended. You would think that such a seemingly basic functionality would have a simple answer. You’d be wrong. Monty Widenius answered like this:

Why not simply have a counter in your transaction object for how start_stmt – reset(); When this is 0 then you know stmnt ended.

In Maria we count number of calls to external_lock() and when the sum goes to 0 we know the transaction has ended.

To this, Mark Callaghan responded:

Why does the solution need to be so obscure?

Monty answered (emphasis mine):

Historic reasons.

MySQL never kept a count of which handlers are used by a transaction, only which tables.

So the original logic was that external_lock(lock/unlock) is called for each usage of the table, which is normally more than enough information for a handler to know when a statement starts/ends.

The one case this didn’t work was in the case someone does lock tables as then external_lock is not called per statement. It was to satisfy this case that we added a call to start_stmt() for each table.

It’s of course possible to change things so that start_stmt() / end_stmt() would be called once per used handler, but this would be yet another overhead for the upper level to do which the current handlers that tracks call to external_lock() doesn’t need.

Well, in Drizzle-land, we aren’t beholden to “historic reasons” So, after looking through the in-need-of-attention transaction processing code in the kernel, I decided that I would clean up the API so that storage engines did not have to jump through hoops to notify the kernel they participate in a transaction or just to figure out when a statement and a transaction started and ended.

The resulting changes to the API are quite dramatic I think, but I’ll leave it to the storage engine developers to tell me if the changes are good or not. The following is a summary of the changes to the storage engine API that I committed in the last few weeks.

`plugin::StorageEngine` Split Into Subclasses

The very first thing I did was to split the enormous base plugin class for a storage engine, plugin::StorageEngine, into two other subclasses containing transactional elements. plugin::TransactionalStorageEngine is now the base class for all storage engines which implement SQL transactions:

/**
 * A type of storage engine which supports SQL transactions.
 *
 * This class adds the SQL transactional API to the regular
 * storage engine.  In other words, it adds support for the
 * following SQL statements:
 *
 * START TRANSACTION;
 * COMMIT;
 * ROLLBACK;
 * ROLLBACK TO SAVEPOINT;
 * SET SAVEPOINT;
 * RELEASE SAVEPOINT;
 */
class TransactionalStorageEngine :public StorageEngine
{
public:
  TransactionalStorageEngine(const std::string name_arg,
                             const std::bitset<HTON_BIT_SIZE> &flags_arg= HTON_NO_FLAGS);
 
  virtual ~TransactionalStorageEngine();
...
private:
  void setTransactionReadWrite(Session& session);
 
  /*
   * Indicates to a storage engine the start of a
   * new SQL transaction.  This is called ONLY in the following
   * scenarios:
   *
   * 1) An explicit BEGIN WORK/START TRANSACTION is called
   * 2) After an explicit COMMIT AND CHAIN is called
   * 3) After an explicit ROLLBACK AND RELEASE is called
   * 4) When in AUTOCOMMIT mode and directly before a new
   *    SQL statement is started.
   */
  virtual int doStartTransaction(Session *session, start_transaction_option_t options)
  {
    (void) session;
    (void) options;
    return 0;
  }
 
  /**
   * Implementing classes should override these to provide savepoint
   * functionality.
   */
  virtual int doSetSavepoint(Session *session, NamedSavepoint &savepoint)= 0;
  virtual int doRollbackToSavepoint(Session *session, NamedSavepoint &savepoint)= 0;
  virtual int doReleaseSavepoint(Session *session, NamedSavepoint &savepoint)= 0;
 
  /**
   * Commits either the "statement transaction" or the "normal transaction".
   *
   * @param[in] The Session
   * @param[in] true if it's a real commit, that makes persistent changes
   *            false if it's not in fact a commit but an end of the
   *            statement that is part of the transaction.
   * @note
   *
   * 'normal_transaction' is also false in auto-commit mode where 'end of statement'
   * and 'real commit' mean the same event.
   */
  virtual int doCommit(Session *session, bool normal_transaction)= 0;
 
  /**
   * Rolls back either the "statement transaction" or the "normal transaction".
   *
   * @param[in] The Session
   * @param[in] true if it's a real commit, that makes persistent changes
   *            false if it's not in fact a commit but an end of the
   *            statement that is part of the transaction.
   * @note
   *
   * 'normal_transaction' is also false in auto-commit mode where 'end of statement'
   * and 'real commit' mean the same event.
   */
  virtual int doRollback(Session *session, bool normal_transaction)= 0;
  virtual int doReleaseTemporaryLatches(Session *session)
  {
    (void) session;
    return 0;
  }
  virtual int doStartConsistentSnapshot(Session *session)
  {
    (void) session;
    return 0;
  }
};

As you can see, plugin::TransactionalStorageEngine inherits from plugin::StorageEngine and extends it with a series of private pure virtual methods that implement the SQL transaction parts of a query — doCommit(), doRollback(), etc. Implementing classes simply inherit from plugin::TransactionalStorageEngine and implement their internal transaction processing in these private methods.

In addition to the SQL transaction, however, is the concept of an XA transaction, which is for distributed transaction coordination. The XA protocol is a two-phase commit protocol because it implements a PREPARE step before a COMMIT occurs. This XA API is exposed via two other classes, plugin::XaResourceManager and plugin::XaStorageEngine. plugin::XaResourceManager derived classes implement the resource manager API of the XA protocol. plugin::XaStorageEngine is a storage engine subclass which, while also implementing SQL transactions, also implements XA transactions.

Here is the plugin::XaResourceManager class:

/**
 * An abstract interface class which exposes the participation
 * of implementing classes in distributed transactions in the XA protocol.
 */
class XaResourceManager
{
public:
  XaResourceManager() {}
  virtual ~XaResourceManager() {}
...
private:
  /**
   * Does the COMMIT stage of the two-phase commit.
   */
  virtual int doXaCommit(Session *session, bool normal_transaction)= 0;
  /**
   * Does the ROLLBACK stage of the two-phase commit.
   */
  virtual int doXaRollback(Session *session, bool normal_transaction)= 0;
  /**
   * Does the PREPARE stage of the two-phase commit.
   */
  virtual int doXaPrepare(Session *session, bool normal_transaction)= 0;
  /**
   * Rolls back a transaction identified by a XID.
   */
  virtual int doXaRollbackXid(XID *xid)= 0;
  /**
   * Commits a transaction identified by a XID.
   */
  virtual int doXaCommitXid(XID *xid)= 0;
  /**
   * Notifies the transaction manager of any transactions
   * which had been marked prepared but not committed at
   * crash time or that have been heurtistically completed
   * by the storage engine.
   *
   * @param[out] Reference to a vector of XIDs to add to
   *
   * @retval
   *  Returns the number of transactions left to recover
   *  for this engine.
   */
  virtual int doXaRecover(XID * append_to, size_t len)= 0;
};

and here is the plugin::XaStorageEngine class:

/**
 * A type of storage engine which supports distributed
 * transactions in the XA protocol.
 */
class XaStorageEngine :public TransactionalStorageEngine,
                       public XaResourceManager
{
public:
  XaStorageEngine(const std::string name_arg,
                  const std::bitset<HTON_BIT_SIZE> &flags_arg= HTON_NO_FLAGS);
 
  virtual ~XaStorageEngine();
  ...
};

Pretty clear. A plugin::XaStorageEngine inherits from both plugin::TransactionStorageEngine and plugin::XaResourceManager because it implements both SQL transactions and XA transactions. The InnobaseEngine is a plugin which inherits from plugin::XaStorageEngine because InnoDB supports SQL transactions as well as XA.

Explicit Statement and Transaction Boundaries

The second major change I made addressed the problem that Mark Callaghan noted in asking why finding out when a statement starts and ends was so obscure. I added two new methods to plugin::StorageEngine called doStartStatement() and doEndStatement(). The kernel now explicitly tells storage engines when a SQL statement starts and ends. This happens before any calls to Cursor::external_lock() happen, and there are no exception cases. In addition, the kernel now always tells transactional storage engines when a new SQL transaction is starting. It does this via an explicit call to plugin::TransactionalStorageEngine::doStartTransaction(). No exceptions, and yes, even for DDL operations.

What this means is that for a transactional storage engine, it no longer needs to “count the calls to Cursor::external_lock()” in order to know when a statement or transaction starts and ends. For a SQL transaction, this means that there is a clear code call path and there is no need for the storage engine to track whether the session is in AUTOCOMMIT mode or not. The kernel does all that work for the storage engine. Imagine a Session executes a single INSERT statement against an InnoDB table while in AUTOCOMMIT mode. This is what the call path looks like:

 drizzled::Statement::Insert::execute()
 |
 -> drizzled::mysql_lock_tables()
    |
    -> drizzled::TransactionServices::registerResourceForTransaction()
       |
       -> drizzled::plugin::TransactionalStorageEngine::startTransaction()
          |
          -> InnobaseEngine::doStartTransaction()
       |
       -> drizzled::plugin::StorageEngine::startStatement()
          |
          -> InnobaseEngine::doStartStatement()
       |
       -> drizzled::plugin::StorageEngine::getCursor()
          |
          -> drizzled::Cursor::write_row()
             |
             -> InnobaseCursor::write_row()
       |
       -> drizzled::TransactionServices::autocommitOrRollback()
          |
          -> drizzled::plugin::TransactionStorageEngine::commit()
             |
             -> InnobaseEngine::doCommit()

I think this will come as a welcome change to storage engine developers working with Drizzle.

No More Need for Engine to Call `trans_register_ha()`

There was an interesting comment in the original documentation for the transaction processing code. It read:

Roles and responsibilities
————————–

The server has no way to know that an engine participates in
the statement and a transaction has been started
in it unless the engine says so. Thus, in order to be
a part of a transaction, the engine must “register” itself.
This is done by invoking trans_register_ha() server call.
Normally the engine registers itself whenever handler::external_lock()
is called. trans_register_ha() can be invoked many times: if
an engine is already registered, the call does nothing.
In case autocommit is not set, the engine must register itself
twice — both in the statement list and in the normal transaction
list.

That comment, and I’ve read it dozens of times, always seemed strange to me. I mean, does the server really not know that an engine participates in a statement or transaction unless the engine tells it? Of course not.

So, I removed the need for a storage engine to “register itself” with the kernel. Now, the transaction manager inside the Drizzle kernel (implemented in the TransactionServices component) automatically monitors which engines are participating in an SQL transaction and the engine doesn’t need to do anything to register itself.

In addition, due to the break-up of the plugin::StorageEngine class and the XA API into plugin::XaResourceManager, Drizzle’s transaction manager can now coordinate XA transactions from plugins other than storage engines. Yep, that’s right. Any plugin which implements plugin::XaResourceManager can participate in an XA transaction and Drizzle will act as the transaction manager. What’s the first plugin that will do this? Drizzle’s transaction log. The transaction log isn’t a storage engine, but it is able to participate in an XA transaction, so it will implement plugin::XaResourceManager but not plugin::StorageEngine.

Performance Impact of Code Changes

So, that “yet another overhead” Monty talked about in the quote above? There wasn’t any noticeable impact in performance or scalability at all. So much for optimize-first coding.

What’s Next?

The next thing I’m working on is removing the notion of the “statement transaction”, which is also a historical by-product, this time because of BerkeleyDB. Gee, I’ve got a lot of work ahead of me…

^[1] Actually, there is a way that a transaction that was rolled back can get written to the transaction log. For bulk operations, the server can cut a Transaction message into multiple segments, and if the SQL transaction is rolled back, a special RollbackStatement message is written to the transaction log.

Ronald Bradford - 42SQL: Understanding Drizzle user authentication options Part 1

Fri, 12 Mar 2010 22:46:09 +0000

A key differentiator in Drizzle from it’s original MySQL roots is user based authentication. Gone is the host/user and schema/table/column model that was stored in the MyISAM based mysql.user table.

Authentication is now completely pluggable, leveraging existing systems such as PAM, LDAP via PAM and Http authentication.

In this post I’ll talk about PAM authentication which is effectively your current Linux based user security.

This information is based on the current build 1317.

Compiling for PAM support

Your Drizzle environment needs to be compiled with PAM support. You would have received the following warning during a configure.

$ ./configure
...
checking for libpam... no
configure: WARNING: Couldn't find PAM development support, pam_auth will not be built. On Debian, libpam is in libpam0g-dev. On RedHat it's in pam-devel.

The solution is provided in the warning message which is another great thing about Drizzle. The pre checks for dependencies and the optional messages like these far exceed the MySQL equivalent compilation process. In my case:

$ sudo yum install pam-devel

When correctly configured, it should look like:

checking for libpam... yes
checking how to link with libpam... -lpam

Working with PAM

You need to enable the PAM authentication plugin at drizzled startup.

sbin/drizzled --plugin_add=auth_pam &

Unfortunately connecting fails to work with

time sbin/drizzle --user=testuser --password=***** --port=4427
real 0m0.003s
user 0m0.003s
sys 0m0.001s

A look into the source at src/drizzle-2010.03.1317/plugin/auth_pam/auth_pam.cc shows a needed config file

117     retval= pam_start("check_user", userinfo.name, &conv_info, &pamh);

Configuring PAM

In order to enable PAM with Drizzle you need to have the following system configuration.

$ cat /etc/pam.d/check_user
auth    required        pam_unix.so
account required        pam_unix.so

$ time sbin/drizzle --user=testuser --password=***** --port=4427
ERROR 1045 (28000): Access denied for user 'testuser'@'127.0.0.1' (using password: YES)

real 0m2.055s
user 0m0.002s
sys 0m0.002s

This did some validation but still failed.

It seems Bug #484069 may fix this problem, however this is not currently in the main line!

Stay Tuned!

Ronald Bradford - 42SQL: Understanding Drizzle user authentication options Part 2

Fri, 12 Mar 2010 22:45:26 +0000

Authentication is now completely pluggable, leveraging existing systems such as PAM, LDAP via PAM and Http authentication.

In this post I’ll talk about HTTP authentication which requires an external http server to implement successfully. You can look at Part 1 for PAM authentication.

Compiling for http auth support

By default during compilation you may find.

checking for libcurl... no
configure: WARNING: libcurl development lib not found: not building auth_http plugin. On Debian this is found in libcurl4-gnutls-dev. On RedHat it's in libcurl-devel.

In my case I needed:

$ sudo yum install curl-devel

NOTE: Bug #527255 talks about issues of the message being incorrect for libcurl-devel however this appears it may be valid in Fedora Installs

After successfully installing the necessary pre-requisite you should see.

checking for libcurl... yes
checking how to link with libcurl... -lcurl
checking if libcurl has CURLOPT_USERNAME... no

HTTP Authentication

We need to enable the plugin at server startup.

$ sbin/drizzled --mysql-protocol-port=3399 --plugin_add=auth_http &

You need to ensure the auth_http plugin is active by checking the data dictionary plugin table.

drizzle> select * from data_dictionary.plugins where plugin_name='auth_http';
+-------------+----------------+-----------+-------------+
| PLUGIN_NAME | PLUGIN_TYPE    | IS_ACTIVE | MODULE_NAME |
+-------------+----------------+-----------+-------------+
| auth_http   | Authentication | TRUE      |             |
+-------------+----------------+-----------+-------------+

The auth_http plugin also has the following system variables.

drizzle> SHOW GLOBAL VARIABLES LIKE '%http%';
+------------------+-------------------+
| Variable_name    | Value             |
+------------------+-------------------+
| auth_http_enable | OFF               |
| auth_http_url    | http://localhost/ |
+------------------+-------------------+
2 rows in set (0 sec)

In order to configure Http authentication, you need to have the following settings added to your drizzled.cnf file. For example:

$ cat etc/drizzled.cnf
[drizzled]
auth_http_enable=TRUE
auth_http_url=http://thedrizzler.com/auth

NOTE: Replace the domain name with something you have, even localhost.

A Drizzle restart gives us

$ bin/drizzle -e "SHOW GLOBAL VARIABLES LIKE 'auth_http%'"
+------------------+-----------------------------+
| Variable_name    | Value                       |
+------------------+-----------------------------+
| auth_http_enable | ON                          |
| auth_http_url    | http://thedrizzler.com/auth |
+------------------+-----------------------------+

By default, currently if the settings result in an invalid url, then account validation does not fail and you can still login. It is recommended that you always configure pam authentication as well as a fall back.

$ wget -O tmp http://thedrizzler.com/auth
--17:32:32--  http://thedrizzler.com/auth
Resolving thedrizzler.com... 208.43.73.220
Connecting to thedrizzler.com|208.43.73.220|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
17:32:32 ERROR 404: Not Found.

$ bin/drizzle
drizzle > exit

Configuring passwords

To correctly configured your web server to perform the HTTP auth, you can use this Apache syntax as an example.

The following is added to the VirtualHost entry in your web browser.

<Directory /var/www/drizzle/auth>
AllowOverride FileInfo All AuthConfig
AuthType Basic
AuthName "Drizzle Access Only"
AuthUserFile /home/drizzle/.authentication
Require valid-user
</Directory>

$ sudo su -
$ mkdir /var/www/drizzle/auth
$ touch /var/www/drizzle/auth/index.htm
$ apachectl graceful

We check we now need permissions for the URL.

$ wget -O tmp http://thedrizzler.com/auth
--17:35:48--  http://thedrizzler.com/auth
Resolving thedrizzler.com... 208.43.73.220
Connecting to thedrizzler.com|208.43.73.220|:80... connected.
HTTP request sent, awaiting response... 401 Authorization Required
Authorization failed.

You need to create the username/password for access.

$ htpasswd -cb /home/drizzle/.authentication testuser sakila
$ cat /home/drizzle/.authentication
testuser:85/7CbdeVql4E

Confirm that the http auth with correct user/password works.

$ wget -O tmp http://thedrizzler.com/auth --user=testuser --password=sakila
--17:37:45--  http://thedrizzler.com/auth
Resolving thedrizzler.com... 208.43.73.220
Connecting to thedrizzler.com|208.43.73.220|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently

Drizzle HTTP Authentication in action

By default we now can’t login

$ bin/drizzle
ERROR 1045 (28000): Access denied for user ''@'127.0.0.1' (using password: NO)

$ bin/drizzle --user=testuser --password=sakila999
ERROR 1045 (28000): Access denied for user 'testuser'@'127.0.0.1' (using password: YES)

$ bin/drizzle --user=testuser --password=sakila
Welcome to the Drizzle client..  Commands end with ; or \g.
Your Drizzle connection id is 6
Server version: 7 Source distribution (trunk)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

drizzle>

Stephen O'Grady: The View from NoSQL Live: Three Takeaways

Fri, 12 Mar 2010 17:10:42 +0000

Crossroads, Inroads, Pitfalls & Bylaws

View more presentations from Tim Anglade.

NoSQL Live was a very different show than I’ve been to in recent months. It had very little in common with, for example, HadoopWorld, where the audience was largely already intimately familiar with the technology and value proposition. The NoSQL Live audience, by contrast, to judge from the questions, was mostly there to learn. With many of the usual suspects from the NoSQL world in attendance, along with substantial representation from projects like Cassandra, HBase, memcache, Riak, Voldemort and so on, the show certainly did not lack for subject matter expertise.

But the number of those generally unfamiliar with NoSQL was as surprising as it was gratifying. Gratifying because it serves as a proxy for interest: besides the experts, there were a substantial number of people there looking to get up to speed on the space. Which they certainly had an opportunity to do.

Adam Marcus, a graduate student at MIT’s Computer Science and Artificial Intelligence Laboratory, did a much better job that I could have taking notes on the show here, so I won’t rehash that. Instead, three quick takeaways from the show.

The NoSQL Term is a Problem

In his remarks to open the show, Dwight Merriman – the CEO at 10gen (the company behind MongoDB) – asserted that while the term “NoSQL” had problems, for better or for worse, the name had stuck. Which might indeed be true, and if so the projects may as well make the best of the situation, as he suggested. But if that’s true, the so-called NoSQL projects – all of them – are going to have problems.

Witness Merriman’s definition of NoSQL: no joins and light transactional semantics. Even were we to accept that definition – and even that is problematic as the support varies from project to project – we still have issues. Clearly column databases are differentiated from graph databases, just as both are differentiated from key value stores and document databases.

Currently, however, they are all referred to – marketed, even – under the blanket NoSQL. Hence some of the confusion heard from users yesterday, who were struggling to grasp why all these NoSQL tools had seemingly nothing in common with one another.

The good news is that there is – as evidenced by this and other NoSQL events – substantial interest and traction in data storage software that is not a relational database. The problem is that the naming is likely to become a serious problem if it isn’t already.

Consider slide 13 of Tim Anglade’s excellent presentation embedded above. If he’s correct, and we’re just this side of the Gartner’s trough of disillusionment – and I believe that’s a reasonable assertion – the NoSQL term is going to be one of the reasons for the fall. Most of the current NoSQL adopters are sufficiently up to date on developments in the data persistence space that the name is not much of an issue. The next wave of adopters is guaranteed to be less familiar with the distinctions between the project approaches and more frustrated by the inherent educational challenges therein.

I know quibbling over a name seems inane to a great many technologists out there, but you’d be surprised at how much difference a name makes in this industry. Remember what the mere application of the term Ajax did to discussion of that technical approach? Now consider if Ajax had attempted to encompass that and native client side development. That’s what NoSQL is doing at present, and it’s a problem.

MySQL is a Target

Mark Callaghan recently said:

I think that MySQL+memcached is still the default choice and I don’t think it is going away in the high-scale market.

Eric Day, Drizzle developer, likewise said that that project is complementary to many NoSQL efforts when I spoke to him on Monday. Clearly his new employer (and yes, more on that later) believes that, significantly contributing as it does to both NoSQL (Cassandra) and SQL (Drizzle) projects.

I think they’re right. The maturity of the MySQL ecosystem and its basic ubiquity will not easily be thrown over, if ever. That said, the commentary from Twitter’s Ryan King was really eye-opening.

It’s no secret that Twitter has been moving slowly towards Cassandra and away from MySQL. This is from an interview that King did previously with myNoSQL, describing the motivations:

We have a lot of data, the growth factor in that data is huge and the rate of growth is accelerating.

We have a system in place based on shared mysql + memcache but its quickly becoming prohibitively costly (in terms of manpower) to operate. We need a system that can grow in a more automated fashion and be highly available.

After yesterday, we can add to that some numbers. Twitter’s Cassandra infrastructure is at 45 nodes, which is handling – in parallel with the MySQL/memcached infrastructure – some 600/700 Tweets (i.e. writes) /second (50M/day) with massive spikes (like for SXSW, for example) and nine or ten billion rows.

The MySQL infrastructure – largely thanks to a massive memcached presence, according to what we heard yesterday – was still handling this load. But much of the real pain comes apparently in manageability. The MySQL cluster could, in the words of King, “never be taken down,” as the restarts were too painful. The Cassandra nodes, meanwhile, are rebooted regularly with rolling restarts.

What does this mean? Nothing, yet. Twitter is a traffic outlier than 99% of MySQL or NoSQL users will never see. But the number of higher traffic properties that are leaving MySQL based infrastructure for NoSQL alternatives is worth monitoring, just as was their original takeup of MySQL.

NoSQL and the Cloud

One of the panels yesterday was on the subject of NoSQL in the Cloud. The panelists were Benjamin Day (consultant with Azure experience), Jonathan Ellis (Rackspace, Cassandra lead), Adam Kocoloski (Cloudant, a CouchDB vendor), and Daniel Rinehart (Allurent, a startup using AWS’ SimpleDB).

Predictably, opinions varied on the suitability of NoSQL technologies for the cloud. The vendors offering or leveraging NoSQL services in the cloud – Allure/Cloudant/Rackspace – were more or less positive on the concept. Ellis, meanwhile, was less enthusiastic, urging workload based deployment: elastic, transient needs to the cloud, general, sustained workloads to the datacenter.

What I was surprised to hear little about, except from a questioner, is the question of operations. For many cloud users, questions of workload or the suitability for a given technology such as NoSQL to the cloud come second. The primary concern is operational costs, or the lackthereof. Put simply: the cloud takes important operational elements and makes them someone else’s problem. This may be even more compelling with NoSQL, because the relative immaturity of the projects means that they are often suboptimally packaged. Being able to spin up a prebuilt image on AWS or Rackspace is likely to be significantly preferable to an alternative of hand assembling all of the necessary pieces of your NoSQL infrastructure.

This is why I find services like Bradford Stephens’ Drawn to Scale interesting (again, more on them later): being able to offload the operational costs – both in dollars and learning curve – of software that can more efficiently attack large or unstructured datasets is likely to be an interesting proposition.

Whether NoSQL technologies are ideally suited to multi-tenant cloud environments, then, seems to be besides the point: they will be used there – heavily – regardless. If they’re not well suited to that, from a customer’s perspective, that’s the provider’s problem.

Anyway, thanks to the folks from 10gen, Cloudant, Hashrocket, O’Reilly, GigaOm, myNoSQL et al who put on the conference. Well worth the trip down.

Pythian Group: Log Buffer #182, a Carnival of the Vanities for DBAs

Fri, 12 Mar 2010 17:01:59 +0000

This is the 182nd edition of Log Buffer, the weekly review of database blogs. Make sure to read the whole edition so you do not miss where to submit your SQL limerick!

This week started out with me posting about International Women’s Day, and has me personally attending Confoo (Montreal) which is an excellent conference I hope to return to next year. I learned a lot from confoo, especially the blending nosql and sql session I attended.

This week was also the Hotsos Symposium. Doug’s Oracle Blog has a series of posts about Hotsos. If all this talk about conferences has gotten you excited, Joshua Drake notes that 14 days and the hotel is almost full for postgresql conference east which is March 25th-28th in Philadelphia. And the Oracle database insider notes that the Oracle OpenWorld call for papers is now open.

According to Susan Visser this week (ending tomorrow) is also read an e-book week. So if you have not already done so, read an e-book! She links a coupon for an e-book in the post.

Craig Mullins notes that the mainframe is a good career choice in Mainframes: The Safe IT Career Choice. He notes that the mainframe is still not dead:

People having been predicting the death of the mainframe since the advent of client/server in the late 1980s. That is more than 20 years! Think of all the things that have died in that timespan while the mainframe keeps on chugging away: IBM’s PC business, Circuit City, Koogle peanut butter, public pay phones, Johnny Cash… the list is endless.

In other career-related news, Antonio Cangiano is looking for [2] top-notch student hackers for a 16-month internship at IBM in Toronto starting in May. All the details, including how to apply, are in Cangiano’s blog post.

Willie Favero wants to know how you “solve the batch dilemma” for issues like “shrinking your batch window, designing your batch to play nicely with … OLTP” in how’s your batch workload doing? Perhaps Favero should read the updated batch best practices posted by Anthony Shorten.

Bryan Smith surveys a more personal question by asking if you go both ways and “manage both DB2 for Linux, UNIX, and Windows and DB2 for z/OS” in don’t ask, don’t tell, bi-platform DBAs. This week’s Log Buffer editor admits to being a tri-platform DBA — she has tried many platforms, and in fact, many databases (MySQL, Oracle, DB2, SQL Server, Sybase, Postgres and Ingres)!

Hari Prasanna Srinivasan promotes a patching survey in Oracle really wants to hear from you! Patching Survey.

Henrik Loeser explains what a deadlock and a hot spot are by using a real life analogy taken from a police report in deadlock and hot spot in real life.

Jamie Thomson asks why do you abbreviate schema names?. Shlomi Noach tries to solve the issue that “there is no consistent convention as for how to write [about table aliases in] an SQL query” in proper sql table alias use conventions. Noach also gives us a tip: faster than truncate.

Leons Petrazickis reminds us that “rulesets are chains” and it is important to have your rulesets in the proper order in iptables firewall pitfall.

Anyone interested in the history of MySQL AB will be informed after reading Dries Buytaert’s article.
Gavin Towey shares his software that helps centrally manage 120 MySQL servers in qsh.pl: distributed query tool For those who want to learn more about column-oriented databases, particularly in MySQL, Robin Schumacher of the InfiniDB blog announces that there is a MySQL University session recording on MySQL column databases now available. MySQL join-fu expert Jay Pipes has moved his blog to www.joinfu.com and starts with An SQL Puzzle and of course a follow up on the sql puzzle.

Ivan Zoratti is happy that finally, slides posted for the MySQL DW breakfast. Venu Anuganti gives you tips on one of the most common MySQL frustrations: optimizing subqueries in how to improve subqueries derived tables performance. Justin Swanhart posts the way in which he Gets Linux performance information from your MySQL database without shell access and emulates a ‘top’ CPU summary using /proc/stat and MySQL using the same method.

The Oracle Apps blog has an introduction to Oracle user productivity kit (UPK). Even though in this editor’s opinion the article is very sales-pitchy, it has valuable information, and does indeed live up to its promise:

UPK is a software tool that can capture all the steps in a system process. It records every keystroke, every click of the mouse, each menu option chosen and each button pressed. All this is done in the UPK Recorder by going through the transaction and pressing “printscreen” after every user action. From this, without any further effort from the developer, UPK builds a number of valuable outputs.

Allen White gives a great tip on how to optimize queries in keep your data clean.

Mike Dietrich reminds you to remove “old” parameters and events from your init.ora when upgrading, “as keeping them will definitely slow down the database performance in the new release.” He shows evidence of slowness when this is not done. Dietrich also shows how you can be gathering workload statistics “to give the optimizer some good knowledge about how powerful your IO-system might be”, especially “a few days after upgrading to the new release…while a real workload is running.”

Brian Aker shows the exciting features coming soon in Drizzle in Drizzle, Cherry, Roadmap for our Next Release.

Maybe you are thinking of migrating, not upgrading…..The O’Reilly Radar shows how to asses an Oracle to MySQL migration in MySQL migration and risk management. Actually, that article interviews Ronald Bradford on the subject — Bradford has been prolific lately, updating free my.cnf advice series and “Don’t Assume”: MySQL for the Oracle DBA series. Nick Quarmby also talks about migrating Oracle, but not to a new database, just to a new platform, in his primer on migrating Oracle Applications to new platforms. And the big news comes from Carlos of dataprix that Twitter will migrate from MySQL to Cassandra DB.

Paul S. Randal explains his way of benchmarking: 1 Tb table population on SQL Server.

Pete Finnigan shares his slides from a webinar on how to secure oracle, and Denis Pilipchuk shares his approaches for discovering security vulnerabilities in software applications.

Jeff Davis shares his thoughts about scalability and the relational model. Robert Treat responds actually, the relational model doesn’t scale and Baron Schwartz counters with NoSQL doesn’t mean non-relational.

Buck Woody explains “whenever you want to know something about SQL Server’s configuration, whether that’s the Instance itself or a database, you have a few options” — and of course what those options are — in system variables, stored procedures or functions for meta data.

This week’s T-SQL Tuesday topic was I/O. There are many links to great blog posts in the comments; three random posts I chose to highlight: Michael Zilberstein talks about IO capacity planning, while Kalen Delaney talks about using STATISTICS IO in I/O, you know, and Merrill Aldrich chimes in with information on real world SSD’s. Aldrich also begs folks not to waste resources and make more work for developers and DBAs in dear ISV, you’re keeping me awake nights with your VARCHAR() dates.

And we end with a bit of fin: Paul Nielsen wants us all to have a bit of fun; he has posted an SQL limerick and asks readers to create there own in there once was in Dublin a query.

Brian Aker: The Art of "What is going on inside of my database?"

Fri, 12 Mar 2010 16:50:02 +0000

Yesterday we were having a conversation on IRC about the need for more useful information about the internals of the database.

"SHOW STATUS" is just too primitive in its design to provide the sort of detailed information you need to do operations. Yesterday we got a bug request over the number of "open tables" found after a particular query. The user had assumed the number was off, but what they hadn't realized was that the number was accurate (in this particular case, MySQL fudges a number on open tables because it can't handle count its derived tables).

One of the patches coming into the tree right now fully exposes the contents of the table cache and table definition cache to the user. You can see who holds what locks on what tables, and you can see the actual count on table access per table.


drizzle> select * from TABLE_DEFINITION_CACHE;
+-----------------+------------------------+---------+-------------+----------------+
| TABLE_SCHEMA    | TABLE_NAME             | VERSION | TABLE_COUNT | IS_NAME_LOCKED |
+-----------------+------------------------+---------+-------------+----------------+
| data_dictionary | schema_names           |       1 |           1 | FALSE          |
| data_dictionary | table_definition_cache |       1 |           1 | FALSE          |
| data_dictionary | show_tables            |       1 |           1 | FALSE          |
+-----------------+------------------------+---------+-------------+----------------+
3 rows in set (0 sec)

drizzle> select * from TABLE_CACHE;
+------------+-----------------+------------------------+-----------+----------------+---------+----------------+------+----------------+------------+----------------+
| SESSION_ID | TABLE_SCHEMA    | TABLE_NAME             | ARCHETYPE | ENGINE         | VERSION | IS_NAME_LOCKED | ROWS | AVG_ROW_LENGTH | TABLE_SIZE | AUTO_INCREMENT |
+------------+-----------------+------------------------+-----------+----------------+---------+----------------+------+----------------+------------+----------------+
|          0 | data_dictionary | schema_names           | FUNCTION  | FunctionEngine |       1 | FALSE          |  100 |            260 |          0 |              0 |
|          0 | data_dictionary | show_tables            | FUNCTION  | FunctionEngine |       1 | FALSE          |  100 |            260 |          0 |              0 |
|          1 | data_dictionary | table_cache            | FUNCTION  | FunctionEngine |       1 | FALSE          |  100 |           1113 |          0 |              0 |
|          0 | data_dictionary | table_definition_cache | FUNCTION  | FunctionEngine |       1 | FALSE          |  100 |            559 |          0 |              0 |
+------------+-----------------+------------------------+-----------+----------------+---------+----------------+------+----------------+------------+----------------+
4 rows in set (0 sec)

drizzle> select * from TABLE_DEFINITION_CACHE WHERE TABLE_COUNT > 1;
Empty set (0 sec)

drizzle> select * from TABLE_DEFINITION_CACHE WHERE TABLE_COUNT > 0;
+-----------------+------------------------+---------+-------------+----------------+
| TABLE_SCHEMA    | TABLE_NAME             | VERSION | TABLE_COUNT | IS_NAME_LOCKED |
+-----------------+------------------------+---------+-------------+----------------+
| data_dictionary | schema_names           |       1 |           1 | FALSE          |
| data_dictionary | table_cache            |       1 |           1 | FALSE          |
| data_dictionary | table_definition_cache |       1 |           1 | FALSE          |
| data_dictionary | show_tables            |       1 |           1 | FALSE          |
+-----------------+------------------------+---------+-------------+----------------+
4 rows in set (0 sec)

drizzle> select count(*) from TABLE_DEFINITION_CACHE WHERE TABLE_COUNT  > 0;
+----------+
| count(*) |
+----------+
|        4 |
+----------+
1 row in set (0 sec)

The term "ARCHETYPE" is the base primitive about what sort of table was used. It is more detailed then the ANSI "TABLE_TYPE" that exists in I_S. We still have a debate on what exactly this term should mean. One of the things I enjoy about working on Drizzle? I am not stuck in a room full of people who will spend hours on this sort of bike shed decisions.

Version gives you the current definition count for the table. Right now that number is still based on "since opened" but we will soon be storing the metadata for this so you will know how many times in the life of an object it has been changed.

We are still working out the details to SHOW TABLE STATUS. Our SHOW commands are just query rewrites to tables.

Here is a partial example of the new table that is outputted from a SHOW TABLE STATUS:

+---------+--------+-------------------+-----------
| Session | Schema | Name              | Type
+---------+--------+-------------------+-----------
|       0 | Schema | b                 | STANDARD
|       0 | Schema | show_tables       | FUNCTION
|       1 | Schema | show_table_status | FUNCTION
|       0 | Schema | schema_names      | FUNCTION
|       0 | Schema | dfsdf             | STANDARD
|       1 | Schema | b                 | TEMPORARY
|       1 | Schema | a                 | TEMPORARY
+---------+--------+-------------------+-----------+
7 rows in set (0 sec)

Notice Session? Notice that type can be Temporary? With our system you can see the current owner of the open table and we now include whatever temporary tables you have in your own session. We have also included a larger table which you can see just your own temporary tables (and most likely I will soon create a table so that you can see all temporary tables open across all sessions). The current "Type" in SHOW TABLE STATUS is an Archetype so we will be changing that so that terms match up across the database. Consistency in design is an awesome thing :)

There is a lot more to come!

P.S. Just wait until I push the code for tracking locks in Drizzle, I demoed it at SCALE and got a lot of both positive and constructive feedback on it.

Official Drizzle Blog: Location for Drizzle Developer Day 2010

Fri, 12 Mar 2010 03:12:13 +0000

We finally have our location for Drizzle Developer Day 2010, this year it will be at the Santa Clara Convention Center, rooms 209 and 210. As we previously announced the event will be Friday April 16th right after the MySQL User Conference, 9:30 a.m. to 5:00 p.m.

Please make sure and sign up at http://drizzle.org/wiki/Drizzle_Developer_Day_2010_signup. Space will be limited and it is filling up fast.

Also, to make this day a success, please provide your input on discussion topics at http://drizzle.org/wiki/Drizzle_Developer_Day_2010

Hope to see you there!

Jay Pipes: A Follow Up on the SQL Puzzle

Thu, 11 Mar 2010 22:12:35 +0000

Or…What the Heck is Wrong with `CREATE TABLE IF NOT EXISTS ... SELECT`?

So, earlier this week, I blogged about an SQL puzzle that had come up in my current work on Drizzle’s new transaction log.

I posed the question to readers what the “correct” result of the following would be:

CREATE TABLE t1 (a int, b int);
INSERT INTO t1 VALUES (1,1),(1,2);
CREATE TEMPORARY TABLE t2 (a int, b int, PRIMARY KEY (a));
BEGIN;
INSERT INTO t2 VALUES (100,100);
CREATE TEMPORARY TABLE IF NOT EXISTS t2 (PRIMARY KEY (a)) SELECT * FROM t1;
 
# The above statement will correctly produce an ERROR 23000: Duplicate entry '1' for key 'PRIMARY'
# What should the below result be?
 
SELECT * FROM t2;
COMMIT;

A number of readers responded, and, to be fair, most everyone was “correct” in their own way. Why? Well, because the way that MySQL deals with calls to CREATE TABLE ... SELECT, CREATE TABLE IF NOT EXISTS ... SELECT and their temporary-table counterparts is completely stupid, as I learned this week. Rob Wultsch essentially sums up my feelings about the behaviour of DDL statements in regards to transactions in a session:

Implicit commit is evil and stupid. Ideally we the server should error and roll back, imho.

The Officially Correct Answer (at least in MySQL)

OK, so here’s the “official” correct answer:

CREATE TABLE IF NOT EXISTS ... SELECT does not first check for the existence of the table in question. Instead, if the table in question does exist, CREATE TABLE IF NOT EXISTS ... SELECT behaves like an INSERT INTO ... SELECT statement. Yep, you heard right. So, instead of throwing a warning when it notices that the table exists, MySQL instead attempts to insert rows from the SELECT query into the existing table.

Here is the official MySQL explanation:

For CREATE TABLE … SELECT, if IF NOT EXISTS is given and the table already exists, MySQL handles the statement as follows:
* The table definition given in the CREATE TABLE part is ignored. No error occurs, even if the definition does not match that of the existing table.
* If there is a mismatch between the number of columns in the table and the number of columns produced by the SELECT part, the selected values are assigned to the rightmost columns. For example, if the table contains n columns and the SELECT produces m columns, where m n, the selected values are assigned to the m rightmost columns in the table. Each of the initial n – m columns is assigned its default value, either that specified explicitly in the column definition or the implicit column data type default if the definition contains no default. If the SELECT part produces too many columns (m > n), an error occurs.
* If strict SQL mode is enabled and any of these initial columns do not have an explicit default value, the statement fails with an error.

So, given the above manual explanation, the correct answer to the original blog post is:

  a  |  b
100 | 100

partly because there is an implicit COMMIT directly before the CREATE TABLE is executed (committing the 100,100 record to the table) and the primary key violation kills off the INSERTs of 1,1 in InnoDB. For a MyISAM table, the 1,1 record would be in the table, since MyISAM has no idea what a ROLLBACK is.

I Think Drizzle Should Follow PostgreSQL’s Example Here

On implicit commits before DDL operations, I believe they should all go bye-bye. DDL should be transactional in Drizzle and if a statement cannot be executed in a transaction, it should throw an error if there is an active transaction. Period.

For behaviour of CREATE TABLE ... SELECT acting like an INSERT INTO ... SELECT, that entire code path should be ripped out.

PostgreSQL’s DDL operations, IMHO, are sane. Sane is good. PostgreSQL allows quite a bit of flexibility by implementing the SQL standard’s CREATE TABLE and CREATE TABLE AS statements. I believe Drizzle should scrap all the DDL table-creation code and instead implement PostgreSQL’s much-nicer DDL methods. There, I said it. Slashdot MySQL haters, there ya go.

Ronald Bradford - 42SQL: Drizzles Data Dictionary and Global Status

Thu, 11 Mar 2010 21:33:14 +0000

With the recent news by Brian about the Data Dictionary in Drizzle replacing the INFORMATION_SCHEMA, I was looking into the server status variables (aka INFORMATION_SCHEMA.GLOBAL_STATUS) and I came across an interesting discovery.

select * from data_dictionary.global_status;
...
| Table_locks_immediate      | 0              |
| Table_locks_waited         | 0              |
| Threads_connected          | 8134064        |
| Uptime                     | 332            |
| Uptime_since_flush_status  | 332            |
+----------------------------+----------------+
51 rows in set (0 sec)

This only retrieved 51 rows, which is way less then previous. What I wanted was clearly missing, all the old com_ status variables. Looking at what the data_dictionary actually has available revealed a new table.

drizzle> select * from data_dictionary.global_statements;
+-----------------------+----------------+
| VARIABLE_NAME         | VARIABLE_VALUE |
+-----------------------+----------------+
| admin_commands        | 0              |
| alter_db              | 0              |
| alter_table           | 0              |
| analyze               | 0              |
| begin                 | 0              |
| change_db             | 1              |
| check                 | 0              |
| checksum              | 0              |
| commit                | 0              |
| create_db             | 0              |
| create_index          | 0              |
| create_table          | 0              |
| delete                | 0              |
| drop_db               | 0              |
| drop_index            | 0              |
| drop_table            | 0              |
| empty_query           | 0              |
| flush                 | 0              |
| insert                | 0              |
| insert_select         | 0              |
| kill                  | 0              |
| load                  | 0              |
| release_savepoint     | 0              |
| rename_table          | 0              |
| replace               | 0              |
| replace_select        | 0              |
| rollback              | 0              |
| rollback_to_savepoint | 0              |
| savepoint             | 0              |
| select                | 10             |
| set_option            | 0              |
| show_create_db        | 0              |
| show_create_table     | 0              |
| show_errors           | 0              |
| show_warnings         | 0              |
| truncate              | 0              |
| unlock_tables         | 0              |
| update                | 0              |
+-----------------------+----------------+
38 rows in set (0 sec)

Kudos to this. Looking at list I saw an obvious omission, of “ping”. Something that caught me out some years ago with huge (300-500 per second admin_commands). I’m also a fan of Mark’s recent work An evening hack – Com_ping in MySQL.

Stewart Smith: Writing A Storage Engine for Drizzle, Part 2: CREATE TABLE

Thu, 11 Mar 2010 14:27:34 +0000

The DDL code paths for Drizzle are increasingly different from MySQL. For example, the embedded_innodb StorageEngine CREATE TABLE code path is completely different than what it would have to be for MySQL. This is because of a number of reasons, the primary one being that Drizzle uses a protobuf message to describe the table format instead of several data structures and a FRM file.

We are pretty close to having the table protobuf message format being final (there’s a few bits left to clean up, but expect them done Real Soon Now (TM)). You can see the definition (which is pretty simple to follow) in drizzled/message/table.proto. Also check out my series of blog posts on the table message (more posts coming, I promise!).

Drizzle allows either your StorageEngine or the Drizzle kernel to take care of storage of table metadata. You tell the Drizzle kernel that your engine will take care of metadata itself by specifying HTON_HAS_DATA_DICTIONARY to the StorageEngine constructor. If you don’t specify HTON_HAS_DATA_DICTIONARY, the Drizzle kernel stores the serialized Table protobuf message in a “table_name.dfe” file in a directory named after the database. If you have specified that you have a data dictionary, you’ll also have to implement some other methods in your StorageEngine. We’ll cover these in a later post.

If you ever dealt with creating a table in MySQL, you may recognize this method:

virtual int create(const char *name, TABLE *form, HA_CREATE_INFO *info)=0;

This is not how we do things in Drizzle. We now have this function in StorageEngine that you have to implement:

int doCreateTable(Session* session, const char *path,
                  Table& table_obj,
                  drizzled::message::Table& table_message)

The existence of the Table parameter is largely historic and at some point will go away. In the Embedded InnoDB engine, we don’t use the Table parameter at all. Shortly we’ll also get rid of the path parameter, instead having the table schema in the Table message and helper functions to construct path names.

Methods name “doFoo” (such as doCreateTable) mean that there is a method named foo() (such as createTable()) in the base class. It does some base work (such as making sure the table_message is filled out and handling any errors) while the “real” work is done by your StorageEngine in the doCreateTable() method.

The Embedded InnoDB engine goes through the table message and constructs a data structure for the Embedded InnoDB library to create a table. The ARCHIVE storage engine is much simpler, and it pretty much just creates the header of the ARZ file, mostly ignoring the format of the table. The best bet is to look at the code from one of these engines, depending on what type of engine you’re working on. This code, along with the table message definition should be more than enough

Official Drizzle Blog: And the answer is ….. Rackspace Cloud!

Thu, 11 Mar 2010 12:46:31 +0000

A lot of you have been wondering where the Drizzle developers who were working at Sun have landed after the Oracle acquisition. As I’m sure you have seen by now we are very happy to now be working on Drizzle at Rackspace Cloud. Check out the blogs from Eric, Jay, Monty and Stewart.

-Lee

Stewart Smith: Continuing the journey

Thu, 11 Mar 2010 07:42:46 +0000

A couple of months ago (December 1st for those playing along at home) it marked five years to the day that I started at MySQL AB (~~now Sun~~, now Oracle). A good part of me is really surprised it was for that long and other parts surprised it wasn’t longer. Through MySQL and Sun, I met some pretty amazing people, worked with some really smart ones and formed really solid and awesome friendships. Of course, not everything was perfect (sometimes not even close), but we did have some fun.

Up until November 2008 (that’s 3 years and 11 months for those playing at home) I worked on MySQL Cluster. Still love the product and love how much better we’re making Drizzle so it’ll be the best SQL interface to NDB :)

The ideas behind Drizzle had been talked about for a while… and with my experience with internals of the MySQL server, I thought that some change and dramatic improvement was sorely needed.

Then, in 2008, Brian created a tree. I was soon sending in patches at nights, we announced to the whole world at OSCON and it captured a lot of attention.

Since November 2008 I’ve been working on Drizzle full time. It was absolutely awesome that I had the opportunity to spend all my days hacking on Drizzle – both directly with fantastic people and for fantastic people.

But… the Sun set… which was exciting and sad at the same time.

Never to fear! There were plenty of places wanting Drizzle hackers (and MySQL hackers). For me, it came down to this: “real artists ship”. While there were other places where I would no doubt be happy and work on something really cool, the only way I could end up working out where I should really be was: what is the best way to have Drizzle make a stable release that we’d see be suitable for deployment? So, Where Am I Now?

Rackspace.

Where I’ll again be spending all my time hacking Drizzle.

Monty Taylor: [RH]acker

Monty Taylor — Thu, 11 Mar 2010 00:39:14 +0000

As I'm sure everyone has figured out by now, I've joined Rackspace where I will continue to work on Drizzle. I'm honestly thrilled with my new home, and there are a myriad of reasons for that. I think the one that I'm most excited about is that they are already the thing that all of the hype was about MySQL and RedHat and IBM wanting to become:

A Service Company

Rackspace doesn't want you to run Rackspace-Apache or RackspaceDB or EC-Rackspace. They want you to be able to run bog-standard Apache. And Linux. And MySQL. And PHP. And Drizzle. Then, Rackspace wants to be the best at providing you the service you need around those.

No ludicrous MySQL Enterprise "we'll sell you a license to a free product, and then we'll include bundled with that a subscription a piece of non-free monitoring software" upselling. Rackspace actually wants to provide you a valuable service, and they want to do such a good job at it that you will happily pay them to do it.

For developers, there is a wonderful upside to this: Rackspace doesn't want a special internal Rackspace-only version of anything. It has no value that way. They want the good software to be ubiquitous so that they can compete in the service arena. This means that they don't want assignment of copyright. This means they don't have crazy policies about what Free Software projects you can and cannot contribute to.

Rackspace goes one step further than "do no evil" ... they actually want you to try to improve the state of the art - which goes right to the core of why I'm involved with Free Software in the first place.

I truly believe that Free will always win over Restricted, that Open beats Closed and that Sharing will always improve the world before Hoarding. I've always contended that a company can be successful and make the world a better place and that the two are not mutually exclusive.

I am thrilled to now be a part of a company where I can do my best to prove it.

Brian Aker: Google Summer of Code projects, Drizzle

Wed, 10 Mar 2010 18:25:25 +0000

I've been doing Google Summer of Code projects with students since its creation. As far as intern programs go, it has been one of the most successful I have ever worked with.

Last year was particularly awesome in that with Drizzle we were able to have students work on projects that made it back into Drizzle. While I have always seen good work created, it has always been hit or miss on whether the student's work has made it back into the project. Last year though we got more code in then ever before and I believe this year will be the same. We have had students go on to jobs thanks to the work they did on Drizzle.

Interning gives you real experience, and it provides resume material which differentiates students who are going on to work in the software engineering field. Working on open source means that you have real experience on your resume, experience that an employer can see. There are many positions open in the Drizzle/MySQL ecosystem and students who have real world experience should have any easy time finding work with the knowledge you will gain from this program.

For Drizzle we have worked out a partial list for this year:
http://drizzle.org/wiki/Soc

Don't see anything you like? I am happy to add new projects or work with students on libmemcached or Gearman.

Are you interested in working on a different project? Apache, Linux, Postgres? Talk to those projects and ask them to either participate or suggest ideas on projects to them.

Padraig O'Sullivan: Out of Tree Plugins in Drizzle

Wed, 10 Mar 2010 08:00:00 +0000

This week I've been working on porting the prototype MySQL storage engine developed at Akiban to Drizzle. While doing this, I discovered that in Drizzle, it is possible to build a plugin out of tree. When I say out of tree, I mean that I can develop a plugin for drizzle and build it without having a copy of the drizzle source code. This is amazingly awesome and is mostly due to the awesome build system that Monty has put together. This build system is called Pandora Build and if you are ever working on a project that needs to use autoconf related tools, you should really check it out. Its friggin awesome. It lets you concentrate on development instead of having to spend a bunch of time trying to get a good build environment set up.
Anyway, here I am going to go through an example of how to build a drizzle plugin out of tree. The code is available at lp:~posulliv/drizzle/out-of-tree-example if anyone is interested in looking at it. I am going to take an existing plugin in the drizzle source tree I developed and show how to build it out of tree. The plugin I'm going to work with is the memcached_stats plugin.
Before starting, its worth noting that Monty is working on creating a one-step tool for taking a plugin that is currently in drizzle's source tree (that is, in the plugin directory of a drizzle tree) and making it possible to build that plugin out of tree. His goal is that there need be no changes in content between a directory that's in the drizzle source tree and one that's outside the source tree.
For this post, we will assume that we are working in a directory named mc-stats-plugin. Before starting. this directory just contains source files. We will be adding all the build-related files that are needed to build it.
The first thing that is needed is a plugin.ini file for a plugin. For an out-of-tree plugin, a name and url is required. Thus, the plugin.ini file for this plugin will look like:

[plugin]
name=memcached_stats
title=Memcached Stats in DATA_DICTIONARY tables
description=Some DATA_DICTIONARY tables that provide Memcached stats
url=http://memcached.org/
version=0.1
disabled=yes
load_by_default=no
author=Padraig O Sullivan
license=PLUGIN_LICENSE_BSD
headers=stats_table.h analysis_table.h sysvar_holder.h
sources=memcached_stats.cc stats_table.cc analysis_table.cc
build_conditional="${ac_cv_libmemcached}" = "yes" -a "x${MEMCACHED_BINARY}" != "xno"
ldflags=${LTLIBMEMCACHED}

Once that's done, we need to create a config directory and copy a few files from drizzle's trunk:

$ cp $DRIZZLE_SRC_ROOT/config/config.rpath ./config/.
$ cp $DRIZZLE_SRC_ROOT/config/pandora-plugin ./config/.
$ cp -R $DRIZZLE_SRC_PORT/m4 .

Like I said before, Monty is working on a tool that will automate the steps above. Now, we can proceed and start compiling our plugin:

$ ./config/pandora-plugin
$ autoreconf -i
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `config'.
libtoolize: copying file `config/config.guess'
libtoolize: copying file `config/config.sub'
libtoolize: copying file `config/install-sh'
libtoolize: copying file `config/ltmain.sh'
libtoolize: putting macros in `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
libtoolize: Remember to add `LT_INIT' to configure.ac.
libtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])' to configure.ac and
libtoolize: rerunning libtoolize, to keep the correct libtool macros in-tree.
configure.ac:7: installing `config/compile'
configure.ac:7: installing `config/missing'
Makefile.am: installing `config/depcomp'
$ ./configure
...
$ make
make  all-am
make[1]: Entering directory `/home/posulliv/repos/drizzle/mc-stats-plugin'
  CXX    libmemcached_stats_plugin_la-memcached_stats.lo
  CXX    libmemcached_stats_plugin_la-stats_table.lo
  CXX    libmemcached_stats_plugin_la-analysis_table.lo
  CXXLD  libmemcached_stats_plugin.la
make[1]: Leaving directory `/home/posulliv/repos/drizzle/mc-stats-plugin'
$

Now, our plugin is built. To install it, we simply do a make install and give the --plugin_add=memcached_stats option to drizzled when we start the server.
I just think this process makes my life a whole lot easier and I wanted to bring some attention to how easy drizzle makes developing plugins.

Stewart Smith: Drizzle BoF at the MySQL Conference and Expo

Tue, 09 Mar 2010 04:46:39 +0000

At the 2010 O’Reilly MySQL Conference and Expo there will be a Drizzle BoF!

It’s currently scheduled for 7pm on April 13th.

Come along, it will be awesome.

Eric Day: Drizzling from the Rackspace Cloud

Mon, 08 Mar 2010 16:57:20 +0000

Since I left Sun back in January, folks have been asking what was next. I’m happy to say that I’m going to continue hacking on open source projects like Drizzle and Gearman, but now at the Rackspace Cloud. Not only will I be there, but I get to continue working closely with a few of the amazing Drizzle hackers who have also joined, including Monty Taylor, Jay Pipes, Stewart Smith, and Lee Bieber.

Why Rackspace Cloud? Late last year I was considering what I wanted to do next with the Oracle acquisition looming near, and this was one of the options that presented itself. Rackspace had been a supporter of Drizzle from early on by offering virtual machines to develop and test on, and when talking to some folks more closely, something really hit home. Rackspace provides first-class service and “fanatical” support – they are not a software company. One might ask why an open source software developer would be interested in a company that doesn’t create software or vice-versa, and the answer is that Rackspace wants to find ways to offer the best possible service now and into the future. What better way than to help develop the next generation of service software and get a jump start into integrating this into their architecture? Both the open source community and Rackspace win.

Another thing I learned while talking with Rackspace is that one of their core principles is transparency. This applies to both customer and employees, and anyone within an open source community can appreciate this. The more I learned about the company and the folks within it, the more impressed I was at the lack of internal barriers or “need-to-know” information. One of Drizzle’s core goals is also transparency, from discussing design decisions on public mailing lists and IRC, to having the entire project management infrastructure hosted out in the open at Launchpad.

What does this mean for the Drizzle project? It means continued support for a number of core developers, more infrastructure for development, and most importantly in my eyes, more context. One of the Drizzle tag-lines is “A Lightweight SQL Database for Cloud and Web,” so what better place to develop a database designed for the cloud than on one of the fastest growing cloud platforms. We’ll get a detailed look at the demands, get feedback from cloud customers, and have the perfect test bed for offering new services. We’ll also be able to work closely with a top-notch group of DBAs, developers, and sysadmins in one of the most demanding service architectures out there. This invaluable context will help the Drizzle developers make more informed decisions moving forward, which also means better software for the community.

Personally, this also means getting back to my hosting roots. Before Sun, I worked at Concentric for almost 10 years in a clustered hosting environment. I’m very familiar with many of the multi-tenant scalability concerns Rackspace has, and I’m excited to be working in this type of environment again. We’ve already been working closely with the MySQL DBAs at Rackspace to learn what the biggest pain points are for a multi-tenant architecture, and we’ll be taking steps to address these as it will help anyone wanting to run Drizzle in a cloud-like environment. Drizzle’s modular architecture has already proved useful, as some of these concerns are easily answered with “oh, we have a plugin point for that.”

I’m excited, this is going to be a fun ride.

Jay Pipes: Happiness is a Warm Cloud

Mon, 08 Mar 2010 14:31:34 +0000

Although a few folks knew about where I and many of the Sun Drizzle team had ended up, we’ve waited until today to “officially” tell folks what’s up. We — Monty Taylor, Eric Day, Stewart Smith, Lee Bieber, and myself — are all now “Rackers”, working at Rackspace Cloud. And yep, we’re still workin’ on Drizzle. That’s the short story. Read on for the longer one

An Interesting Almost 3 Years at MySQL

I left my previous position of Community Relations Manager at MySQL to begin working on Brian Aker’s newfangled Drizzle project in October 2008.

Many people at MySQL still think that I abandoned MySQL when I did so. I did not. I merely had gotten frustrated with the slow pace of change in the MySQL engineering department and its resistance to transparency. Sure, over the 3 years I was at MySQL, the engineering department opened up a bit, but it was far from the ideal level of transparency I had hoped to inspire when I joined MySQL.

For almost 3 years, I had sent numerous emails to the MySQL internal email discussion lists asking the engineering and marketing departments, both headed by Zack Urlocker, to recognize the importance and necessity of major refactoring of the MySQL kernel, and the need to modularize the kernel or risk having more modular databases overtake MySQL as the key web infrastructure database. The focus was always on the short term; on keeping up with the Jones’ as far as features went, and I railed against this kind of roadmap, instead pushing the idea of breaking up the server into modules that could be blackboxed and developed independently of the kernel. My ideas were met with mostly kind responses, but nothing ever materialized as far as major refactoring efforts were concerned.

I remember Jim Winstead casually responding to one of my emails, “Congratulations, you’ve just reinvented Apache 2.0″. And, yes, Jim, that was kind of the point…

The MySQL source code base had gotten increasingly unmaintainable over the years, and key engineers were extremely resistant to changing the internals of MySQL and modernizing it. There were some good reasons for being resistant, and some poor reasons (such as “this is the way we’ve always done it”). Overall, it’s tough to question the strategy that Zack, Marten Mickos, and others had regarding the short term gains. After all, they managed to maneuver MySQL into a winning position that Sun Microsystems thought was worth one billion dollars. Because of this, it’s tough to argue with them.

Working on Drizzle since October 2008 (officially)

I’m not the kind of person which likes to wait for years to see change, and so the Drizzle project interested me because it was not concerned with backwards compatibility with MySQL, it wasn’t concerned with having a roadmap that was dependent on the whims of a few big customers, and it was very much interested in challenging the assumptions built into a 20 year-old code base. This is a project I could sink my teeth into. And I did.

Many folks have said that the only reason Drizzle is still around is because Sun continued to pay for a number of engineers to work on Drizzle as “an experiment of sorts” and that Drizzle has no customers and therefore nothing to lose and everything to gain. This was true, no doubt about it. At Sun CTO Labs, the few of us did have the ability to code on Drizzle without the pressure-cooker of product marketing and sales demands. We were lucky.

4 6 9 10 Months in Purgatory

So, around rolls April 2009. The stock market and worldwide economy had collapsed and recession was in the air. There’s one thing that is absolutely certain in recession economies: companies that have poor leadership and direction and are beholden to the interests of a large stockholder will seek an end to their misery through acquisition by a larger, stronger firm.

And Sun Microsystems was no different. JAVA stock plummeted to two dollars a share, and Jonathan Schwartz and the Sun board began shopping Sun around to the highest bidder. IBM was courted along with other tech giants. So was Oracle.

And it was with a bit of a hangover that I awoke at the MySQL conference in April 2009 to the news that Oracle had purchased Sun Microsystems. Joy. We’d just gone through 14 months of ongoing integration with Sun Microsystems and now it was going to start all over again.

Anyone who follows PlanetMySQL knows about the ensuing battle in the European Commission’s court regarding monopoly of Oracle in the database market with its acquisition of MySQL. Monty Widenius, Eben Moglen, even Richard Stallman, weighed in on the pros and cons of Oracle’s impending control over MySQL.

All the while, us Sun Microsystems employees had to hold our tongues and try to keep our jobs as Sun laid off thousands more workers while the EC battle ensued. Not fun. It was the employment equivalent of purgatory. And the time just dragged on, with many employees, including myself and the Sun Drizzle team, not having a clue as to what would happen to us. Management was completely silent about future plans. Oracle made zero attempts to outline its future strategy regarding software, and thus most software employees simply kept on doing their work not knowing if the pink slip was arriving tomorrow or not. Lots of fun that was.

Oracle Doesn’t Need Our Services — Larry Don’t Need No Stinkin’ Cloud

The acquisition finally closed and very shortly afterwards, I got a call from my boss, Lee Bieber, that Oracle wouldn’t be needing our services. Monty, Eric, and Stewart had already resigned; none of them had any desire to work for Oracle. Lee and I had decided to see what they had in mind for us. Apparently, not much.

Larry Ellison has gone on record that the whole “cloud thing” is faddish. I don’t know whether Larry understands that cloud computing and infrastructure-as-a-service, platform-as-a-service, and database-as-a-service will eventually put his beloved Oracle cash cow in its place or not. I don’t know whether Oracle is planning on embracing the cloud environments which will continue to eat up the market share of more traditional in-house environments upon which their revenue streams depend. I really don’t.

But what I do know is that Rackspace is betting that providing these services is what the future of technology will be about.

Happiness is a Warm Cloud

Our team has landed at Rackspace Cloud. I’ve now been down to San Antonio twice to meet with key individuals with whom we’ll be working closely. Rackspace is not shy about why the wanted to acquire our team. They see Drizzle as a database that will provide them an infrastructure piece that will be modular and scalable enough to meet the needs of their very diverse Cloud customers, of which there are many tens of thousands.

Rackspace recognizes that the pain points they feel with traditional MySQL cannot be solved with simple hacks and workarounds, and that to service the needs of so many customers, they will need a database server that thinks of itself as a friendly piece of their infrastructure and not the driver of its applications. Drizzle’s core principles of flexibility and focus on scalability align with the goals Rackspace Cloud has for its platform’s future.

Rackspace is also heavily invested in Cassandra, and sees integration of Drizzle and Cassandra as being a key way to add value to its platforms and therefore for its customers.

Rackspace is all about the customers, and this is a really cool thing to experience. It’s typical for companies to claim they are all about the customer — in fact, every company I’ve ever worked for has claimed this. Rackspace is the first company I’ve worked for where you actually feel this spirit, though. You can see the fanaticism of Rackers and how they view what they do always in terms of service to the customer. It’s infectious, and I’m pretty psyched to be on their team.

Anyway, that’s my story and I’m stickin’ to it. See y’all on the nets.

Eric Day: C++, or Something Like It

Sat, 06 Mar 2010 03:23:17 +0000

I’ve developed primarily in C most of my career, and recently decided to give C++ a shot as my “primary language” due to hacking on Drizzle and MySQL. The past few months I’ve read and experimented with most features C++ provides over C, including reading Scott Meyer’s excellent “Effective” series books (highly recommended). Along the way I’ve been developing a project I’ve wanted to write for a while, and I’m finding some features to be problematic. I thought I’d share these issues so others can be aware of them and perhaps I can learn better workarounds.

The project I’ve been working on uses dynamic shared object loading at runtime (using dlopen() and friends), is threaded, and has about every strict compiler warning on you can find and being treated as errors (thanks to Monty Taylor’s pandora-build project). I’m also testing on various architectures and compilers, including Linux, OpenSolaris, and OSX. I also have been trying my best to avoid any dependencies on large C++ libraries like Boost and just stick to the standard language and STL. With these requirements in mind, here are the issues I’ve run into:

Can’t Reliably Use Exceptions

My first pass relied on exceptions, but this proved problematic on some architectures as soon as custom exceptions were being throw across module boundaries. This comes down to ABI issues for some shared object formats generated by some compiler versions. While you can make it work in some environments, it’s not going to be portable. This means I’ve had to catch exceptions closer to where they are throw, requiring a lot more try/catch blocks, and not being able to take full advantage of automatic stack cleanup. This also means resorting back to the C way or handling exceptions: returning and checking return codes while generating error strings. To be completely exception safe, this means not using std::string for error returns since they can throw exceptions while building useful error messages. Not using exceptions has had a viral effect throughout the rest of the design of the code, making it look more like C. I was a bit disappointed by this, as not having to check every function’s return code was keeping the code very clean. :)

Limited Use of the STL and std::string

I was excited to take advantage of the STL, as writing things like doubly-linked lists and hash tables for every C struct was getting a bit old (I did have a set of macros I used, but they were not the most popular in some circles because of certain C-preprocessor features). When I learned more about the internals of the STL, and how it relies heavily on copying objects, my heart sunk a little. It completely makes sense in the design, it’s just not as efficient as it could be (especially coming from a place where I would optimize to reduce pointer copies in C). No worries, I just created private copy constructors/assignment operators and only used pointers to objects. This came with it’s own set of issues with pointer management and avoiding leaks if the ‘new’ operator were to fail. Once working out the memory management issues, there were still exceptions to watch out for, including figuring out all the methods that may throw (due to an internal allocation usually). This is especially annoying when doing simple std::string operations like assignment or concatenation, and having to always catch around those. With other annoyances like the reference-to-reference issues and std::unary_function having a non-virtual destuctor, I’ve ended up using a watered down set of STL algorithms and resorted to a mix of non-STL containers and custom algorithms for some things. The lack of thread safety concerns in STL containers and differences in implementations have also lead me to not use STL containers for thread communication (using a mutex for every access is not efficient).

Conclusion

For the sake of consistency, I’ve wondered if it’s worth incorporating STL components? Is it better to have a mix or none at all? This would leave only inheritance, polymorphism, member protections, namespaces, and automatic object destruction the only C++ features being used. These are still very good reasons to use C++, but I’ve found the transition to not be as productive as I had hoped. I am very curious to hear other folks thoughts on their experience with any of the issues above.

Official Drizzle Blog: Roadmap for our next Drizzle release (Cherry)

Fri, 05 Mar 2010 22:27:35 +0000

Check out Brian’s blog which details all the features we are targeting for our next release, code name Cherry. Actually in launchpad these are termed “series” and within each series we are defining two week milestones to track our progress. Currently looking to complete Cherry by the end of April.

Brian Aker: Drizzle, Cherry, Roadmap for our Next Release

Fri, 05 Mar 2010 17:24:47 +0000

Its March already? How times flies.

The last quarter has been a pretty exciting time for all of us working on Drizzle. We completed the Bell milestone back in January and began immediately plowing into Cherry, our next release. Somewhere in the last couple of months we have seen transition of all of the Drizzle developers at Sun moving into full time positions at other companies working on Drizzle, and we have seen the non-Sun developers move into other positions as well (most note'ably Padraig who went to work for Akiban).

So what is happening for the Cherry release?

Authorization with LDAP

We are adding direct support for LDAP in the next release. With this we be able to fetch authorization information from LDAP. All of this work is being done via our plugin system and builds on the work we have done for PAM and HTTP Auth. We know that companies already have identity systems and don't want to have to deal with "yet another system".

Support for the MySQL Protocol

Will you support the MySQL protocol? The answer is that we will be supporting the MySQL protocol. Today we support our own protocol but we have decided to go back and support the MiniSQL/MySQL protocol. So how are we going about doing this? There will be a plugin you can load, that we will be testing, that will allow Drizzle to speak the MySQL protocol. So you can leverage all of the drivers you use today. Please note that I said "will be testing". Up till now no one has ever written a set of unit tests or a "please crash the server" sort of test on that protocol. For us to feel good about supporting the protocol, we have put together a set of tests to do exactly this. You can run the tests on any of the M/M protocol speaking servers that are being shipped today. We have licensed the test system using the BSD license. We did this so that anyone who ships M/M servers today can feel comfortable about making use and extending what we have made available. In the Memcached world we created Memcapble in order to allow end users to self certify their Memcached servers. We believe that this same sort of process needs to happen around the M/M protocol as well.

Embedded Innodb

Stewart Smith is working on moving us off the Innodb plugin, and onto the Innodb Embedded Server. We continue to support Innodb and we believe that moving to the embedded server is the best way to support end users longterm.

Replication Logshipping

Our ongoing work to enhance replication continues. We are working on adding log shipping to the next version of Drizzle. We will also be removing RAW SQL transport of DDL operations. This allows us to easily take Google Protobuffer messages and translate our schemas into schemas for other databases. We know that data centers are not homogeneous, and that we need to play nicely with other databases.

Query Rewriting

The first version of query rewriting went into the server last week. This is the first step to us having a full query rewrite system in Drizzle. This allows storage engine vendors to easily adapt to Drizzle in the Data Analytics world, and will allow us to eventually add support for VIEWS. The first version allows for user level rewrites but the second generation that is coming will allow for parsed tree rewrites.

Schemas as real objects.

In the past a "schema", aka a "databases", was just a directory. Now they are fully supported as objects in our system and have their own caching layer. This allows for better integration with engines and provides support for engine writers to do fully transactional DDL on schemas. This is very exciting since it really means that engines are no longer locked into the "MyISAM" shoe.

Information Schema becomes the Data Dictionary

Goodbye information schema and all of its bugs! No more materialization, no hidden execution paths, and no locks on table and schema lookups. We have transitioned to a system where we now have Table Functions. These allow us to expose more data to the outside world with none of the costs that were associated with the old system. "SHOW" commands are now just query rewrites, and we have been able to further shrink the size of our parser.

The above sums up all of the big stuff going on for this release, and doesn't include the incoming work for developers or partners who haven't committed to the release. Cheery will be completed by the time we hit the O'Reilly MySQL User's Conference in April. I'll be giving a keynote there on the State of Drizzle. The conference should make for an exciting event!

Monty Taylor: Hudson Parameterized Matrix Builds

Monty Taylor — Wed, 03 Mar 2010 01:43:58 +0000

I've been making some improvements to our use of Hudson recently that have been really helpful.

The first was starting to use a set of parameterized builds. These use the Parameterized Trigger plugin, which allows you to pass parameters to triggered additional jobs when a job finishes. So using these we make a job which checks out the latest source (which should just about always succeed) and then fire off a whole host of jobs running on all of our build hosts. It has a single "build now" submit form which takes a bzr branch location as the branch to build. With this all of our developers can check their tree against the build farm before submitting the branch for merge.

That's great, but as we got more and more build hosts, we had to set up a job on each of them - and then having 4 machines which were all amd64 running Ubuntu 9.10 didn't help - it ran on all of them.

Enter PlatformLabeler

Robert Collins wrote a plugin for Hudson which checks a build slave for OS details when it connects (architecture, OS and OS Release) and adds Labels to the slave based on the values it finds. With that - instead of targetting a job to say, hades.inaugust.com, I can target that job to x64_64-Mac-10.4.1. I can easily see both platform coverage, and I can take advantage of having multiple machines with the same config, since it'll run the job on only one of them sharing that label.

The next problem with that setup is that there is not a tight binding between the parent and child jobs. You can see that the parent job triggered the child, but you can't really get a single snapshot of "how did this job work everywhere?" If you're doing this pre-merge request, it might be nice to get a URL you can refer to saying "hey! check it out, it worked"

Enter Matrix Builds.

Matrix Builds are a single job. They allow you to define how you run the job, and then you can select which hosts to run it on. You can also set up arbitrary sets of flags and hudson will build a grid. So in our case, I set up a job (shortened for brevity)

 ./config/autorun.sh

 ./configure

 make distcheck

Then I told hudson to build it both with and without the --with-debug flag to configure, and on a set of our build hosts. Now when I kick it off, I get a page with an overview of how that particular build went. Also, having one build description means I don't accidentally do something different on one host.

Matrix Builds also have a nice feature for a sentinel or canary build, which let you build first on a host where you're pretty darned sure it's going to build, and then build everywhere else. In our case, if it doesn't build on 64-bit Ubuntu 9.10, it's very unlikely to build anywhere, so we don't bother trying.

I still use the Parameterized Trigger from the Matrix build to kick of jobs for things like Valgrind and Sysbench when the build completes cleanly. Those jobs themselves can be launched independently (if you only wanted to check how valgrind is treating your new branch)

Moving forward, I would love to migrate our main set of build branches to Matrix builds, but Matrix currently lacks support for launching an individual sub-build. If it doesn't show up soon, perhaps I'll add it... ah the joy of open source!

See the Hudson link at http://hudson.drizzle.org/view/Drizzle-param/job/drizzle-param/ for more details and examples.

Brian Aker: Data Dictionary Fun in Drizzle

Tue, 02 Mar 2010 19:56:01 +0000

How often do people get indexing wrong? All the time. This is a sample table in Drizzle, I'll show how you can view indexes with the new data dictionary code.

drizzle> show create table f;
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                      |
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| f     | CREATE TABLE `f` (
   `a` int NOT NULL,
   `b` int NOT NULL,
   `c` int NOT NULL,
   PRIMARY KEY (`a`,`b`,`c`),
   KEY `c` (`c`),
   KEY `b` (`b`,`c`)
)  |
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)

drizzle> select TABLE_NAME, COLUMN_NAME, IS_INDEXED, IS_USED_IN_PRIMARY, IS_UNIQUE, IS_MULTI, IS_FIRST_IN_MULTI, INDEXES_FOUND_IN from data_dictionary.columns WHERE TABLE_NAME="f";
+------------+-------------+------------+--------------------+-----------+----------+-------------------+------------------+
| TABLE_NAME | COLUMN_NAME | IS_INDEXED | IS_USED_IN_PRIMARY | IS_UNIQUE | IS_MULTI | IS_FIRST_IN_MULTI | INDEXES_FOUND_IN |
+------------+-------------+------------+--------------------+-----------+----------+-------------------+------------------+
| f          | a           | TRUE       | TRUE               | TRUE      | TRUE     | TRUE              |                1 |
| f          | b           | TRUE       | TRUE               | TRUE      | TRUE     | TRUE              |                2 |
| f          | c           | TRUE       | TRUE               | TRUE      | TRUE     | FALSE             |                3 |
+------------+-------------+------------+--------------------+-----------+----------+-------------------+------------------+
3 rows in set (0 sec)

What is so cool about the above? Often people over index columns. They will over index the first part of keys not realizing that the first part of key can be used without creating a standalone key.

These are a few of the design goals in the above:

1) Give someone an easy view into whether a column is a part of a primary key.

2) Find out how many times you are indexes one column.

3) From a glance see what keys are first inline for a multipart key.

4) Allow a DBA to work with the database to find out problems (don't force them into tools until they need to see the state of clusters).

Anyone who has ever managed a MySQL database can pretty quickly see how valuable the above is. Doing a JOIN against a log table will quickly show you what queries are running without the advantage of indexes.

What is especially cool about the above?

1) None of that data is materialized. We generate the data through table functions which federate data from multiple engines (all using the new Storage Engine interface).

2) We open zero tables for the above queries. You can query the data dictionary without blowing through the table cache, or anyway affecting your running queries. In the past the information schema would force tables into the table cache in order to get data from them. In our world? We don't require that at all. If you want schema data we can provide it without undermining your active query sessions.

3) All of the above information is stored in our Google Protobuffer table format. We do no translations, so what you see is always what you get.

Pretty cool :)

Official Drizzle Blog: Drizzle build 1317 source tarball has been released

Tue, 02 Mar 2010 17:46:43 +0000

Drizzle source tarball based on build 1317 has been released. There was a lot of clean up and removal of dead code from Stewart as he continues to work on integrating embedded innodb. Brian added a data_dictionary schema and also did a complete rewrite of the SHOW commands.

The Drizzle download file and change log can be found here

Padraig O'Sullivan: Schema-Free Drizzle!

Tue, 02 Mar 2010 08:00:00 +0000

I came across this post from Ilya Grigorik on Hacker News yesterday and I figured I just had to implement this in Drizzle now with the new query rewriting interface that I mentioned yesterday. The awesome thing about Drizzle is that I can try all these ideas out easily by just implementing a plugin.
Any SQL statements we want to use on our schema-free constructs, we have to prefix with the string 'nos'. With that said, here is a session demonstrating this query rewriting plugin:

Your Drizzle connection id is 2
Server version: 7 Source distribution (schema-less)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

drizzle> use test;
Database changed
drizzle> nos create table widgets;
Query OK, 0 rows affected (0.06 sec)

drizzle> nos insert into widgets (id,name) values ('a', 'apple');
Query OK, 1 row affected (0.19 sec)

drizzle> nos insert into widgets (id,name,type) values ('b', 'blackberry', 'phone');
Query OK, 1 row affected (0.21 sec)

drizzle> nos select * from widgets;
+------+------------+-------+
| id   | name       | type  |
+------+------------+-------+
| a    | apple      | NULL  | 
| b    | blackberry | phone | 
+------+------------+-------+
2 rows in set (0 sec)

drizzle> nos select * from widgets where id = 'a';
+------+-------+------+
| id   | name  | type |
+------+-------+------+
| a    | apple | NULL | 
+------+-------+------+
1 row in set (0 sec)

drizzle>

The code for this is available on Launchpad (lp:~posulliv/drizzle/schema-less). I threw this together in a few hours today for fun so it is what it is.

Stewart Smith: Writing A Storage Engine for Drizzle, Part 1: Plugin basics

Mon, 01 Mar 2010 13:40:27 +0000

So, you’ve decided to write a Storage Engine for Drizzle. This is excellent news! The API is continually being improved and if you’ve worked on a Storage Engine for MySQL, you’ll notice quite a few differences in some areas.

The first step is to create a skeleton StorageEngine plugin.

You can see my skeleton embedded_innodb StorageEngine plugin in its merge request.

The important steps are:

1. Create the plugin directory

e.g. mkdir plugin/embedded_innodb

2. Create the plugin.ini file describing the plugin

create the plugin.ini file in the plugin directory (so it’s plugin/plugin_name/plugin.ini)
An example plugin.ini for embedded_innodb is.

[plugin]
title=InnoDB Storage Engine using the Embedded InnoDB library
description=Work in progress engine using libinnodb instead of including it in tree.
sources=embedded_innodb_engine.cc
headers=embedded_innodb_engine.h

This gives us a title and description, along with telling the build system what sources to compile and what headers to make sure to include in any source distribution.

3. Add plugin dependencies

Your plugin may require extra libraries on the system. For example, the embedded_innodb plugin uses the Embedded InnoDB library (libinnodb).

Other examples include the MD5 function requiring either openssl or gnutls, the gearman related plugins requiring gearman libraries, the UUID() function requiring libuuid and BlitzDB requiring Tokyo Cabinet libraries.

For embedded_innodb, pandora-build has a macro for finding libinnodb on the system. We want to run this configure check, so we create a plugin.ac file in the plugin directory (i.e. plugin/plugin_name/plugin.ac) and add the check to it.

For embedded_innodb, the plugin.ac file just contains this one line:

PANDORA_HAVE_LIBINNODB

We also want to add two things to plugin.ini; one to tell the build system only to build our plugin if libinnodb was found and the other to link our plugin with libinnodb. For embedded_innodb, it’s these two lines:

build_conditional="x${ac_cv_libinnodb}" = "xyes"
ldflags=${LTLIBINNODB}

Not too hard at all! This should look relatively familiar for those who have seen autoconf and automake in the past.

Some plugins (such as the md5 function) have a bit more custom auto-foo in plugin.ini and plugin.ac (as one of two libraries can be used). You can do pretty much anything with the plugin system, but you’re a lot more likely to keep it simple like we have here.

4. Add skeleton source code for your StorageEngine

While this will change a little bit over time (and is a little long to just paste into here), you can see what I did for embedded_innodb in the skeleton-embedded-innodb-engine tree.

5. Build!

You will need to re-run ./config/autorun.sh so the build system picks up your new plugin. When you run ./configure --help afterwards, you should see options for building with/without your new plugin.

6. Add a test

You will probably want to add a test to see that your plugin loads successfully. When your plugin is built, the test suite automatically picks up any tests you have in the plugin/plugin_name/tests directory. This is in the same format as general MySQL and Drizzle tests: tests go in a t/ directory, expected results in a r/ directory.

Since we are loading a plugin, we will also need some server options to make sure that plugin is loaded. These are stored in the rather inappropriately named test-master.opt file (that’s the test name with “-master.opt” appended to the end instead of “.test“). For the embedded_innodb plugin_load test, we have a plugin/embedded_innodb/tests/t/plugin_load-master.opt file with the following content:

--plugin_add=embedded_innodb

You can have pretty much anything in the plugin_load.test file… if you’re fancy, you’ll have a SELECT query on data_dictionary.plugins to check that the plugin really is there. Be sure to also add a r/plugin_load.result file (My preferred method is to just create an empty result file, run the test suite and examine the rejected output before renaming the .reject file to .result)

Once you’ve added your test, you can run it either by just typing “make test” (which will run the whole test suite), or you can go into the main tests/ directory and run ./test-run.pl --suite=plugin_name (which will just run the tests for your plugin).

7. Check the code in, feel good about self

and you’re done. Well… the start of a Storage Engine plugin is done :)

Padraig O'Sullivan: Query Rewriting Plugin Point for Drizzle

Mon, 01 Mar 2010 08:00:00 +0000

One of the first tasks in my new position at Akiban was to create a plugin point within Drizzle for query rewriting.
The first decision to make was where to insert a plugin point for a query rewriter. The parsed representation of a query would seem like a natural thing to pass to a query rewriter plugin since the plugin would not have to implement its own parser then. However, the parsed representation of a query in Drizzle is not the easiest in the world to deal with right now so passing this to a plugin would make developing a rewriting plugin quite difficult. Thus, I made the decision to create the plugin point before parsing occurs.
This means that if a plugin developer wants to do some complex rewriting, they may need to parse the query in their plugin. It may not be ideal but it does make the plugin API for query rewriting quite simple and opens up a lot of interesting opportunities.
Following the lead of other plugin interfaces such as the replication API developed by Jay, I wanted to keep it as simple and easy to understand as possible. With that in mind, here is the entire API for a query rewriting plugin:

Thus, all a plugin developer needs to do is implement the rewrite() function within their plugin. The query is passed by reference as a std::string so a plugin can do whatever it likes to this string and this string will then be passed to the parser in the Drizzle core kernel for parsing.
This interface opens up a lot of possibilties for interesting plugins. For example, one could develop a plugin to analyze a query for common SQL injection patterns or develop a plugin to rewrite a query based on a set of rules. I would be really interested in hearing other ideas people reading this have for plugins using this interface?

Toru Maesaka: DATE type under the hood in Drizzle/MySQL

Mon, 01 Mar 2010 07:24:39 +0000

Learned something new from my own bug in BlitzDB today. The problem was that writing a DATE column index would always return a duplicate key error (regardless of what I feed it). There are two suspicious candidates that can cause this.

Comparison Function has a defect.
Key Generator has a defect.

The latter suspect was going to be tricky if it was true since BlitzDB currently uses Drizzle’s native “field packer” (except for VARCHAR) inherited from MySQL. This would mean that Drizzle’s field system has a bug in it which was somewhat difficult to believe. Furthermore, you should always blame yourself before you start suspecting other people’s code. So, I decided to look into the comparison function which was completely written by me. Turned out that’s where the bug was.

Comparison Function

Allow me to quickly clarify what I mean by “comparison function” in this context. TC’s B+Tree API has an interface that allows you to provide your own comparison function for all operations that involves traversing.

bool tcbdbsetcmpfunc(TCBDB *bdb, TCCMP cmp, void *cmpop);

What BlitzDB’s comparison function callback does is, it looks at the data type of the values to be compared and performs appropriate processing on the values then compares them. You can also look at it as a long switch statement. For those that are interested, this code is in blitzcmp.cc (blitz_keycmp_cb).

DATE under the hood

After inspecting the “type number” with GDB and looking at the corresponding ha_base_keytype enum, it turns out that the DATE type is internally represented as an unsigned 3 byte integer (HA_KEYTYPE_UINT24). This was pleasant to discover since I’ve been wondering what a 3 byte integer is still used for in Drizzle. The problem I had was that I didn’t take this type into account in the comparator and it also showed how silly I am since the answer was always there.

10.5. Data Type Storage Requirements

Now, the question is should it be kept this way? Respect alignment or reduce total I/O and space by keeping it this way? This should hopefully be a fun discussion to have in the Drizzle community :)

P.S. My two cents is that it should respect alignment since folks that seek performance should have most of their data on memory. Respecting alignment in this environment should make some difference. Although, I can only say this after benchmarking it of course.

Pythian Group: Log Buffer #180: a Carnival of the Vanities for DBAs

Fri, 26 Feb 2010 18:04:21 +0000

Hello and welcome to Log Buffer #180. Time’s a-wastin’, so let’s go!

Oracle

There was so much Oracle stuff this week that I’ve decided to cram a little more of it into Log Buffer by providing a little less context than usual.

Jonathan Lewis shares an explication of aliases: “I was asked the following question recently: ‘Does the use of table aliases affect performance?’ To which the best answer is probably ‘Yes, though in general you probably won’t notice the difference and there are reasons more imporant [sic] than performance for using table aliases.’”

Doug Burns continues his most recent series: Statistics on Partitioned Tables – Part 2, and Statistics on Partitioned Tables – Part 3.

Charles Schultz demonstrates how VPD + bad ANYDATA practices can really bite: “The point of my blog was that using CAST can really screw up your data. Oracle Support is filing a bug on this behavior, as it looks like an overflow problem.”

Pythian’s Gleb Otochkin begins a series on Oracle GoldenGate installation.

Guy Harrison provides a thorough introduction and recommendations on memory management for Oracle databases on VMWare ESX.

Robert Vollman returns to blogging and offers his 10-point plan on improving your SQL queries.

Jared Still sheds some light on a cool but unknown RMAN feature.

Richard Foote knocks holes in another myth: “One of the great myths in Oracle is that bitmap indexes are only suitable and should only be used with columns that have so-called low cardinality (few distinct) values.”

Alexander Kornbrust shares a link to a really good whitepaper about “Hacking Oracle from the Web” by Sumit Siddarth.

Eddie Awad shares a link to a SQL injection prevention cheat sheet.

Charles Hooper answers the question, What is the meaning of the %CPU column in an explain plan?.

Meanwhile, Harald van Breederode does the same for this one: Why does the size of my ORACLE_HOME increase?

SQL Server

Thomas LaRock gives an recap of MS’s 2010 MVP Summit. Quotable take-away: “If I had to compare SQL 2008 R2 to SQL Server 4.0, I would say the difference is the same as comparing an F1 race car to a Chevy Vega.”

Half a world away, there is the SQLSocial Event – London March 16th, as advertised by Simon Sabin.

Simon also shares a script to get indexes and their included columns, beginning, “I get increasingly frustrated with the lack of visibility of included columns in management studio and from the system stored procedures sp_… This is a query that returns all indexes and there key and include columns[.]”

Andy Leonard throws us another nourishing SSIS snack: conditional split outputs.

Here’s Rob Farley with a book review of an oldie but a goodie: Inside SQL 2005 Query Tuning and Optimization, by Kalen Delaney et al. “If you spend any time tuning SQL Server databases, then this book will feel much thicker than it really is, and you’ll be finding useful information on just about every page.”

Thomas LaRock, meanwhile, writes that SQL Server 2008 Query Performance Tuning Distilled is
a good way to start your day. “Each morning, while I wait for my desktop to boot, I pick up their book, turn to any page, and just start reading.”

MySQL

Sticking with the theme a little longer, here is Baron Schwartz with a review of Understanding MySQL Internals by Sasha Pachev. “I should have read this book a long time ago, and it’s my loss that I didn’t. . . . Overall, this book is easily a high 4 stars on a scale of 5, and again, anyone seriously using MySQL should have it.”

Baron also shares a link to Oracle guy Cary Millsap’s Thinking Clearly about Performance paper.

Brian “Krow” Aker starts an extensive conversation with his post, Protocols, The GPL, Influences from MySQL. His thesis, “MySQL was the company that had the most influence on how companies and investors viewed the GPL.”

Paul Vallée of Pythian responds with his ideas on product management, effective developers, and the future of MySQL. “ . . . the future of MySQL, Drizzle, Monty Program, the Percona fork, etc.” to be more precise.

Colin Charles provides news of what’s been happening recently in MariaDB #1.

Mohammad Lahlouh wonders, can I use latin1 to store utf8 data? and gets several answers from his readers.

He might have asked Ronald Bradford, who knows this stuff. Here is his post on migrating MySQL latin1 to utf8 – character set options.

Pursuing a similar matter (collations), Roland Bouman opines, the best stored routine is the one you don’t write.

PostgreSQL

Baron Schwartz again! He announces, mk-query-digest now supports Postgres logs.

David Fetter says, part(ition)ing is such sweet sorrow. “There are excellent references on partitioning tables that depend on one table, but what happens when you need to partition the referenced table? Let’s find out!”

Bruce Momjian is here with news on the Python driver confusion.

Jon Jensen of End Point’s Blog posts a HOWTO on PostgreSQL EC2/EBS/RAID 0 snapshot backup.

NoSQL, Etc.

Chen Shapira has been at the compass and protractor, mapping the NoSQL space and returns from terra incognita unscathed.

Ronald Bradford has been getting started with Cassandra, one of the outposts on Chen’s map, and shares his steps.

Arnie Rowland says, “Mark your calendar! Portland SQLSaturday/CodeCamp/Barcamp 2010 is scheduled for May 22, 2010, at the University of Portland campus. . . . Portland SQLSaturday is encouraging presentations related to interoperability of any of the SQL platforms, including T-SQL (SQL Server), PostgreSQL, MySQL, and PL-SQL. Abstracts for Platform specific sessions are also encouraged.”

Okay, that is all for this edition. You guys are running me ragged! Fortunately, Gary Myers picks it up next week on his Sydney Oracle Lab. Till then!

Padraig O'Sullivan: Installing SystemTap on Ubuntu

Fri, 26 Feb 2010 08:00:00 +0000

I'm presenting at the MySQL user's conference this year and one of my talks is on using SystemTap and DTrace with MySQL and Drizzle. I'm also doing a tutorial with Jay Pipes on developing replication plugins for Drizzle and that should be a lot of fun.
I wanted to write some posts before the conference that I can reference within my talk which detail how to install SystemTap and configure Drizzle and MySQL for use with SystemTap. Thus, this post is on how to install SystemTap on Ubuntu while my next post will go in to details about how to configure MySQL and Drizzle for use with SystemTap.
Before starting, its worth noting that this post is specific to Ubuntu 9.10. The procedure to follow may be different on other versions so its worth keeping that in my mind. The first thing we do is install systemtap and some associated packages which will be needed by Drizzle and MySQL:

$ sudo apt-get install systemtap
$ sudo apt-get install systemtap-sdt-dev

Now, being used to Ubuntu, you would think you are good to go now. Unfortunately, attempting to run SystemTap will probably give you the following error:

$ stap -e 'probe kernel.function("sys_open") {log("hello world") exit()}'
semantic error: libdwfl failure (missing x86_64 kernel/module debuginfo under
'/lib/modules/2.6.31-19-generic/build'): No such file or directory while resolving probe point
kernel.function("sys_open")
semantic error: no probes found
Pass 2: analysis failed.  Try again with another '--vp 01' option.
$

The above error occurs because SystemTap needs to have a debug version of the kernel available. Unfortunately, installing the debug information for a kernel on ubuntu is not a trivial operation to perform. In fact, there is a bug on Launchpad about this issue. Thus, we will build a kernel debug package from source ourselves. This can be done as follows:

$ cd $HOME
$ sudo apt-get install dpkg-dev debhelper gawk
$ mkdir tmp
$ cd tmp
$ sudo apt-get build-dep --no-install-recommends linux-image-$(uname -r)
$ apt-get source linux-image-$(uname -r)
$ cd linux-2.6.31 (this is currently the kernel version of 9.10)
$ fakeroot debian/rules clean
$ AUTOBUILD=1 fakeroot debian/rules binary-generic skipdbg=false
$ sudo dpkg -i ../linux-image-debug-2.6.31-19-generic_2.6.31-19.56_amd64.ddeb

This builds a debug image of the kernel and so will take quite a while. Once we have the above completed, we can try running our hello world example with SystemTap again. In order to get some output, you should open or create some file on the system in another terminal window. In this example, I backgrounded the stap process and created a file:

$ sudo stap -e 'probe kernel.function("sys_open") {log("hello world") exit()}' &
[1] 951
$ touch /tmp/padraig
$ hello world
$ [1]+ Done

Installing SystemTap on CentOS is significantly easier since it is primarily developed by Red Hat. A good article on how to install it on CentOS is available here.
In my next post on the topic, I'll explain how to configure MySQL and Drizzle for SystemTap and give some simple examples of using SystemTap with them.

Stephen O'Grady: Data vs Dual Licensing: Which Will Make More Money?

Thu, 25 Feb 2010 22:00:00 +0000

“By 2012, at least 70% of the revenue from commercial OSS will come from vendor-centric projects with dual-license business models.

80% probability. This is may true today but the lack of revenue among broader market OSS products compared to Linux isn’t large enough yet to make this one a done deal. What is clear is that the overwhelming majority of ‘commercial oss’ efforts are based on a dual license model – vendor prefer the ‘open core’ moniker because it sounds more OSS friendly but its essentially the same thing

.”
- Mark Driver, Gartner, Open Source Predictions for 2010

This prediction from Gartner’s Mark Driver confused me, I’ll admit, when I first read it. Baffled me, actually. Looking at the market, it seemed clear to me that the practice of dual licensing was, if anything, in decline. I couldn’t see how we could look at the same market and come to such different conclusions. My view was similar to Brian Aker’s (as is Cloudera’s Mike Olson’s, notably), most recently of MySQL/Sun:

When MySQL pushed dual licensing, investors looked for this hook in every business model. I remember standing outside of a conference room in SF a couple of years ago and talking to one of the Mozilla Foundation people. Their question to me was “Is the nonsense over dual licensing being the future over yet?”. The fact is, there are few, and growing fewer, opportunities to make money on dual licensing. Dual licensing is one of the areas where open source can often commoditize other open source right out of the market. The dearth of companies following in MySQL’s dual licensing footsteps to riches, belabors the point of how niche this solution was.

Even for MySQL, long the standard bearer for the approach, the logistics of dual licensing were and are becoming increasingly problematic over time:

For smaller firms, the primary limitation [of dual licensing] is the development. Unlike non-dual licensed projects which need only concern themselves with the quality and provenance of code contributions from external parties, dual-license vendors need also consider the question of copyright ownership. Because dual licensing depends on ownership of copyright for the entirety of the asset in question, third parties must either assign or be willing to jointly hold the copyright for any potential contributions. Early in a project’s lifecycle, this is a minor concern because the project owner likely employs most of those qualified to improve it. As a project matures and becomes more popular, however, this is a more pressing issue. First, because it acts to inhibit community participation (see slide 18 of this deck produced by Monty), but second – and more problematically – it means that third parties can, in practical terms, offer a more complete product.

Jeremy Zawodny made reference to the practical implications of the dual license in a post from December of last year entitled “The New MySQL Landscape.” In it, he made the assertion that “You can get a ‘better’ MySQL than the one Sun/MySQL gives you today. For free.” This is the cost of the dual licensing model: in return for the right to exclusively relicense the code, you forfeit a.) the right to amortize your development costs across a wide body of contributors, and b.) the right to uniformly integrate the best patches/fixes/etc that are made available under the original license because you cannot always acquire the copyright.

This doesn’t mean that dual licensing is a uniformly bad strategy, but it does imply that it has costs, and that those costs escalate over time. This situation is the inevitable result of the dual license model over time as applied to a successful project. For those looking for perspective from a MySQL and Drizzle developer, I’d recommend reading Brian Aker’s piece here.

Even setting aside the disincentives to pursuing a dual licensing strategy, the basic math of the 70% argument didn’t work for me. Even at MySQL, remember, a fraction of the revenue is derived from the issuance of dual licenses. And even if we assumed, for the sake of argument, that the entire revenue stream was the product of dual licensing, that still wouldn’t be enough to meet the 70% projection. Not nearly so.

As Driver notes when he says “the lack of revenue among broader market OSS products compared to Linux isn’t large enough yet.” Linux, it seems clear, is the largest single open source commercial ecosystem, and due to the lack of centralized copyright ownership, it cannot be dual licensed by anyone. What Driver is saying, in other words, is that the open source commercial ecosystem has to be big enough that Linux doesn’t comprise more than 30% of it.

Consider the following back of the envelope calculations. Red Hat’s revenues in the year ending 2009 were $652 million and change. We know that, for copyright and licensing reasons, none of that money may derive from dual licensing revenues. If we assumed, counterfactually, that Red Hat represented all of the non-dual license revenue of the market – the leftover 30%, my math says that the total revenue picture would be around $2.17B. Meaning that we need a little more than two Red Hat’s more worth of revenue to emerge from dual licensees like MySQL.

Personally, I’m skeptical that that would happen, even with the hybrid source trend.

Part of the problem is, I believe, semantics. Driver seems to be conflating what is sometimes referred to as “open core” licensing with dual licensing. Personally, I believe they are distinct. The former tends to refer to varying combinations of open source and proprietary codebases, while the latter is more generally used in conjunction with copyright mechanisms as they apply to a single open source codebase. This view is supported by my analyst colleagues over at the 451 Group.

Were we to grant Driver the more expansive definition of dual licensing, however, I still think that figure is wrong. Based on the conversations we’re having with vendors in the space, it seems more likely that revenue growth and expansion will come not from quote unquote dual licensing, but derived intelligence from gathered data and telemetry.

Judging by the almost universally poor conversion metrics – that is, the number of users of a given open source tool that are converted to paying customers – it seems reasonable to assert that there are ongoing and systemic issues in the commercialization of open source software. Hence the proliferation of alternative revenue models such as dual licensing, open source and even SaaS. It is far from clear, however, that these models satisfactorily align customer and vendor interests such that conversion percentage will elevate to levels where they are competitive with proprietary software.

At the end of the day, open source customers are generally paying for one or more of a.) break/fix/integration/support/etc services that they hope not to need, b.) withheld features that they need to pay to gain access to, or c.) the right to not observe the terms and conditions of the original license. The relative distribution of revenue within this set is skewed by the size and scope of the Linux community towards A, with B being the raison d’etre for open core and C the same for dual licensing.

But what if open source vendors could leverage their primary strength – distribution – more effectively as a direct revenue stream? I’ve been predicting for three years or so that they would do just that, via data aggregation and analytics. The alignment of customer and vendor goals is better in this model than in virtually any other. The simplest example of this model outside of open source is Google, who provides users with search at no cost, receiving in return massive volumes of data which they monetize both directly (contextual ad placement) and indirectly (algorithmic improvement, machine learning, intelligence for product planning strategy, etc). Why couldn’t software vendors employ a similar model, trading free software for user generated telemetry data? The answer is, they can. SpiceWorks, for one, is doing just that now, quite successfully, albeit not with open source software.

The strength of open source is in its ubiquity, and the volume it commands ensures that the telemetry returned would have substantial – potentially immense, depending on the project – value. Importantly, however, the value lies in the aggregation. A single user’s telemetry is likely to be relatively uninteresting. A hundred users’ telemetry, more interesting. A thousand users’, that much more so, and so on. Users, therefore, wouldn’t be surrendering anything of material value to a would-be vendor in the transaction. Better, analysis of the aggegrate could have enormous value to customers. How is my infrastructure performing relative to similar environments? What are the types of conditions that indicate a potential problem? What differentiates my architecture from the Top 10 best performing? These are answerable questions…if you have a big enough dataset. Most customers would not have that; an open source software provider aggregating and analyzing their combined telemetry would.

Privacy and trust will certainly be concerns, but if the right data is offered as an incentive and the appropriate anonymization assured, those can be addressed for most customers. And for those that remain concerned, they should have the ability to opt out understanding that they will in turn have no access to the resulting analytics, and might therefore be at a disadvantage relative to their competitors who were using the intelligence.

This direction seems nothing less than inevitable for me, and so it is no surprise that we’re beginning to see (and help) a variety of open source vendors move in this direction. Free and open source data has a bright future regardless of the revenue model, but as we see successful projects better leverage their traction via analytics, the result should be a win for ecosystems and customers alike.

Whether you believe as I do, however, that the money is ultimately going to come from data more than code, it seems clear to me that it is not going from what is commonly considered to be dual licensing. Because while it is not true that I am an enemy of that particular approach, I do believe it’s in decline. Not least because it’s poorly aligned with customers needs.

Unlike data.

Pythian Group: Product management, effective developers, and the future of MySQL

Tue, 23 Feb 2010 19:38:13 +0000

I am writing because Sheeri sent me a note about a blog post written by Brian Aker, where Brian concludes, quite correctly, that (in Sheeri’s words not Brian’s)

MySQL is now just a branch (the official branch, but a branch nonetheless, and a bunch of trademark (logo) and copyright (docs) ownerships).

This is exactly true. No denying it. Why bother. It’s true. It’s also true for the vast majority of open-source projects, by the way.

I replied to Sheeri:

There's no denying that. The product direction will be set by whoever sets the best product management strategy backed by the most effective development effort. And there can be multiple winners. -Paul

Well, this is the kind of quality output I can be relied on. It might not fit on twitter, but it’s not blogworthy. Sheeri’s word of encouragement:

See, now that would be a nice blog post with a positive outlook that both Oracle Corp and MySQL community would agree and be happy with, because both Oracle Corp and the MySQL community feel they can set "the best product management strategy backed by the most effective development effort." -Sheeri

God. My reply was embarassing but maybe I should include it for humour value:

Go for it. Its a tweet for me at the most. No time to expand that thinking into a blog worthy of the blog today. -Paul

and then, right away,

ah censored it i'll do it. it'll be short. -paul

You are now reading the result of this very modest effort.

Here’s the future of MySQL, Drizzle, Monty Program, the Percona fork, etc.

The best product management strategies… should we be lightweight for the web, plug-in oriented like Drizzle? Should we follow Monty’s giant-killing roadmap? Should we focus on performance-oriented patches? The best product management strategies will win.

They can’t win alone. Will they be backed by appropriate investments from effective developers? Effective developers are the ones who convert winning product management strategies into working products. You can’t get there without them and I’ve seen lots of great strategies fail that test (including my own actually).

And there can be more than one winner.

It’s doesn’t matter what roadmap Oracle plots for MySQL. If it’s not the roadmap the community wants, it will lose ground and open an opportunity for another fork. If it is, however, (and NEVER, NEVER underestimate Oracle’s product management because it is outstanding and a big component of their historical success), if it is, however, Oracle can win the long-term hearts and minds, because they can resource quality developers in a way that I don’t think any of the competing forks are capitalized to do (yet.)

Either way, it’s going to be fun to watch.

And more than one player can win.

And regardless, the community wins. Big time.

Brian Aker: Drizzle, Licensing, Having Honest Conversations with your Community

Tue, 23 Feb 2010 17:27:50 +0000

I pulled this from a quote on yesterday's Slashdot story about MySQL Licensing where the author of the quote mentions Drizzle's licensing terms:

"you require the code to be under BSD"

This is actually a myth, we don't.

If you look through the Drizzle codebase you will note that very few files have BSD headers, and all that do?

They are a part of new systems that have been written since the fork of the project, and not all of these are BSD.

Why is this?

A large part of Drizzle is derived work from MySQL, and in all cases there we inherited its GPLv2 license on files. Bug fixes and code refactoring projects all fall under the umbrella of "derived work". In all of those cases the work was made under the GPL but no copyright assignment ever occurred. To understand ownership there you would need to look at the revision history to the code. We have never made any effort to track anything more then "where did the code come from".

I am a big opponent of copyright assignment in open source. How do you have an honest conversation with someone where you say "yes, I will need the work you did for free, to be assigned over to me, so that I can make money on it"?

Or better put:

"Here is $100 for the code you did, would you like some trinkets and beads to go along with that?"

We never did copyright assignment for Drizzle, and in no other project I have run personally have I ever done it. It is not a conversation I can have with a straight face with anyone. If you need commercial rights to your work and you want to do a true "quid pro quo"? License your code under the BSD license in the first place, that way both you and the contributor are on equal footing.

Does this mean we are a free for all when it comes to code? No, we track ownership. We know where every single line of code came from thanks to bazaar. If someone wanted to "poison" the codebase we would know exactly who that was. We would point the copyright owner to the offender, remove the code, and help with the prosecution of said individual. We have an active discussion going on at the moment about the future of our copyright headers, and one option we are considering is just replacing the copyright owner notice with a "Please see revision history for complete ownership information".

Are all of the files in Drizzle GPL? No, we do have some which are BSD.

I am a big fan of the BSD license and I typically suggest to developers that they use it for certain projects. As a license it links well with other code, and by establishing new plugins as BSD the original author can pull from any bug fixes that are made to the code. As an example Patrick Galbraith established the built in drivers that allows Drizzle to speak with Memcached. He has pulled code from the Drizzle driver and used that for his MySQL drivers. If those drivers had been done as GPL he would have not been able to pull code back into the MySQL Memcached drivers (which are BSD).

The GPL works just fine as the license for the kernel of Drizzle. I don't see that changing. In discussions with the other core authors, there has never been a push to "rewrite" the entire internals just to change the license. It is more work then it is worth, and it is not needed. The micro-kernel design means that going forward most of the work is pushed out to the libraries that link us to other systems. Work in the kernel is all about making those interfaces robust and creating more opportunity for plugin writers.

A final note on licensing and direction.

We never had any ambition to aim Drizzle at the deeply embedded market. Taking Drizzle, compiling it into a library, and linking it against another application is not a goal that the core team has ever had. If we had ever wanted to go into the world of deeply embedded databases we would have needed to have done code assignment or switched the entire license of the program.

In the deeply embedded world SQLite reigns supreme. SQLite does an awesome job in that space, and we see zero reasons to go head to head with it. If I needed a deeply embedded database I would just use SQLite, I wouldn't bother to write a new one.

Drizzle's core will stay GPL, and we continue to take contributions without code assignment. If you are a programmer I believe you can appreciate the merits of why we do this.

If you are a developer and you find yourself in the peculiar position of being asked to sign over your work? I would encourage you to take a hard look at why this is being asked, and see how comfortable you are with the value you are getting in return for your work.

Pythian Group: Announcing: Monday night community dinner at Pedro’s during the O’Reilly MySQL Conference & Expo

Mon, 22 Feb 2010 17:12:43 +0000

Just the facts:
What: MySQL user community dinner
Who: me, you, and many MySQL community members
When: Monday, April 12th – Meet at 6:30 at the Hyatt Santa Clara or at 7 pm at the restaurant
Where: Pedro’s Restaurant and Cantina – 3935 Freedom Circle, Santa Clara, CA 95054
How: Comment on this blog post to add your name to the list of probable attendees

I was sad that last year there was no community dinner, and I missed the one the year before when Jonathan Schwartz and Rich Green made an appearance. This year I am determined not to miss it, and so I am calling for a community (pay-your-own-way) dinner on Monday, April 12th, at Pedro’s – a Mexican restaurant that has vegetarian and vegan options. I think Monday is a better time because many folks arrive Sunday evening, or even Monday morning (there are tutorials on Monday, but not everyone attends).

Pedro’s can handle large groups of people, but we would like to have a vague idea of how many people are attending — while you are not required to RSVP, we would like to make an accurate reservation at Pedro’s….In 2008, there was a wiki page with a list of attendees, and I was disappointed because there were so many people on that list I wanted to see.

Meet us at 6:30 pm on Monday in the lobby of the Hyatt Santa Clara, or at 7 pm at Pedro’s. If you want to come later, just show up at Pedro’s whenever you can.

Since commenting on this blog does not require registration (as the wiki does), I invite folks to comment on this blog post and I’ll add you to the list of attendees:

Sheeri K. Cabral (The Pythian Group)
Paul Vallee (The Pythian Group)
Rob Hamel (The Pythian Group)
Giuseppe Maxia (Sun)
Brian Aker (Drizzle)
Konstantin Osipov (Sun)
Mark Callaghan (Facebook) (will arrive later)
Wagner Bianchi (EAC Software, Brazil)
Roland Bouman (BI wizard)
Bill Karwin (Karwin Software Solutions)
Maxim Volkov (OpenCandy)
Brian Moon (DealNews) – note: Monday Apr 12th is Brian’s birthday!
Rob Peck (DealNews)
Arjen Lentz (OpenQuery)
Vadim Tkachenko (Percona)
Rohit Nadhani (WebYog)
Paul McCullagh (PrimeBase)
Monty Widenius (Monty Program)
Sergei Golubchik (Monty Program)
Kristian Nielsen (Monty Program)
Henrik Ingo (Monty Program)
Nick Westerlund

(Tuenti)

Sergey Petrunya (Monty Program)

Baron Schwartz (Percona) (Tentative yes)

Baron Schwartz: A review of Understanding MySQL Internals by Sasha Pachev

Fri, 19 Feb 2010 23:13:46 +0000

Understanding MySQL Internals

Understanding MySQL Internals. By Sasha Pachev, O’Reilly 2007. Page count: about 227 pages. (Here’s a link to the publisher’s site).

I should have read this book a long time ago, and it’s my loss that I didn’t. Although the title makes it sound like it should only benefit those who’ll be changing the MySQL server’s own code, that’s not true. To the contrary, at least parts of this book should be required reading for DBAs and developers who use MySQL, after they gain a moderate level of familiarity with how to use the server.

The book does indeed start off with a few chapters on the source code, how to work with it, and the core structures that make up the MySQL server at the source code level. However, even these topics hold value for users such as DBAs. If you don’t know how the server really works, you are lost when you are faced with a problem or asked to understand some behavior. Peter Zaitsev refers to this as “X-Ray Vision,” something a good DBA or consultant needs. I think the first few chapters of this book are a great way to develop that X-Ray Vision into MySQL.

The next couple of chapters are on the client/server API, configuration variables, and thread-based architecture. Although the first is probably not a core competency for DBAs, the others are. I sure wish I’d had the client/server protocol chapter handy when I was working with the protocol, though. It is variously more useful than, and a good supplement to, the internals document on the MySQL wiki.

The following chapters cover the storage engine interface, the server-level lock manager and how it interacts with the storage-engine locking, and the parser and optimizer. These are absolutely core knowledge for DBAs and developers in my opinion. The server/storage-engine division is one of the things that makes MySQL different from other databases, and is mandatory to understand deeply. This applies equally well to the rest of the chapters, which cover the parser, optimizer, various storage engines (as opposed to just the server’s interface to them), transactions, and replication. Mandatory, every one.

What’s missing? I found that the book is kind of funny in one major way. It doesn’t talk much about MySQL 5.0. Instead, it delves into 4.x and 5.1. Most of the new features in 5.0 are not mentioned at all. Stored procedures, the INFORMATION_SCHEMA, triggers, and so on are absent, as are most discussions of changes to the optimizer and so forth. Some 5.0 topics are covered: index merge, for example. But by and large, there’s not a lot of coverage here. The 5.1-specific topics are those such as the new storage engine API and row-based binary logging. Events are not covered, nor are changes to other types of logging. Honestly, I feel this is appropriate in a book this size; the stuff that hasn’t changed since 4.x days is more important to understand.

There’s little discussion of exactly how certain features work, such as the different sorting algorithms. But that’s OK. These are covered pretty well by the MySQL manual, and even by my own book High Performance MySQL 2nd Edition. I think some other major topics might be missing, but I can’t quite think of them now.

The book is really well written. I expected it to be dry but it’s not at all. It’s actually engaging and interesting. I also found a curious thing happening as I read: I became more aware of how much legacy cruft there is inside MySQL, and how much that has contributed to various shortcomings. This made me actually feel sad, and made me yearn for the bright pure clean exciting vision that Drizzle strives towards. But at the same time I kind of felt nostalgic, kind of fell a little more in love with MySQL, for its strengths and for the countless hours of work and the really monumental genius that it embodies, warts and all. It was quite a cognitive dissonance experience, to tell the truth!

For those who have any inclination to reading it, I’d say: do it. It’ll benefit you a lot more than you think. And if possible, do it with a copy of the MySQL source code available and actually take the time to look through it and explore the things Sasha suggests. I read this book on an airplane, far from a computer, and I need to read it again and look at source code as I do so. I am positive I’ll get more than twice as much benefit from this second reading than I did from the first. I say that because I have a shin-deep exposure to the MySQL source code myself, so I knew just enough about it to recognize that I really would get a lot more from going and looking at the code Sasha cross-referenced. It was a bit like speaking Spanish without a dictionary, but having had a few weeks of intensive instruction ten years ago. I remembered some things well, other things just hazily.

Overall, this book is easily a high 4 stars on a scale of 5, and again, anyone seriously using MySQL should have it.

A review of MySQL Administrator’s Bible MySQL Admi
A review of Get it Done with MySQL 5&6 by Peter Brawley and Arthur Fuller Get it Don
A review of Pentaho Solutions by Roland Bouman and Jos van Dongen Pentaho So

Related posts brought to you by Yet Another Related Posts Plugin.

Official Drizzle Blog: Drizzle build 1304 source tarball has been released

Fri, 19 Feb 2010 22:48:02 +0000

Drizzle source tarball based on build 1304 has been released. This marks the beginning of our Cherry series of milestones. Most notably in this release is the beginning changes to replace Information Schema with table functions. See Brian’s blog for more details.

The Drizzle download file and change log can be found here

Brian Aker: Drizzle, Look at the last row...

Fri, 19 Feb 2010 19:44:18 +0000

drizzle> select * from plugins WHERE PLUGIN_TYPE="storageengine";
+--------------------+---------------+-----------+-------------+
| PLUGIN_NAME        | PLUGIN_TYPE   | IS_ACTIVE | MODULE_NAME |
+--------------------+---------------+-----------+-------------+
| ARCHIVE            | StorageEngine | TRUE      |             |
| BLACKHOLE          | StorageEngine | TRUE      |             |
| CSV                | StorageEngine | TRUE      |             |
| FunctionEngine     | StorageEngine | TRUE      |             |
| INFORMATION_ENGINE | StorageEngine | TRUE      |             |
| InnoDB             | StorageEngine | TRUE      |             |
| MEMORY             | StorageEngine | TRUE      |             |
| MyISAM             | StorageEngine | TRUE      |             |
| schema             | StorageEngine | TRUE      |             |
+--------------------+---------------+-----------+-------------+
9 rows in set (0 sec)

The payoff for making Drizzle a plugin based architecture? Development goes much faster.

Brian Aker: Gearman 0.12 Released

Fri, 19 Feb 2010 16:36:32 +0000

The highlights:

Fixed bug where memory loss occured if data was too large.

Added gearman_strerror().

Fixed bug where setting an option off in mass would not trip any triggers on the option (for both worker and client).

Options that are internal can no longer be set by external callers.

Deprecated gearman_client_set_event_watch_fn() and gearman_worker_set_event_watch_fn.

gearman_job_handle() and gearman_job_function_name() now return const char* pointers

gearman_worker_unregister now returns GEARMAN_NO_REGISTERED_FUNCTION if the function does not exist (or is being removed)

Added gearman_worker_function_exist()

Trying to send too large of a piece of data will result in GEARMAN_ARGUMENT_TOO_LARGE.

Added support for gearmand command client to daemonize and create a pid file.

Fixed job handle comparison bug with WORK_FAIL responses.

Fixed disable assert configure option.

TokyoCabinet support added.

Updates to Drizzle support.

Updates to MySQL support.

Build system updates.

And a lot more then what I will write here...

0.12 was a cleanup release for the most part. Numerous bugs were fixed and we cleaned up the API to make it a bit safer for the future.

For the next release the plan is to put more effort into the server, which will including additional logging options and updates to the server code around its internals (mainly C++ work over C).

You can download 0.12 from here:
https://launchpad.net/gearmand

Mark Schoonover: Thoughts on Oracle's Purchase of Sun

Mark Schoonover — Fri, 19 Feb 2010 04:52:25 +0000

With a first day announcement at the MySQL 2009 User Conference, Oracle had purchased Sun! Reading the tweets and blogs, it's understandable that some think it's the end for MySQL, some think it's not - I think it could be both.

MySQL itself is going through a second corporate purchase inside of 18 months. When Sun purchased MySQL, some of the internal talent left. Just recently we saw the exit of Marten, and just before that Monty. What made MySQL what it was in 2008 is the people behind the database, from corporate to the community.

MySQL the database server itself is safe since it's GPLed. Anyone can have access to the code and create their own distribution. We've already seen this happen within the community, Jeremy Cole, Percona, OurDelta to name a few. The main concern that could slow or stop MySQL progress is all the programming talent leaving en masse. Put this together with the departure of all the original corporate management and MySQL is no longer MySQL the company we remember. The part we don't know for sure is Oracle's plans for MySQL. Just seems to be getting worse.

Drizzle could also be impacted since Brian Aker is a Sun employee, so is Jay Pipes, and I'm sure there are others. Some companies support Open Source Software by paying their employees to work on these projects. We don't know if Oracle will allow this to continue.

Who could be impacted by Oracle's purchase? That huge computer company in Redmond. For years, their database team has had access to the operating system source code. Gives a huge advantage to support and performance. Oracle not only gains the source code to Solaris, but also their hardware too. No company in my 22 years in IT has had access to everything - hardware, operating system and the database. It's going to be a wild ride....

The other company I think could be greatly impacted is Kickfire. Now that Oracle has all that Sun hardware talent on tap, they could also design a database appliance as well.

It's not all gloom and doom.

I see great opportunity for 3rd party companies that support MySQL. Some of these companies have former MySQL employees on staff too, and have already started contributing code to MySQL with their own patches and enhancements. This could be an opportunity for a consortium of companies to come together and support MySQL. Something like the Eclipse Foundation might work. I could also see David, Monty and Marten coming together again under a new company. Plenty of opportunities exist.

What advice can I give? First thing - don't panic. Your MySQL database servers are not going to stop working tomorrow. I'd research 3rd party companies like Percona and Pythian. Be diligent, don't wait until Oracle announces something. Like any great DBA, be proactive.

Mark Schoonover: Last Week in Drizzle - Vol 3

Mark Schoonover — Fri, 19 Feb 2010 04:52:25 +0000

This is the third post in the weekly series "Last Week in Drizzle" where we summarize the efforts of various folks in the Drizzle community over the past week. This edition covers Aug 18th through the 24th. As with the week before, a number of developers and community advocates continue to refactor the code base, come together in discussions on the mailing list, and brainstorm on how to solve the tough problems that Drizzle is trying to address. It sounds like many in the community have been swamped this week, but there's still plenty to report. Jay Pipes and myself are tag teaming Last Week in Drizzle.

Continued Growth in the Drizzle Community

Drizzle mailing list has 191 members, up from 148 last week As I type this, there are 42 folks hanging out on the #drizzle Freenode channel. The Drizzle wiki has been steadily growing in its content, there have been a number of entries added or updated:

* Updated FAQ
* Simple Replication
* Table Definitions

Ongoing Conversations on the Mailing List

We've had less Bikeshed stuff this past week. Biggest topics discussed this week:

Community Blogs

Jay Pipes has written up two guides about Getting a Working C/C++ Development Environment for Developing Drizzle and A Contributor's Guide to Launchpad.net - Part 1 - Getting Started

Domas Mituzas blogs about his thoughts on Drizzle and modularity.

Brian "krow" Aker writes: Removal of code for views, some code removed for unireg and definitions. Table objects also created.

Drizzle Build Farm

Still seeking additional Build Bots. You can also find out realtime status about our buildfarm on irc #Drizzle. Details at http://ronaldbradford.com/blog/interacting-with-buildbot-using-irc-2008-08-18/

From last week:

Ronald has continued to champion the Drizzle Build Farm, and writes the following update and request to the community:

The Build Farm has helped us in identifying problems but we are still seeking people to contribute different platforms to our automated process become better.

Currently we cover the following platforms

- Ubuntu 8.04 - 32 & 64 bit - Debian 5 - 32 & 64 bit - Gentoo 8 - 32 & 64 bit - CentOS 5 - 64 bit - Fedora 8 - 32 & 64 bit - SUSE 11 - 32 bit - Mac OS/X 10.5 - 64 bit

See the wiki for how to contribute to the Drizzle Build Farm.

Final Words

That wraps up this week's entry. My apologies to anyone I missed in this edition. Feel free to add errata and additions to the comments of the entry and I will update the blog post accordingly.

Mark Schoonover: Dell/Ubuntu Inspiron 1525 Review

Mark Schoonover — Fri, 19 Feb 2010 04:52:25 +0000

I recently purchased a new Dell 1525 laptop running Ubuntu 8.04. I've been in need of a new laptop for about a year, and decided to take the plunge and see how good a Linux based laptop from Dell really is. My needs are fairly modest. I'm not a gamer, and don't watch many videos. Mainly it's used for blogging, internet, OpenOffice, IRC/IM, listening to podcasts and programming. Portablity is only somewhat important, mostly it'll be used 80% on a desk with the remaining time mobile.

Model: Dell Inspiron 1525N
Price as reviewed: $749
CPU: Intel Core 2 Duo T7250 (2 Ghz/800 Mhz FSB/2MB Cache)
OS: Ubuntu 8.04 with DVD playback
Screen: Glossy 15.4" widescreen (1280x800)
Video: Intel Graphics Media Accelerator X3100
Memory: 3GB Shared Dual Channel DDR2 at 667Mhz
Hard Drive: 120GB SATA (5400 RPM)
Combo Drive: CD/DVD Writer (DVD+/-RW)
Wireless: Intel 3945 a/g Mini-card
Camera: N/A
Battery: 6 cell Li/Io
Sound: High Definition Audio 2.0
Network: Integrated 10/100
Weight: ~6 pounds

Initial Impressions

I ordered this laptop on a Tues and according to Dell, it was going to take 7-10 working days then 3-5 days shipping before I'd see my new laptop here in Southern California. Much to my surprise, it shipped two days later, and arrived at my home two more days after that! I'd say Dell is very conservative in estimating dates. For once, it was great to see a laptop that didn't have all kinds of extra packaging. Dell packaged the laptop with a very minimum amount of packaging material. My box arrived via DHL intact. The contents of the box were very minimal as well, only including the power supply and Ubuntu DVD. A quick start guide and manual was also included, but not really referenced. It's a laptop after all, there's not much to put together! The included manual was for Windows OS only, there were no directions on using Ubuntu.

Upon power up, it was cool to see the Ubuntu splash screen. It went through some hardware configuration, found my wireless network and I was online in under 10 minutes from opening the box! A very flawless start! Ubuntu is installed in mostly a default configuration, and after a few minutes of being online, it had almost 700MB of updates for me to download. I transferred my data from my aging desktop system on a USB thumbdrive to the Dell. I was 100% working off the Dell in about 30 mins from the time I opened the box. The 700MB of updates would wait until I went to bed before downloading and updating.

Performance

Performance is very subjective, but for my needs, the 1525N has been great. Firefox 3.0.1 performance has been very acceptable. OpenOffice also has very acceptable load times too. I downloaded the latest branch of Drizzle and compiled it in about 8 mins. Under normal use, the CPU fan isn't running very fast, but during the compile, it was running at fullspeed. Graphics has been very good for my needs. Watching the occasional video, it was crisp with good colors. The Intel GMA runs at 500Mhz and uses up to 384MB of shared RAM. Using that amount of system RAM, it's best to install as much RAM as you can afford. I rank installing the most RAM you can afford as the #1 upgrade you should do to this laptop. Upgrading to a faster CPU or harddisk may not be a great return on your investment. YYMV of course.

I tested out the DVD writer by creating a restore disk using the Dell/Ubuntu Rescue DVD. I used standard Memorex DVD-R disks, and it burned about 1.8GB of data in around 3 minutes. I made two DVDs, and when trying to burn the second disk, it took a few ejects before the system would recognize a blank DVD was inserted.

Wireless has been flawless. My WAP is a Buffalo WHR-G54S flashed with dd-wrt. The laptop hasn't dropped the link at anytime, unlike my work HP/Compaq 6710b which drops WIFI every few mins.

Sound is a tad weak, it's not very loud. The speakers are upward facing, and Ubuntu responds to the hardware volume controls just fine. Sound quality is what you'd expect from built-in speakers, kinda rough. Plugging in headphones is the high quality option.

The touch pad is just OK. The default settings are ,very sensitive, causing text selection to occur when that's not the intended action. Double clicking by double tapping doesn't work reliably either. This could be due to Ubuntu settings, something I haven't investigated yet.

Keyboard has a nice tactile feel to it. I use the Dvorak layout, so I might see about rearranging the keys from QWERTY layout. By default, the backspace and delete keys are not configured to autorepeat.

Battery life is about what I expected. I'm seeing around 2 hours and 45 mins runtime with the 6 cell battery. I haven't made any configuration changes to Ubuntu for power saving modes, this run time is based on the stock Ubuntu configuration and "general" useage.

Not Tested

Modem
8-in-1 memory card reader
HDMI port
S-video output
Express card slot

Conclusion

For less than $750, this is a fantastic laptop. I'm very happy with it. My other choice was an IBM Thinkpad T61, but I'm glad I went with the Dell. I've only had the laptop about 5 days now, but it's performed flawlessly for my needs. I think the Dell/Ubuntu laptops are a great combination. Only a few small negatives, but they are not a major concern for this laptop. I think Dell should include some kind of Ubuntu quick start guide instead of the manual that ships. If you're already experienced with Ubuntu, this won't be such a big issue, it's more of a concern for users that are new to Ubuntu.

Mark Schoonover: Drizzle Report Vol 1 No 1

Mark Schoonover — Fri, 19 Feb 2010 04:52:25 +0000

The Drizzle Report is a weekly synopsis on drizzle development. This first report will contain more than just a weeks worth of development in order to catch up.

What has been removed from the initial MySQL source tree can be found at the MySQL Differences wiki page. Some of the highlights were the removal of the mysql database, TINY/MED/LONG BLOB, TINY/MED/LONG TEXT thespatial data types, and FULLTEXT indexes. Certain keywords were removed too like ENGINES, CLIENT and CONTRIBUTORS. Even drizzleadmin has been stripped down to just ping and shutdown.The long-standing MySQL ACL has been ripped out, and initial PAM authentication has been started.

Monty Taylor reports RIP: errmsg.sys, in its place is gettext which is a standard for outputting strings.

Comparing drizzle to SQLite has been commonplace. An initial attempt at comparing the two architectures can be found on the Drizzle compared to SQLite wiki page.

Drizzle features have been gleaned from Brian Aker's OSCON 2008 video and are now listed on the Drizzle Features wiki page. Brian has also been interviewed on FLOSS Weekly #35 and Linux Cast. Be sure to listen in.

More information on Drizzle can be found on the Drizzle wiki. IRC and mailing list are also available. If there's something additional you'd like to see in the Drizzle Report, just let me know.

Mark Schoonover: Drizzle Near San Diego?

Mark Schoonover — Fri, 19 Feb 2010 04:52:25 +0000

Not really, it's about 90 degrees out and sunny here near San Diego. For a couple of years, I've been looking around to volunteer on an OSS project, and Drizzle really fits the bill. I've used MySQL for years, and have done some community support as well. I've been wanting to gain greater experience in technical writing, so Drizzle is the perfect project.

I'll be working on development & user documentation, and the weekly Drizzle Report. The Drizzle Report is a weekly synopsis on development and other progress to be posted on my blog Sunday evenings.

Be sure to check out the Drizzle Wiki. Launchpad is the home of the Drizzle project.

Toru Maesaka: Speaking at the MySQL Conference 2010

Thu, 18 Feb 2010 09:31:30 +0000

I’m a little behind in announcing this but I’m going to be speaking at O’Reilly’s MySQL Conference this year. My presentation is a three hour tutorial titled, Drizzle Storage Engine Development. Practical Example with BlitzDB. Three hours is a long time but I assure you that there will be a break.

This session isn’t solely about going through Drizzle’s Storage Engine API. Various performance topics like B+Tree structure, memory handling and concurrency control will be covered. I will also go through BlitzDB’s design concept and it’s internal stuff. So, needless to say I’ll talk a lot about Tokyo Cabinet and it’s internals as well.

Hopefully those that come along will walk out of the tutorial standing far ahead of the start line. It will help you get started on reading the implementation of other storage engines in the MySQL ecosystem (MyISAM, InnoDB, PBXT, Federated and so forth). Better yet you will start writing one.

Looking forward to seeing you there :)

Eric Day: MySQL Conf & Drizzle Dev Day

Wed, 17 Feb 2010 22:41:21 +0000

I’m glad to announce that we’ll be having a Drizzle developer day again this year on the Friday after the MySQL Conference! Be sure to sign up and add any topic ideas you may have so we know what folks are interested in. Space is limited!

While at the MySQL Conference, I’ll be speaking with Monty Taylor on “Using Drizzle.” This will take a non-developer approach to the project, so everyday DBAs and web developers should find this interesting. I’ll also be teaming up with Giuseppe Maxia to talk about Gearman in three sessions. These include:

We’re also going to have a combo Drizzle/Gearman booth in the expo hall, so be sure to stop by and chat. See you there!

Official Drizzle Blog: Drizzle Developer Day 2010 is coming

Mon, 15 Feb 2010 18:45:30 +0000

We are very happy to announce Drizzle Developer Day is coming again this year. Similar to last year it will be the Friday after the MySQL users conference, April 16 – 9:30 a.m. to 5:00 p.m. We are still working on the exact location and will let you know as soon as possible, most likely it will be at or very close to the Santa Clara convention center. We invite anyone interested in providing feedback, implementing new features, helping to fix bugs, or just wanting to learn more about Drizzle to attend.

Please sign up at http://drizzle.org/wiki/Drizzle_Developer_Day_2010_signup. Space will be limited.

Also, to make this day a success, please provide your input on discussion topics at http://drizzle.org/wiki/Drizzle_Developer_Day_2010

Hope to see you there!
-Lee

Padraig O'Sullivan: New Job at Akiban

Sat, 06 Feb 2010 08:00:00 +0000

I just finished my first week at my new position as a software engineer at Akiban Technologies in Boston.
I'm really excited about working here. Akiban is a small startup developing some really cool technology that I believe will get people talking about the relational model in a good way again. We are currently based in the South End of Boston. The building where we are located is pretty awesome and not at all what I pictured an office to be like. There is a resident artist in the building who hangs his paintings on the walls and they seem to move to different places at random times. Its a strange feeling to walk in to work in the morning and smell fresh paint as I go to my desk. Definitely not something I expected!
But besides all that, one of the best things for me about working here is that I get paid to contribute to open source. I've been pretty involved with Drizzle for the last year while still a student and it was always something I really enjoyed which I never thought someone would pay me to work on. The community around the project is awesome and I was just happy to be involved with it. Now that I get paid to contribute, it's nice to know that I can still be part of that community without having to worry about how I'm going to make a living. It's weird to be paid for something that I would still be doing anyway without the pay! I'm not complaining though, it's a nice change!
I'll be presenting at the MySQL conference in April, lots of awesome work is happening in the Drizzle project and Akiban will be out of stealth mode by the conference so there are some exciting times ahead!

Eric Day: Linux Conf AU 2010

Thu, 04 Feb 2010 00:54:25 +0000

I was really excited when I had my Gearman talk accepted to Linux Conf AU 2010 because I had never been out that far in the Pacific (only Hawaii). Of course it wasn’t in Australia this year, and instead in Wellington, New Zealand. My wife came too, and we also made a vacation out of the down times we had around the conference. It turned out Brian couldn’t make it this year so Monty, Stewart, and I gave the Drizzle talk. It was great to see some familiar faces, including Mark Atwood, Giuseppe Maxia, Josh Berkus, and Selena Deckelmann. Josh actually ended up being on the same flight out, so we got to catch up while going through New Zealand customs at 5am after a 13 hour flight. :)

New Zealand is an amazing place. We flew in and out of Auckland and took the train to Wellington. The train ride mostly consisted of grazing sheep once out of the metro areas, did you know there are more sheep than people in NZ? Beyond the sheep, there were great views along the way, especially in the middle near the larger mountains and volcanoes. We stopped for a day to hike the Tongariro Alpine Crossing. It was sunny when we started, but it it was raining with 40mph winds at the top, so we didn’t get to see as much as we hoped. There were still beautiful views on each side though.

The conference was very well run, thanks to anyone who had a hand in it! The speakers dinner was at this great museum nearby on the waterfront and included live Maori singing and dancing. The vegan options were tasty, and I got to meet a few interesting folks there (like the folks from Dreamwidth, a LiveJournal-like blogging service). Some notable sessions during the conference were “The World’s Worst Inventions” by Paul Fenwick, “Anti-features” Keynote by Benjamin Mako Hill, “The Hydras GCC Static Analysis Plugins” by Taras Glek, and “Simplicity Through Optimization” by Paul McKenney. There were many other great sessions, and some I wish I could have attended.

I’m certainly going to try to go again next year, which if you didn’t hear will be in Brisbane!

Brian Aker: Fun with Table Functions

Wed, 03 Feb 2010 20:33:51 +0000

I just about have all of the INFORMATION_SCHEMA replaced with Table Functions!

The big wins:

One Execution path (less bugs)

Simple interface, which means more langauges

Zero materialization happening

Less Code. This allows us to remove a lot of code (and single shot passes for particular use cases).

The data dictionary operates entirely off the proto system, so what you see is what we have. We use the table names stored within the proto so no translation ever happens. This is pretty handy for filesystems which do not preserve case (and we don't have to do anything to support them any longer).

You can also type "SELECT * FROM DATA_DICTIONARY.SCHEMAS".

There is no longer a "SCHEMATA" tables, just SCHEMAS. Want INDEXES? SELECT FROM INDEXES.

We will also be able to split up the tables that are from the SQL ANSI standard one from the ones that are not. The advantage is that we can provide an ANSI compliant INFORMATION_SCHEMA, and still have plenty of room to add additional tables as we see fit (yes, like the tables we provide today which house the information on the active Memcached cluster information).

drizzle> use data_dictionary;
Database changed

drizzle> select * from plugins;
+-------------------------------------------+-----------------------+-----------+-------------+
| PLUGIN_NAME                               | PLUGIN_TYPE           | IS_ACTIVE | MODULE_NAME |
+-------------------------------------------+-----------------------+-----------+-------------+
| ARCHIVE                                   | StorageEngine         | TRUE      | TRUE        | 
| ascii                                     | Function              | TRUE      | TRUE        | 
| benchmark                                 | Function              | TRUE      | TRUE        | 
| BLACKHOLE                                 | StorageEngine         | TRUE      | TRUE        | 
| char_length                               | Function              | TRUE      | TRUE        | 
| character_length                          | Function              | TRUE      | TRUE        | 
| CHARACTER_SETS                            | TableFunction         | TRUE      | TRUE        | 
| COLLATIONS                                | TableFunction         | TRUE      | TRUE        | 
| COLUMNS                                   | TableFunction         | TRUE      | TRUE        | 
| compress                                  | Function              | TRUE      | TRUE        | 
| connection_id                             | Function              | TRUE      | TRUE        | 
| console                                   | Listen                | TRUE      | TRUE        | 
| crc32                                     | Function              | TRUE      | TRUE        | 
| CSV                                       | StorageEngine         | TRUE      | TRUE        | 
| default_replicator                        | TransactionReplicator | TRUE      | TRUE        | 
| drizzle_protocol                          | Listen                | TRUE      | TRUE        | 
| Error_message_stderr                      | ErrorMessage          | TRUE      | TRUE        | 
| FunctionEngine                            | StorageEngine         | TRUE      | TRUE        | 
| GLOBAL_STATEMENTS                         | TableFunction         | TRUE      | TRUE        | 
| GLOBAL_STATUS                             | TableFunction         | TRUE      | TRUE        | 
| GLOBAL_VARIABLES                          | TableFunction         | TRUE      | TRUE        | 
| hello_world                               | Function              | TRUE      | TRUE        | 
| INDEX_PARTS                               | TableFunction         | TRUE      | TRUE        | 
| INDEXES                                   | TableFunction         | TRUE      | TRUE        |

Official Drizzle Blog: Drizzle build 1273 source tarball has been released

Thu, 28 Jan 2010 20:40:46 +0000

Drizzle source tarball based on build 1273 has been released. This marks completion of our Bell milestone. We made a lot of great progress and improvements during the Bell milestone, one of the main goals was to make sure data loss does not occur so that active testing can now occur. While Bell has been completed, upgrading will continue to require dump/load , so hopefully we will nail that down soon and let you know when it is ready.

Stay tuned for more information on goals and features for the our next major milestone which will be code named Cherry.

The Drizzle download file and change log can be found here

Robert Hodges: Tungsten 1.2.2 Release is Out - Faster, More Stable, More Fun

Robert Hodges — Thu, 28 Jan 2010 07:01:53 +0000

Release 1.2.2 of Tungsten Clustering is available on SourceForge as well as through the Continuent website. The release contains mostly bug fixes in the open source version but there are also two very significant improvements of interest to all users.

The manager and monitoring capabilities of Tungsten are completely integrated on the same group communications channel. This fixes a number of problems that caused data sources not to show up properly in older versions.
We are officially supporting a new Tungsten Connector capability for MySQL called pass-through mode, which allows us to proxy connections by transferring network blocks directly rather than translating native request protocol to JDBC calls. Our tests show that it speeds up throughput by as much as 200% in some cases.

The commercial version has additional features like PostgreSQL warm standby clustering, add-on rules to manage master virtual IP addresses and other niceties. If you are serious about replication and clustering it is worth a look.

This is a good time to give a couple of reminders for Tungsten users. First, Tungsten is distributed as a single build that integrates replication, management, monitoring, and connectivity. The old Tungsten Replicator and Myosotis builds are going away. Second, we have a single set of docs on the Continuent website that covers both open source and commercial distributions.

With that, enjoy the new release. If you are using the open source edition, please post your experiences in the Tungsten community forums or write a blog article. We would love to hear from you.

P.s., We have added Drizzle support thanks to a patch from Marcus Eriksson but it's not in 1.2.2. For that you need to build directly from the SVN trunk. Drizzle support will be out in binary builds as part of Tungsten version 1.3.

Toru Maesaka: Notes on HEAP/MyISAM Index Key Handling on WRITE

Tue, 26 Jan 2010 08:57:05 +0000

Disclaimer: This post is based on HEAP/MyISAM’s sourcecode in Drizzle.

Here are my brief notes on investigating how index keys are generated in HEAP and MyISAM. I lurked through these because I’ve started preparing for decent index support in BlitzDB. I also wrote this to assist my biological memory for later grepping (I have terrible memory for names). I’m only going to cover key generation on write in this post. Otherwise this post is going to be massive.

HEAP Engine

The index structure of HEAP can be either BTREE or HASH (in MySQL doc terms). Like other engines HEAP has a structure for keeping Key definition (parts, type, logic and etc). This structure is called HP_KEYDEF and it contains function pointers for write, delete, and getting the length of the key. These function pointers are assigned to at table creation or when the table is opened. The assigned function depends on the data structure of the index and it can be either of the following:

BTREE

hp_rb_write_key()
hp_rb_delete_key()

HASH

hp_write_key()
hp_delete_key()

As for get_key_length(), either of the following functions are used for both data structures.

hp_rb_var_key_length()
hp_rb_null_key_length()
hp_rb_key_length()

When writing a row to the tree, HEAP writes to the index using a key generated by hp_rb_make_key(). Note that it does not use this for the hash index. The generated key is populated inside ‘recbuffer’ in HEAP’s handler object (HP_INFO structure).

From my understanding, it loops through the key segments (I suspect it is similar the internal KEY_PART_INFO structure) and appropriately copies each key field value to the output buffer. By meaning “appropriately” it respects the characteristics of the data type when packing the buffer. For example, for a variable length field, it will only copy the actual data and not the max possible size of it. The final byte that is copied to the buffer is the address of the chunk where the record lives.

MyISAM Engine

The upper layer of key handling in MyISAM looks somewhat similar to HEAP so you can really tell that it was written by the same people. Things are nicely wrapped together by the MYISAM_SHARE structure so it’s relatively easy to follow. BlitzDB has a class called BlitzShare for the same purpose (This is based off Archive Engine’s ArchiveShare class).

Like HEAP, MyISAM has a structure for individual key definition called MI_KEYDEF (it’s defined in myisam.h). There are more function pointers in this structure than HEAP.

bin_search()
get_key()
pack_key()
store_key()
ck_insert()
ck_delete()

In Drizzle, _mi_ck_write() is assigned to ck_insert() which is the entry point to writing a MyISAM index. The key that MyISAM uses to write to the index is generated by _mi_make_key(). Like HEAP, it will loop through the key segments and pack the relevant fields accordingly to the characteristic of the data type. The output buffer belongs to MyISAM’s hander (lastkey2).

From Here

I’ve actually written a naive key generator for BlitzDB already based on Drizzle/MySQL’s internal KEY_PART_INFO array. It seems to be working on EXACT MATCH but I still need to implement an index scanner which looks much harder to pull off than a table scanner. What I’m really worried about is supporting composite indexes (namely reading/searching on it) but hopefully I’ll understand how this area of the storage system works soon.

Marcus Eriksson: Replicating transactions directly to RabbitMQ

Marcus Eriksson — Mon, 25 Jan 2010 15:43:10 +0000

Previously RabbitReplication tailed the transaction log provided by Drizzle and then the Java application sent the protobuf serialized transaction to RabbitMQ. Now it is possible to skip the transaction log file and send the transaction directly to the RabbitMQ server without the extra step of storing it in a file first.

The code is available at https://code.launchpad.net/~krummas/drizzle/rabbitmq_log and to build it with rabbitmq support you need to install librabbitmq which is a bit tricky;

Installing librabbitmq

Install mercurial
Branch the librabbitmq code: hg clone http://hg.rabbitmq.com/rabbitmq-c/
Branch the rabbitmq codegen repo into a subdirectory called codegen in the rabbitmq-c directory:

cd rabbitmq-c
hg clone http://hg.rabbitmq.com/rabbitmq-codegen/ codegen

Run autoconf like this: autoreconf -i
Run the configure script
make
make install

Build Drizzle
When librabbitmq is installed, build drizzle like this:

bzr branch lp:~krummas/drizzle/rabbitmq-log
config/autorun.sh
./configure --with-rabbitmq-log-plugin
make
make install

and it is done!

Start Drizzle with RabbitMQ support

First, you can run drizzled with a --help flag to see the options available, they are all prefixed with --rabbitmq-log-XYZ.

The default values for the parameters makes drizzle connect to localhost as "guest" and replicate to an exchange called ReplicationExchange. Start it like this to replicate changes to a rabbitmq on localhost:

$ sbin/drizzled --default-replicator-enable --rabbitmq-log-enable

The other available options are described in --help

Brian Aker: From Scratch, Get set, Go!

Fri, 22 Jan 2010 21:37:55 +0000

[brian@gaz fix_is]$ ./drizzled/drizzled --console-enable 
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins.
100122 13:12:00  InnoDB: highest supported file format is Barracuda.
100122 13:12:00 InnoDB Plugin 1.0.4 started; log sequence number 44600
Listening on 0.0.0.0:3306
Listening on :::3306
Listening on 0.0.0.0:4427
Listening on :::4427
./drizzled/drizzled: Forcing close of thread 0 user: ''
./drizzled/drizzled: ready for connections.
Version: '2010.01.1273' Source distribution (fix_is)

drizzled> use data_dictionary;
OK

drizzled> show tables;
Tables_in_data_dictionary	
MODULES	
PLUGINS	
PROCESSLIST	

drizzled> desc PLUGINS;
Field	Type	Null	Key	Default	Extra	

PATH ./data_dictionary/plugins
ID	bigint	YES		NULL		
USER	varchar(16)	YES		NULL		
HOST	varchar(64)	YES		NULL		
DB	varchar(64)	YES		NULL		
COMMAND	varchar(16)	YES		NULL		
TIME	bigint	YES		NULL		
STATE	varchar(64)	YES		NULL		
INFO	varchar(100)	YES		NULL		
drizzled>

Monty Taylor: Using libnotify for error messages

Monty Taylor — Thu, 21 Jan 2010 00:34:05 +0000

Any good piece of infrastructure software should be able to pop up little windows on your desktop. :)

Yesterday at LCA in Stewart's Hacking Drizzle talk, it occurred to me that error messages should pop up little windows on my desktop. So I wrote an error message plugin which uses libnotifymm to send Gnome pop-up window messages.

I don't always get to post screenshots, so here you go:

Mark Atwood: Machines Plus Minds - Welcome

Mark Atwood — Wed, 20 Jan 2010 20:37:25 +0000

Welcome!

I write about the techne that is the amazing machine we call "The Net".

At present, I am mostly interested in Memcached, the Drizzle DB fork of MySQL, NoSQL, and in open standards, but I will be writing about other stuff as well.

I used to do my public geek blogging in my personal LiveJournal, but for various reasons it makes sense to separate my public writings about technology, my other public writings, and writings that are of interest only to my closer friends and family.

My day job title is "Director of Community Development" for Gear6, which means that my main paid interest is memcached and the memcached community. I read and write technical articles, research nosql databases and other web-scale open source software, and go to conferences to attend and to speak.

Stewart Smith: You have already lost

Mon, 18 Jan 2010 02:43:13 +0000

When the following code introduces a valgrind warning… you are in a world of pain and loss:

=== modified file 'drizzled/field/blob.h'
--- drizzled/field/blob.h	2009-12-21 08:16:13 +0000
+++ drizzled/field/blob.h	2010-01-18 01:36:48 +0000
@@ -32,6 +32,7 @@
  */
 class Field_blob :public Field_str {
 protected:
+  uint32_t assassass;
   uint32_t packlength;
   String value;				// For temporaries
 public:

Marcus Eriksson: Drizzle to Infinispan replication and a small code walkthrough

Marcus Eriksson — Sat, 16 Jan 2010 20:39:49 +0000

This post aims to explain how to build your own key/value-store applier in RabbitReplication by walking through the new Infinispan support as an example.

Infinispan
Infinispan is a "distributed, highly available data grid platform" and it exposes a REST interface where it is possible to manipulate the data in infinispan. For example it is a simple HTTP PUT method to store new data and a HTTP DELETE does exactly what you expect. The data is stored under resources, for RabbitReplication the data is stored under /<schema.table>/<key>. This means you can view the data in infinispan using a browser.

When I implemented the Infinispan support, I simply dropped the .war file in the webapps directory of a jetty installation and started it. Since Infinispan has a REST interface, the client library can be any HTTP client, I picked the Jersey REST client since it is incredibly easy to use.

Customizing RabbitReplication
RabbitReplication uses Guice internally for dependency injection, so, to use another KeyValue store you need to create your own Module for configuration. I'll show an example below.

To add support for a new key/value store, you need to implement an interface, org.drizzle.replication.transformer.objectstores.KeyValueStoreClient (here). It is a quite straight-forward add/get/remove interface. Look at the infinispan implementation (here) for an example. Note the @Inject on the constructor, it tells guice that the WebResource parameter should be injected when it constructs the object. You will need to put the rabbitreplication.jar file on your classpath when building your stuff.

Guice is configured in modules where you bind() an interface to an implementation, so, to configure guice to use a new KeyValueStore, we need to bind() the KeyValueStoreClient interface to the new implementation. Look at the infinispan module (here) for an example how to do it, the method annotated with @Provides is called by guice when it needs to create a KeyValueStoreClient.

To tell RabbitReplication to use your new module, you simply edit your configuration file, and set where the new Module is located, see this example. If you need to configure your new client, simply add the properties in the config file, guice will bind every property in the file to @Named(...) strings, check the Infinispan module provider method for an example how to use it.

Now, build your code into a jar, drop the jar in the lib/ directory of rabbitreplication and start RabbitReplication like this; bin/start.sh config/someconf.properties - the config file should have the custom_module set to the name of your new module.

Downloading and installing
Best way to use rabbitreplication is still to branch the code from launchpad (lp:rabbitreplication) and then write ant in the base directory, this will create a .zip and a .tgz in the dist directory. You can also download the binaries here.

Unpack the distribution file
Copy a config file from .sample to .properties in the config directory and edit the file to your liking. objectslave.properties.sample is the sample you want to look at if you want to try out Infinispan.
Start rabbitreplication by executing bin/start.sh config/yourconf.properties

Look at the previous posts on rabbitreplication to find out how to start a master etc.

Toru Maesaka: Further thoughts on BlitzDB’s Index Handling

Fri, 15 Jan 2010 09:05:30 +0000

I’ve been thinking quite a bit about collation handling in BlitzDB for the last couple of days. The more I think about it, the more stuck I’ve been getting with BlitzDB’s index design. I’m actually so frustrated with myself at the moment that I want to hit my head against a wall or something.

So, I’m writing this entry to clear up my mind. Heh, my blog is slowly becoming BlitzDB’s design document draft. This should hopefully be good though since by blogging it, people can tell me whether I’m moving towards a stupid direction or not.

Collation Importance

When writing database software that is intended for International use, it is important to handle textual data by respecting collation order. It is arguable that most people are only interested in English lexicographic ordering but unfortunately the world is not so standard.

Internal Primary Key Handling

I want to motivate people to actively define a PRIMARY KEY with BlitzDB. I plan to make this attractive by providing the best performance when PK is defined. In December 2009, my answer to this was to write the PK value as the key for the data dictionary (where actual rows are stored in BlitzDB). This allows BlitzDB to do a direct lookup on the data dictionary for PK based lookup, instead of consulting the B+Tree index. I’m still fond of this lookup optimization approach but it introduces problems too.

Problem 1. Consider the following textual keys: “key” and “KEY”. They obviously have different binary representations but in certain cases they can be logically equivalent. Because the data dictionary is a hash database, this is a problem. The solution that instantly pops up is to normalize the key before writing or reading it. This however, causes a problem in cases where the two keys are inequivalent. Perhaps Drizzle/MySQL provides an internal normalization function that respects this. I still need to study this area of the storage subsystem.

Problem 2. Directly writing a PK to the data dictionary means fast lookup but because of the data structure, it’s not possible to fetch the next “logical” key, meaning I can’t implement index scanning on PK as it is. Quick solution for users is to create an index on the PK column (this would create a separate B+Tree for it) but this is not so friendly because it requires the user to have prior knowledge of all this. So, my plan is to provide the best of both worlds. I’ll elaborate on how I’m planning on tackling this problem next.

Current Primary Key Read/Write Behavior

In general, keys of BlitzDB’s data dictionary is a unique 8 byte integer. The idea is that BlitzDB writes this unique ID along with the key to the B+Tree Index so that it can later identify that row. The difference with PK is that, if a PK is present in a table, BlitzDB will not generate an internal unique ID and use PK for the data dictionary’s key instead. BlitzDB won’t create a B+Tree index for PK at the time I wrote this blog entry.

Next Step

Create a B+Tree index for PK anyway. BlitzDB will still use the PK value as the key for data dictionary if it exists. For PK based lookup requests, BlitzDB will look directly at the data dictionary and for PK based requests that involve index scanning, BlitzDB will look at the B+Tree index.

This approach can consume more space when textual data is used for keys but I think it’s worth it. At the same time, you can save space if you use use types that are smaller than 8 bytes for PK. For example, using a 4 byte integer would reduce BlitzDB’s key space by 50%.

Hmm, I think my mind has cleared a little.

Toru Maesaka: Drizzle, BlitzDB and HTON_STATS_RECORDS_IS_EXACT

Wed, 13 Jan 2010 13:23:34 +0000

Recently I enabled HTON_STATS_RECORDS_IS_EXACT in BlitzDB to let the optimizer know that BlitzDB can instantaneously return the number of rows in a specified table. As a result, the Drizzle kernel can directly call the Cursor::info() function to get the row count. To users, it means that SELECT COUNT statements can be executed in O(1). So it’s a great thing in general.

Something Broke

After I enabled HTON_STATS_RECORDS_IS_EXACT, I noticed that issuing SELECT statement on a table with 1 row would no longer return a resultset. Weird indeed! after investigating with GDB, I noticed that rnd_next() is only called once instead of twice on a table with 1 row (second time is to find EOF) when HTON_STATS_RECORDS_IS_EXACT is enabled. This makes sense because the kernel knows that there is only 1 row and therefore it doesn’t need to keep scanning for EOF. However, this made me scratch my head since this shouldn’t break BlitzDB’s table scanner.

Remedy

Logically, I was confident that BlitzDB’s table scanner was functioning properly so I decided to look at what was going on beyond the engine API. Turns out that join_read_system() in sql_select.cc looks at the table->status value and decides that it’s an error if 0 isn’t assigned to it. What’d you know? I realized that I wasn’t assigning anything to the status variable. It’s more that I didn’t know that I was meant to update an internal structure. You’d think that engine developers aren’t meant to touch those. It’s not mentioned in the Engine Documentation at MySQL Forge either. Nevertheless, the important thing is that it works now. Oh and SELECT COUNT is fast now too.

Eye Opener

This experience among other occasions where I had to read the kernel’s source made me think that it would be nice to provide an intensive up to date documentation on how to develop storage engines for Drizzle in the future (when the API becomes stable). Needless to say, this would be co-ordinated within the Drizzle community. I’m not a license person but it should hopefully be provided with a freely available license too.

Planet Drizzle

Jay Pipes: Recent Work on Improving Drizzle’s Storage Engine API

From Commands to Transactions

If Only It Were Simple…

plugin::StorageEngine Split Into Subclasses

Explicit Statement and Transaction Boundaries

No More Need for Engine to Call trans_register_ha()

Performance Impact of Code Changes

What’s Next?

Ronald Bradford - 42SQL: Understanding Drizzle user authentication options Part 1

Compiling for PAM support

Working with PAM

Configuring PAM

Ronald Bradford - 42SQL: Understanding Drizzle user authentication options Part 2

Compiling for http auth support

HTTP Authentication

Configuring passwords

Drizzle HTTP Authentication in action

Stephen O'Grady: The View from NoSQL Live: Three Takeaways

The NoSQL Term is a Problem

MySQL is a Target

NoSQL and the Cloud

Pythian Group: Log Buffer #182, a Carnival of the Vanities for DBAs

Brian Aker: The Art of "What is going on inside of my database?"

Official Drizzle Blog: Location for Drizzle Developer Day 2010

Jay Pipes: A Follow Up on the SQL Puzzle

Or…What the Heck is Wrong with CREATE TABLE IF NOT EXISTS ... SELECT?

The Officially Correct Answer (at least in MySQL)

I Think Drizzle Should Follow PostgreSQL’s Example Here

Ronald Bradford - 42SQL: Drizzles Data Dictionary and Global Status

Stewart Smith: Writing A Storage Engine for Drizzle, Part 2: CREATE TABLE

Official Drizzle Blog: And the answer is ….. Rackspace Cloud!

Stewart Smith: Continuing the journey

Monty Taylor: [RH]acker

Brian Aker: Google Summer of Code projects, Drizzle

Padraig O'Sullivan: Out of Tree Plugins in Drizzle

Stewart Smith: Drizzle BoF at the MySQL Conference and Expo

Eric Day: Drizzling from the Rackspace Cloud

Jay Pipes: Happiness is a Warm Cloud

An Interesting Almost 3 Years at MySQL

Working on Drizzle since October 2008 (officially)

4 6 9 10 Months in Purgatory

Oracle Doesn’t Need Our Services — Larry Don’t Need No Stinkin’ Cloud

Happiness is a Warm Cloud

Eric Day: C++, or Something Like It

Official Drizzle Blog: Roadmap for our next Drizzle release (Cherry)

Brian Aker: Drizzle, Cherry, Roadmap for our Next Release

Monty Taylor: Hudson Parameterized Matrix Builds

Brian Aker: Data Dictionary Fun in Drizzle

Official Drizzle Blog: Drizzle build 1317 source tarball has been released

Padraig O'Sullivan: Schema-Free Drizzle!

Stewart Smith: Writing A Storage Engine for Drizzle, Part 1: Plugin basics

1. Create the plugin directory

2. Create the plugin.ini file describing the plugin

3. Add plugin dependencies

4. Add skeleton source code for your StorageEngine

5. Build!

6. Add a test

7. Check the code in, feel good about self

Padraig O'Sullivan: Query Rewriting Plugin Point for Drizzle

Toru Maesaka: DATE type under the hood in Drizzle/MySQL

Comparison Function

DATE under the hood

Pythian Group: Log Buffer #180: a Carnival of the Vanities for DBAs

Oracle

SQL Server

MySQL

PostgreSQL

NoSQL, Etc.

Padraig O'Sullivan: Installing SystemTap on Ubuntu

Stephen O'Grady: Data vs Dual Licensing: Which Will Make More Money?

Pythian Group: Product management, effective developers, and the future of MySQL

Brian Aker: Drizzle, Licensing, Having Honest Conversations with your Community

Pythian Group: Announcing: Monday night community dinner at Pedro’s during the O’Reilly MySQL Conference & Expo

Baron Schwartz: A review of Understanding MySQL Internals by Sasha Pachev

Official Drizzle Blog: Drizzle build 1304 source tarball has been released

Brian Aker: Drizzle, Look at the last row...

Brian Aker: Gearman 0.12 Released

Mark Schoonover: Thoughts on Oracle's Purchase of Sun

Mark Schoonover: Last Week in Drizzle - Vol 3

`plugin::StorageEngine` Split Into Subclasses

No More Need for Engine to Call `trans_register_ha()`

Or…What the Heck is Wrong with `CREATE TABLE IF NOT EXISTS ... SELECT`?