Planet Drizzle

Drizzle BoF at the MySQL Conference and Expo

Posted by: Stewart Smith, March 09, 2010 04:46 AM

At the 2010 O’Reilly MySQL Conference and Expo there will be a Drizzle BoF!

It’s currently scheduled for 7pm on April 13th.

Come along, it will be awesome.

Drizzling from the Rackspace Cloud

Posted by: Eric Day, March 08, 2010 04:57 PM

Since I left Sun back in January, folks have been asking what was next. I’m happy to say that I’m going to continue hacking on open source projects like Drizzle and Gearman, but now at the Rackspace Cloud. Not only will I be there, but I get to continue working closely with a few of the amazing Drizzle hackers who have also joined, including Monty Taylor, Jay Pipes, Stewart Smith, and Lee Bieber.

Why Rackspace Cloud? Late last year I was considering what I wanted to do next with the Oracle acquisition looming near, and this was one of the options that presented itself. Rackspace had been a supporter of Drizzle from early on by offering virtual machines to develop and test on, and when talking to some folks more closely, something really hit home. Rackspace provides first-class service and “fanatical” support – they are not a software company. One might ask why an open source software developer would be interested in a company that doesn’t create software or vice-versa, and the answer is that Rackspace wants to find ways to offer the best possible service now and into the future. What better way than to help develop the next generation of service software and get a jump start into integrating this into their architecture? Both the open source community and Rackspace win.

Another thing I learned while talking with Rackspace is that one of their core principles is transparency. This applies to both customer and employees, and anyone within an open source community can appreciate this. The more I learned about the company and the folks within it, the more impressed I was at the lack of internal barriers or “need-to-know” information. One of Drizzle’s core goals is also transparency, from discussing design decisions on public mailing lists and IRC, to having the entire project management infrastructure hosted out in the open at Launchpad.

What does this mean for the Drizzle project? It means continued support for a number of core developers, more infrastructure for development, and most importantly in my eyes, more context. One of the Drizzle tag-lines is “A Lightweight SQL Database for Cloud and Web,” so what better place to develop a database designed for the cloud than on one of the fastest growing cloud platforms. We’ll get a detailed look at the demands, get feedback from cloud customers, and have the perfect test bed for offering new services. We’ll also be able to work closely with a top-notch group of DBAs, developers, and sysadmins in one of the most demanding service architectures out there. This invaluable context will help the Drizzle developers make more informed decisions moving forward, which also means better software for the community.

Personally, this also means getting back to my hosting roots. Before Sun, I worked at Concentric for almost 10 years in a clustered hosting environment. I’m very familiar with many of the multi-tenant scalability concerns Rackspace has, and I’m excited to be working in this type of environment again. We’ve already been working closely with the MySQL DBAs at Rackspace to learn what the biggest pain points are for a multi-tenant architecture, and we’ll be taking steps to address these as it will help anyone wanting to run Drizzle in a cloud-like environment. Drizzle’s modular architecture has already proved useful, as some of these concerns are easily answered with “oh, we have a plugin point for that.”

I’m excited, this is going to be a fun ride.

Happiness is a Warm Cloud

Posted by: nospam@example.com (Jay Pipes), March 08, 2010 04:31 PM

Although a few folks knew about where I and many of the Sun Drizzle team had ended up, we've waited until today to "officially" tell folks what's up. We — Monty Taylor, Eric Day, Stewart Smith, Lee Bieber, and myself — are all now "Rackers", working at Rackspace Cloud. And yep, we're still workin' on Drizzle. That's the short story. Read on for the longer one :-)

An Interesting Almost 3 Years at MySQL

I left my previous position of Community Relations Manager at MySQL to begin working on Brian Aker's newfangled Drizzle project in October 2008.

Many people at MySQL still think that I abandoned MySQL when I did so. I did not. I merely had gotten frustrated with the slow pace of change in the MySQL engineering department and its resistance to transparency. Sure, over the 3 years I was at MySQL, the engineering department opened up a bit, but it was far from the ideal level of transparency I had hoped to inspire when I joined MySQL.

For almost 3 years, I had sent numerous emails to the MySQL internal email discussion lists asking the engineering and marketing departments, both headed by Zack Urlocker, to recognize the importance and necessity of major refactoring of the MySQL kernel, and the need to modularize the kernel or risk having more modular databases overtake MySQL as the key web infrastructure database. The focus was always on the short term; on keeping up with the Jones' as far as features went, and I railed against this kind of roadmap, instead pushing the idea of breaking up the server into modules that could be blackboxed and developed independently of the kernel. My ideas were met with mostly kind responses, but nothing ever materialized as far as major refactoring efforts were concerned.

I remember Jim Winstead casually responding to one of my emails, "Congratulations, you've just reinvented Apache 2.0". And, yes, Jim, that was kind of the point...

The MySQL source code base had gotten increasingly unmaintainable over the years, and key engineers were extremely resistant to changing the internals of MySQL and modernizing it. There were some good reasons for being resistant, and some poor reasons (such as "this is the way we've always done it"). Overall, it's tough to question the strategy that Zack, Marten Mickos, and others had regarding the short term gains. After all, they managed to maneuver MySQL into a winning position that Sun Microsystems thought was worth one billion dollars. Because of this, it's tough to argue with them. :|

Working on Drizzle since October 2008 (officially)

I'm not the kind of person which likes to wait for years to see change, and so the Drizzle project interested me because it was not concerned with backwards compatibility with MySQL, it wasn't concerned with having a roadmap that was dependent on the whims of a few big customers, and it was very much interested in challenging the assumptions built into a 20 year-old code base. This is a project I could sink my teeth into. And I did.

Many folks have said that the only reason Drizzle is still around is because Sun continued to pay for a number of engineers to work on Drizzle as "an experiment of sorts" and that Drizzle has no customers and therefore nothing to lose and everything to gain. This was true, no doubt about it. At Sun CTO Labs, the few of us did have the ability to code on Drizzle without the pressure-cooker of product marketing and sales demands. We were lucky.

4 6 9 10 Months in Purgatory

So, around rolls April 2009. The stock market and worldwide economy had collapsed and recession was in the air. There's one thing that is absolutely certain in recession economies: companies that have poor leadership and direction and are beholden to the interests of a large stockholder will seek an end to their misery through acquisition by a larger, stronger firm.

And Sun Microsystems was no different. JAVA stock plummeted to two dollars a share, and Jonathan Schwartz and the Sun board began shopping Sun around to the highest bidder. IBM was courted along with other tech giants. So was Oracle.

And it was with a bit of a hangover that I awoke at the MySQL conference in April 2009 to the news that Oracle had purchased Sun Microsystems. Joy. We'd just gone through 14 months of ongoing integration with Sun Microsystems and now it was going to start all over again.

Anyone who follows PlanetMySQL knows about the ensuing battle in the European Commission's court regarding monopoly of Oracle in the database market with its acquisition of MySQL. Monty Widenius, Eben Moglen, even Richard Stallman, weighed in on the pros and cons of Oracle's impending control over MySQL.

All the while, us Sun Microsystems employees had to hold our tongues and try to keep our jobs as Sun laid off thousands more workers while the EC battle ensued. Not fun. It was the employment equivalent of purgatory. And the time just dragged on, with many employees, including myself and the Sun Drizzle team, not having a clue as to what would happen to us. Management was completely silent about future plans. Oracle made zero attempts to outline its future strategy regarding software, and thus most software employees simply kept on doing their work not knowing if the pink slip was arriving tomorrow or not. Lots of fun that was.

Oracle Doesn't Need Our Services — Larry Don't Need No Stinkin' Cloud

The acquisition finally closed and very shortly afterwards, I got a call from my boss, Lee Bieber, that Oracle wouldn't be needing our services. Monty, Eric, and Stewart had already resigned; none of them had any desire to work for Oracle. Lee and I had decided to see what they had in mind for us. Apparently, not much.

Larry Ellison has gone on record that the whole "cloud thing" is faddish. I don't know whether Larry understands that cloud computing and infrastructure-as-a-service, platform-as-a-service, and database-as-a-service will eventually put his beloved Oracle cash cow in its place or not. I don't know whether Oracle is planning on embracing the cloud environments which will continue to eat up the market share of more traditional in-house environments upon which their revenue streams depend. I really don't.

But what I do know is that Rackspace is betting that providing these services is what the future of technology will be about.

Happiness is a Warm Cloud

Our team has landed at Rackspace Cloud. I've now been down to San Antonio twice to meet with key individuals with whom we'll be working closely. Rackspace is not shy about why the wanted to acquire our team. They see Drizzle as a database that will provide them an infrastructure piece that will be modular and scalable enough to meet the needs of their very diverse Cloud customers, of which there are many tens of thousands.

Rackspace recognizes that the pain points they feel with traditional MySQL cannot be solved with simple hacks and workarounds, and that to service the needs of so many customers, they will need a database server that thinks of itself as a friendly piece of their infrastructure and not the driver of its applications. Drizzle's core principles of flexibility and focus on scalability align with the goals Rackspace Cloud has for its platform's future.

Rackspace is also heavily invested in Cassandra, and sees integration of Drizzle and Cassandra as being a key way to add value to its platforms and therefore for its customers.

Rackspace is all about the customers, and this is a really cool thing to experience. It's typical for companies to claim they are all about the customer — in fact, every company I've ever worked for has claimed this. Rackspace is the first company I've worked for where you actually feel this spirit, though. You can see the fanaticism of Rackers and how they view what they do always in terms of service to the customer. It's infectious, and I'm pretty psyched to be on their team.

Anyway, that's my story and I'm stickin' to it. See y'all on the nets.

C++, or Something Like It

Posted by: Eric Day, March 06, 2010 03:23 AM

I’ve developed primarily in C most of my career, and recently decided to give C++ a shot as my “primary language” due to hacking on Drizzle and MySQL. The past few months I’ve read and experimented with most features C++ provides over C, including reading Scott Meyer’s excellent “Effective” series books (highly recommended). Along the way I’ve been developing a project I’ve wanted to write for a while, and I’m finding some features to be problematic. I thought I’d share these issues so others can be aware of them and perhaps I can learn better workarounds.

The project I’ve been working on uses dynamic shared object loading at runtime (using dlopen() and friends), is threaded, and has about every strict compiler warning on you can find and being treated as errors (thanks to Monty Taylor’s pandora-build project). I’m also testing on various architectures and compilers, including Linux, OpenSolaris, and OSX. I also have been trying my best to avoid any dependencies on large C++ libraries like Boost and just stick to the standard language and STL. With these requirements in mind, here are the issues I’ve run into:

Can’t Reliably Use Exceptions

My first pass relied on exceptions, but this proved problematic on some architectures as soon as custom exceptions were being throw across module boundaries. This comes down to ABI issues for some shared object formats generated by some compiler versions. While you can make it work in some environments, it’s not going to be portable. This means I’ve had to catch exceptions closer to where they are throw, requiring a lot more try/catch blocks, and not being able to take full advantage of automatic stack cleanup. This also means resorting back to the C way or handling exceptions: returning and checking return codes while generating error strings. To be completely exception safe, this means not using std::string for error returns since they can throw exceptions while building useful error messages. Not using exceptions has had a viral effect throughout the rest of the design of the code, making it look more like C. I was a bit disappointed by this, as not having to check every function’s return code was keeping the code very clean. :)

Limited Use of the STL and std::string

I was excited to take advantage of the STL, as writing things like doubly-linked lists and hash tables for every C struct was getting a bit old (I did have a set of macros I used, but they were not the most popular in some circles because of certain C-preprocessor features). When I learned more about the internals of the STL, and how it relies heavily on copying objects, my heart sunk a little. It completely makes sense in the design, it’s just not as efficient as it could be (especially coming from a place where I would optimize to reduce pointer copies in C). No worries, I just created private copy constructors/assignment operators and only used pointers to objects. This came with it’s own set of issues with pointer management and avoiding leaks if the ‘new’ operator were to fail. Once working out the memory management issues, there were still exceptions to watch out for, including figuring out all the methods that may throw (due to an internal allocation usually). This is especially annoying when doing simple std::string operations like assignment or concatenation, and having to always catch around those. With other annoyances like the reference-to-reference issues and std::unary_function having a non-virtual destuctor, I’ve ended up using a watered down set of STL algorithms and resorted to a mix of non-STL containers and custom algorithms for some things. The lack of thread safety concerns in STL containers and differences in implementations have also lead me to not use STL containers for thread communication (using a mutex for every access is not efficient).

Conclusion

For the sake of consistency, I’ve wondered if it’s worth incorporating STL components? Is it better to have a mix or none at all? This would leave only inheritance, polymorphism, member protections, namespaces, and automatic object destruction the only C++ features being used. These are still very good reasons to use C++, but I’ve found the transition to not be as productive as I had hoped. I am very curious to hear other folks thoughts on their experience with any of the issues above.

Roadmap for our next Drizzle release (Cherry)

Posted by: lbieber, March 05, 2010 10:27 PM

Check out Brian’s blog which details all the features we are targeting for our next release, code name Cherry. Actually in launchpad these are termed “series” and within each series we are defining two week milestones to track our progress. Currently looking to complete Cherry by the end of April.

Drizzle, Cherry, Roadmap for our Next Release

March 05, 2010 05:24 PM

Its March already? How times flies.

The last quarter has been a pretty exciting time for all of us working on Drizzle. We completed the Bell milestone back in January and began immediately plowing into Cherry, our next release. Somewhere in the last couple of months we have seen transition of all of the Drizzle developers at Sun moving into full time positions at other companies working on Drizzle, and we have seen the non-Sun developers move into other positions as well (most note'ably Padraig who went to work for Akiban).

So what is happening for the Cherry release?

  • Authorization with LDAP

    We are adding direct support for LDAP in the next release. With this we be able to fetch authorization information from LDAP. All of this work is being done via our plugin system and builds on the work we have done for PAM and HTTP Auth. We know that companies already have identity systems and don't want to have to deal with "yet another system".

  • Support for the MySQL Protocol

    Will you support the MySQL protocol? The answer is that we will be supporting the MySQL protocol. Today we support our own protocol but we have decided to go back and support the MiniSQL/MySQL protocol. So how are we going about doing this? There will be a plugin you can load, that we will be testing, that will allow Drizzle to speak the MySQL protocol. So you can leverage all of the drivers you use today. Please note that I said "will be testing". Up till now no one has ever written a set of unit tests or a "please crash the server" sort of test on that protocol. For us to feel good about supporting the protocol, we have put together a set of tests to do exactly this. You can run the tests on any of the M/M protocol speaking servers that are being shipped today. We have licensed the test system using the BSD license. We did this so that anyone who ships M/M servers today can feel comfortable about making use and extending what we have made available. In the Memcached world we created Memcapble in order to allow end users to self certify their Memcached servers. We believe that this same sort of process needs to happen around the M/M protocol as well.

  • Embedded Innodb

    Stewart Smith is working on moving us off the Innodb plugin, and onto the Innodb Embedded Server. We continue to support Innodb and we believe that moving to the embedded server is the best way to support end users longterm.

  • Replication Logshipping

    Our ongoing work to enhance replication continues. We are working on adding log shipping to the next version of Drizzle. We will also be removing RAW SQL transport of DDL operations. This allows us to easily take Google Protobuffer messages and translate our schemas into schemas for other databases. We know that data centers are not homogeneous, and that we need to play nicely with other databases.

  • Query Rewriting

    The first version of query rewriting went into the server last week. This is the first step to us having a full query rewrite system in Drizzle. This allows storage engine vendors to easily adapt to Drizzle in the Data Analytics world, and will allow us to eventually add support for VIEWS. The first version allows for user level rewrites but the second generation that is coming will allow for parsed tree rewrites.

  • Schemas as real objects.

    In the past a "schema", aka a "databases", was just a directory. Now they are fully supported as objects in our system and have their own caching layer. This allows for better integration with engines and provides support for engine writers to do fully transactional DDL on schemas. This is very exciting since it really means that engines are no longer locked into the "MyISAM" shoe.

  • Information Schema becomes the Data Dictionary

    Goodbye information schema and all of its bugs! No more materialization, no hidden execution paths, and no locks on table and schema lookups. We have transitioned to a system where we now have Table Functions. These allow us to expose more data to the outside world with none of the costs that were associated with the old system. "SHOW" commands are now just query rewrites, and we have been able to further shrink the size of our parser.

    The above sums up all of the big stuff going on for this release, and doesn't include the incoming work for developers or partners who haven't committed to the release. Cheery will be completed by the time we hit the O'Reilly MySQL User's Conference in April. I'll be giving a keynote there on the State of Drizzle. The conference should make for an exciting event!
  • Hudson Parameterized Matrix Builds

    Posted by: mordred@inaugust.com (Monty Taylor), March 03, 2010 01:43 AM

    I've been making some improvements to our use of Hudson recently that have been really helpful.

    The first was starting to use a set of parameterized builds. These use the Parameterized Trigger plugin, which allows you to pass parameters to triggered additional jobs when a job finishes. So using these we make a job which checks out the latest source (which should just about always succeed) and then fire off a whole host of jobs running on all of our build hosts. It has a single "build now" submit form which takes a bzr branch location as the branch to build. With this all of our developers can check their tree against the build farm before submitting the branch for merge.

    That's great, but as we got more and more build hosts, we had to set up a job on each of them - and then having 4 machines which were all amd64 running Ubuntu 9.10 didn't help - it ran on all of them.

    Enter PlatformLabeler

    Robert Collins wrote a plugin for Hudson which checks a build slave for OS details when it connects (architecture, OS and OS Release) and adds Labels to the slave based on the values it finds. With that - instead of targetting a job to say, hades.inaugust.com, I can target that job to x64_64-Mac-10.4.1. I can easily see both platform coverage, and I can take advantage of having multiple machines with the same config, since it'll run the job on only one of them sharing that label. 

    The next problem with that setup is that there is not a tight binding between the parent and child jobs. You can see that the parent job triggered the child, but you can't really get a single snapshot of "how did this job work everywhere?" If you're doing this pre-merge request, it might be nice to get a URL you can refer to saying "hey! check it out, it worked"

    Enter Matrix Builds.

    Matrix Builds are a single job. They allow you to define how you run the job, and then you can select which hosts to run it on. You can also set up arbitrary sets of flags and hudson will build a grid. So in our case, I set up a job (shortened for brevity)

     ./config/autorun.sh
     ./configure
     make distcheck

    Then I told hudson to build it both with and without the --with-debug flag to configure, and on a set of our build hosts. Now when I kick it off, I get a page with an overview of how that particular build went. Also, having one build description means I don't accidentally do something different on one host.

    Matrix Builds also have a nice feature for a sentinel or canary build, which let you build first on a host where you're pretty darned sure it's going to build, and then build everywhere else. In our case, if it doesn't build on 64-bit Ubuntu 9.10, it's very unlikely to build anywhere, so we don't bother trying. 

    I still use the Parameterized Trigger from the Matrix build to kick of jobs for things like Valgrind and Sysbench when the build completes cleanly. Those jobs themselves can be launched independently (if you only wanted to check how valgrind is treating your new branch)

    Moving forward, I would love to migrate our main set of build branches to Matrix builds, but Matrix currently lacks support for launching an individual sub-build. If it doesn't show up soon, perhaps I'll add it... ah the joy of open source! 

    See the Hudson link at http://hudson.drizzle.org/view/Drizzle-param/job/drizzle-param/ for more details and examples.

    Data Dictionary Fun in Drizzle

    March 02, 2010 07:56 PM

    How often do people get indexing wrong? All the time. This is a sample table in Drizzle, I'll show how you can view indexes with the new data dictionary code.

    drizzle> show create table f;
    +-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Table | Create Table                                                                                                                                                      |
    +-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | f     | CREATE TABLE `f` (
       `a` int NOT NULL,
       `b` int NOT NULL,
       `c` int NOT NULL,
       PRIMARY KEY (`a`,`b`,`c`),
       KEY `c` (`c`),
       KEY `b` (`b`,`c`)
    )  |
    +-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    1 row in set (0.01 sec)
    


    drizzle> select TABLE_NAME, COLUMN_NAME, IS_INDEXED, IS_USED_IN_PRIMARY, IS_UNIQUE, IS_MULTI, IS_FIRST_IN_MULTI, INDEXES_FOUND_IN from data_dictionary.columns WHERE TABLE_NAME="f";
    +------------+-------------+------------+--------------------+-----------+----------+-------------------+------------------+
    | TABLE_NAME | COLUMN_NAME | IS_INDEXED | IS_USED_IN_PRIMARY | IS_UNIQUE | IS_MULTI | IS_FIRST_IN_MULTI | INDEXES_FOUND_IN |
    +------------+-------------+------------+--------------------+-----------+----------+-------------------+------------------+
    | f          | a           | TRUE       | TRUE               | TRUE      | TRUE     | TRUE              |                1 |
    | f          | b           | TRUE       | TRUE               | TRUE      | TRUE     | TRUE              |                2 |
    | f          | c           | TRUE       | TRUE               | TRUE      | TRUE     | FALSE             |                3 |
    +------------+-------------+------------+--------------------+-----------+----------+-------------------+------------------+
    3 rows in set (0 sec)
    
    


    What is so cool about the above? Often people over index columns. They will over index the first part of keys not realizing that the first part of key can be used without creating a standalone key.

    These are a few of the design goals in the above:

    1) Give someone an easy view into whether a column is a part of a primary key.

    2) Find out how many times you are indexes one column.

    3) From a glance see what keys are first inline for a multipart key.

    4) Allow a DBA to work with the database to find out problems (don't force them into tools until they need to see the state of clusters).

    Anyone who has ever managed a MySQL database can pretty quickly see how valuable the above is. Doing a JOIN against a log table will quickly show you what queries are running without the advantage of indexes.

    What is especially cool about the above?

    1) None of that data is materialized. We generate the data through table functions which federate data from multiple engines (all using the new Storage Engine interface).

    2) We open zero tables for the above queries. You can query the data dictionary without blowing through the table cache, or anyway affecting your running queries. In the past the information schema would force tables into the table cache in order to get data from them. In our world? We don't require that at all. If you want schema data we can provide it without undermining your active query sessions.

    3) All of the above information is stored in our Google Protobuffer table format. We do no translations, so what you see is always what you get.

    Pretty cool :)
    Drizzle build 1317 source tarball has been released

    Posted by: lbieber, March 02, 2010 05:46 PM

    Drizzle source tarball based on build 1317 has been released. There was a lot of clean up and removal of dead code from Stewart as he continues to work on integrating embedded innodb. Brian added a data_dictionary schema and also did a complete rewrite of the SHOW commands.

    The Drizzle download file and change log can be found here

    Schema-Free Drizzle!

    March 02, 2010 08:00 AM

    I came across this post from Ilya Grigorik on Hacker News yesterday and I figured I just had to implement this in Drizzle now with the new query rewriting interface that I mentioned yesterday. The awesome thing about Drizzle is that I can try all these ideas out easily by just implementing a plugin.
    Any SQL statements we want to use on our schema-free constructs, we have to prefix with the string 'nos'. With that said, here is a session demonstrating this query rewriting plugin:
    Your Drizzle connection id is 2
    Server version: 7 Source distribution (schema-less)
    
    Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
    
    drizzle> use test;
    Database changed
    drizzle> nos create table widgets;
    Query OK, 0 rows affected (0.06 sec)
    
    drizzle> nos insert into widgets (id,name) values ('a', 'apple');
    Query OK, 1 row affected (0.19 sec)
    
    drizzle> nos insert into widgets (id,name,type) values ('b', 'blackberry', 'phone');
    Query OK, 1 row affected (0.21 sec)
    
    drizzle> nos select * from widgets;
    +------+------------+-------+
    | id   | name       | type  |
    +------+------------+-------+
    | a    | apple      | NULL  | 
    | b    | blackberry | phone | 
    +------+------------+-------+
    2 rows in set (0 sec)
    
    drizzle> nos select * from widgets where id = 'a';
    +------+-------+------+
    | id   | name  | type |
    +------+-------+------+
    | a    | apple | NULL | 
    +------+-------+------+
    1 row in set (0 sec)
    
    drizzle>
    

    The code for this is available on Launchpad (lp:~posulliv/drizzle/schema-less). I threw this together in a few hours today for fun so it is what it is.
    Writing A Storage Engine for Drizzle, Part 1: Plugin basics

    Posted by: Stewart Smith, March 01, 2010 01:40 PM

    So, you’ve decided to write a Storage Engine for Drizzle. This is excellent news! The API is continually being improved and if you’ve worked on a Storage Engine for MySQL, you’ll notice quite a few differences in some areas.

    The first step is to create a skeleton StorageEngine plugin.

    You can see my skeleton embedded_innodb StorageEngine plugin in its merge request.

    The important steps are:

    1. Create the plugin directory

    e.g. mkdir plugin/embedded_innodb

    2. Create the plugin.ini file describing the plugin

    create the plugin.ini file in the plugin directory (so it’s plugin/plugin_name/plugin.ini)
    An example plugin.ini for embedded_innodb is.

    [plugin]
    title=InnoDB Storage Engine using the Embedded InnoDB library
    description=Work in progress engine using libinnodb instead of including it in tree.
    sources=embedded_innodb_engine.cc
    headers=embedded_innodb_engine.h

    This gives us a title and description, along with telling the build system what sources to compile and what headers to make sure to include in any source distribution.

    3. Add plugin dependencies

    Your plugin may require extra libraries on the system. For example, the embedded_innodb plugin uses the Embedded InnoDB library (libinnodb).

    Other examples include the MD5 function requiring either openssl or gnutls, the gearman related plugins requiring gearman libraries, the UUID() function requiring libuuid and BlitzDB requiring Tokyo Cabinet libraries.

    For embedded_innodb, pandora-build has a macro for finding libinnodb on the system. We want to run this configure check, so we create a plugin.ac file in the plugin directory (i.e. plugin/plugin_name/plugin.ac) and add the check to it.

    For embedded_innodb, the plugin.ac file just contains this one line:

    PANDORA_HAVE_LIBINNODB

    We also want to add two things to plugin.ini; one to tell the build system only to build our plugin if libinnodb was found and the other to link our plugin with libinnodb. For embedded_innodb, it’s these two lines:

    build_conditional="x${ac_cv_libinnodb}" = "xyes"
    ldflags=${LTLIBINNODB}
    Not too hard at all! This should look relatively familiar for those who have seen autoconf and automake in the past.

    Some plugins (such as the md5 function) have a bit more custom auto-foo in plugin.ini and plugin.ac (as one of two libraries can be used). You can do pretty much anything with the plugin system, but you’re a lot more likely to keep it simple like we have here.

    4. Add skeleton source code for your StorageEngine

    While this will change a little bit over time (and is a little long to just paste into here), you can see what I did for embedded_innodb in the skeleton-embedded-innodb-engine tree.

    5. Build!

    You will need to re-run ./config/autorun.sh so the build system picks up your new plugin. When you run ./configure --help afterwards, you should see options for building with/without your new plugin.

    6. Add a test

    You will probably want to add a test to see that your plugin loads successfully. When your plugin is built, the test suite automatically picks up any tests you have in the plugin/plugin_name/tests directory. This is in the same format as general MySQL and Drizzle tests: tests go in a t/ directory, expected results in a r/ directory.

    Since we are loading a plugin, we will also need some server options to make sure that plugin is loaded. These are stored in the rather inappropriately named test-master.opt file (that’s the test name with “-master.opt” appended to the end instead of “.test“). For the embedded_innodb plugin_load test, we have a plugin/embedded_innodb/tests/t/plugin_load-master.opt file with the following content:

    --plugin_add=embedded_innodb

    You can have pretty much anything in the plugin_load.test file… if you’re fancy, you’ll have a SELECT query on data_dictionary.plugins to check that the plugin really is there. Be sure to also add a r/plugin_load.result file (My preferred method is to just create an empty result file, run the test suite and examine the rejected output before renaming the .reject file to .result)

    Once you’ve added your test, you can run it either by just typing “make test” (which will run the whole test suite), or you can go into the main tests/ directory and run ./test-run.pl --suite=plugin_name (which will just run the tests for your plugin).

    7. Check the code in, feel good about self

    and you’re done. Well… the start of a Storage Engine plugin is done :)

    Query Rewriting Plugin Point for Drizzle

    March 01, 2010 08:00 AM

    One of the first tasks in my new position at Akiban was to create a plugin point within Drizzle for query rewriting.
    The first decision to make was where to insert a plugin point for a query rewriter. The parsed representation of a query would seem like a natural thing to pass to a query rewriter plugin since the plugin would not have to implement its own parser then. However, the parsed representation of a query in Drizzle is not the easiest in the world to deal with right now so passing this to a plugin would make developing a rewriting plugin quite difficult. Thus, I made the decision to create the plugin point before parsing occurs.
    This means that if a plugin developer wants to do some complex rewriting, they may need to parse the query in their plugin. It may not be ideal but it does make the plugin API for query rewriting quite simple and opens up a lot of interesting opportunities.
    Following the lead of other plugin interfaces such as the replication API developed by Jay, I wanted to keep it as simple and easy to understand as possible. With that in mind, here is the entire API for a query rewriting plugin:

    Thus, all a plugin developer needs to do is implement the rewrite() function within their plugin. The query is passed by reference as a std::string so a plugin can do whatever it likes to this string and this string will then be passed to the parser in the Drizzle core kernel for parsing.
    This interface opens up a lot of possibilties for interesting plugins. For example, one could develop a plugin to analyze a query for common SQL injection patterns or develop a plugin to rewrite a query based on a set of rules. I would be really interested in hearing other ideas people reading this have for plugins using this interface?
    DATE type under the hood in Drizzle/MySQL

    Posted by: Toru Maesaka, March 01, 2010 07:24 AM

    Learned something new from my own bug in BlitzDB today. The problem was that writing a DATE column index would always return a duplicate key error (regardless of what I feed it). There are two suspicious candidates that can cause this.

    • Comparison Function has a defect.
    • Key Generator has a defect.

    The latter suspect was going to be tricky if it was true since BlitzDB currently uses Drizzle’s native “field packer” (except for VARCHAR) inherited from MySQL. This would mean that Drizzle’s field system has a bug in it which was somewhat difficult to believe. Furthermore, you should always blame yourself before you start suspecting other people’s code. So, I decided to look into the comparison function which was completely written by me. Turned out that’s where the bug was.

    Comparison Function

    Allow me to quickly clarify what I mean by “comparison function” in this context. TC’s B+Tree API has an interface that allows you to provide your own comparison function for all operations that involves traversing.

    bool tcbdbsetcmpfunc(TCBDB *bdb, TCCMP cmp, void *cmpop);

    What BlitzDB’s comparison function callback does is, it looks at the data type of the values to be compared and performs appropriate processing on the values then compares them. You can also look at it as a long switch statement. For those that are interested, this code is in blitzcmp.cc (blitz_keycmp_cb).

    DATE under the hood

    After inspecting the “type number” with GDB and looking at the corresponding ha_base_keytype enum, it turns out that the DATE type is internally represented as an unsigned 3 byte integer (HA_KEYTYPE_UINT24). This was pleasant to discover since I’ve been wondering what a 3 byte integer is still used for in Drizzle. The problem I had was that I didn’t take this type into account in the comparator and it also showed how silly I am since the answer was always there.

    Now, the question is should it be kept this way? Respect alignment or reduce total I/O and space by keeping it this way? This should hopefully be a fun discussion to have in the Drizzle community :)

    P.S. My two cents is that it should respect alignment since folks that seek performance should have most of their data on memory. Respecting alignment in this environment should make some difference. Although, I can only say this after benchmarking it of course.

    Log Buffer #180: a Carnival of the Vanities for DBAs

    Posted by: David Edwards, February 26, 2010 06:04 PM

    Hello and welcome to Log Buffer #180. Time’s a-wastin’, so let’s go!

    Oracle

    There was so much Oracle stuff this week that I’ve decided to cram a little more of it into Log Buffer by providing a little less context than usual.

    Jonathan Lewis shares an explication of aliases: “I was asked the following question recently: ‘Does the use of table aliases affect performance?’ To which the best answer is probably ‘Yes, though in general you probably won’t notice the difference and there are reasons more imporant [sic] than performance for using table aliases.’”

    Doug Burns continues his most recent series: Statistics on Partitioned Tables – Part 2, and Statistics on Partitioned Tables – Part 3.

    Charles Schultz demonstrates how VPD + bad ANYDATA practices can really bite: “The point of my blog was that using CAST can really screw up your data. Oracle Support is filing a bug on this behavior, as it looks like an overflow problem.”

    Pythian’s Gleb Otochkin begins a series on Oracle GoldenGate installation.

    Guy Harrison provides a thorough introduction and recommendations on memory management for Oracle databases on VMWare ESX.

    Robert Vollman returns to blogging and offers his 10-point plan on improving your SQL queries.

    Jared Still sheds some light on a cool but unknown RMAN feature.

    Richard Foote knocks holes in another myth: “One of the great myths in Oracle is that bitmap indexes are only suitable and should only be used with columns that have so-called low cardinality (few distinct) values.

    Alexander Kornbrust shares a link to a really good whitepaper about “Hacking Oracle from the Web” by Sumit Siddarth.

    Eddie Awad shares a link to a SQL injection prevention cheat sheet.

    Charles Hooper answers the question, What is the meaning of the %CPU column in an explain plan?.

    Meanwhile, Harald van Breederode does the same for this one: Why does the size of my ORACLE_HOME increase?

    SQL Server

    Thomas LaRock gives an recap of MS’s 2010 MVP Summit. Quotable take-away: “If I had to compare SQL 2008 R2 to SQL Server 4.0, I would say the difference is the same as comparing an F1 race car to a Chevy Vega.”

    Half a world away, there is the SQLSocial Event – London March 16th, as advertised by Simon Sabin.

    Simon also shares a script to get indexes and their included columns, beginning, “I get increasingly frustrated with the lack of visibility of included columns in management studio and from the system stored procedures sp_… This is a query that returns all indexes and there key and include columns[.]”

    Andy Leonard throws us another nourishing SSIS snack: conditional split outputs.

    Here’s Rob Farley with a book review of an oldie but a goodie: Inside SQL 2005 Query Tuning and Optimization, by Kalen Delaney et al. “If you spend any time tuning SQL Server databases, then this book will feel much thicker than it really is, and you’ll be finding useful information on just about every page.”

    Thomas LaRock, meanwhile, writes that SQL Server 2008 Query Performance Tuning Distilled is
    a good way to start your day. “Each morning, while I wait for my desktop to boot, I pick up their book, turn to any page, and just start reading.”

    MySQL

    Sticking with the theme a little longer, here is Baron Schwartz with a review of Understanding MySQL Internals by Sasha Pachev. “I should have read this book a long time ago, and it’s my loss that I didn’t.  . . .  Overall, this book is easily a high 4 stars on a scale of 5, and again, anyone seriously using MySQL should have it.”

    Baron also shares a link to Oracle guy Cary Millsap’s Thinking Clearly about Performance paper.

    Brian “Krow” Aker starts an extensive conversation with his post, Protocols, The GPL, Influences from MySQL. His thesis, “MySQL was the company that had the most influence on how companies and investors viewed the GPL.”

    Paul Vallée of Pythian responds with his ideas on product management, effective developers, and the future of MySQL. “ . . . the future of MySQL, Drizzle, Monty Program, the Percona fork, etc.” to be more precise.

    Colin Charles provides news of what’s been happening recently in MariaDB #1.

    Mohammad Lahlouh wonders, can I use latin1 to store utf8 data? and gets several answers from his readers.

    He might have asked Ronald Bradford, who knows this stuff. Here is his post on migrating MySQL latin1 to utf8 – character set options.

    Pursuing a similar matter (collations), Roland Bouman opines, the best stored routine is the one you don’t write.

    PostgreSQL

    Baron Schwartz again! He announces, mk-query-digest now supports Postgres logs.

    David Fetter says, part(ition)ing is such sweet sorrow. “There are excellent references on partitioning tables that depend on one table, but what happens when you need to partition the referenced table? Let’s find out!”

    Bruce Momjian is here with news on the Python driver confusion.

    Jon Jensen of End Point’s Blog posts a HOWTO on PostgreSQL EC2/EBS/RAID 0 snapshot backup.

    NoSQL, Etc.

    Chen Shapira has been at the compass and protractor, mapping the NoSQL space and returns from terra incognita unscathed.

    Ronald Bradford has been getting started with Cassandra, one of the outposts on Chen’s map, and shares his steps.

    Arnie Rowland says, “Mark your calendar! Portland SQLSaturday/CodeCamp/Barcamp 2010 is scheduled for May 22, 2010, at the University of Portland campus.  . . .  Portland SQLSaturday is encouraging presentations related to interoperability of any of the SQL platforms, including T-SQL (SQL Server), PostgreSQL, MySQL, and PL-SQL. Abstracts for Platform specific sessions are also encouraged.”

    Okay, that is all for this edition. You guys are running me ragged! Fortunately, Gary Myers picks it up next week on his Sydney Oracle Lab. Till then!

    Installing SystemTap on Ubuntu

    February 26, 2010 08:00 AM

    I'm presenting at the MySQL user's conference this year and one of my talks is on using SystemTap and DTrace with MySQL and Drizzle. I'm also doing a tutorial with Jay Pipes on developing replication plugins for Drizzle and that should be a lot of fun.
    I wanted to write some posts before the conference that I can reference within my talk which detail how to install SystemTap and configure Drizzle and MySQL for use with SystemTap. Thus, this post is on how to install SystemTap on Ubuntu while my next post will go in to details about how to configure MySQL and Drizzle for use with SystemTap.
    Before starting, its worth noting that this post is specific to Ubuntu 9.10. The procedure to follow may be different on other versions so its worth keeping that in my mind. The first thing we do is install systemtap and some associated packages which will be needed by Drizzle and MySQL:
    $ sudo apt-get install systemtap
    $ sudo apt-get install systemtap-sdt-dev
    
    Now, being used to Ubuntu, you would think you are good to go now. Unfortunately, attempting to run SystemTap will probably give you the following error:
    $ stap -e 'probe kernel.function("sys_open") {log("hello world") exit()}'
    semantic error: libdwfl failure (missing x86_64 kernel/module debuginfo under
    '/lib/modules/2.6.31-19-generic/build'): No such file or directory while resolving probe point
    kernel.function("sys_open")
    semantic error: no probes found
    Pass 2: analysis failed.  Try again with another '--vp 01' option.
    $
    
    The above error occurs because SystemTap needs to have a debug version of the kernel available. Unfortunately, installing the debug information for a kernel on ubuntu is not a trivial operation to perform. In fact, there is a bug on Launchpad about this issue. Thus, we will build a kernel debug package from source ourselves. This can be done as follows:
    $ cd $HOME
    $ sudo apt-get install dpkg-dev debhelper gawk
    $ mkdir tmp
    $ cd tmp
    $ sudo apt-get build-dep --no-install-recommends linux-image-$(uname -r)
    $ apt-get source linux-image-$(uname -r)
    $ cd linux-2.6.31 (this is currently the kernel version of 9.10)
    $ fakeroot debian/rules clean
    $ AUTOBUILD=1 fakeroot debian/rules binary-generic skipdbg=false
    $ sudo dpkg -i ../linux-image-debug-2.6.31-19-generic_2.6.31-19.56_amd64.ddeb
    
    This builds a debug image of the kernel and so will take quite a while. Once we have the above completed, we can try running our hello world example with SystemTap again. In order to get some output, you should open or create some file on the system in another terminal window. In this example, I backgrounded the stap process and created a file:
    $ sudo stap -e 'probe kernel.function("sys_open") {log("hello world") exit()}' &
    [1] 951
    $ touch /tmp/padraig
    $ hello world
    $ [1]+ Done
    
    Installing SystemTap on CentOS is significantly easier since it is primarily developed by Red Hat. A good article on how to install it on CentOS is available here.
    In my next post on the topic, I'll explain how to configure MySQL and Drizzle for SystemTap and give some simple examples of using SystemTap with them.
    Data vs Dual Licensing: Which Will Make More Money?

    Posted by: sogrady, February 25, 2010 10:00 PM

    By 2012, at least 70% of the revenue from commercial OSS will come from vendor-centric projects with dual-license business models.

    80% probability. This is may true today but the lack of revenue among broader market OSS products compared to Linux isn’t large enough yet to make this one a done deal. What is clear is that the overwhelming majority of ‘commercial oss’ efforts are based on a dual license model – vendor prefer the ‘open core’ moniker because it sounds more OSS friendly but its essentially the same thing

    .”
    - Mark Driver, Gartner, Open Source Predictions for 2010

    This prediction from Gartner’s Mark Driver confused me, I’ll admit, when I first read it. Baffled me, actually. Looking at the market, it seemed clear to me that the practice of dual licensing was, if anything, in decline. I couldn’t see how we could look at the same market and come to such different conclusions. My view was similar to Brian Aker’s (as is Cloudera’s Mike Olson’s, notably), most recently of MySQL/Sun:

    When MySQL pushed dual licensing, investors looked for this hook in every business model. I remember standing outside of a conference room in SF a couple of years ago and talking to one of the Mozilla Foundation people. Their question to me was “Is the nonsense over dual licensing being the future over yet?”. The fact is, there are few, and growing fewer, opportunities to make money on dual licensing. Dual licensing is one of the areas where open source can often commoditize other open source right out of the market. The dearth of companies following in MySQL’s dual licensing footsteps to riches, belabors the point of how niche this solution was.

    Even for MySQL, long the standard bearer for the approach, the logistics of dual licensing were and are becoming increasingly problematic over time:

    For smaller firms, the primary limitation [of dual licensing] is the development. Unlike non-dual licensed projects which need only concern themselves with the quality and provenance of code contributions from external parties, dual-license vendors need also consider the question of copyright ownership. Because dual licensing depends on ownership of copyright for the entirety of the asset in question, third parties must either assign or be willing to jointly hold the copyright for any potential contributions. Early in a project’s lifecycle, this is a minor concern because the project owner likely employs most of those qualified to improve it. As a project matures and becomes more popular, however, this is a more pressing issue. First, because it acts to inhibit community participation (see slide 18 of this deck produced by Monty), but second – and more problematically – it means that third parties can, in practical terms, offer a more complete product.

    Jeremy Zawodny made reference to the practical implications of the dual license in a post from December of last year entitled “The New MySQL Landscape.” In it, he made the assertion that “You can get a ‘better’ MySQL than the one Sun/MySQL gives you today. For free.” This is the cost of the dual licensing model: in return for the right to exclusively relicense the code, you forfeit a.) the right to amortize your development costs across a wide body of contributors, and b.) the right to uniformly integrate the best patches/fixes/etc that are made available under the original license because you cannot always acquire the copyright.

    This doesn’t mean that dual licensing is a uniformly bad strategy, but it does imply that it has costs, and that those costs escalate over time. This situation is the inevitable result of the dual license model over time as applied to a successful project. For those looking for perspective from a MySQL and Drizzle developer, I’d recommend reading Brian Aker’s piece here.

    Even setting aside the disincentives to pursuing a dual licensing strategy, the basic math of the 70% argument didn’t work for me. Even at MySQL, remember, a fraction of the revenue is derived from the issuance of dual licenses. And even if we assumed, for the sake of argument, that the entire revenue stream was the product of dual licensing, that still wouldn’t be enough to meet the 70% projection. Not nearly so.

    As Driver notes when he says “the lack of revenue among broader market OSS products compared to Linux isn’t large enough yet.” Linux, it seems clear, is the largest single open source commercial ecosystem, and due to the lack of centralized copyright ownership, it cannot be dual licensed by anyone. What Driver is saying, in other words, is that the open source commercial ecosystem has to be big enough that Linux doesn’t comprise more than 30% of it.

    Consider the following back of the envelope calculations. Red Hat’s revenues in the year ending 2009 were $652 million and change. We know that, for copyright and licensing reasons, none of that money may derive from dual licensing revenues. If we assumed, counterfactually, that Red Hat represented all of the non-dual license revenue of the market – the leftover 30%, my math says that the total revenue picture would be around $2.17B. Meaning that we need a little more than two Red Hat’s more worth of revenue to emerge from dual licensees like MySQL.

    Personally, I’m skeptical that that would happen, even with the hybrid source trend.

    Part of the problem is, I believe, semantics. Driver seems to be conflating what is sometimes referred to as “open core” licensing with dual licensing. Personally, I believe they are distinct. The former tends to refer to varying combinations of open source and proprietary codebases, while the latter is more generally used in conjunction with copyright mechanisms as they apply to a single open source codebase. This view is supported by my analyst colleagues over at the 451 Group.

    Were we to grant Driver the more expansive definition of dual licensing, however, I still think that figure is wrong. Based on the conversations we’re having with vendors in the space, it seems more likely that revenue growth and expansion will come not from quote unquote dual licensing, but derived intelligence from gathered data and telemetry.

    Judging by the almost universally poor conversion metrics – that is, the number of users of a given open source tool that are converted to paying customers – it seems reasonable to assert that there are ongoing and systemic issues in the commercialization of open source software. Hence the proliferation of alternative revenue models such as dual licensing, open source and even SaaS. It is far from clear, however, that these models satisfactorily align customer and vendor interests such that conversion percentage will elevate to levels where they are competitive with proprietary software.

    At the end of the day, open source customers are generally paying for one or more of a.) break/fix/integration/support/etc services that they hope not to need, b.) withheld features that they need to pay to gain access to, or c.) the right to not observe the terms and conditions of the original license. The relative distribution of revenue within this set is skewed by the size and scope of the Linux community towards A, with B being the raison d’etre for open core and C the same for dual licensing.

    But what if open source vendors could leverage their primary strength – distribution – more effectively as a direct revenue stream? I’ve been predicting for three years or so that they would do just that, via data aggregation and analytics. The alignment of customer and vendor goals is better in this model than in virtually any other. The simplest example of this model outside of open source is Google, who provides users with search at no cost, receiving in return massive volumes of data which they monetize both directly (contextual ad placement) and indirectly (algorithmic improvement, machine learning, intelligence for product planning strategy, etc). Why couldn’t software vendors employ a similar model, trading free software for user generated telemetry data? The answer is, they can. SpiceWorks, for one, is doing just that now, quite successfully, albeit not with open source software.

    The strength of open source is in its ubiquity, and the volume it commands ensures that the telemetry returned would have substantial – potentially immense, depending on the project – value. Importantly, however, the value lies in the aggregation. A single user’s telemetry is likely to be relatively uninteresting. A hundred users’ telemetry, more interesting. A thousand users’, that much more so, and so on. Users, therefore, wouldn’t be surrendering anything of material value to a would-be vendor in the transaction. Better, analysis of the aggegrate could have enormous value to customers. How is my infrastructure performing relative to similar environments? What are the types of conditions that indicate a potential problem? What differentiates my architecture from the Top 10 best performing? These are answerable questions…if you have a big enough dataset. Most customers would not have that; an open source software provider aggregating and analyzing their combined telemetry would.

    Privacy and trust will certainly be concerns, but if the right data is offered as an incentive and the appropriate anonymization assured, those can be addressed for most customers. And for those that remain concerned, they should have the ability to opt out understanding that they will in turn have no access to the resulting analytics, and might therefore be at a disadvantage relative to their competitors who were using the intelligence.

    This direction seems nothing less than inevitable for me, and so it is no surprise that we’re beginning to see (and help) a variety of open source vendors move in this direction. Free and open source data has a bright future regardless of the revenue model, but as we see successful projects better leverage their traction via analytics, the result should be a win for ecosystems and customers alike.

    Whether you believe as I do, however, that the money is ultimately going to come from data more than code, it seems clear to me that it is not going from what is commonly considered to be dual licensing. Because while it is not true that I am an enemy of that particular approach, I do believe it’s in decline. Not least because it’s poorly aligned with customers needs.

    Unlike data.

    by-nc-sa
    Product management, effective developers, and the future of MySQL

    Posted by: Paul Vallee, February 23, 2010 07:38 PM

    I am writing because Sheeri sent me a note about a blog post written by Brian Aker, where Brian concludes, quite correctly, that (in Sheeri’s words not Brian’s)


    MySQL is now just a branch (the official branch,
    but a branch nonetheless, and a bunch of trademark (logo) and
    copyright (docs) ownerships).

    This is exactly true. No denying it. Why bother. It’s true. It’s also true for the vast majority of open-source projects, by the way.

    I replied to Sheeri:


    There's no denying that. The product direction will be set by whoever sets the best product management strategy backed by the most effective development effort. And there can be multiple winners.
    -Paul

    Well, this is the kind of quality output I can be relied on. It might not fit on twitter, but it’s not blogworthy. Sheeri’s word of encouragement:


    See, now that would be a nice blog post with a positive outlook that
    both Oracle Corp and MySQL community would agree and be happy with,
    because both Oracle Corp and the MySQL community feel they can set
    "the best product management strategy backed by the most effective
    development effort."
    -Sheeri

    God. My reply was embarassing but maybe I should include it for humour value:


    Go for it. Its a tweet for me at the most. No time to expand that thinking into a blog worthy of the blog today.
    -Paul

    and then, right away,


    ah censored it i'll do it.
    it'll be short.
    -paul

    You are now reading the result of this very modest effort.

    Here’s the future of MySQL, Drizzle, Monty Program, the Percona fork, etc.

    The best product management strategies… should we be lightweight for the web, plug-in oriented like Drizzle? Should we follow Monty’s giant-killing roadmap? Should we focus on performance-oriented patches? The best product management strategies will win.

    They can’t win alone. Will they be backed by appropriate investments from effective developers? Effective developers are the ones who convert winning product management strategies into working products. You can’t get there without them and I’ve seen lots of great strategies fail that test (including my own actually).

    And there can be more than one winner.

    It’s doesn’t matter what roadmap Oracle plots for MySQL. If it’s not the roadmap the community wants, it will lose ground and open an opportunity for another fork. If it is, however, (and NEVER, NEVER underestimate Oracle’s product management because it is outstanding and a big component of their historical success), if it is, however, Oracle can win the long-term hearts and minds, because they can resource quality developers in a way that I don’t think any of the competing forks are capitalized to do (yet.)

    Either way, it’s going to be fun to watch.

    And more than one player can win.

    And regardless, the community wins. Big time.

    Drizzle, Licensing, Having Honest Conversations with your Community

    February 23, 2010 05:27 PM

    I pulled this from a quote on yesterday's Slashdot story about MySQL Licensing where the author of the quote mentions Drizzle's licensing terms:

    "you require the code to be under BSD"

    This is actually a myth, we don't.

    If you look through the Drizzle codebase you will note that very few files have BSD headers, and all that do?

    They are a part of new systems that have been written since the fork of the project, and not all of these are BSD.

    Why is this?

    A large part of Drizzle is derived work from MySQL, and in all cases there we inherited its GPLv2 license on files. Bug fixes and code refactoring projects all fall under the umbrella of "derived work". In all of those cases the work was made under the GPL but no copyright assignment ever occurred. To understand ownership there you would need to look at the revision history to the code. We have never made any effort to track anything more then "where did the code come from".

    I am a big opponent of copyright assignment in open source. How do you have an honest conversation with someone where you say "yes, I will need the work you did for free, to be assigned over to me, so that I can make money on it"?

    Or better put:

    "Here is $100 for the code you did, would you like some trinkets and beads to go along with that?"
    


    We never did copyright assignment for Drizzle, and in no other project I have run personally have I ever done it. It is not a conversation I can have with a straight face with anyone. If you need commercial rights to your work and you want to do a true "quid pro quo"? License your code under the BSD license in the first place, that way both you and the contributor are on equal footing.

    Does this mean we are a free for all when it comes to code? No, we track ownership. We know where every single line of code came from thanks to bazaar. If someone wanted to "poison" the codebase we would know exactly who that was. We would point the copyright owner to the offender, remove the code, and help with the prosecution of said individual. We have an active discussion going on at the moment about the future of our copyright headers, and one option we are considering is just replacing the copyright owner notice with a "Please see revision history for complete ownership information".

    Are all of the files in Drizzle GPL? No, we do have some which are BSD.

    I am a big fan of the BSD license and I typically suggest to developers that they use it for certain projects. As a license it links well with other code, and by establishing new plugins as BSD the original author can pull from any bug fixes that are made to the code. As an example Patrick Galbraith established the built in drivers that allows Drizzle to speak with Memcached. He has pulled code from the Drizzle driver and used that for his MySQL drivers. If those drivers had been done as GPL he would have not been able to pull code back into the MySQL Memcached drivers (which are BSD).

    The GPL works just fine as the license for the kernel of Drizzle. I don't see that changing. In discussions with the other core authors, there has never been a push to "rewrite" the entire internals just to change the license. It is more work then it is worth, and it is not needed. The micro-kernel design means that going forward most of the work is pushed out to the libraries that link us to other systems. Work in the kernel is all about making those interfaces robust and creating more opportunity for plugin writers.

    A final note on licensing and direction.

    We never had any ambition to aim Drizzle at the deeply embedded market. Taking Drizzle, compiling it into a library, and linking it against another application is not a goal that the core team has ever had. If we had ever wanted to go into the world of deeply embedded databases we would have needed to have done code assignment or switched the entire license of the program.

    In the deeply embedded world SQLite reigns supreme. SQLite does an awesome job in that space, and we see zero reasons to go head to head with it. If I needed a deeply embedded database I would just use SQLite, I wouldn't bother to write a new one.

    Drizzle's core will stay GPL, and we continue to take contributions without code assignment. If you are a programmer I believe you can appreciate the merits of why we do this.

    If you are a developer and you find yourself in the peculiar position of being asked to sign over your work? I would encourage you to take a hard look at why this is being asked, and see how comfortable you are with the value you are getting in return for your work.
    Announcing: Monday night community dinner at Pedro’s during the O’Reilly MySQL Conference & Expo

    Posted by: Sheeri Cabral, February 22, 2010 05:12 PM

    Just the facts:
    What: MySQL user community dinner
    Who: me, you, and many MySQL community members
    When: Monday, April 12th – Meet at 6:30 at the Hyatt Santa Clara or at 7 pm at the restaurant
    Where: Pedro’s Restaurant and Cantina – 3935 Freedom Circle, Santa Clara, CA 95054
    How: Comment on this blog post to add your name to the list of probable attendees

    I was sad that last year there was no community dinner, and I missed the one the year before when Jonathan Schwartz and Rich Green made an appearance. This year I am determined not to miss it, and so I am calling for a community (pay-your-own-way) dinner on Monday, April 12th, at Pedro’s – a Mexican restaurant that has vegetarian and vegan options. I think Monday is a better time because many folks arrive Sunday evening, or even Monday morning (there are tutorials on Monday, but not everyone attends).

    Pedro’s can handle large groups of people, but we would like to have a vague idea of how many people are attending — while you are not required to RSVP, we would like to make an accurate reservation at Pedro’s….In 2008, there was a wiki page with a list of attendees, and I was disappointed because there were so many people on that list I wanted to see.

    Meet us at 6:30 pm on Monday in the lobby of the Hyatt Santa Clara, or at 7 pm at Pedro’s. If you want to come later, just show up at Pedro’s whenever you can.

    Since commenting on this blog does not require registration (as the wiki does), I invite folks to comment on this blog post and I’ll add you to the list of attendees:

    1. Sheeri K. Cabral (The Pythian Group)
    2. Paul Vallee (The Pythian Group)
    3. Rob Hamel (The Pythian Group)
    4. Giuseppe Maxia (Sun)
    5. Brian Aker (Drizzle)
    6. Konstantin Osipov (Sun)
    7. Mark Callaghan (Facebook) (will arrive later)
    8. Wagner Bianchi (EAC Software, Brazil)
    9. Roland Bouman (BI wizard)
    10. Bill Karwin (Karwin Software Solutions)
    11. Maxim Volkov (OpenCandy)
    12. Brian Moon (DealNews) – note: Monday Apr 12th is Brian’s birthday!
    13. Rob Peck (DealNews)
    14. Arjen Lentz (OpenQuery)
    15. Vadim Tkachenko (Percona)
    16. Rohit Nadhani (WebYog)
    17. Paul McCullagh (PrimeBase)
    18. Monty Widenius (Monty Program)
    19. Sergei Golubchik (Monty Program)
    20. Kristian Nielsen (Monty Program)
    21. Henrik Ingo (Monty Program)
    22. Nick Westerlund

    (Tuenti)
  • Sergey Petrunya (Monty Program)
  • Baron Schwartz (Percona) (Tentative yes)
  • A review of Understanding MySQL Internals by Sasha Pachev

    Posted by: Xaprb, February 19, 2010 11:13 PM

    Understanding MySQL Internals

    Understanding MySQL Internals

    Understanding MySQL Internals. By Sasha Pachev, O’Reilly 2007. Page count: about 227 pages. (Here’s a link to the publisher’s site).

    I should have read this book a long time ago, and it’s my loss that I didn’t. Although the title makes it sound like it should only benefit those who’ll be changing the MySQL server’s own code, that’s not true. To the contrary, at least parts of this book should be required reading for DBAs and developers who use MySQL, after they gain a moderate level of familiarity with how to use the server.

    The book does indeed start off with a few chapters on the source code, how to work with it, and the core structures that make up the MySQL server at the source code level. However, even these topics hold value for users such as DBAs. If you don’t know how the server really works, you are lost when you are faced with a problem or asked to understand some behavior. Peter Zaitsev refers to this as “X-Ray Vision,” something a good DBA or consultant needs. I think the first few chapters of this book are a great way to develop that X-Ray Vision into MySQL.

    The next couple of chapters are on the client/server API, configuration variables, and thread-based architecture. Although the first is probably not a core competency for DBAs, the others are. I sure wish I’d had the client/server protocol chapter handy when I was working with the protocol, though. It is variously more useful than, and a good supplement to, the internals document on the MySQL wiki.

    The following chapters cover the storage engine interface, the server-level lock manager and how it interacts with the storage-engine locking, and the parser and optimizer. These are absolutely core knowledge for DBAs and developers in my opinion. The server/storage-engine division is one of the things that makes MySQL different from other databases, and is mandatory to understand deeply. This applies equally well to the rest of the chapters, which cover the parser, optimizer, various storage engines (as opposed to just the server’s interface to them), transactions, and replication. Mandatory, every one.

    What’s missing? I found that the book is kind of funny in one major way. It doesn’t talk much about MySQL 5.0. Instead, it delves into 4.x and 5.1. Most of the new features in 5.0 are not mentioned at all. Stored procedures, the INFORMATION_SCHEMA, triggers, and so on are absent, as are most discussions of changes to the optimizer and so forth. Some 5.0 topics are covered: index merge, for example. But by and large, there’s not a lot of coverage here. The 5.1-specific topics are those such as the new storage engine API and row-based binary logging. Events are not covered, nor are changes to other types of logging. Honestly, I feel this is appropriate in a book this size; the stuff that hasn’t changed since 4.x days is more important to understand.

    There’s little discussion of exactly how certain features work, such as the different sorting algorithms. But that’s OK. These are covered pretty well by the MySQL manual, and even by my own book High Performance MySQL 2nd Edition. I think some other major topics might be missing, but I can’t quite think of them now.

    The book is really well written. I expected it to be dry but it’s not at all. It’s actually engaging and interesting. I also found a curious thing happening as I read: I became more aware of how much legacy cruft there is inside MySQL, and how much that has contributed to various shortcomings. This made me actually feel sad, and made me yearn for the bright pure clean exciting vision that Drizzle strives towards. But at the same time I kind of felt nostalgic, kind of fell a little more in love with MySQL, for its strengths and for the countless hours of work and the really monumental genius that it embodies, warts and all. It was quite a cognitive dissonance experience, to tell the truth!

    For those who have any inclination to reading it, I’d say: do it. It’ll benefit you a lot more than you think. And if possible, do it with a copy of the MySQL source code available and actually take the time to look through it and explore the things Sasha suggests. I read this book on an airplane, far from a computer, and I need to read it again and look at source code as I do so. I am positive I’ll get more than twice as much benefit from this second reading than I did from the first. I say that because I have a shin-deep exposure to the MySQL source code myself, so I knew just enough about it to recognize that I really would get a lot more from going and looking at the code Sasha cross-referenced. It was a bit like speaking Spanish without a dictionary, but having had a few weeks of intensive instruction ten years ago. I remembered some things well, other things just hazily.

    Overall, this book is easily a high 4 stars on a scale of 5, and again, anyone seriously using MySQL should have it.

    Related posts:

    1. A review of MySQL Administrator’s Bible MySQL Admi
    2. A review of Get it Done with MySQL 5&6 by Peter Brawley and Arthur Fuller Get it Don
    3. A review of Pentaho Solutions by Roland Bouman and Jos van Dongen Pentaho So

    Related posts brought to you by Yet Another Related Posts Plugin.

    Drizzle build 1304 source tarball has been released

    Posted by: lbieber, February 19, 2010 10:48 PM

    Drizzle source tarball based on build 1304 has been released. This marks the beginning of our Cherry series of milestones. Most notably in this release is the beginning changes to replace Information Schema with table functions. See Brian’s blog for more details.

    The Drizzle download file and change log can be found here

    Drizzle, Look at the last row...

    February 19, 2010 07:44 PM

    drizzle> select * from plugins WHERE PLUGIN_TYPE="storageengine";
    +--------------------+---------------+-----------+-------------+
    | PLUGIN_NAME        | PLUGIN_TYPE   | IS_ACTIVE | MODULE_NAME |
    +--------------------+---------------+-----------+-------------+
    | ARCHIVE            | StorageEngine | TRUE      |             |
    | BLACKHOLE          | StorageEngine | TRUE      |             |
    | CSV                | StorageEngine | TRUE      |             |
    | FunctionEngine     | StorageEngine | TRUE      |             |
    | INFORMATION_ENGINE | StorageEngine | TRUE      |             |
    | InnoDB             | StorageEngine | TRUE      |             |
    | MEMORY             | StorageEngine | TRUE      |             |
    | MyISAM             | StorageEngine | TRUE      |             |
    | schema             | StorageEngine | TRUE      |             |
    +--------------------+---------------+-----------+-------------+
    9 rows in set (0 sec)
    
    


    The payoff for making Drizzle a plugin based architecture? Development goes much faster.
    The highlights:

  • Fixed bug where memory loss occured if data was too large.
  • Added gearman_strerror().
  • Fixed bug where setting an option off in mass would not trip any triggers on the option (for both worker and client).
  • Options that are internal can no longer be set by external callers.
  • Deprecated gearman_client_set_event_watch_fn() and gearman_worker_set_event_watch_fn.
  • gearman_job_handle() and gearman_job_function_name() now return const char* pointers
  • gearman_worker_unregister now returns GEARMAN_NO_REGISTERED_FUNCTION if the function does not exist (or is being removed)
  • Added gearman_worker_function_exist()
  • Trying to send too large of a piece of data will result in GEARMAN_ARGUMENT_TOO_LARGE.
  • Added support for gearmand command client to daemonize and create a pid file.
  • Fixed job handle comparison bug with WORK_FAIL responses.
  • Fixed disable assert configure option.
  • TokyoCabinet support added.
  • Updates to Drizzle support.
  • Updates to MySQL support.
  • Build system updates.
  • And a lot more then what I will write here...

    0.12 was a cleanup release for the most part. Numerous bugs were fixed and we cleaned up the API to make it a bit safer for the future.

    For the next release the plan is to put more effort into the server, which will including additional logging options and updates to the server code around its internals (mainly C++ work over C).

    You can download 0.12 from here:
    https://launchpad.net/gearmand
  • Thoughts on Oracle's Purchase of Sun

    Posted by: Mark Schoonover (noreply@blogger.com), February 19, 2010 04:52 AM

    With a first day announcement at the MySQL 2009 User Conference,  Oracle had purchased Sun! Reading the tweets and blogs, it's understandable that some think it's the end for MySQL, some think it's not - I think it could be both.

    MySQL itself is going through a second corporate purchase inside of 18 months. When Sun purchased MySQL, some of the internal talent left. Just recently we saw the exit of Marten, and just before that Monty. What made MySQL what it was in 2008 is the people behind the database, from corporate to the community. 

    MySQL the database server itself is safe since it's GPLed. Anyone can have access to the code and create their own distribution. We've already seen this happen within the community, Jeremy Cole, Percona, OurDelta to name a few. The main concern that could slow or stop MySQL progress is all the programming talent leaving en masse. Put this together with the departure of all the original corporate management and MySQL is no longer MySQL the company we remember. The part we don't know for sure is Oracle's plans for MySQL. Just seems to be getting worse.

    Drizzle could also be impacted since Brian Aker is a Sun employee, so is Jay Pipes, and I'm sure there are others. Some companies support Open Source Software by paying their employees to work on these projects. We don't know if Oracle will allow this to continue.

    Who could be impacted by Oracle's purchase? That huge computer company in Redmond. For years, their database team has had access to the operating system source code. Gives a huge advantage to support and performance. Oracle not only gains the source code to Solaris, but also their hardware too. No company in my 22 years in IT has had access to everything - hardware, operating system and the database. It's going to be a wild ride....

    The other company I think could be greatly impacted is Kickfire. Now that Oracle has all that Sun hardware talent on tap, they could also design a database appliance as well.

    It's not all gloom and doom.

    I see great opportunity for 3rd party companies that support MySQL. Some of these companies have former MySQL employees on staff too, and have already started contributing code to MySQL with their own patches and enhancements. This could be an opportunity for a consortium of companies to come together and support MySQL. Something like the Eclipse Foundation might work. I could also see David, Monty and Marten coming together again under a new company. Plenty of opportunities exist.

    What advice can I give? First thing - don't panic. Your MySQL database servers are not going to stop working tomorrow. I'd research 3rd party companies like Percona and Pythian. Be diligent, don't wait until Oracle announces something. Like any great DBA, be proactive.

    This is the third post in the weekly series "Last Week in Drizzle" where we summarize the efforts of various folks in the Drizzle community over the past week. This edition covers Aug 18th through the 24th. As with the week before, a number of developers and community advocates continue to refactor the code base, come together in discussions on the mailing list, and brainstorm on how to solve the tough problems that Drizzle is trying to address. It sounds like many in the community have been swamped this week, but there's still plenty to report. Jay Pipes and myself are tag teaming Last Week in Drizzle.

    Continued Growth in the Drizzle Community

    Drizzle mailing list has 191 members, up from 148 last week As I type this, there are 42 folks hanging out on the #drizzle Freenode channel. The Drizzle wiki has been steadily growing in its content, there have been a number of entries added or updated:

    * Updated FAQ
    * Simple Replication
    * Table Definitions

    Ongoing Conversations on the Mailing List

    We've had less Bikeshed stuff this past week. Biggest topics discussed this week:



    Community Blogs

    Jay Pipes has written up two guides about
    Getting a Working C/C++ Development Environment for Developing Drizzle and A Contributor's Guide to Launchpad.net - Part 1 - Getting Started

    Domas Mituzas blogs about his thoughts on Drizzle and modularity.

    Brian "krow" Aker writes: Removal of code for views, some code removed for unireg and definitions. Table objects also created.

    Drizzle Build Farm

    Still seeking additional Build Bots. You can also find out realtime status about our buildfarm on irc #Drizzle. Details at http://ronaldbradford.com/blog/interacting-with-buildbot-using-irc-2008-08-18/

    From last week:

    Ronald has continued to champion the Drizzle Build Farm, and writes the following update and request to the community:

    The Build Farm has helped us in identifying problems but we are still seeking people to contribute different platforms to our automated process become better.

    Currently we cover the following platforms

    - Ubuntu 8.04 - 32 & 64 bit - Debian 5 - 32 & 64 bit - Gentoo 8 - 32 & 64 bit - CentOS 5 - 64 bit - Fedora 8 - 32 & 64 bit - SUSE 11 - 32 bit - Mac OS/X 10.5 - 64 bit

    See the wiki for how to contribute to the Drizzle Build Farm.


    Final Words

    That wraps up this week's entry. My apologies to anyone I missed in this edition. Feel free to add errata and additions to the comments of the entry and I will update the blog post accordingly.


    I recently purchased a new Dell 1525 laptop running Ubuntu 8.04. I've been in need of a new laptop for about a year, and decided to take the plunge and see how good a Linux based laptop from Dell really is. My needs are fairly modest. I'm not a gamer, and don't watch many videos. Mainly it's used for blogging, internet, OpenOffice, IRC/IM, listening to podcasts and programming. Portablity is only somewhat important, mostly it'll be used 80% on a desk with the remaining time mobile.

    Model: Dell Inspiron 1525N
    Price as reviewed: $749
    CPU: Intel Core 2 Duo T7250 (2 Ghz/800 Mhz FSB/2MB Cache)
    OS: Ubuntu 8.04 with DVD playback
    Screen: Glossy 15.4" widescreen (1280x800)
    Video: Intel Graphics Media Accelerator X3100
    Memory: 3GB Shared Dual Channel DDR2 at 667Mhz
    Hard Drive: 120GB SATA (5400 RPM)
    Combo Drive: CD/DVD Writer (DVD+/-RW)
    Wireless: Intel 3945 a/g Mini-card
    Camera: N/A
    Battery: 6 cell Li/Io
    Sound: High Definition Audio 2.0
    Network: Integrated 10/100
    Weight: ~6 pounds

    Initial Impressions

    I ordered this laptop on a Tues and according to Dell, it was going to take 7-10 working days then 3-5 days shipping before I'd see my new laptop here in Southern California. Much to my surprise, it shipped two days later, and arrived at my home two more days after that! I'd say Dell is very conservative in estimating dates. For once, it was great to see a laptop that didn't have all kinds of extra packaging. Dell packaged the laptop with a very minimum amount of packaging material. My box arrived via DHL intact. The contents of the box were very minimal as well, only including the power supply and Ubuntu DVD. A quick start guide and manual was also included, but not really referenced. It's a laptop after all, there's not much to put together! The included manual was for Windows OS only, there were no directions on using Ubuntu.

    Upon power up, it was cool to see the Ubuntu splash screen. It went through some hardware configuration, found my wireless network and I was online in under 10 minutes from opening the box! A very flawless start! Ubuntu is installed in mostly a default configuration, and after a few minutes of being online, it had almost 700MB of updates for me to download. I transferred my data from my aging desktop system on a USB thumbdrive to the Dell. I was 100% working off the Dell in about 30 mins from the time I opened the box. The 700MB of updates would wait until I went to bed before downloading and updating.

    Performance

    Performance is very subjective, but for my needs, the 1525N has been great. Firefox 3.0.1 performance has been very acceptable. OpenOffice also has very acceptable load times too. I downloaded the latest branch of Drizzle and compiled it in about 8 mins. Under normal use, the CPU fan isn't running very fast, but during the compile, it was running at fullspeed. Graphics has been very good for my needs. Watching the occasional video, it was crisp with good colors. The Intel GMA runs at 500Mhz and uses up to 384MB of shared RAM. Using that amount of system RAM, it's best to install as much RAM as you can afford. I rank installing the most RAM you can afford as the #1 upgrade you should do to this laptop. Upgrading to a faster CPU or harddisk may not be a great return on your investment. YYMV of course.

    I tested out the DVD writer by creating a restore disk using the Dell/Ubuntu Rescue DVD. I used standard Memorex DVD-R disks, and it burned about 1.8GB of data in around 3 minutes. I made two DVDs, and when trying to burn the second disk, it took a few ejects before the system would recognize a blank DVD was inserted.

    Wireless has been flawless. My WAP is a Buffalo WHR-G54S flashed with dd-wrt. The laptop hasn't dropped the link at anytime, unlike my work HP/Compaq 6710b which drops WIFI every few mins.

    Sound is a tad weak, it's not very loud. The speakers are upward facing, and Ubuntu responds to the hardware volume controls just fine. Sound quality is what you'd expect from built-in speakers, kinda rough. Plugging in headphones is the high quality option.

    The touch pad is just OK. The default settings are ,very sensitive, causing text selection to occur when that's not the intended action. Double clicking by double tapping doesn't work reliably either. This could be due to Ubuntu settings, something I haven't investigated yet.

    Keyboard has a nice tactile feel to it. I use the Dvorak layout, so I might see about rearranging the keys from QWERTY layout. By default, the backspace and delete keys are not configured to autorepeat.

    Battery life is about what I expected. I'm seeing around 2 hours and 45 mins runtime with the 6 cell battery. I haven't made any configuration changes to Ubuntu for power saving modes, this run time is based on the stock Ubuntu configuration and "general" useage.

    Not Tested
    Conclusion

    For less than $750, this is a fantastic laptop. I'm very happy with it. My other choice was an IBM Thinkpad T61, but I'm glad I went with the Dell. I've only had the laptop about 5 days now, but it's performed flawlessly for my needs. I think the Dell/Ubuntu laptops are a great combination. Only a few small negatives, but they are not a major concern for this laptop. I think Dell should include some kind of Ubuntu quick start guide instead of the manual that ships. If you're already experienced with Ubuntu, this won't be such a big issue, it's more of a concern for users that are new to Ubuntu.

    The Drizzle Report is a weekly synopsis on drizzle development. This first report will contain more than just a weeks worth of development in order to catch up.

    What has been removed from the initial MySQL source tree can be found at the MySQL Differences wiki page. Some of the highlights were the removal of the mysql database, TINY/MED/LONG BLOB, TINY/MED/LONG TEXT thespatial data types, and FULLTEXT indexes. Certain keywords were removed too like ENGINES, CLIENT and CONTRIBUTORS. Even drizzleadmin has been stripped down to just ping and shutdown.The long-standing MySQL ACL has been ripped out, and initial PAM authentication has been started.

    Monty Taylor reports RIP: errmsg.sys, in its place is gettext which is a standard for outputting strings.

    Comparing drizzle to SQLite has been commonplace. An initial attempt at comparing the two architectures can be found on the Drizzle compared to SQLite wiki page.

    Drizzle features have been gleaned from Brian Aker's OSCON 2008 video and are now listed on the Drizzle Features wiki page. Brian has also been interviewed on FLOSS Weekly #35 and Linux Cast. Be sure to listen in.

    More information on Drizzle can be found on the Drizzle wiki. IRC and mailing list are also available. If there's something additional you'd like to see in the Drizzle Report, just let me know.

    Not really, it's about 90 degrees out and sunny here near San Diego. For a couple of years, I've been looking around to volunteer on an OSS project, and Drizzle really fits the bill. I've used MySQL for years, and have done some community support as well. I've been wanting to gain greater experience in technical writing, so Drizzle is the perfect project.

    I'll be working on development & user documentation, and the weekly Drizzle Report. The Drizzle Report is a weekly synopsis on development and other progress to be posted on my blog Sunday evenings.

    Be sure to check out the Drizzle Wiki. Launchpad is the home of the Drizzle project.

    Speaking at the MySQL Conference 2010

    Posted by: Toru Maesaka, February 18, 2010 09:31 AM

    I’m a little behind in announcing this but I’m going to be speaking at O’Reilly’s MySQL Conference this year. My presentation is a three hour tutorial titled, Drizzle Storage Engine Development. Practical Example with BlitzDB. Three hours is a long time but I assure you that there will be a break.

    This session isn’t solely about going through Drizzle’s Storage Engine API. Various performance topics like B+Tree structure, memory handling and concurrency control will be covered. I will also go through BlitzDB’s design concept and it’s internal stuff. So, needless to say I’ll talk a lot about Tokyo Cabinet and it’s internals as well.

    Hopefully those that come along will walk out of the tutorial standing far ahead of the start line. It will help you get started on reading the implementation of other storage engines in the MySQL ecosystem (MyISAM, InnoDB, PBXT, Federated and so forth). Better yet you will start writing one.

    Looking forward to seeing you there :)

    MySQL Conf & Drizzle Dev Day

    Posted by: Eric Day, February 17, 2010 10:41 PM

    I’m glad to announce that we’ll be having a Drizzle developer day again this year on the Friday after the MySQL Conference! Be sure to sign up and add any topic ideas you may have so we know what folks are interested in. Space is limited!

    While at the MySQL Conference, I’ll be speaking with Monty Taylor on “Using Drizzle.” This will take a non-developer approach to the project, so everyday DBAs and web developers should find this interesting. I’ll also be teaming up with Giuseppe Maxia to talk about Gearman in three sessions. These include:

    We’re also going to have a combo Drizzle/Gearman booth in the expo hall, so be sure to stop by and chat. See you there!

    Drizzle Developer Day 2010 is coming

    Posted by: lbieber, February 15, 2010 06:45 PM

    We are very happy to announce Drizzle Developer Day is coming again this year. Similar to last year it will be the Friday after the MySQL users conference, April 16 – 9:30 a.m. to 5:00 p.m. We are still working on the exact location and will let you know as soon as possible, most likely it will be at or very close to the Santa Clara convention center. We invite anyone interested in providing feedback, implementing new features, helping to fix bugs, or just wanting to learn more about Drizzle to attend.

    Please sign up at http://drizzle.org/wiki/Drizzle_Developer_Day_2010_signup. Space will be limited.

    Also, to make this day a success, please provide your input on discussion topics at http://drizzle.org/wiki/Drizzle_Developer_Day_2010

    Hope to see you there!
    -Lee

    New Job at Akiban

    February 06, 2010 08:00 AM

    I just finished my first week at my new position as a software engineer at Akiban Technologies in Boston.
    I'm really excited about working here. Akiban is a small startup developing some really cool technology that I believe will get people talking about the relational model in a good way again. We are currently based in the South End of Boston. The building where we are located is pretty awesome and not at all what I pictured an office to be like. There is a resident artist in the building who hangs his paintings on the walls and they seem to move to different places at random times. Its a strange feeling to walk in to work in the morning and smell fresh paint as I go to my desk. Definitely not something I expected!
    But besides all that, one of the best things for me about working here is that I get paid to contribute to open source. I've been pretty involved with Drizzle for the last year while still a student and it was always something I really enjoyed which I never thought someone would pay me to work on. The community around the project is awesome and I was just happy to be involved with it. Now that I get paid to contribute, it's nice to know that I can still be part of that community without having to worry about how I'm going to make a living. It's weird to be paid for something that I would still be doing anyway without the pay! I'm not complaining though, it's a nice change!
    I'll be presenting at the MySQL conference in April, lots of awesome work is happening in the Drizzle project and Akiban will be out of stealth mode by the conference so there are some exciting times ahead!
    Linux Conf AU 2010

    Posted by: Eric Day, February 04, 2010 12:54 AM

    I was really excited when I had my Gearman talk accepted to Linux Conf AU 2010 because I had never been out that far in the Pacific (only Hawaii). Of course it wasn’t in Australia this year, and instead in Wellington, New Zealand. My wife came too, and we also made a vacation out of the down times we had around the conference. It turned out Brian couldn’t make it this year so Monty, Stewart, and I gave the Drizzle talk. It was great to see some familiar faces, including Mark Atwood, Giuseppe Maxia, Josh Berkus, and Selena Deckelmann. Josh actually ended up being on the same flight out, so we got to catch up while going through New Zealand customs at 5am after a 13 hour flight. :)

    New Zealand is an amazing place. We flew in and out of Auckland and took the train to Wellington. The train ride mostly consisted of grazing sheep once out of the metro areas, did you know there are more sheep than people in NZ? Beyond the sheep, there were great views along the way, especially in the middle near the larger mountains and volcanoes. We stopped for a day to hike the Tongariro Alpine Crossing. It was sunny when we started, but it it was raining with 40mph winds at the top, so we didn’t get to see as much as we hoped. There were still beautiful views on each side though.

    The conference was very well run, thanks to anyone who had a hand in it! The speakers dinner was at this great museum nearby on the waterfront and included live Maori singing and dancing. The vegan options were tasty, and I got to meet a few interesting folks there (like the folks from Dreamwidth, a LiveJournal-like blogging service). Some notable sessions during the conference were “The World’s Worst Inventions” by Paul Fenwick, “Anti-features” Keynote by Benjamin Mako Hill, “The Hydras GCC Static Analysis Plugins” by Taras Glek, and “Simplicity Through Optimization” by Paul McKenney. There were many other great sessions, and some I wish I could have attended.

    I’m certainly going to try to go again next year, which if you didn’t hear will be in Brisbane!

    Fun with Table Functions

    February 03, 2010 08:33 PM

    I just about have all of the INFORMATION_SCHEMA replaced with Table Functions!

    The big wins:
  • One Execution path (less bugs)
  • Simple interface, which means more langauges
  • Zero materialization happening
  • Less Code. This allows us to remove a lot of code (and single shot passes for particular use cases).

    The data dictionary operates entirely off the proto system, so what you see is what we have. We use the table names stored within the proto so no translation ever happens. This is pretty handy for filesystems which do not preserve case (and we don't have to do anything to support them any longer).

    You can also type "SELECT * FROM DATA_DICTIONARY.SCHEMAS".

    There is no longer a "SCHEMATA" tables, just SCHEMAS. Want INDEXES? SELECT FROM INDEXES.

    We will also be able to split up the tables that are from the SQL ANSI standard one from the ones that are not. The advantage is that we can provide an ANSI compliant INFORMATION_SCHEMA, and still have plenty of room to add additional tables as we see fit (yes, like the tables we provide today which house the information on the active Memcached cluster information).

    drizzle> use data_dictionary;
    Database changed
    
    drizzle> select * from plugins;
    +-------------------------------------------+-----------------------+-----------+-------------+
    | PLUGIN_NAME                               | PLUGIN_TYPE           | IS_ACTIVE | MODULE_NAME |
    +-------------------------------------------+-----------------------+-----------+-------------+
    | ARCHIVE                                   | StorageEngine         | TRUE      | TRUE        | 
    | ascii                                     | Function              | TRUE      | TRUE        | 
    | benchmark                                 | Function              | TRUE      | TRUE        | 
    | BLACKHOLE                                 | StorageEngine         | TRUE      | TRUE        | 
    | char_length                               | Function              | TRUE      | TRUE        | 
    | character_length                          | Function              | TRUE      | TRUE        | 
    | CHARACTER_SETS                            | TableFunction         | TRUE      | TRUE        | 
    | COLLATIONS                                | TableFunction         | TRUE      | TRUE        | 
    | COLUMNS                                   | TableFunction         | TRUE      | TRUE        | 
    | compress                                  | Function              | TRUE      | TRUE        | 
    | connection_id                             | Function              | TRUE      | TRUE        | 
    | console                                   | Listen                | TRUE      | TRUE        | 
    | crc32                                     | Function              | TRUE      | TRUE        | 
    | CSV                                       | StorageEngine         | TRUE      | TRUE        | 
    | default_replicator                        | TransactionReplicator | TRUE      | TRUE        | 
    | drizzle_protocol                          | Listen                | TRUE      | TRUE        | 
    | Error_message_stderr                      | ErrorMessage          | TRUE      | TRUE        | 
    | FunctionEngine                            | StorageEngine         | TRUE      | TRUE        | 
    | GLOBAL_STATEMENTS                         | TableFunction         | TRUE      | TRUE        | 
    | GLOBAL_STATUS                             | TableFunction         | TRUE      | TRUE        | 
    | GLOBAL_VARIABLES                          | TableFunction         | TRUE      | TRUE        | 
    | hello_world                               | Function              | TRUE      | TRUE        | 
    | INDEX_PARTS                               | TableFunction         | TRUE      | TRUE        | 
    | INDEXES                                   | TableFunction         | TRUE      | TRUE        | 
    
    
  • Drizzle build 1273 source tarball has been released

    Posted by: lbieber, January 28, 2010 08:40 PM

    Drizzle source tarball based on build 1273 has been released. This marks completion of our Bell milestone. We made a lot of great progress and improvements during the Bell milestone, one of the main goals was to make sure data loss does not occur so that active testing can now occur. While Bell has been completed, upgrading will continue to require dump/load , so hopefully we will nail that down soon and let you know when it is ready.

    Stay tuned for more information on goals and features for the our next major milestone which will be code named Cherry.

    The Drizzle download file and change log can be found here

    Tungsten 1.2.2 Release is Out - Faster, More Stable, More Fun

    Posted by: Robert Hodges (noreply@blogger.com), January 28, 2010 07:01 AM

    Release 1.2.2 of Tungsten Clustering is available on SourceForge as well as through the Continuent website.  The release contains mostly bug fixes in the open source version but there are also two very significant improvements of interest to all users.
    • The manager and monitoring capabilities of Tungsten are completely integrated on the same group communications channel.  This fixes a number of problems that caused data sources not to show up properly in older versions.  
    • We are officially supporting a new Tungsten Connector capability for MySQL called pass-through mode, which allows us to proxy connections by transferring network blocks directly rather than translating native request protocol to JDBC calls.  Our tests show that it speeds up throughput by as much as 200% in some cases. 
    The commercial version has additional features like PostgreSQL warm standby clustering, add-on rules to manage master virtual IP addresses and other niceties.   If you are serious about replication and clustering it is worth a look.

    This is a good time to give a couple of reminders for Tungsten users.  First, Tungsten is distributed as a single build that integrates replication, management, monitoring, and connectivity.   The old Tungsten Replicator and Myosotis builds are going away.   Second, we have a single set of docs on the Continuent website that covers both open source and commercial distributions.

    With that, enjoy the new release.  If you are using the open source edition, please post your experiences in the Tungsten community forums or write a blog article.  We would love to hear from you.

    P.s., We have added Drizzle support thanks to a patch from Marcus Eriksson but it's not in 1.2.2.  For that you need to build directly from the SVN trunk.  Drizzle support will be out in binary builds as part of Tungsten version 1.3.
    Notes on HEAP/MyISAM Index Key Handling on WRITE

    Posted by: Toru Maesaka, January 26, 2010 08:57 AM

    Disclaimer: This post is based on HEAP/MyISAM’s sourcecode in Drizzle.

    Here are my brief notes on investigating how index keys are generated in HEAP and MyISAM. I lurked through these because I’ve started preparing for decent index support in BlitzDB. I also wrote this to assist my biological memory for later grepping (I have terrible memory for names). I’m only going to cover key generation on write in this post. Otherwise this post is going to be massive.

    HEAP Engine

    The index structure of HEAP can be either BTREE or HASH (in MySQL doc terms). Like other engines HEAP has a structure for keeping Key definition (parts, type, logic and etc). This structure is called HP_KEYDEF and it contains function pointers for write, delete, and getting the length of the key. These function pointers are assigned to at table creation or when the table is opened. The assigned function depends on the data structure of the index and it can be either of the following:

    BTREE

    • hp_rb_write_key()
    • hp_rb_delete_key()

    HASH

    • hp_write_key()
    • hp_delete_key()

    As for get_key_length(), either of the following functions are used for both data structures.

    • hp_rb_var_key_length()
    • hp_rb_null_key_length()
    • hp_rb_key_length()

    When writing a row to the tree, HEAP writes to the index using a key generated by hp_rb_make_key(). Note that it does not use this for the hash index. The generated key is populated inside ‘recbuffer’ in HEAP’s handler object (HP_INFO structure).

    From my understanding, it loops through the key segments (I suspect it is similar the internal KEY_PART_INFO structure) and appropriately copies each key field value to the output buffer. By meaning “appropriately” it respects the characteristics of the data type when packing the buffer. For example, for a variable length field, it will only copy the actual data and not the max possible size of it. The final byte that is copied to the buffer is the address of the chunk where the record lives.

    MyISAM Engine

    The upper layer of key handling in MyISAM looks somewhat similar to HEAP so you can really tell that it was written by the same people. Things are nicely wrapped together by the MYISAM_SHARE structure so it’s relatively easy to follow. BlitzDB has a class called BlitzShare for the same purpose (This is based off Archive Engine’s ArchiveShare class).

    Like HEAP, MyISAM has a structure for individual key definition called MI_KEYDEF (it’s defined in myisam.h). There are more function pointers in this structure than HEAP.

    • bin_search()
    • get_key()
    • pack_key()
    • store_key()
    • ck_insert()
    • ck_delete()

    In Drizzle, _mi_ck_write() is assigned to ck_insert() which is the entry point to writing a MyISAM index. The key that MyISAM uses to write to the index is generated by _mi_make_key(). Like HEAP, it will loop through the key segments and pack the relevant fields accordingly to the characteristic of the data type. The output buffer belongs to MyISAM’s hander (lastkey2).

    From Here

    I’ve actually written a naive key generator for BlitzDB already based on Drizzle/MySQL’s internal KEY_PART_INFO array. It seems to be working on EXACT MATCH but I still need to implement an index scanner which looks much harder to pull off than a table scanner. What I’m really worried about is supporting composite indexes (namely reading/searching on it) but hopefully I’ll understand how this area of the storage system works soon.

    Replicating transactions directly to RabbitMQ

    Posted by: Marcus Eriksson (krummas@gmail.com), January 25, 2010 03:43 PM

    Previously RabbitReplication tailed the transaction log provided by Drizzle and then the Java application sent the protobuf serialized transaction to RabbitMQ. Now it is possible to skip the transaction log file and send the transaction directly to the RabbitMQ server without the extra step of storing it in a file first.

    The code is available at https://code.launchpad.net/~krummas/drizzle/rabbitmq_log and to build it with rabbitmq support you need to install librabbitmq which is a bit tricky;

    Installing librabbitmq

    1. Install mercurial
    2. Branch the librabbitmq code: hg clone http://hg.rabbitmq.com/rabbitmq-c/ 
    3. Branch the rabbitmq codegen repo into a subdirectory called codegen in the rabbitmq-c directory:
      1. cd rabbitmq-c
      2. hg clone http://hg.rabbitmq.com/rabbitmq-codegen/ codegen
    4. Run autoconf like this: autoreconf -i 
    5. Run the configure script
    6. make
    7. make install
    Build Drizzle
    When librabbitmq is installed, build drizzle like this:

    1. bzr branch lp:~krummas/drizzle/rabbitmq-log
    2. config/autorun.sh
    3. ./configure --with-rabbitmq-log-plugin
    4. make
    5. make install
    and it is done! 

    Start Drizzle with RabbitMQ support
    First, you can run drizzled with a --help flag to see the options available, they are all prefixed with --rabbitmq-log-XYZ. 

    The default values for the parameters makes drizzle connect to localhost as "guest" and replicate to an exchange called ReplicationExchange. Start it like this to replicate changes to a rabbitmq on localhost:
    $ sbin/drizzled --default-replicator-enable --rabbitmq-log-enable

    The other available options are described in --help
    From Scratch, Get set, Go!

    January 22, 2010 09:37 PM

    [brian@gaz fix_is]$ ./drizzled/drizzled --console-enable 
    InnoDB: The InnoDB memory heap is disabled
    InnoDB: Mutexes and rw_locks use GCC atomic builtins.
    100122 13:12:00  InnoDB: highest supported file format is Barracuda.
    100122 13:12:00 InnoDB Plugin 1.0.4 started; log sequence number 44600
    Listening on 0.0.0.0:3306
    Listening on :::3306
    Listening on 0.0.0.0:4427
    Listening on :::4427
    ./drizzled/drizzled: Forcing close of thread 0 user: ''
    ./drizzled/drizzled: ready for connections.
    Version: '2010.01.1273' Source distribution (fix_is)
    
    drizzled> use data_dictionary;
    OK
    
    drizzled> show tables;
    Tables_in_data_dictionary	
    MODULES	
    PLUGINS	
    PROCESSLIST	
    
    drizzled> desc PLUGINS;
    Field	Type	Null	Key	Default	Extra	
    
    PATH ./data_dictionary/plugins
    ID	bigint	YES		NULL		
    USER	varchar(16)	YES		NULL		
    HOST	varchar(64)	YES		NULL		
    DB	varchar(64)	YES		NULL		
    COMMAND	varchar(16)	YES		NULL		
    TIME	bigint	YES		NULL		
    STATE	varchar(64)	YES		NULL		
    INFO	varchar(100)	YES		NULL		
    drizzled> 
    
    
    
    Using libnotify for error messages

    Posted by: mordred@inaugust.com (Monty Taylor), January 21, 2010 12:34 AM

    Any good piece of infrastructure software should be able to pop up little windows on your desktop. :)

    Yesterday at LCA in Stewart's Hacking Drizzle talk, it occurred to me that error messages should pop up little windows on my desktop. So I wrote an error message plugin which uses libnotifymm to send Gnome pop-up window messages.

    I don't always get to post screenshots, so here you go:

    Screenshot of libnotify error message 

    Machines Plus Minds - Welcome

    Posted by: Mark Atwood (me@mark.atwood.name), January 20, 2010 08:37 PM

    Welcome!

    I write about the techne that is the amazing machine we call "The Net".

    At present, I am mostly interested in Memcached, the Drizzle DB fork of MySQL, NoSQL, and in open standards, but I will be writing about other stuff as well.

    I used to do my public geek blogging in my personal LiveJournal, but for various reasons it makes sense to separate my public writings about technology, my other public writings, and writings that are of interest only to my closer friends and family.


    My day job title is "Director of Community Development" for Gear6, which means that my main paid interest is memcached and the memcached community.  I read and write technical articles, research nosql databases and other web-scale open source software, and go to conferences to attend and to speak.


    You have already lost

    Posted by: Stewart Smith, January 18, 2010 02:43 AM

    When the following code introduces a valgrind warning… you are in a world of pain and loss:

    === modified file 'drizzled/field/blob.h'
    --- drizzled/field/blob.h	2009-12-21 08:16:13 +0000
    +++ drizzled/field/blob.h	2010-01-18 01:36:48 +0000
    @@ -32,6 +32,7 @@
      */
     class Field_blob :public Field_str {
     protected:
    +  uint32_t assassass;
       uint32_t packlength;
       String value;				// For temporaries
     public:
    Drizzle to Infinispan replication and a small code walkthrough

    Posted by: Marcus Eriksson (krummas@gmail.com), January 16, 2010 08:39 PM

    This post aims to explain how to build your own key/value-store applier in RabbitReplication by walking through the new Infinispan support as an example.

    Infinispan
    Infinispan is a "distributed, highly available data grid platform" and it exposes a REST interface where it is possible to manipulate the data in infinispan. For example it is a simple HTTP PUT method to store new data and a HTTP DELETE does exactly what you expect. The data is stored under resources, for RabbitReplication the data is stored under /<schema.table>/<key>. This means you can view the data in infinispan using a browser.


    When I implemented the Infinispan support, I simply dropped the .war file in the webapps directory of a jetty installation and started it. Since Infinispan has a REST interface, the client library can be any HTTP client, I picked the Jersey REST client since it is incredibly easy to use.


    Customizing RabbitReplication
    RabbitReplication uses Guice internally for dependency injection, so, to use another KeyValue store you need to create your own Module for configuration. I'll show an example below.

    To add support for a new key/value store, you need to implement an interface, org.drizzle.replication.transformer.objectstores.KeyValueStoreClient (here). It is a quite straight-forward add/get/remove interface. Look at the infinispan implementation (here) for an example. Note the @Inject on the constructor, it tells guice that the WebResource parameter should be injected when it constructs the object. You will need to put the rabbitreplication.jar file on your classpath when building your stuff.


    Guice is configured in modules where you bind() an interface to an implementation, so, to configure guice to use a new KeyValueStore, we need to bind() the KeyValueStoreClient interface to the new implementation. Look at the infinispan module (here) for an example how to do it, the method annotated with @Provides is called by guice when it needs to create a KeyValueStoreClient.

    To tell RabbitReplication to use your new module, you simply edit your configuration file, and set where the new Module is located, see this example. If you need to configure your new client, simply add the properties in the config file, guice will bind every property in the file to @Named(...) strings, check the Infinispan module provider method for an example how to use it.

    Now, build your code into a jar, drop the jar in the lib/ directory of rabbitreplication and start RabbitReplication like this; bin/start.sh config/someconf.properties - the config file should have the custom_module set to the name of your new module.

    Downloading and installing
    Best way to use rabbitreplication is still to branch the code from launchpad (lp:rabbitreplication) and then write ant in the base directory, this will create a .zip and a .tgz in the dist directory. You can also download the binaries here.

    • Unpack the distribution file
    • Copy a config file from .sample to .properties in the config directory and edit the file to your liking. objectslave.properties.sample is the sample you want to look at if you want to try out Infinispan.
    • Start rabbitreplication by executing bin/start.sh config/yourconf.properties
    Look at the previous posts on rabbitreplication to find out how to start a master etc.
    Further thoughts on BlitzDB’s Index Handling

    Posted by: Toru Maesaka, January 15, 2010 09:05 AM

    I’ve been thinking quite a bit about collation handling in BlitzDB for the last couple of days. The more I think about it, the more stuck I’ve been getting with BlitzDB’s index design. I’m actually so frustrated with myself at the moment that I want to hit my head against a wall or something.

    So, I’m writing this entry to clear up my mind. Heh, my blog is slowly becoming BlitzDB’s design document draft. This should hopefully be good though since by blogging it, people can tell me whether I’m moving towards a stupid direction or not.

    Collation Importance

    When writing database software that is intended for International use, it is important to handle textual data by respecting collation order. It is arguable that most people are only interested in English lexicographic ordering but unfortunately the world is not so standard.

    Internal Primary Key Handling

    I want to motivate people to actively define a PRIMARY KEY with BlitzDB. I plan to make this attractive by providing the best performance when PK is defined. In December 2009, my answer to this was to write the PK value as the key for the data dictionary (where actual rows are stored in BlitzDB). This allows BlitzDB to do a direct lookup on the data dictionary for PK based lookup, instead of consulting the B+Tree index. I’m still fond of this lookup optimization approach but it introduces problems too.

    Problem 1. Consider the following textual keys: “key” and “KEY”. They obviously have different binary representations but in certain cases they can be logically equivalent. Because the data dictionary is a hash database, this is a problem. The solution that instantly pops up is to normalize the key before writing or reading it. This however, causes a problem in cases where the two keys are inequivalent. Perhaps Drizzle/MySQL provides an internal normalization function that respects this. I still need to study this area of the storage subsystem.

    Problem 2. Directly writing a PK to the data dictionary means fast lookup but because of the data structure, it’s not possible to fetch the next “logical” key, meaning I can’t implement index scanning on PK as it is. Quick solution for users is to create an index on the PK column (this would create a separate B+Tree for it) but this is not so friendly because it requires the user to have prior knowledge of all this. So, my plan is to provide the best of both worlds. I’ll elaborate on how I’m planning on tackling this problem next.

    Current Primary Key Read/Write Behavior

    In general, keys of BlitzDB’s data dictionary is a unique 8 byte integer. The idea is that BlitzDB writes this unique ID along with the key to the B+Tree Index so that it can later identify that row. The difference with PK is that, if a PK is present in a table, BlitzDB will not generate an internal unique ID and use PK for the data dictionary’s key instead. BlitzDB won’t create a B+Tree index for PK at the time I wrote this blog entry.

    Next Step

    Create a B+Tree index for PK anyway. BlitzDB will still use the PK value as the key for data dictionary if it exists. For PK based lookup requests, BlitzDB will look directly at the data dictionary and for PK based requests that involve index scanning, BlitzDB will look at the B+Tree index.

    This approach can consume more space when textual data is used for keys but I think it’s worth it. At the same time, you can save space if you use use types that are smaller than 8 bytes for PK. For example, using a 4 byte integer would reduce BlitzDB’s key space by 50%.

    Hmm, I think my mind has cleared a little.

    Drizzle, BlitzDB and HTON_STATS_RECORDS_IS_EXACT

    Posted by: Toru Maesaka, January 13, 2010 01:23 PM

    Recently I enabled HTON_STATS_RECORDS_IS_EXACT in BlitzDB to let the optimizer know that BlitzDB can instantaneously return the number of rows in a specified table. As a result, the Drizzle kernel can directly call the Cursor::info() function to get the row count. To users, it means that SELECT COUNT statements can be executed in O(1). So it’s a great thing in general.

    Something Broke

    After I enabled HTON_STATS_RECORDS_IS_EXACT, I noticed that issuing SELECT statement on a table with 1 row would no longer return a resultset. Weird indeed! after investigating with GDB, I noticed that rnd_next() is only called once instead of twice on a table with 1 row (second time is to find EOF) when HTON_STATS_RECORDS_IS_EXACT is enabled. This makes sense because the kernel knows that there is only 1 row and therefore it doesn’t need to keep scanning for EOF. However, this made me scratch my head since this shouldn’t break BlitzDB’s table scanner.

    Remedy

    Logically, I was confident that BlitzDB’s table scanner was functioning properly so I decided to look at what was going on beyond the engine API. Turns out that join_read_system() in sql_select.cc looks at the table->status value and decides that it’s an error if 0 isn’t assigned to it. What’d you know? I realized that I wasn’t assigning anything to the status variable. It’s more that I didn’t know that I was meant to update an internal structure. You’d think that engine developers aren’t meant to touch those. It’s not mentioned in the Engine Documentation at MySQL Forge either. Nevertheless, the important thing is that it works now. Oh and SELECT COUNT is fast now too.

    Eye Opener

    This experience among other occasions where I had to read the kernel’s source made me think that it would be nice to provide an intensive up to date documentation on how to develop storage engines for Drizzle in the future (when the API becomes stable). Needless to say, this would be co-ordinated within the Drizzle community. I’m not a license person but it should hopefully be provided with a freely available license too.

    Drizzle build 1263 and libdrizzle 0.7 source tarballs have been released

    Posted by: lbieber, January 12, 2010 10:27 PM

    For Drizzle build 1263 this release includes continuing general code clean up, bug fixes and improvements as we get closer to our Bell milestone. The Drizzle download file and change log can be found here

    For libdrizzle version 0.7 this release includes:

  • Added test coverage reports using lcov
  • Updated autoconf build system
  • Updated RPM packaging
  • The libdrizzle download file and change log can be found here
  • Documentation for libdrizzle can be found at API and Development
  • Moving On

    Posted by: Eric Day, January 11, 2010 07:06 PM

    Friday was my last day at Sun Microsystems, and today is the first day at my new job (location coming soon). I’ve had a great time at Sun, and thank them for all the opportunities given to me there. I’ll be doing mostly the same work at the new gig, working on projects like Drizzle, but with a slightly different focus. For the most part my day-to-day won’t change much.

    Right now I’m focusing on libdrizzle again and am implementing the prepared statement API, cleaning up the MySQL protocol support a little, and also implementing the new Drizzle client/server protocol. I’ll continue to work on Gearman as well, especially where it is relevant to Drizzle. I also need to start blogging again with specific topics in the projects I’m working on, I’ve been fairly quiet lately.

    I’ll be in New Zealand next week at Linux Conf AU (yes, it’s not in AU this year). I have a talk on Gearman, and it looks like I’ll also be helping out with the Drizzle talk. It will be really nice to escape the Portland, OR winter for a bit. :)

    Same Bat-time, Same Bat-channel

    Posted by: mordred@inaugust.com (Monty Taylor), January 11, 2010 06:31 PM

    Friday was my last day working for Sun Microsystems, which means it was also the end of the job that started when I was hired by MySQL, Inc a little over four years ago. For those of you who didn't know me before that, it should be noted that the other than periods of time when I have started companies and worked for myself, the next longest I've ever been with one company was 10 months. I mention that as one of the few quantifiable metrics I could cite in describing what a wonderful experience it has been to work for MySQL and then Sun. The people I've worked with have been nothing short of stellar.

    I will be continuing my work as a core Drizzle developer, so this blog post notwithstanding, I doubt most people will notice the change. The only real outward difference I can think of is that now that I'm not a Sun employee, I am free to comment on the merger and on what some people see as the need to save MySQL from Oracle. I don't know if I will, but at least now I can.

    I would say I will miss everyone, but I imagine I will continue to see most people with about the same frequency as before. One of the wonderful things about the Open Source world is that changes in business and legal entities don't often have very much effect on the coding that we do day in and day out... so let's keep up the good work everybody! See you on IRC...

    The future of open source SQL databases (as I see it)

    Posted by: mike, January 10, 2010 06:58 AM

    With the whole MySQL/Oracle issue going on, I find myself looking into the future and how I see it. As far as I'm concerned, MySQL will start to lose it's popularity as the landscape changes. As far as I am concerned, there will be two key players in the MySQL replacement market, those being Drizzle and MariaDB.

    I am not just saying Drizzle just because I help out with the project in various ways, however, that should be a good sign that I believe in it if I am willing to put any effort into it. With people behind it like Brian Aker, Eric Day, Monty Taylor, Stewart Smith, Jay Pipes, you've got a coding powerhouse that could solve the cancer issue if it was up to software development to fix it. These guys work around the clock and have been refactoring and re-examining everything inside of MySQL. What's going to be left ideally is a superfast microkernel that supports plugins for everything - leveraging the best options out there for replication, messaging, storage engines, etc. Growing apart from the monolithic huge distribution model that MySQL currently follows.

    The second key player is MariaDB. Another fork off of MySQL, led by Monty Widenius himself and with other MySQL key players behind it, there is no doubt it will continue Monty's legacy as being able to spin success out of a tiny little open source product. I believe it will stay more traditional in-line with MySQL, but will provide more advanced functionality and scalability as it is developed further.

    I won't get into other options like PostgreSQL as I don't follow the rest of the community there much.

    Also, we'll see more NoSQL (did we ever bottom out on a better term for that?) options. CouchDB and MongoDB (both of which from a 50,000 foot view look identical from a usage model) and options like Cassandra will also become important and your data needs will become the decision maker for going with a SQL or a NoSQL database. Both of which offer advantages. However, I see Drizzle as making huge strides in leveling the playing field (or attempting to) with it's replication work to make it as scalable as NoSQL databases seem to be with their ability to scale out and replicate changes easily (which to me are their main selling point right now...)

    Anyway, this is from a user perspective, not a developer perspective, and from what I've seen from #drizzle on freenode, a few SQL and open source conferences, blog talk and my own gut feelings.

    I should make a note that I still use MySQL and will probably continue for some time. Neither Drizzle nor MariaDB are production-friendly yet. However, I believe 2010 should see the first "production capable" release of Drizzle (not sure of MariaDB.)

    It is an exciting time though as we're starting to be presented with more options by the day, in fact there are so many various NoSQL databases now, key/value stores, and even a few more SQL databases that it's too hard to keep track of them anymore. There's a lot of code being written and with this whole Oracle possibly inheriting MySQL depending on the EU's judgement, it could ultimately help usher in some of these smaller projects into the spotlight quicker depending on what Oracle does with MySQL...

    Multi threaded replication appliers

    Posted by: Marcus Eriksson (krummas@gmail.com), January 08, 2010 08:58 PM

    Lately I've been working on a transaction reducer to be able to multi thread the applier. Basic idea is to reduce the transaction to only affect one row with one statement, when that is the case, we can have a thread pool doing the actual persisting of the statements (of course it has some drawbacks as well, more about those later). This approach is particularly interesting for NoSQL appliers.

    Reducing transactions
    The transaction log in drizzle contains a list of statements. Each statement contains a list of records, where each record contains information about what changed on one row in the master. One example transaction could look like something like this:


    transaction_context {
      server_id: 1
      transaction_id: 10
      start_timestamp: 1262812100381445
      end_timestamp: 1262812153799963
    }
    statement {
      type: INSERT
      start_timestamp: 1262812100381446
      end_timestamp: 1262812134317464
      insert_header {
        table_metadata {
          schema_name: "unittests"
          table_name: "test1"
        }
        field_metadata {
          type: INTEGER
          name: "id"
        }
        field_metadata {
          type: VARCHAR
          name: "test"
        }
        field_metadata {
          type: VARCHAR
          name: "ignored"
        }
      }
      insert_data {
        segment_id: 1
        end_segment: true
        record {
          insert_value: "78"
          insert_value: "a"
          insert_value: "b"
        }
        record {
          insert_value: "79"
          insert_value: "a"
          insert_value: "b"
        }
    ...
        record {
          insert_value: "87"
          insert_value: "a"
          insert_value: "b"
        }
      }
    }
    statement {
      type: UPDATE
      start_timestamp: 1262812134317466
      end_timestamp: 1262812151669387
      update_header {
        table_metadata {
          schema_name: "unittests"
          table_name: "test1"
        }
        key_field_metadata {
          type: INTEGER
          name: "id"
        }
        set_field_metadata {
          type: VARCHAR
          name: "test"
        }
      }
      update_data {
        segment_id: 1
        end_segment: true
        record {
          key_value: "85"
          after_value: "test"
        }
      }
    }
    statement {
      type: DELETE
      start_timestamp: 1262812151669389
      end_timestamp: 1262812153799963
      delete_header {
        table_metadata {
          schema_name: "unittests"
          table_name: "test1"
        }
        key_field_metadata {
          type: INTEGER
          name: "id"
        }
      }
      delete_data {
        segment_id: 1
        end_segment: true
        record {
          key_value: "81"
        }
      }
    }

    Or in SQL:
    BEGIN;
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (78, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (79, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (80, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (81, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (82, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (83, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (84, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (85, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (86, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (87, "a","b");
    UPDATE unittests.table1 set test = "test" WHERE id = 85;
    DELETE FROM unittests.table1 WHERE id = 81;
    COMMIT;


    I.e. a number of inserted rows, one updated and one deleted. If we could exploit this and make sure that every record in the transaction only affects one row then each row is totally independent from all other records in the transaction and we could then have a pool of threads applying the transaction.

    So, if i reduce the above transaction like this:

    TransactionReducer reducer = new DrizzleTransactionReducer();
    TransactionMessage.Transaction txn = getNextTransaction();
    TransactionMessage.Transaction t = reducer.reduce(txn);

    I get this transaction:


    transaction_context {
      server_id: 1
      transaction_id: 10
      start_timestamp: 1262812100381445
      end_timestamp: 1262812153799963
    }
    statement {
      type: INSERT
      start_timestamp: 1262812100381446
      end_timestamp: 1262812134317464
      insert_header {
        table_metadata {
          schema_name: "unittests"
          table_name: "test1"
        }
        field_metadata {
          type: INTEGER
          name: "id"
        }
        field_metadata {
          type: VARCHAR
          name: "test"
        }
        field_metadata {
          type: VARCHAR
          name: "ignored"
        }
      }
      insert_data {
        segment_id: 1
        end_segment: true
        record {
          insert_value: "78"
          insert_value: "a"
          insert_value: "b"
        }
        record {
          insert_value: "79"
          insert_value: "a"
          insert_value: "b"
        }
        record {
          insert_value: "80"
          insert_value: "a"
          insert_value: "b"
        }
        record {
          insert_value: "82"
          insert_value: "a"
          insert_value: "b"
        }
        record {
          insert_value: "83"
          insert_value: "a"
          insert_value: "b"
        }
        record {
          insert_value: "84"
          insert_value: "a"
          insert_value: "b"
        }
        record {
          insert_value: "85"
          insert_value: "test"
          insert_value: "b"
        }
        record {
          insert_value: "86"
          insert_value: "a"
          insert_value: "b"
        }
        record {
          insert_value: "87"
          insert_value: "a"
          insert_value: "b"
        }
      }
    }
    Or, in SQL:
    BEGIN;
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (78, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (79, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (80, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (82, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (83, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (84, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (85, "test","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (86, "a","b");
    INSERT INTO unittests.table1 (id, test, ignored) VALUES (87, "a","b");
    COMMIT;

    I.e. a list of only inserts. This means we can apply the transaction using several threads. Another benefit is that we reduce the total number of records to insert meaning better performance.

    Updates and deletes that affect rows outside the transaction are also reduced, for example if, within one transaction, one external row is updated twice, then deleted, only the delete statement will be executed.

    Drawbacks
    There are, of course some drawbacks, for example, if this is used when applying to a database, the applier will not be able to apply the transaction as a transaction since there is no way of sharing transaction context between several client threads.

    Using it
    To try it out, branch this repository: lp:~krummas/+junk/transactionreducer and look at the tests file. Should be straight forward to add more tests to see how it handles them.

    It will also be available in RabbitReplication as a configuration option on the slave. The size of the thread pool and number of client connections will also be configurable.

    Performance
    In theory it should be faster to apply fewer statements using more threads, and the time spent reducing the transaction should easily be less then the time spent doing network I/O etc. I've done a few non-scientific benchmarks using a multi threaded cassandra applier and it takes approximately half the time applying reduced transactions using the multi threaded applier. I will make some proper benchmarks when everything is in place in RabbitReplication.
    Describing Drizzle's Development Process

    Posted by: nospam@example.com (Jay Pipes), January 06, 2010 04:17 PM

    Yesterday, I was working on a survey that Selena Deckelmann put together for open source databases. She will be presenting the results at Linux.conf.au this month.

    One of the questions on the survey was this:

    How would you describe your development process?

    followed by these answer choices:

    • Individuals request features
    • Large/small group empowered to make decisions
    • Benevolent dictator
    • Other, please specify:____________

    I thought a bit about the question and then answered the following in the "Other, please specify:" area:

    Bit of a mix between all three above.

    The more I think about it, the more I really do feel that Drizzle's development process is indeed a mixture of individuals, groups, and a Benevolent dictator. And I think it works pretty well. :-) Here's some of the reasons why I believe our development process is effective in enabling contributions by being a mix of the above three styles.

    Who's the Benevolent Dictator of Drizzle?

    First, let me get the BDFL question out of the way. We've made a big deal in the Drizzle community and mailing lists that anyone and everyone is encouraged to participate in the development process — so why would I say that Drizzle has a benevolent dictator?

    Well, although he would probably disagree with the tile of BDFL, Brian Aker does have some dictator-like abilities with regards to the development process, and rightfully so. Brian came up with many of the concepts that Drizzle aspires to be, and Brian has more experience working on the code base than any other contributor.

    After having worked closely with Brian now for 18 months or so, I can definitively say that Brian's brain works in a very, well, interesting way. Those of us who work with him understand that sometimes his brain works so fast, his typing fingers struggle to keep up, resulting in something I call "Krowspeak". It's kinda funny sometimes trying to translate :-)

    With this wonderfully unique noodle, Brian tends to knock out large chunks of code at a time, and often he wants to push these chunks of code into our build and regression system and into trunk to see the results of his work quickly. Sometimes, this can cause other branches to get out of sync and get merge conflicts, and Brian will inform branch owners of the conflicts and work with them to resolve them.

    So, regarding dictator-like development processes, I suppose we have Brian acting as the merge dictator because he's got a lot of experience and understands best how both his code and other's code integrates. We tried a little while back having myself and Monty Taylor be merge captains, but that distribution of merge work actually created a number of other problems and we've since gone back to Brian being the merge captain by himself, with Lee, Monty, and myself improving our automated build and regression system to help Brian with the repetitive work.

    That said, what Brian does not do is make decisions in a dictator-like way. Decisions about the code style, reviews, features, syntax changes, etc are made on the mailing list by consensus vote. If a consensus is not reached, generally, no change is made which would depend on the decision. Brian does not influence the direction of the software or the source code style any more than anyone else on the mailing list which expresses an opinion about an issue; and for this, I greatly respect his wisdom to seek consensus in an open and community-oriented way.

    Groups Empowered to Make Decisions

    I'm assuming that what Selena's "large/small group empowered to make decisions" answer meant was what is sometimes called "Cabal Leadership" of a project. In other words, there is some group which steers the project and makes decisions about the project which affect the rest of the project's contributors.

    Drizzle has at least one such group, the Sun Microsystems Drizzle Team, which is composed of Brian, Monty Taylor, Lee Bieber, Eric Day, Stewart Smith, and myself. One might call us the core committers for Drizzle.

    However, while the Sun Drizzle team certainly is empowered to guide development, it is no different than any other group of developers that choose to contribute to Drizzle. There isn't a "what the Sun Drizzle team decides" rule in effect. Our "power" in the development process is no greater or less than any other group of contributors. We act merely as a team of individuals who work on the Drizzle code and advocate for the project's goals.

    Individuals Empowered to Make Decisions

    One thing I've been impressed with in the past 18 months is how the Drizzle community has embraced the opinions and work of individual contributors. I believe Toru Maesaka, Andrew Hutchings, Diego Medina and Padraig O'Sullivan were among the first individuals to begin actively contributing to Drizzle. Since then, dozens of others have joined the developer and advocate community, with each individual carving out a piece of the source code or community activities that they want to work on.

    I have learned much from all these individuals over the last year or so, and I've tried my best to share knowledge and encourage others to do the same. Our IRC channel and mailing list are active places of discussion. Our code reviews are always completely open to the public for comments and discussed transparently on Launchpad, and this code review process has been a great mixing bowl of opinion, discussion, learning and debate. I love it.

    More and more we have developers showing up and taking ownership of a bug, a blueprint, or just a part of the code that interests them. And nobody stands in their way and says "Oh, no, you shouldn't work on that because <insert another contributor's name> owns that code." Instead, what you will more likely see on the lists or on IRC is a response like "hey, that's awesome! be sure to chat with <insert another contributor's name>. They are interested in that code, too, and you should share ideas!" This is incredibly refreshing to see.

    In short, the Drizzle developer process is a nice mix of empowered individuals and groups, and a dash of dictatorship just to keep things moving efficiently. It's open, transparent, and fun to work on Drizzle. Come join us :-)

    2010, Life

    January 05, 2010 09:42 PM

    Things that happened in 2009:

  • We got a major chunk of MySQL refactored into Drizzle. Thanks to Jay we have a new replication system that we are shaking out. Monty Taylor has created an impressive bit of porting kung-fu that has made porting cake. Eric has the the Listener plugin work in shape which makes NULL plugins, drizzled console, and the future "whatever a server side script is" cake to work on. We have a BSD based libdrizzle so the "well if you link like this..." license questions all go away, along with any historical FUD that has been created around them. Lee, Jay, and community have put together an impressive regression system which is far beyond anything we ever had for MySQL. Stewart killed FRM which has always been one of our "we could do this, if we could replace FRM...". We have full time developers now in multiple companies and contributions continue to come in.

  • It took three years but I completed my "lets see if we can be a bit healthier if we lose some weight". I started off in 2005 at about 182 and nowadays I stick to about 150. What changed? I cut out eating bread in large part and I switched to eating when I felt like I needed to eat. I get a lot of "you look healthier" or "were you sick?". It is amazing how people see weight loss. People's reactions can really be across the board. For me though? I rarely have issues with my asthma now and I have a lot more energy in general.

  • I spent more time in Seattle then I have since 2005. I am still traveling, but I made a point of really trying to live in Seattle and be here more. This spring a friend commented on my being around by saying "It has really been great to see you at events again". Comments like that made it completely worth the effort I have been putting in to be here.

  • Completed the "puppy acquisition". I am always happiest when I have a dog in my life. I've been watching Rosalynd get old and while I know she has a few more years in her, I really wanted to keep a constant flow of dogs in my life.

  • We built the Worm Tube. For Burning Man this year I worked out a 2800 square foot structure that weighs under 500 pounds and cost less then $800 to build.

  • Libmemcached is alive and well. Its user base continues to grow and we did a lot of exciting things. It is now distributed in all major Linux distributions. We have a new library that wraps the Memcached protocol. Using it you can create add Memcached compatible existing services to existing projects. Schooner provided a new memslap in December that greatly enhances testing.

    What about 2010?

  • Put more effort into personal relationships. Something I walked away from Burning Man this year was a need to enhance the personal relationships I have. Focus on what matters and put more effort into the core of my social group/family/etc. There are some awesome people in my life and I should be spending a considerable amount of my time nurturing the relationships with them.

  • Is Drizzle production yet? Whenever I ask an audience if anyone is using Drizzle in production I get a few hands up in the air. I asked one group "what does production mean to you?". I got back an answer of "will we lose data?". My answer was, "if you trust Innodb today, then you are fine". We have our bugs and there are issues at hand but I feel like the gap is narrowing on that question. This year we will shake out the replication system, and it should be interesting to see what all happens there. We have seen plugins being written to replicate to many of the NoSQL solutions and I believe the flexibility that we have put into the system will solve a lot of the mixed shop problems that we run into. A database is not an island unto itself, it is just another piece of infrastructure and it needs to play well with other pieces. I believe this year we will see a Windows Binary. I would like to say we would double the number of major contributors we have in multiple companies, but I can't keep track any more of where all people come from. I should find this out so that I can create a metric :)

  • For Libmemcached we will see a restructuring of its guts to do more asynchronous work. We have a few years under our belts now of knowledge and we can improve on performance. I am going to be adding support for range queries, which is specified in the protocol. I am hoping to see at least one vendor adopt the protocol library. I believe we should finally support Windows directly out of the box. I have patches now that will allow us to do more centralized management of Memcached clusters. I am looking forward to pushing those. Some of this work will be done in the current version, but I expect to work on some of this in a version 2.0. We hit trillions of transactions some time ago, so claiming 2.0 seems humble enough :)

  • Raise puppy. This is not a small amount of work.

  • Build the Monorail :)

  • Spend about the same amount of time in Seattle. I want to take a couple of trips overseas this year but I want to spend less time in general away, especially weekends.

  • Be a better advocate for Open Source, both for the projects and the business models. There is a lot of FUD that has been spread over the years and I want to spend a bit of time being more vocal squashing it. We have seen coalition's built around cripple source concepts, license BS, developer model hijinks, etc... I spent a couple of hours in one country just explaining the differences between open source and free software to some government administrators after a certain luminary confused the hell out of them. There is no advantage too damning one license at the expense of another.

  • Advocate for open source databases. One thing I have been doing in the last year has been trying to help people pick a database. There are a lot of good open source databases right now and none of them are a one size fit all solution. It has been refreshing to make suggestions or provide answers based on need. When we worked out the original FAQ for Drizzle we made a point of mentioning different databases, I was very happy about that.
  • 2009 and looking forward to 2010

    Posted by: Marcus Eriksson (krummas@gmail.com), January 01, 2010 01:30 PM

    So, this is yet another 2009 retrospect with some goals for 2010, I'll do it in list form so someone might actually browse it;

    2009:
    • Main event of 2009 was that i got kid nr 2, Teo.
    • Ran 1000K despite injuries
    • Started Drizzle JDBC.
    • Got excited about programming again and realized I need a new job
    • Got Drizzle JDBC to version 0.6, not many users yet, so can't say much about the quality (or, one could look at it from another angle, it is bug free! *cough*)
    • Went to JavaOne, great stuff, probably the last one.
    • Started a very long parental leave!
    • Started working a lot on replication-related stuff:
    • "Learned" Haskell and started looking at Erlang.
    • Read some great books; Java Concurrency In Practice, Real World Haskell, Effective java (like every year) ...
    2010 goals:
    • Continue my long, sweet parental leave (living in Sweden has its benefits)
    • Run 1500K
    • Get myself a new job, main requirements:
      • Most importantly, has a high paced startup-feel to it, I want to build stuff, not have meetings about the stuff we could build
      • Uses open source products
      • Contributes stuff back to open source communities (or, best of all, has some open source products of their own)
    • Make RabbitReplication into a proper product
      • Create web page
      • Make regular releases with good documentation
      • Build support for more column/key-value stores
      • Document how to extend it
      • Clean up the code
    • Get Drizzle-JDBC to "1.0" with someone using it in production (given drizzle makes it there). I'm guessing that when Drizzle itself is production ready, the user (and bug-) count will increase.
    • Learn at least one new programming language (Erlang, I'm looking at you), and build something with it.
    • Go to at least one conference
    • Blog more, November and December frequencies have been ok, mainly because I actually had time to build stuff worth blogging about.
    • Read more books (current reading queue: Erlang programming, SICP, Reread Distributed Systems - Concepts and design, Java Generics and Collections)
    • Invent a clock with 30 hrs a day to actually manage all the above goals
    Drizzle now running dbt2 benchmark

    Posted by: lbieber, December 31, 2009 05:19 PM

    We recently added support for running the dbt2 benchmark as part of our drizzle automation suite. dbt2 is an OLTP transactional performance test. It simulates a wholesale parts supplier where several workers access a database, update customer information and check on parts inventories. We currently are using the defaults (10 warehouses, 5 minute test runs and running various number of connections for each run up to 1024. The initial runs exposed a race condition in our TemporalFormat::match() code at 1024 connections.

    We now have a very nice suite of tools to help us with tracking performance and scalability of Drizzle, besides dbt2 we also are running sysbench, sqlbench, crash-me and randgen for all of our builds in the staging branch of Drizzle. Of course we are always looking for more, so if you have any suggestions on other benchmarks or tools to add, please let us know.

    If you want to receive all of the various benchmark results you can subscribe to the mailing list.

    Note also we are in the process of getting our changes to dbt2 merged to the dbt2 trunk, it should be there very soon.

    Happy New Year to all!

    -Lee

    Drizzle replication to WebSockets

    Posted by: Marcus Eriksson (krummas@gmail.com), December 28, 2009 10:03 PM

    I just pushed a WebSocket applier to RabbitReplication, and yes, it is as crazy as it sounds. It works pretty much like all appliers - it consumes drizzle transactions from  RabbitMQ, converts them into objects by inspecting annotations, marshalls the object to JSON, and then stores the JSON string. In this case it stores it to a set of websockets. RabbitReplication is deployed as a war file to Jetty 7.01 which supports websockets.


    I set the demo up on my server at home (in Sweden) on a DSL line, so it might be slow, but it should show the idea, all operations are instant when latency is low (if anyone wants to host it at a better place, please let me know). Of course, it requires a WebSocket capable browser and the only one I know of is Google Chrome.

    It works like this:

    1. INSERT is executed from the "drizzle client" webapp - totally separate webapp that uses drizzle jdbc to insert/update/delete data.
    2. Drizzle stores the transaction in the database and in the transaction log.
    3. Master extractor extracts the transaction and publishes it to RabbitMQ
    4. Slave applier consumes the transaction from RabbitMQ
    5. Applier transforms the transaction to JSON
    6. Applier writes the JSON to a set of websockets
    7. Javascript voodoo is performed to make it visible
    Possible real usecases
    The demo app just shows what is possible, but a real use case could be that someone has a drizzle backed forum and want to add some real time post-updates to some front page somewhere. This would be real easy, simply start a new slave configured for WebSocket application (of course RabbitReplication is already used for other replication needs :) ), convert the JSON to something that makes sense and they are set! If someone has a cool usecase, please let me know and i'll build a more realistic demo app!
    new drizzle low-hanging-fruit milestones

    Posted by: mordred@inaugust.com (Monty Taylor), December 24, 2009 11:16 PM

    I've got some code in lp:drizzle/staging right now that's on its way (barring major catastrophes) to trunk. It's not code that does anything sexy as far as the actual running server is concerned. It's a code cleanup branch.

    Anyway - short story being - everything from mysys and mystrings that is actually part of public APIs has been moved into drizzled/ proper. Everything else has been moved into drizzled/internal. None of the headers from drizzled/internal are installed... so none of the headers in drizzled/ should be using any of them. Combine this with the past week's removal of both server_includes.h and global.h, and we're getting pretty close to having fully consumable headers.

    Which brings me to:

    In doing this, I noticed a bunch of things that either need to be fixed, still need to be deleted, or need to be put behind a namespace so that including our headers doesn't strangely and unexpectedly b0rk someone's application. To that end, I've added a bunch of new low-hanging-fruit blueprints. As always, you can look at the full list on Launchpad but I'm including links to the new ones here.

    New Drizzle Low Hanging Fruit:

    http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-consolidate-charset-headers

    http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-merge-decimal

    http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-merge-error

    http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-move-sql-alloc

    http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-types-into-namespace

    http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-mem-root-namespace

    http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-internal-namespace

    http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-remove-myisam-depends

    http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-remove-my-files

    http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-remove-sql-list

    End of Year Progress on BlitzDB

    Posted by: Toru Maesaka, December 24, 2009 09:47 AM

    FURTHER UPDATE: Further thoughts on BlitzDB’s Index Handling

    My open source friends might have noticed that I’ve been working quite a bit on BlitzDB lately. To tell the truth, I had a hidden goal to get Version-1 done by Christmas. Unfortunately it doesn’t look like I can reach that goal. However, looking at the brightside I got a lot done in the past few weeks so allow me to “journal” it in this blog post.

    Agony of Knowing

    The more I understood Drizzle’s storage mechanism and Tokyo Cabinet’s internals, the more I disliked what I previously had. This led me to spending quite a bit of time rewriting BlitzDB’s codebase. I was using pthread’s rwlock for concurrency control but I decided to design and write BlitzDB’s own lock mechanism to get the best out of TC (in terms of concurrency). I also rewrote the entire table scan code which is something you’d hope won’t be executed that often (people should use indexes!) but needless to say, it’s an important component of a relational storage engine so I’ve put in a lot of effort there.

    Rewriting the Table Scanner

    In the process of rewriting the table scanner, Jay Pipes’ gave me a fantastic advise on using Drizzle’s internal atomic type (drizzled::atomics). He gave me this advise because he noticed that my atomic ID generator was securing atomicity with pthread’s mutex. It is debatable that this mutex was only enabled for only few CPU instructions but the philosophy of using the most efficient method on the platform where BlitzDB is to be run was appealing enough for me to use drizzled::atomics. Mikio did some experiments on this and found that in a competitive/congested environment, using the compiler’s builtin function can gain you 3x throughput.

    Hacking on Index Support

    I’ve finally started hacking on index support and I just finished supporting basic operations on a primary key. By design, BlitzDB’s index is a dense clustered b+tree but in the first release I am going to limit PK to only be a HASH index. This is because I want BlitzDB to treat all PKs as direct keys inside the data dictionary (hash database where the actual rows are stored). So in other words, I want people to use PK for “needle in a haystack” like queries only. An example of a needle in a haystack like query is:

    SELECT * FROM TABLE WHERE primary_key_column = whatever;

    Saying that, I don’t like to force people to do things the way I like so I plan on providing best of both worlds by supporting both data structures for PKs in Version-2:

    CREATE TABLE t1 (id int, PRIMARY KEY(id) USING btree) ENGINE=blitzdb;
    CREATE TABLE t1 (id int, PRIMARY KEY(id) USING hash) ENGINE=blitzdb;

    BlitzDB’s default configuration will use PK as a “direct” data dictionary index. If you wish to do range queries on PK, the solution is to create a index on the PK column.

    Primary Key lookup Performance

    So, how does my implementation perform? Here’s a quick benchmark with a test-run that randomly fetches 100 thousand rows from a BlitzDB table with 1 million rows. This is the table I used:

    CREATE TABLE t1 (id int PRIMARY KEY, a int, b int) ENGINE=blitzdb;

    and the query looks like this:

    SELECT * FROM t1 WHERE id = random_number_under_one_million;

    The hardware I used is the following commodity server: Intel Quad Xeon E5345 (2×4MB L2 cache), 8GB Memory, 500GB SATA II. Unfortunately I could not prepare a standalone client server today so both the server and the test program were run on the same machine. Yeah… this sucks so I can’t claim that this benchmark is 100% creditable.

    Here is the result I obtained from skyload. Please only view it as a guideline to BlitzDB’s lookup performance. I’ll do a proper benchmark with the Drizzle Community and publish it after I get Version-1 released.

    [ READ LOAD EMULATION RESULT ]
      SQL File               : 100k_select.sql
      Concurrent Connections : 1
      Task Completion Time   : 5.88856 secs
      Number of Queries:     : 100000
      Number of Test Runs:   : 1
     
    [ READ LOAD EMULATION RESULT ]
      SQL File               : 100k_select.sql
      Concurrent Connections : 2
      Task Completion Time   : 6.94474 secs
      Number of Queries:     : 100000
      Number of Test Runs:   : 1
     
    [ READ LOAD EMULATION RESULT ]
      SQL File               : 100k_select.sql
      Concurrent Connections : 4
      Task Completion Time   : 7.04455 secs
      Number of Queries:     : 100000
      Number of Test Runs:   : 1

    As you can see, “needle in a haystack” queries can be executed pretty efficiently in BlitzDB. Looking at the first result, we can observe that it took an average of 0.058 milliseconds to process a query.

    Future Plans

    Admittedly, primary key support isn’t completely done so I’ll continue working on it. After that, I will start hacking on b+tree indexes and write more tests as I go. Once I support at least two indexes, I’ll ask the Drizzle Community to consider merging BlitzDB into Drizzle’s trunk. This is my goal for BlitzDB at the moment.

    I also happen to own blitzdb.com so I’m planning on putting user documentation (including tutorial) and architectural notes there. This is currently not so high on my TODO list so I suspect it won’t happen until I get Version-1 released. All I can say about the release schedule at the moment is, “before the MySQL conference in april”.

    So, that’s all I have to summarize for now. Thanks for reading this far. Merry Christmas and have a Happy New Year. Don’t trip on ice :)

    Getting Drizzle Started -or- “Know thy Path!”

    Posted by: Kent Bozlinski, December 23, 2009 05:12 PM

    After being gone a while for the day job I have finally gotten around to setting up my storage file locations and permissions in Drizzle. Stuart Smith helped me get past some problems, mostly associated with not knowing basic syntax, but I also ran into one or two that are due to changes ongoing in Drizzle. After spending a lot of time not knowing what I didn’t know (one of the quirks of teaching yourself), a little chat finally straightened me out.

    So after Drizzle is installed on Ubuntu you obviously have to have a place to store your data. To avoid all the time we spent figuring out what was going on, just pay attention to the last lines after you run your “make install” so you know where the damn thing is.

    Run your sudo command then add your user group where drizzle is the name of the group you want to have access to the drizzle server:

    $ sudo groupadd drizzle

    then create a new user within that group, replacing USER with your user name.

    $ sudo useradd -g drizzle USER

    After running this procedure a couple times I found that on one install the directory was created for me, on another I had to create it. Either way, make sure you have a new directory for drizzle and then a data directory in that:

    $ sudo mkdir /home/USER/drizzle
    $ sudo mkdir /home/USER/drizzle/data

    Now set up the permission on that directory for your user:

    $ sudo chown -R drizzle:USER /home/drizzle/data

    Finally start up the Drizzle Server with:

    $ sudo -u USER /usr/local/sbin/drizzled --datadir=/home/USER/drizzle/data

    You should get a bunch of lines of code saying the Drizzle server and InnoDB are now running and what ports it is listening too. Now you can start up the Drizzle Client:

    sudo -u USER /usr/local/bin/drizzle

    and you should see:

    Welcome to the Drizzle client.. Commands end with ; or \g.
    Your Drizzle connection id is 1
    Server version: 2009.12.1251 Source distribution (drizzle)
    Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
    drizzle>

    I found that if I installed this on a server that already had MySQL running on it, the start up command would return an error where the InnoDB (a storage engine drizzle uses) was already using the port that Drizzle wanted to use. I had to shut down MySQL on port 3306 to get it up and running. Mark Atwood helped me get past that issue.

    Everything seemed to work much cleaner installing on a clean system but that I guess is to be expected. I don’t know all the ways Drizzle will overlap and conflict with other programs and I expect MySQL is the first and most frequent one I will notice.

    Drizzle build 1251 source tarball has been released

    Posted by: lbieber, December 22, 2009 09:23 PM

    For Drizzle build 1251 this release includes:

  • Support for building out of tree plugins
  • Now using ICC compiler on some build machines to clean up even more warnings!
  • The Drizzle download file and change log can be found here
  • Better replication from drizzle to cassandra

    Posted by: Marcus Eriksson (krummas@gmail.com), December 16, 2009 07:47 PM



    Introduction
    This article describes how one of the replication appliers work in rabbitreplication, namely the HashTransformer which transforms each INSERT/UPDATE into a hashmap which is then stored in a column-family based storage, currently Cassandra. For a better overview of RabbitReplication, go check out earlier posts on the subject here: http://developian.blogspot.com


    Configuration
    This example replicates changes done to a table called test1 in the schema called unittests. RabbitReplication is configured to only replicate the columns id and test (yes, good example, I know...). The column id is used as a key. The following slave configuration is used for this use case:


    replication_role = hashslave


    rabbitmq.host = 10.100.100.50
    rabbitmq.queuename = ReplicationQueue
    rabbitmq.exchangename = ReplicationExchange
    rabbitmq.routingkey = ReplicationRoutingKey
    rabbitmq.password =
    rabbitmq.username =
    rabbitmq.virtualhost =
    hashstore.host = localhost:9160
    hashstore.type = cassandra


    hashreplicator.replicate.unittests.test1 = id,test
    hashreplicator.key.unittests.test1 = id
    hashreplicator.keycolseparator.unittests.test1 = .


    The hashreplicator rows are the interresting ones, they describe what columns to replicate, what columns are the primary key and what separator to use between the columns when the key is multi column.


    Example
    Replicating an insert:
    drizzle> use unittests
    Reading table information for completion of table and column names
    You can turn off this feature to get a quicker startup with -A


    Database changed
    drizzle> desc test1;
    +---------+-------------+------+-----+---------+-------+
    | Field   | Type        | Null | Key | Default | Extra |
    +---------+-------------+------+-----+---------+-------+
    | id      | int         | NO   | PRI | NULL    |       | 
    | test    | varchar(10) | YES  |     | NULL    |       | 
    | ignored | varchar(10) | YES  |     | NULL    |       | 
    +---------+-------------+------+-----+---------+-------+
    3 rows in set (0.02 sec)


    drizzle> insert into test1 (id, test) values (300, "firstins");
    Query OK, 1 row affected (0 sec)


    Results in the following on the Cassandra side:
    cassandra> get unittests.test1['300']
      (column=test, value=firstins; timestamp=1260985298391)
      (column=id, value=300; timestamp=1260985298385)
    Returned 2 rows.


    Update:
    drizzle> update test1 set test = "updated" where id = 300;
    Query OK, 1 row affected (0 sec)
    Rows matched: 1  Changed: 1  Warnings: 0


    Gives this in cassandra:
    cassandra> get unittests.test1['300']
      (column=test, value=updated; timestamp=1260985526210)
      (column=id, value=300; timestamp=1260985298385)
    Returned 2 rows.


    Note that the timestamp for the id column is not updated (only changes are updated, not entire rows).


    Delete:
    drizzle> delete from test1 where id = 300;
    Query OK, 1 row affected (0 sec)
    And in Cassandra:
    cassandra> get unittests.test1['300']
    Returned 0 rows.


    That is it, go to http://launchpad.net/rabbitreplication to check out the code, report bugs or suggest features!