Planet Drizzle

Code coverage – now with branches!

2010-09-03T18:44:53+00:00

We have now upgraded our lcov testing to take advantage of lcov 1.9‘s branch coverage.

While code coverage isn’t the be-all-end-all of testing, it is a very useful tool in helping us target areas that could use more tender, loving care (by which I mean beating them mercilessly with our test tools). It doesn’t prove completeness of testing – it merely helps us move in that direction : ) The addition of branch-level coverage gives us another dimension to help us expand our testing. We’re also making use of the –demangle-cpp option to produce neater function names for the function-level coverage.

You can check out the updated reports here. We gather the code coverage of our test suite with every push to trunk and store the data for general analysis.

In other news, Brian has added two new functions for testing. These changes were a part of another blueprint to remove print_stack_trace and stack dump. While I haven’t had a chance to do anything with them yet, I think these will be very useful and look forward to seeing what kinds of tests our community will be able to cook up : )

Crash the server:

% ./drizzled/drizzled –plugin-add=crash_function

select crash();

Shutdown the server:

% ./drizzled/drizzled –plugin-add=shutdown_function

select shutdown()

Finally, I have been playing with the syslog plugin and it is awesome! It is easy to setup and is very easy to use. The data produced will be very useful in testing as well. An example produced via a randgen run for your enjoyment:

Sep 2 20:06:16 mahmachine drizzled[4340]: thread_id=4 query_id=200380 db=”test” query=”SELECT SUM( table1 . `col_int_key` ) AS field1 FROM c AS table1 RIGHT JOIN g AS table2 ON table1 . `col_varchar_1024` = table2 . `col_varchar_1024_key` WHERE table1 . `col_int` <> table1 . `col_int_key` ORDER BY field1″ command=”Query” t_connect=1283472376187915 t_start=486 t_lock=367 rows_sent=1 rows_examined=2 tmp_table=0 total_warn_count=0

Well, that’s it for now. We have some exciting work coming up and we look forward to seeing what kinds of awesome plugins are being developed out there. Now back to hacking.

Testing the data dictionary in a concurrent environment

2010-09-02T19:06:41+00:00

So, Brian wrote a bug the other day, asking me to do more testing of Drizzle’s data dictionary. Specifically, we wanted to look for how things behaved in a concurrent environment as this is often a killer for table functions / what people are most likely to forget.

What we came up with was the following plan:

1) Generate a test that only looked at data dictionary tables, with several users generating the same queries.

2) If step 1 looks good, we will slowly introduce background workloads (SELECT / UPDATE / etc) while we continue with the workload from step 1.

This resulted in a couple of new randgen grammars:

data_dict_concurrent_drizzle.yy – this grammar generates nothing but queries against the data dictionary tables. At present, these are mostly of the variety:

SELECT * FROM data_dictionary_table
SHOW PROCESSLIST | VARIABLE | TABLE STATUS | etc

This is designed to stress the data dictionary, either alone or with another randgen process generating a background workload.

proclist_subquery_drizzle.yy – this grammar is the same as optimizer_subquery_drizzle (generating *nasty*, subquery-heavy SELECTs), but also allows for SHOW PROCESSLIST commands. This is mainly designed to stress the server / PROCESSLIST. This grammar is nice as it is a single test that can just be run with several threads.

I am happy to report that in a data dictionary-only environment, the server was able to handle things very well. I was running up to 100 connections, 100k queries per connection and things looked good.

The other tests are another matter. While these are somewhat simple tests, they have proven highly effective so far:

Bug #627733: Crash in InnodbTrxTool::Generator::populate_innodb_locks (this=0x7f26140046f0) at plugin/innobase/handler/data_dictionary.cc:269
Bug #627742: Assertion failed – in drizzled::plugin::TableFunction::Generator::push (this=0x23876c0, arg=<value optimized out>, length=<value optimized out>) at drizzled/plugin/table_function.cc:185
Bug #628398: Crash / segfault in copy_fields (join=0x284f868, end_of_records=false) at drizzled/sql_select.cc:6228
Bug #628891: Crash / assertion failed – in drizzled::Diagnostics_area::set_eof_status (this=0x7f9f3c2c4258, session=0x7f9f3c2c3b10) at drizzled/diagnostics_area.cc:120

For each of these bugs, data dictionary queries were being executed while another query was also being processed. It should be noted that our newest team member, Andrew Hutchings, had Bug#627742 fixed in less than 24 hours : )

I still have a few more scenarios to run through, but it appears that we have shaken out most of the bugs in this area. Our next steps will be to install such tests in our build and test system to prevent regressions / catch new bugs and to fix the remaining crashes noted above.

Drizzle build 1742 tarball has been released

2010-08-31T20:03:40+00:00

Drizzle source tarball based on build 1742 has been released.

In this release:

*Continued work on logging statistics - added a table to the scoreboard that gives memory usage and size
print stack trace and stack dump have been removed. As we use gcc for a majority of our builds, we are now using the built-in backtrace ability. Added crash and shutdown functions for use in testing. See blueprint for more details
FOREIGN KEY info is now put into the table proto on CREATE TABLE, no longer just passed to the engine
SHOW CREATE TABLE now uses the table proto
Packages available for Debian and Ubuntu
Removed static instances from plugins
Continued code cleanup
Various bug fixes

The Drizzle download file can be found here

Welcome Andrew Hutchings to the Rackspace Drizzle team

2010-08-30T18:13:13+00:00

Please join me in welcoming Andrew Hutchings to the Rackspace Drizzle team. Andrew has been a very active community contributor to the Drizzle project so we are very excited to have him on board now working on Drizzle. Well ok, his first official day is tomorrow but he has been very busy with contributions over the past number of weeks!

Andrew lives in a small town in Northamptonshire, England with his wife and two sons. Known as LinuxJedi to many online, he is a community code contributor to several open source projects. He was previously a Senior MySQL Support Engineer for Sun/Oracle specialising in MySQL Cluster. He is also the co-author of a book on the MySQL Plugin API. You may have already seen his blog on Planet Drizzle.

Welcome to the team Andrew!

Drizzle Transaction Message Limit

2010-08-29T18:56:11+00:00

Some recent changes I made have recently been pushed to Drizzle trunk that affect the size of the Transaction protobuf message that any replication stream will see (e.g., the transaction log). This was necessary to fix bug 600795.

Without a Transaction message size limit, for any bulk operations, like LOAD DATA, we would have ended up with a Transaction message that could possibly contain a very large Statement message that contained all of the INSERT data for the bulk load. This obviously could eat up a large amount of memory if we kept allowing the Statement to grow without bounds. The Drizzle kernel, when it can, keeps appending the values to INSERT onto the same record.

To circumvent this, we now allow multiple Transaction records for a single database transaction. Each Transaction GPB message representing a single database transaction will all have the same transaction ID, and only the last Transaction message will have the Statement's end_segment attribute set to true.

Here is an example of this change that you might now see in the transaction log:

transaction_context {
  server_id: 1
  transaction_id: 3
  start_timestamp: 1283118092815781
  end_timestamp: 1283118092815869
}
statement {
  type: INSERT
  start_timestamp: 1283118092815782
  end_timestamp: 1283118092815868
  insert_header {
   table_metadata {
   schema_name: "test"
   table_name: "t"
   }
   field_metadata {
   type: INTEGER
   name: "id"
   }
   field_metadata {
   type: VARCHAR
   name: "a"
   }
  }
  insert_data {
   segment_id: 1
   end_segment: false
   record {
   insert_value: "2"
   insert_value: "abc"
   is_null: false
   is_null: false
   }
   record {
   insert_value: "3"
   insert_value: "def"
   is_null: false
   is_null: false
   }
  }
}

transaction_context {

server_id: 1

transaction_id: 3

start_timestamp: 1283118092816250

end_timestamp: 1283118092816725

}

statement {

type: INSERT

start_timestamp: 1283118092816251

end_timestamp: 1283118092816724

insert_header {

table_metadata {

schema_name: "test"

table_name: "t"

}

field_metadata {

type: INTEGER

name: "id"

}

field_metadata {

type: VARCHAR

name: "a"

}

insert_data {

segment_id: 1

end_segment: true

record {

insert_value: "4"

insert_value: "ghi"

is_null: false

}

record {

insert_value: "5"

insert_value: "jkl"

is_null: false

}

This example is a bit contrived as there is no need to split up such a small transaction, but you can see the basic changes here. We have two Transaction messages, both with the same transaction ID. You can see that the Statement's end_segment is set to false in the first message, while the Statement within the second Transaction message has end_segment set to true.

So, in case it isn't obvious, there are now two ways to determine when you should commit if you are a replication stream TransactionApplier, or if you are reading from the transaction log:

If the transaction ID changes, COMMIT.
Or, if the current Transaction has all Statement messages with end_segment set to true, COMMIT.

Choose which ever method of the two best suits your needs.

Currently, if a Transaction message crosses the 1M threshold, the kernel will create a new Transaction message. Why did I choose 1M? Well, the Google Protobuf documentation says:

Protocol Buffers are not designed to handle large messages. As a general rule of thumb, if you are dealing in messages larger than a megabyte each, it may be time to consider an alternate strategy.

So 1M seemed to be a reasonable default. I'll change this in the near future to be a configurable value once we get some changes to our sys var stuff merged.

Adding a Drizzle Executable in Xcode

2010-08-25T12:23:42+00:00

In my last post, I explained how to setup Drizzle under an Xcode project. This allows you to take advantage of Xcode's features while developing on Drizzle (or any other project of your own choosing). The one thing we weren't able to do was debug the Drizzle executable. This post remedies that.

So it turns out that this is an easy fix. But for Drizzle, there are a few extra hoops you have to jump through in order to get it to work.

The basic steps we need to do for Drizzle are:

Add a custom executable in Xcode
Setup any arguments you want to pass to the executable
Setup any environment variables needed for the executable to run properly

For most other projects, you can probably just get away with #1 and possibly #2. For Drizzle, though, we need to do #3 so that it can find its libraries.

Step #1 is easy. In the Groups & Files window, right click on Executables and then Add -> New Custom Executable...

This will pop up a window where you define where the executable resides. Once that is done, you can define what arguments to pass it, and what environment variables should be set when it runs, among other things. These should be self explanatory, and you should be able to set this up for your particular project. For Drizzle, though, it isn't so intuitive.

In Drizzle, after you run configure and make, the executable lives in a hidden directory within your xcode-branch repo directory. It will actually be in:

$drizzle-repo/xcode-branch/drizzled/.libs/drizzled

The executable you see in the xcode-branch/drizzled subdirectory is actually a shell script that runs the real executable for you. Don't ask me why. So enter the path to the executable in the .libs subdirectory:

Once you enter in the path and click Finish, you are given the chance to edit executable working directory, arguments, environment variables etc. The important part here is the information under the Arguments tab. We need to set the arguments to the executable as well as set the DYLD_LIBRARY_PATH environment variable so that the executable can find its dynamic libraries (otherwise, it looks for them in the installation directory, which if doesn't exist yet, will cause the executable to not start). Here is an example:

Once you have this setup, you should now be able to run your executable (after you've built it, of course), and use the Xcode debugger to merrily do some bug hunting. Make sure that you have unselected the Load symbols lazily option in Xcode Debugging preferences so that your breakpoints will be recognized!

Happy debugging.

WordPress and Drizzle

2010-08-25T10:30:17+00:00

I am a firm believer in practising what you preaching so since I am working in Drizzle my blog should work on Drizzle.

After a few small conversions of table schemas this WordPress blog is now running on a Drizzle database server! I will over the next few weeks migrate the rest of the sites I host as well as other server-side things to use drizzle instead of MySQL.

I also suggest watching this space, I will soon be working on a tool to make such migrations much easier.

Drizzle Clients on Windows - dirzzle and drizzledump

2010-08-24T13:09:00+00:00

All I have to share now is Drizzle client tools (both drizzle command line and drizzledump) for MS Windows are available for Download.
If you have time, please try it and give me feedback.
(Consider this as a temporary location for hosting the binary packages. In future, we shall move to sacred location.)

Kept postponing this post because there is a real dragon is under hatching (CoApp).
See Monty's post.
Help of those tools are required for moving forward because of our dependancy on list of libraries.
Current build is using MinGW.

All started with, Toru's question on my previous blog post. :)
Question was "Does this survive through the tests?" (libdrizzle patches on windows)
Answer is : This is the productive way I can test my libdrizzle patches for windows.

Patches for this windows build of clients are on the way to main trunk.
I am trying to keep the patches as small chunks to ensure proper review and required rework in this earlier stages itself.

What are your favorite MySQL bug reports?

2010-08-23T23:00:51+00:00

Bug reports can be fun. They can also be terrible. Either way they can be entertaining. On the Drizzle IRC channel today I saw a couple references to MySQL bug reports: it is stop working and Does not make toast (which reminds me of the Mozilla bug report about the kitchen sink). Got any other favourites¹?

¹ This one’s for Jay.

LinuxJedi, the new era

2010-08-22T18:38:14+00:00

I have recently moved on from my roll as a Senior MySQL Support Engineer at Oracle to a Drizzle Developer at Rackspace. As such my previous blog is now sitting idle so this is the replacement.

This new blog may have much less MySQL Cluster content, but I do hope to continue the same quality (I shall leave you to judge on how high that is!) of blog posts.

Where to find randgen testing information / documentation

2010-08-20T16:06:35+00:00

I know I write a lot about testing with the randgen. In case anyone is interested, I wanted to share some links to randgen documentation and information. It really is an amazing and powerful tool and has found many bugs in MySQL, MariaDB, and Drizzle.

It can generate queries faster than anyone could ever think of them. This provides breadth of input testing and allows for stress testing (multi-threaded scenarios are supported)
A number of Validators are available to determine query correctness. These include server-server comparison validators for testing result sets, execution time comparison validators for comparing query execution time, and others. Custom validators are implemented fairly easily
Grammar-based testing allows for faster generation of test queries. Rather than thinking of individual queries, one can focus on ‘patterns’ of queries and have the randgen generate hundreds of queries for the same amount of time and effort.

With that little sales pitch out of the way, here are the links to documentation.

MySQL Forge – general documentation
Drizzle wiki – information on how to run randgen tests and what to look for
DrizzleQA – links to information about all Drizzle testing

I will be writing more about how to use this tool in the future, but I’d definitely welcome feedback on what people are interested in. The tool has applications for testing patches for regressions, validating upgrades based on current workloads, stress testing, etc. – feel free to let me know what you’d like to learn about. Discussion is always welcome – stop by IRC (#drizzle) or drop us a line on our mailing list.

SHOW CREATE TABLE using table protobuf message

2010-08-20T03:37:45+00:00

… and really testing the replication code path for CREATE TABLE.

So, for a very long time now, Drizzle has been using a protobuf based structure to describe the structure of tables. The idea was to be able to have engines rather painlessly generate this structure themselves (which several now do). A secondary idea was to use this structure itself for CREATE TABLE (in progress, and embedded_innodb does in fact does only use the table message for its CREATE TABLE codepath). The third idea was to just put the table protobuf message into the replication stream instead of the CREATE TABLE statement (i.e. a SQL string). This means that you could (easily) write a replicator to a DBMS with different SQL syntax, or to a system that doesn’t speak SQL at all.

The final step, to reduce duplicated code functionality, would be to have the code that does SHOW CREATE TABLE to use a common bit of code for turning a table protobuf message back into a SQL string.

We now have that.

Just now, my branch to replace the old SHOW CREATE TABLE code (that was using TableShare and friends) with the statement_transform code (that we also use in converting a replication log to SQL commands) hit trunk.

Yay.

Testing restoration from drizzledump files

2010-08-18T18:28:28+00:00

Currently, we have a lot of uses for drizzledump + restore: upgrading Drizzle, backing up data, etc. Since we are putting people in a position to rely on these tools, it seemed prudent that we improve our testing in this area.

Our plan was to include both test-suite tests as well as more vigorous testing with the randgen. The test-suite tests are pretty straightforward and are in tests/t/drizzledump_restore.test. You can get Drizzle here to check them out if reading test cases is your idea of fun : ) Of course, the randgen tests are the ones that do the interesting, high-volume stuff.

In a nutshell, the randgen test randomly generates any number of test tables in a database. The test tables have random composition (columns and datatypes used) and data population. Once the tables have been generated, we call drizzledump, restore the tables from the dump file to a new database, then do a table-to-table comparison of the original and restored tables. We report error and debugging information if we find a difference, and go on our happy way otherwise.

This test is now a part of our regular build and test system and runs on every push to trunk. We make use of a standard, repeatable set of queries and we also make use of the randgen’s ability to randomize data and queries and use –seed=time for a smaller run of the same test. This gives us a larger spread of tests each run. Fortunately, the randgen is nice enough to report the seed value for repeatability in case we encounter a bug.

# 2010-08-17T21:32:20 Converting –seed=time to –seed=128208074

Feeding the randgen the same seed value for a grammar / gendata file combination will produce the same tables, data, and queries each time.

I’m happy to report that the test currently passes with flying colors. Interestingly enough, I discovered what happens when you try to have several clients running drizzledump, creating and altering test tables, and restoring data from dumpfiles and it isn’t pretty. Please note that our test is designed to run with only a single thread at a time : )

If anyone is interested in testing or anything else related to Drizzle, stop by IRC (#drizzle) or drop us a line on our mailing list.

Storage Engine independent test suite

2010-08-18T01:57:16+00:00

This is something that has never existed in the MySQL world. Nothing to help you start developing your engine.

Sure, you could start running the whole test suite against your engine…. but where it wasn’t specifically MyISAM, you’d certainly hit things that were.. well… as simple as having to change the result file so that SHOW CREATE TABLE masked out the ENGINE= part.

Also, if you were just starting out and were trying to incrementally write your engine, instead of just hacking away for 6 months on “everything” and then testing “everything” and hoping that anything at all worked – you were out of luck.

So when working on the embedded_innodb engine for Drizzle I wanted to finally fix this problem. I was not going to fix it perfectly, or completely. What I wanted was a set of simple tests, that were very short and that tested distinct bits of the engine.

So… if you look in the Drizzle tree, in plugin/embedded_innodb/tests/ you’ll find a bunch of test cases beginning with basic_ (they’re also now in plugin/pbxt/tests). These check very basic functionality and should be useful when starting to write your own storage engine.

They should also port easily to MySQL too (possibly without changes), but I haven’t tried it. Porting to PBXT was simple.. a regex to replace InnoDB with PBXT.

Enjoy.

Drizzle build 1717 tarball has been released

2010-08-17T23:22:50+00:00

Drizzle source tarball based on build 1717 has been released.

NOTE: New dependency introduced!

We now require libboost-thread-dev to build.

Our ppa has been updated to reflect the change.

We apologize for any inconvenience, but this gives us the following benefits:

boost threads have scope safety
boost threads are completely portable (no more worrying about that stuff)
They will allow us to see what is happening with threads through the Performance Schema
They will allow us to eventually get rid of per-session memory, which will allow us to do more with asynchronous scheduling inside the server.

In this release:

continued work on embedded_innodb / HailDB
continued work on logging statistics
continued work on updating the options system
continued work on the filesystem storage engine
code cleanup
various bug fixes

The Drizzle download file can be found here

The last of the “my_long_options structs”

2010-08-16T21:00:14+00:00

As Google Summer of Code 2010 comes to an end my work of re factoring the commandline options and configuration file processing system using boost::program_options also is nearing the grand finale which would probably mean the complete removal of my_getopt which was used to solve the above stated purpose. Over the past couple of weeks [...]

dedcode

2010-08-16T08:38:31+00:00

I just finalized a first version of the query cache plugin for Drizzle. The good news is that the version is functional (YES you can try it out) but I guess it’s just a starting point for finding bugs and adding the new stuff in , in short … make it really functional.

In a previous article I have exposed the plugin query cache interface and hooks in the Drizzle kernel. Few modification were made to that part and I will talk about that and all the technical details in a next article, for now I am just going to give a “how-to” to get you started using the plugin.

First we need to get the branch where the latest code is pushed from my launchpad directory with the command:

$ bzr branch lp:~dedzone/drizzle/query-cache-gsoc

The plugin has two dependencies, Memcached and libmemcached, so make sure have these two installed before going further otherwise the compilation would fail. Then compile the branch and specify the query cache plugin (it’s not by default):

$ ./config/autorun.sh
$ ./configure --with-memcached-query-cache-plugin
$ ./make -j2 && make install

If you wish you can run the test plugin’s test suite (try to have a memcached instance runing on localhost:11211, if the test poses problems)

$./tests/dtr --suite=memcached_query_cache

After that you would have an operational Drizzle server, and since we are going to use the memcached query cache plugin, we will also need a runing instance of memcached (it can be local or remote). Note that the option –add-plugin would cause our plugin to get loaded, again it’s not loaded by default, and don’t forget to create a directory for your data exp: /home/user/db

$ memcached -l 127.0.0.1 -p 11211 -d
$ drizzled --datadir=/home/user/db/ --add-plugin=memcached_query_cache

Now that we are all set, lets try out the memcached query cache plugin by starting a new client (drizzle c++ client or boots) and connect to drizzle. At this point the plugin is loaded and pointing to a default memcached server localhost:11211, if you have a different configuration you can change it dinamically with

drizzle> set global query_cache_servers="localhost:11311 remotehost:11211"

You can also set the expiration time of a memcached entry (default is 0, never), that is a global parameter that everybody will be bound to, but can also be changed dynamically.

drizzle> set global query_cache_expiry= 1000;

So, where is the fun part? I have launched a discussion on Drizzle mailing list to decide on the syntax to cache or not a query. My very first orientation was to extend the sql syntax to and SQL_CACHE and SQL_NO_CACHE hints, but (thanks to all the contributor) switched to a server command that will switch the on or off the cache on demand and per Session.

drizzle>set query_cache_enable= on;

Now your next Select statement will be cached and if you execute the exact same command (including spacing) the results you will get will come out from memcached \o/. Well, not totally true, a Select statement to be cached has to be cacheable, i.e. time functions, random, user, database .. are not deterministic, thus must not be cached. Also user defined functions are not supported (we just don’t know what you’ve done out there), and queries referring to Data_dictionary schema, that will make the server stuck in a time point … not good

But, How would you know if your query is really cached! there is in fact a local cache structure that stores the current cached queries that you can freely query

drizzle> select * from emp;
+----------+------+
| name     | age  |
+----------+------+
| ded      |   28 |
| ded2     |   27 |
| vjsamuel |   19 |
+----------+------+
3 rows in set (0 sec)

drizzle> select * from emp where name like "ded";
+------+------+
| name | age  |
+------+------+
| ded  |   28 |
+------+------+
1 row in set (0 sec)

drizzle> select * from emp where age < 25;
+----------+------+
| name     | age  |
+----------+------+
| vjsamuel |   19 |
+----------+------+
1 row in set (0 sec)

drizzle> select sysdate() from emp where age < 25;
+---------------------+
| sysdate()           |
+---------------------+
| 2010-08-16 03:19:52 |
+---------------------+
1 row in set (0 sec)

drizzle> select * from data_dictionary.query_cache_entries;
+----------------------------------+--------+-----------------------------------------+
| key                              | schema | sql                                     |
+----------------------------------+--------+-----------------------------------------+
| 376555ba10dae3d09ca6df52a0839be6 | cool   | select * from emp where name like "ded" |
| b1a551e4aa3f717798380e41dcee0960 | cool   | select * from emp                       |
| e2e2402c2b29ac0c2fad1769a31d2d0a | cool   | select * from emp where age < 25        |
+----------------------------------+--------+-----------------------------------------+
3 rows in set (0.01 sec)

Here we issued 5 select statement and the 3 of them is in the cache except the ones calling sysdate(), or the data_dictionary. The hash key that you see is an md5 of your query, it’s the sole identification of your query cache in memcached and locally. One can play and look at the local cache content (It is a lovely protobuff message) with the following function:

drizzle> select print_query_cache_meta("e2e2402c2b29ac0c2fad1769a31d2d0a");

key: "e2e2402c2b29ac0c2fad1769a31d2d0a"
schema: "cool"
sql: "select * from emp where age < 25"
select_header {
table_meta {
schema_name: "cool"
table_name: "emp"
}
field_meta {
field_name: "name"
field_alias: "name"
table_name: "emp"
table_alias: "emp"
schema_name: "cool"
}
field_meta {
field_name: "age"
field_alias: "age"
table_name: "emp"
table_alias: "emp"
schema_name: "cool"
}
}

That’s nice, but what happen if the data cached gets tempered with (DML, DDL) ! don’t worry, the invalidation will take that in charge, in fact the plugin will keep a track of all the tables referenced in a query and will build a table similar to query_cache_entries, but will contain the reverse information, thus the table and all the entries of the cache making reference to it, that will avoid the burden of iterating overall the cache.

drizzle> select * from data_dictionary.query_cached_tables;
+---------+--------------------------------------------------------------------------------------------------------+
| Table   | Cache_Keys                                                                                             |
+---------+--------------------------------------------------------------------------------------------------------+
| coolemp | ::b1a551e4aa3f717798380e41dcee0960::376555ba10dae3d09ca6df52a0839be6::e2e2402c2b29ac0c2fad1769a31d2d0a |
+---------+--------------------------------------------------------------------------------------------------------+
1 row in set (0 sec)

Next, if one will change (or drop) the content of the table emp in the schema cool the replication system will cause the invalidation of all the cache entries of the cache:

drizzle> delete from emp where name like "ded1";
Query OK, 1 row affected (0 sec)
drizzle> select * from data_dictionary.query_cached_tables;
Empty set (0 sec)
drizzle> select * from data_dictionary.query_cache_entries;
Empty set (0 sec)

Final capability is the query_cache_flush() function, it will basically wipe out the whole content the local cache and the content of the memcached instances registred (make sure your instances are not used for another purpose)

drizzle> select query_cache_flush();
+---------------------+
| query_cache_flush() |
+---------------------+
|                   1 |
+---------------------+
1 row in set (0 sec)
drizzle> select * from data_dictionary.query_cached_tables;
Empty set (0 sec)
drizzle> select * from data_dictionary.query_cache_entries;
Empty set (0 sec)

Conclusion:

That was a small kinda manual to get you started with the query cache plugin, I hope to extend the functionalities to support fine grained invalidation in the near future. But for now there is still many work to be done to get rid of all the withstanding problems.

Tab completion framework done.

2010-08-15T19:54:00+00:00

I have completed the tab completion framework is now complete. It has been tested with a network updater built on MySQLdb library of python. Both static and network parts of updater mechanism are working.
Now the task will be to finish this module by rewriting this network update framework with boots query framework.

I now think that gsoc work is complete for me. Will now look for more open source work with drizzle.

Future HailDB release planning

2010-08-13T19:15:35+00:00

We’ve started to plan out the next couple of HailDB releases over on the HailDB Launchpad page. We’re looking at each release being small and incremental changes. The next release or two will help us make the release process nice and smooth as well as fixing a few small things.

We are a completely open project and encourage participation from everyone. Currently the best place for discussion would be the drizzle-discuss mailing list (It’ll be great when we have enough traffic to create a HailDB specific list).

Blob Streaming (PBMS) support in drizzle jdbc

2010-08-11T22:18:29+00:00

I just pushed up initial support for PBMS, blob streaming for drizzle jdbc. It is not yet a complete solution for blob streaming, but you can use setBinaryStream and getBinaryStream to stream data to/from Drizzle and MySQL.

To use it, grab a snapshot from hudson (the reason the tests fail is that i have not had time to rewrite INFORMATION_SCHEMA queries to support the new I_S in Drizzle). Then follow this example to get started, important stuff is in bold

Connection connection = 
    DriverManager.getConnection(
           "jdbc:drizzle://localhost:3307/test_units_jdbc?enableBlobStreaming=true");
// inserting a blob:
Statement stmt = connection.createStatement();
stmt.execute("create table bstreaming1 (id int not null primary key auto_increment, test longblob)");
PreparedStatement ps = connection.prepareStatement("insert into bstreaming1 values (null, ?)");
// fake an inputstream with data:
ByteArrayInputStream bais = new ByteArrayInputStream("HEJHEJHEJ".getBytes());
ps.setBinaryStream(1, bais);
ps.executeUpdate();
        
stmt = connection.createStatement();
ResultSet rs = stmt.executeQuery("select * from bstreaming1");
assertEquals(rs.next(), true);
byte[] b = new byte[100];
int l = rs.getBinaryStream("test").read(b);
assertEquals("HEJHEJHEJ",new String(b,0,l));

To get PBMS support in drizzle you need to have a fairly recent build, and start the daemon with --plugin_add=pbms. InnoDB tables are streaming enabled, which means that any LONGBLOB column will be streamed.

Note that in the current version you will get the blob reference and not a String representation of the blob if you do a getString on the column for example, this will change when I get time to do it.

I have not yet tested against MySQL, if you have a PBMS enabled MySQL server, please let me know if it works!

What’s new in the plugin world!

2010-08-10T19:00:13+00:00

Its been more than a month since i started working on re factoring the plugin system in order to support the new option processing system which uses boost::program_options. I ‘ve had tonnes of help from Monty Taylor who had worked on the interface. I ‘ve also had constant help from Brian Aker. I ‘ve slowly [...]

libdrizzle in Visual Studio

2010-08-10T04:22:50+00:00

Thanks to Jobin's work with mingw and getting libdrizzle to compile on Windows at all, I have been able to get it working in Visual Studio natively. The code is in trunk now.

The approach I took, which is how I'm going to approach Windows and Visual Studio for all of our stuff, is to not worry with analogues to things like configure on Windows. Windows is a very different platform from Linux, and there is no need to attempt to duplicate Linux process there. To that end, the goal at least for now will be static VS Solution files and a set of instructions of how to get depends installed so that the Solution can find them.

I'm excited to start poking at Garrett Serack's CoApp Project, which has some tools do do tracing of things like make to help with the initial project creation... and I fully expect to take advantage of his work in terms of creating installable packages.

Jobin has started working on Drizzle itself and has the drizzle client working so far.

Testing Drizzle with the random query generator

2010-08-06T15:50:26+00:00

As some of you may know, part of Drizzle’s testing process involves the use of the random query generator (aka randgen aka RQG). Any patches that want to make it into trunk must survive several randgen tests, among other things.

One of the first things I did upon joining the Drizzle team at Rackspace was to take a look at the RQG tests running in Hudson. My evaluation included tuning up the existing tests and modifying some MySQL test grammars for use with Drizzle. It should be noted that randomness is a double-edged sword – it is extremely powerful for finding bugs, but it also requires a certain amount of diligence to produce useful, valid queries. Once my initial work was complete, we unleashed the new RQG tests and found the following bugs:

At the time of this writing, 6 of these are in ‘Fix Released’ status and 3 more are in work. I’d like to extend a personal “Thank you” to Prafulla Tekawade for his great work – he has supplied the patches for 5 out of the 6 fixed bugs (!). Additionally, thanks for Toru Maesaka for his continued work on BlitzDB (which is part of our regular randgen testing).

If anyone is interested in learning more about using the random query generator, I encourage you to check out launchpad. General information can also be found here and here

Ubuntu packages of HailDB now available

2010-08-04T20:14:59+00:00

In the haildb-core PPA over on launchpad. Thanks to Monty Taylor for packaging.

Drizzle on Windows - Challenges ahead

2010-08-04T18:04:00+00:00

Since Drizzle is Open, I am sure that one day someone will take the mantile to deepest.
As you already know, Drizzle is effectively making use of other opensource libraries available.
(Drizzle developers tell that is because they are lazy. But personally i consider that it is because they (we) are open to the environment and we respect and trust on expertise of other opensource communities)

But same philosophy works against a "builder" and "porter" on non open platform (say windows).
many of this opensource libraries need to be built on (ported???) windows.

Best example is google protobuffer itslef.
latest tarball release of protobuffer won't compile on windows.
(simple googling told me that google already corrected this issue in the branch)
But it is an alarm bell! These powerful libraries are not very well used / tested on Closed Grave yards.

I downloaded tarballs of most of the libraries on which drizzle is dependent on.
all these tarballs expect me to run ./configure and make.
!!!!(puzzled!)...in my language... "dog gets an unbroken coconut as food" (trasilated version)

I think, since windows relys on binary distributions, they are not exepcting us to "make" it.
But there are dedicated people who maintain binary versions; but getting latest version is challenge.
Mingw (Msys) team are doing an amazingly well job here.
No wonder, most of binary versions of these libraries are build on Mingw
I hope thats the way to go deeper.

Drizzle build 1683 tarball has been released

2010-08-04T00:07:42+00:00

Drizzled source tarball based on build 1683 has been released. In this release:

Fixes for several bugs in transaction log
ANSI INFORMATION_SCHEMA now populates itself
Continued code cleanup
Various bug fixes
Filesystem plugin merged in
Merged in intltool

The Drizzle download file can be found here

RPMs of HailDB now available!

2010-08-03T17:14:10+00:00

Over at the Drizzle Wiki, there are instructions on how to add the Unofficial Fedora/RHEL repository to your installation and then you can install the HailDB package.

We hope to at some point soon also submit it to the official repository.

We’ll also have debs for Debian and Ubuntu shortly.

Developing Drizzle with Xcode

2010-08-03T10:52:43+00:00

Even with all of its shortcomings, Xcode has excellent integration with the GNU debugger. It also has some good features for understanding a large code base that you may be unfamiliar with. For these reasons, I like to use Xcode for my development work. Unfortunately, it takes a bit of magic to get it setup to work with an external project that was not created specifically with/for Xcode and that does not use CVS/Subversion/Perforce.

If you want to use Xcode to work on Drizzle, these are the steps you need to take to setup a project. Note that Xcode doesn't support Bazaar so you still need to do some things from the command line. (Come on, Apple. CVS and Subversion just plain suck)

These instructions assume that the Drizzle repo is located in $drizzle-repo, as described here in the Drizzle wiki.

First, we need to create a new branch (we'll call it xcode-branch) of Drizzle trunk to use for our yet to be created Xcode project.

cd $drizzle-repo
bzr branch trunk xcode-branch

Next, we'll go into Xcode and create a new project. Choose the External Build System project. With this type of project, we'll be able to click on Build to run make for us.

Click the Choose... button and save your project some place appropriate (anywhere but inside your $drizzle-repo directory). We'll tell the project where to find the source files in xcode-branch in the following steps.

After saving, your project window should appear. At this point, we have no source code files in our project, so we need to tell Xcode to use our xcode-branch as the location of our source. We do that by right-clicking on our project name in the Groups & Files section of the project window and selecting Add -> Existing Files...

Navigate to your xcode-branch directory, select the directory, and click the Add button. In the window that appears, make sure that Copy items into destination group's folder option is NOT selected. We don't want to copy the source files into the Xcode project directory, we want to use them from within the Bazaar branch. We do this so that after we make changes with Xcode, we can easily open a Terminal window and commit our changes from the command line. (Again, this is because Xcode does not have Bazaar direct integration.) Also, make sure that Recursively create groups for any added folders is selected. So you should have:

Click Add and a folder for your xcode-branch should appear under your project name in the Groups & Files section.

So what have we just done? We basically did two things here:

Created a Bazaar branch for our modifications we will make under Xcode
Created an Xcode project that points to our Bazaar branch.

Our working Xcode project is split up into two separate directories: one for the Xcode project files, and one for the Bazaar source files. This may seem a bit odd if you expected everything to be under a single directory, but we have to do it this way so that we can still use Bazaar commands.

There is one last thing to do now. Xcode is now setup to automatically just call 'make' when we click on our Build button, but it will attempt to run 'make' in the Xcode project directory, not in our xcode-branch directory. This is an easy fix. Double-click on the target underneath Targets in the Groups & Files section of the project window. This will open up the following window:

Set the Directory under Custom Build Command to the location of your xcode-branch. Once you do this, then clicking Build will run 'make' from within the xcode-branch.

That's it! You can now edit the source, use Xcode features to jump to various definitions, search project files, and build the project using Command-B or the Build menu option. Obviously, you'll first have to create and run the configure script in your xcode-branch directory by hand before you can build. And you will not be able to use the other build commands, like Build and Run, or use the debugger until you setup an executable for the project. I'll cover that in a future post.

Switching to HailDB from Embedded InnoDB

2010-08-03T01:22:17+00:00

HailDB is a project that aims to continue the Embedded InnoDB project. That is, provide the excellent, high quality and super reliable InnoDB database engine (as used in MySQL and Drizzle by millions of users all over the world) as a shared library to application developers. We just announced our first release!

We will shortly have packages for various distributions up, along with a source tarball.

It’s incredibly easy to switch to HailDB from Embedded InnoDB for your project. Just look for libhaildb instead of libinnodb and include “haildb.h” instead of “innodb.h”. The rest of the API is exactly the same.

HailDB 2.0.0 released!

2010-08-03T00:59:40+00:00

We’ve made our first HailDB release! We’ve decided to make this a very conservative release. Fixing some minor bugs, getting a lot of compiler warnings fixed and start to make the name change in the source from Embedded InnoDB to HailDB.

Migrating your software to use HailDB is really simple. In fact, for this release, it shouldn’t take more than 5 minutes.

Highlights of this release:

A lot of compiler warnings have been fixed.
The build system is now pandora-build.
some small bugs have been fixed
Header file is now haildb.h instead of innodb.h
We display “HailDB” instead of “Embedded InnoDB”
Library name is libhaildb instead of libinnodb
It is probably binary compatible with the last Embedded InnoDB release, but we don’t have explicit tests for that, so YMMV.

Check out the Launchpad page on 2.0.0 and you can download the tarball either from there or right here:

haildb-2.0.0.tar.gz
MD5: 183b81bfe2303aed435cdc8babf11d2b
SHA1: 065e6a2f2cb2949efd7b8f3ed664bc1ac655cd75

HailDB 2.0.0 released!

2010-08-03T00:52:59+00:00

(Reposted from the HailDB Blog. See also the announcement on the Drizzle Blog.)
We’ve made our first HailDB release! We’ve decided to make this a very conservative release. Fixing some minor bugs, getting a lot of compiler warnings fixed and start to make the name change in the source from Embedded InnoDB to HailDB.

Migrating your software to use HailDB is really simple. In fact, for this release, it shouldn’t take more than 5 minutes.

Highlights of this release:

A lot of compiler warnings have been fixed.
The build system is now pandora-build.
some small bugs have been fixed
Header file is now haildb.h instead of innodb.h
We display “HailDB” instead of “Embedded InnoDB”
Library name is libhaildb instead of libinnodb
It is probably binary compatible with the last Embedded InnoDB release, but we don’t have explicit tests for that, so YMMV.

Check out the Launchpad page on 2.0.0 and you can download the tarball either from there or right here:

haildb-2.0.0.tar.gz
MD5: 183b81bfe2303aed435cdc8babf11d2b
SHA1: 065e6a2f2cb2949efd7b8f3ed664bc1ac655cd75

HailDB version 2.0.0 has been released

2010-08-03T00:49:56+00:00

HailDB source tarball, version 2.0.0 has been released.
In this version:
- A lot of compiler warnings have been fixed.
- The build system is now pandora-build.
- some small bugs have been fixed
- Header file is now haildb.h instead of innodb.h
- We display "HailDB" instead of "Embedded InnoDB"
- Library name is libhaildb instead of libinnodb
- It is probably binary compatible with the last Embedded InnoDB release, but we don't have explicit tests for that, so YMMV

The HailDB download file can be found [https://launchpad.net/haildb/+milestone/release-2.0 here]

Welcome David Shrewsbury to the Rackspace Drizzle team

2010-08-02T15:39:05+00:00

We are very happy to announce that David Shrewsbury is joining the Rackspace Drizzle team today. David has been a steady community contributor to the Drizzle project so we are very excited to have him on board full time working on Drizzle.

David has a M.S. in Computer Science from Appalachian State University and currently lives in Durham, North Carolina. In his previous lives, David has been a developer, database architect, and even worked as a MySQL Support Engineer focusing on Cluster support.

When not trying to wrap his head around some obscure piece of code, David can often be found either on the golf course, reading books from the dustier corners of the library, or sampling the latest microbrew.

Welcome to the team David!

libdrizzle - now with windows

2010-08-01T04:26:33+00:00

Recently, Jobin took up the mantle of starting to poke at Windows support for Drizzle. We decided that step 1 is getting libdrizzle building on Windows - not to mention we could solve a few systemic "how to go about it" problems on a much simpler codebase.

We decided to go the mingw route - for two reasons.

on windows we wouldn't have to solve the build problem yet
on linux we can cross-compile, and then I can spend less time connecting to a windows machine

I'm happy to say that the branch implementing support for building under mingw has been merged in to trunk - so please go hammer at it and tell me that I'm an idiot.

If you're on linux and want to play, you'll need mingw32 and mingw32-pthread. You may notice that mingw-pthread isn't in the Ubuntu archive... that's ok - I added a package for it in the drizzle-developers ppa (add-apt-repository ppa:drizzle-developers/ppa) Once you're set with that, it's as simple as:

 bzr branch lp:libdrizzle

 cd libdrizzle

 ./config/autorun.sh

 ./configure --build=x86_64-unknown-linux-gnu --host=i586-mingw32msvc

 make

And you're off to the races. Now, as for actually doing anything with it, well, I'm expecting some windows person to fill in details on that.

Next up - building under Visual Studio. (oh, and getting Drizzle itself working)

O, hai intltool!

2010-07-31T19:57:59+00:00

Almost no one will ever notice anything about this, but I'm excited, so I thought I'd share.

Instead of using the build infrastructure installed by gettextize to handle message translations, we're now using the more modern replacement, intltool.

Unless you're actually hacking on build, this will mean nothing - translations are still done via launchpad - Drizzle still uses gettext() to translate strings. It's purely a tooling change which means the build is less brittle and I'm slightly less ornery. Oh, and we have many less random generated files committed to bzr. I like deleting files.

Changes in Option Processing

2010-07-30T17:42:24+00:00

Change is in this air this summer in Drizzle-land. One of our GSoC students, Vijay Samuel, has been hard at work replacing the options processing system we inherited from MySQL, my_getopt, with one based on boost::program_options. We've been merging his work in to trunk for a while now, and he's made really excellent progress, so it's probably about time to point out how the new system will be different from the MySQL one. There are three main changes afoot here, and I'm actually pretty pleased with all three of them:

Plugin Option Prefixing
Dashes v. Underscores
Config Files

A few of these changes are still in the middle of their transition, so I'm just going to describe the finished system, but we've essentially got all of the client programs and most of the plugins done at this point.

Plugin Option Prefixing

In the old system, when a module registered a new option or variable, we would concatenate the module's name on to the front of the option name with a "_". So, for instance, the mysql_protocol module has an option for setting the port, which wound up being named "mysql_protocol_port". In the new system, boost actually understands grouping and namespacing of options, and we decided to take advantage of that by mapping a plugin module to a group. This means that the concat character is now a "." (that's a dot if your eyesight is bad)

Dashes v. Underscores

In addition to that, boost doesn't know anything about the long-standing MySQL tradition of going out of the way to allow you to do things as many ways as humanly possible. To Boost, the strings "mysql_protocol" and "mysql-protocol" are, well, different. We considered writing a parser plugin for it to do normalization for us, but then we made the other decision.

Dashes and Underscores are different.

Now this may royal screw some people who may have enjoyed making art by listing an option like "auto-increment_increment" before, and to those people I'm sorry, because I really do like injecting art into day to day life. Nonetheless, from this day forth let the decree be heard...

Options have all been normalized to use no underscores and to use only dashes.

On the off chance you can't follow the logic there, that means that the command line option for setting the MySQL protocol port in Drizzle is now: --mysql-protocol.port=3306

Config File Changes

But wait, there's more...

First of all, we changed from having one master config file for everything with sections in it for different programs to having one config file per program. drizzled will read "/etc/drizzle/drizzled.cnf". The command line clients will also read a few more, since you might want to repeat yourself - so drizzleddump will read ~/.conf/drizzle/drizzledump.cnf, ~/.conf/drizzle/client.cnf, /etc/drizzle/drizzledump.cnf and /etc/drizzle/client.cnf, with files listed first in that list overriding values in files later in the list. (In case you're wondering about the ~/.config part, it's part of the freedesktop.org Base Directory Specification. Since we didn't have a system for this before, it seemed sensible enough to just follow the standard for user/client utilities here.

The files are still mostly ini-style files, except they do not require a base [section] header. So just stick "datadir=/var/lib/drizzle" into /etc/drizzle/drizzled.cnf and you're off to the races. However - remember that grouping thing I mentioned before- in the boost world, section headings translate to option prefixes joined by, you guessed it, a dot. This means you can do:

datadir=/var/lib/drizzle

[innodb]

buffer-pool-size=100G

log-file-size=1M

And be good to go. We're also thinking about having /etc/drizzle/${plugin_name}.cnf added for each plugin... but haven't actually looked at what that will look like yet, so don't hold me to that.

Determining I/O throughput for a system

2010-07-29T15:23:33+00:00

At Kscope this year, I attended a half day in-depth session entitled Data Warehousing Performance Best Practices, given by Maria Colgan of Oracle. In that session, there was a section on how to determine I/O throughput for a system, because in data warehousing I/O per second (iops) is less important than I/O throughput (how much actual data goes through, not just how many reads/writes).

The section contained an Oracle-specific in-database tool, and a standalone tool that can be used on many operating systems, regardless of whether or not a database exists:

If Oracle is installed, run DBMS_RESOURCE_MANAGER.CALIBRATE_IO:

SET SERVEROUTPUT ON
DECLARE
lat INTEGER;
iops INTEGER;
mbps INTEGER;
BEGIN
-- DBMS_RESOURCE_MANAGER.CALIBRATE_IO(<DISKS>, <MAX_LATENCY>,iops,mbps,lat);
DBMS_RESOURCE_MANAGER.CALIBRATE_IO (2, 10, iops, mbps, lat);
dbms_output.put_line('max_mbps = ' || mbps);
end;

For us MySQL folks, or even the Drizzle or NoSQL folks, Oracle offers a free standalone tool called Orion. The example given in the slides was:

./orion –run advanced –testname mytest –num_small 0 –size_large 1024 –type rand –simulate contact –write 0 –duration 60 –matrix column

-num_small is 0 because you don’t usually do small transactions in a dw.
-type rand for random I/O’s because data warehouse queries usually don’t do sequential reads
-write 0 – no writes, because you do not write often to the dw, that is what the ETL is for.
-duration is in seconds
-matrix column shows you how much you can sustain

I would be interested to see how other folks measure I/O throughput, and maybe even do a side-by-side comparison of different tools. Orion is available for:

Linux (x86, x86-64, Itanium, Power)
Solaris (SPARC64)
AIX (PPC64)
zLinux
HPUX (PA RISC, Itanium)
Windows

I am working on a larger write-up of the session itself, which had many concise descriptions of data warehousing issues, but I thought that this merited its own post.

Drizzle, boost::unordered_multimap++

2010-07-28T05:47:16+00:00

=========================================================================================================
REGRESSION REPORT 
=========================================================================================================
MACHINE:  drizzle-build-n01.wc1.dfw1.stabletransit.com
RUN ID:   324
RUN DATE: 2010-07-27T21:48:07.932094
WORKLOAD: innodb_1000K_readonly
SERVER:   drizzled
VERSION:  lp:drizzle/staging
REVISION: 1669
COMMENT:  1669: Brian Aker 2010-07-27 This patch turns the table_cache into boost::unordered_multimap.
=========================================================================================================

TRENDING OVER LAST 5 runs 
Conc   TPS       % Diff from Avg Diff       Norm?          Min        Max        Avg        STD       
=========================================================================================================
16       1993.10          +1.19%      23.37   within norms    1905.44    2010.28    1969.73      26.56
32       2676.28          +1.75%      46.11  outside norms    2568.20    2685.05    2630.17      33.22
64       2622.82          -2.09%     -56.00  outside norms    2613.02    2737.89    2678.82      43.09
128      2621.23          -0.61%     -16.15   within norms    2609.74    2662.30    2637.38      16.41
256      2598.18          +5.50%     135.46  outside norms    2406.50    2599.34    2462.72      68.67
512      2343.83         +52.30%     804.91  outside norms    1318.38    2347.49    1538.92     402.80
1024     1722.08        +104.66%     880.65  outside norms     614.65    1725.52     841.43     440.35

Boost::unordered_multimap versus the hand crafted MySQL HASH?

Boost wins by a long shot. The above is from sysbench. We run it on each and every push that goes into the main tree looking for regression.

Every so often we get to see the opposite happen :)

Thanks to this patch, "Table" now joins the ranks of what we call a "trusted object". This means we can start safely assuming that the destructor on it is working (most of the MySQL codebase was crafted in such a way that you can only use a very limited subset of C++). In all of the new code in Drizzle we can easily make use of C++. There are still a few older bits where we cannot. Having "Table" now work means we can safely work in a number of new areas in the server. One of the biggest changes that will be coming soon is the removal of LOCK_open for a number of new cases.

HOWTO screw up launching a free software project

2010-07-27T18:40:54+00:00

Josh Berkus gave a great talk at linux.conf.au 2010 (the CFP for linux.conf.au 2011 is open until August 7th) entitled “How to destroy your community” (lwn coverage). It was a simple, patented, 10 step program, finely homed over time to have maximum effect. Each step is simple and we can all name a dozen companies that have done at least three of them.

Simon Phipps this past week at OSCON talked about Open Source Continuity in practice – specifically mentioning some open source software projects that were at Sun but have since been abandoned by Oracle and different strategies you can put in place to ensure your software survives, and check lists for software you use to see if it will survive.

So what can you do to not destroy your community, but ensure you never get one to begin with?

Similar to destroying your community, you can just make it hard: “#1 is to make the project depend as much as possible on difficult tools.”

#1 A Contributor License Agreement and Copyright Assignment.

If you happen to be in the unfortunate situation of being employed, this means you get to talk to lawyers. While your employer may well have an excellent Open Source Contribution Policy that lets you hack on GPL software on nights and weekends without a problem – if you’re handing over all the rights to another company – there gets to be lawyer time.

Your 1hr of contribution has now just ballooned. You’re going to use up resources of your employer (hey, lawyers are not cheap), it’s going to suck up your work time talking to them, and if you can get away from this in under several hours over a few weeks, you’re doing amazingly well – especially if you work for a large company.

If you are the kind of person with strong moral convictions, this is a non-starter. It is completely valid to not want to waste your employers’ time and money for a weekend project.

People scratching their own itch, however small is how free software gets to be so awesome.

I think we got this almost right with OpenStack. If you compare the agreement to the Apache License, there’s so much common wording it ends up pretty much saying that you agree you are able to submit things to the project under the Apache license. This (of course) makes the entire thing pretty redundant as if people are going to be dishonest about submitting things under the Apache licnese there’s no reason they’re not going to be dishonest and sign this too.

You could also never make it about people – just make it about your company.

#2 Make it all about the company, and never about the project

People are not going to show up, do free work for you to make your company big, huge and yourself rich.

People are self serving. They see software they want only a few patches away, they see software that serves their company only a few patches away. They see software that is an excellent starting point for something totally different.

I’m not sure why this is down at number three… it’s possibly the biggest one for danger signs that you’re going to destroy something that doesn’t even yet exist…

#3 Open Core

This pretty much automatically means that you’re not going to accept certain patches for reasons of increasing your own company’s short term profit. i.e. software is no longer judged on technical merits, but rather political ones.

There is enough politics in free software as it is, creating more is not a feature.

So when people ask me about how I think the OpenStack launch went, I really want people to know how amazing it can be to just not fuck it up to begin with. Initial damage is very, very hard to ever undo. The number of Open Source software projects originally coming out of a company that are long running, have a wide variety of contributors and survive the original company are much smaller than you think.

PostgreSQL has survived many companies coming and going around it, and is stronger than ever. MySQL only has a developer community around it almost in spite of the companies that have shepherded the project. With Drizzle I think we’ve been doing okay – I think we need to work on some things, but they’re more generic to teams of people working on software in general rather than anything to do with a company.

Drizzle on Windows - Libdrizzle

2010-07-27T11:27:00+00:00

lately I started thinking from a users point of view.
Web developers want to try drizzle. but many of them got windows platform for devleopment.
So they are not able to connect to drizzle server

so started seaching for options. found that i have 3 options.
1. Native compile the source using MinGW.
2. Take the help of cygwin to get posix platform
3. Coss-compile the source from Linux to windows.

Toru, Stewart and Monty responded to my views on drizzle-discuss.
Monty was already trying on libdrizzle for sometime.
and Toru also voted for libdrizzle first.
Monty's approach was to use option 3 with option 1.
(later i realized his approach is better and smart). like:

./configure --build=x86_64-unknown-linux-gnu --host=i586-mingw32msvc

everybody voted for libdrizzle to be available for windows first.
That voting makes sence because developer in windows should be able to connect to server first.

My step was to get libdrizzle compiled using cygwin.
cygwin gives full Linux system calls in a dll. that is a very big offer for me.
i could not deny that.

as expected everything went smooth in cygwin.
libdrizzle got compiled without any effort. and i am able to verify the connectivity from windows xp box.

Next step is very challenging.
(The step to compile libdrizzle on windows natively without help of cygwin1.dll)
for a non-programmer like me it is a herculean task.
Monty's working branch (lp:~mordred/libdrizzle/mingwport)
gave a kick start because he already done all background work.

after adding few more lines to Monty's work, (drizzle.h and conn.c )
I am able to compile files from mingw

it was bit challenging to figure out that following defines are must have (i added them to common.h)

#define WINVER WindowsXP
#define _WIN32_WINNT 0x0501

(otherwise haunting, "undefined reference to `freeaddrinfo'"and undefined reference to `getaddrinfo'"" like messages will come when you try to link)

My effort is to create static library which can be linked to make executable.
since configure and make are giving me trouble on windows, i tried handweave the compilation.
to see what is happening in the make
logged all commands executed when libdrizzle is getting compiled with cygwin
$>make V=1

more over go a good guidence from:
http://www.adp-gmbh.ch/cpp/gcc/create_lib.html

Step 1. compiling the library files:
all below executed in windows command prompt.

"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\drizzle.c  -o libdrizzle\libdrizzle_la-drizzle.o
"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\conn.c  -o libdrizzle\libdrizzle_la-conn.o
"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\conn_uds.c  -o libdrizzle\libdrizzle_la-conn_uds.o
"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\handshake.c   -o libdrizzle\libdrizzle_la-handshake.o
"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\command.c   -o libdrizzle\libdrizzle_la-command.o
"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\query.c   -o libdrizzle\libdrizzle_la-query.o
"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\result.c   -o libdrizzle\libdrizzle_la-result.o
"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\column.c   -o libdrizzle\libdrizzle_la-column.o
"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\row.c   -o libdrizzle\libdrizzle_la-row.o
"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\field.c   -o libdrizzle\libdrizzle_la-field.o
"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\pack.c   -o libdrizzle\libdrizzle_la-pack.o
"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\state.c   -o libdrizzle\libdrizzle_la-state.o
"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\sha1.c   -o libdrizzle\libdrizzle_la-sha1.o

Step 2. get the poll implementation on windows
cd libdrizzle\win32
"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c poll.c   -o poll.o
cd ..\..

Step 3. Create a single static library (libdrizzle.a)
"C:\MinGW\bin\ar.exe" cru libdrizzle/libdrizzle.a  libdrizzle/libdrizzle_la-drizzle.o  libdrizzle/libdrizzle_la-conn.o libdrizzle/libdrizzle_la-conn_uds.o libdrizzle/libdrizzle_la-handshake.o libdrizzle/libdrizzle_la-command.o  libdrizzle/libdrizzle_la-query.o  libdrizzle/libdrizzle_la-result.o  libdrizzle/libdrizzle_la-column.o  libdrizzle/libdrizzle_la-row.o libdrizzle/libdrizzle_la-field.o libdrizzle/libdrizzle_la-pack.o libdrizzle/libdrizzle_la-state.o libdrizzle/libdrizzle_la-sha1.o libdrizzle/win32/poll.o

Step 4. Compile user program and link to the above created library.
"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c examples/simple.c   -o examples/simple.o
"C:\MinGW\bin\gcc.exe" -op -o examples/simple.exe examples/simple.o  ./libdrizzle/libdrizzle.a  -lwsock32 -lws2_32
cd examples

Step 5. Test the user program.
simple.exe -d test -h 10.97.93.243 -q "select * from t1"

I want to thank you Monty for the huge help and guidance i recived.

Resolution to get my hands wet on opensource

2010-07-27T10:48:00+00:00

I took a resolution month back and it is working perfect for me.
being a open source enthusiast i did not contribute to community much.
so "i should concentrate on something for atleast 1 weeks and if it is interesting, work on that in coming weeks until something materializes". that was the resolution.
this started showing me magics.

First one was that i should work on "Firefox extensions".
That was a grant success inside the company appreciations and patches started flowing.
it automates most of my tasks inside the company.
In-fact, just like any Open Source effort, here also i was trying to scratch on my itch.
later i realized, it is not only solves myown problems, but others too..
I taught me a big lesson, most of us require opensource software to solve over problems
and it works even inside a closed system like a company

OSCON and OpenStack

2010-07-26T07:33:14+00:00

The past two weeks have been both exciting and extremely busy, first traveling to Austin, TX for the first OpenStack Design Summit, and then back home to Portland, OR for The O’Reilly Open Source Conference (OSCON) and Community Leadership Summit. The events were great in different ways, and there was some overlap with OpenStack since we announced it on the first day of OSCON and created quite a bit of buzz around the conference. I want to comment on a few things that came up during these two weeks.

New Role

I’m now focusing on OpenStack related projects at Rackspace. I’m no longer working on Drizzle, but I will still be involved in the MySQL and database ecosystems through future projects and conferences (see you at OpenSQL Camp). I will also still be working on a couple of Gearman related projects in my spare time. At OSCON I gave two presentations on Gearman and Drizzle, you can find the slides here.

The Five Steps to Open

One question that came up a few times over the past couple weeks is what the term “Open” means when a business or organization decides to adopt the open source philosophy. It turns out this means many different things to folks, and when an organization decides to go open, they need to make a decision on how open they are willing to be. Here are the various layers we’ve seen over the years:

Open API – You’ve decided to take the first step to being open and released a well documented API to work with your web service or project. Everything behind the API is still a black-box though.
Open Core – Beyond the APIs, you’ve decided to release part of the code open source, but you still keep some of the bits proprietary in an attempt to keep a competitive advantage. This is a hot debate lately on whether it is a viable Open Source business model.
Open Source – You’ve decided keeping some code proprietary doesn’t help, and actually even hurts your project or adoption. You put all of the code out in the open for everyone to see. While everyone can see all of the source code, there still isn’t a whole lot of interaction going on.
Open Development – Putting the source code out wasn’t enough. You want to enable users and external developers to be able to file bugs, submit patches, and track the development process to see what to expect next. This usually involves running your project on a public project site such as github or Launchpad.
Open Decision Making – You’ve postponed the inevitable for long enough. Feature requests and bug reports are pouring in, and the community wants to have a say in what gets prioritized. Should we focus only on stability? Performance? New features? Porting to mobile platforms? Let the community decided the direction of the project.

There have been examples of success for organizations who have stopped at each of these steps. Given the proper environment, any can work. My preference is to work on projects that are fully open, where company and organizational boundaries do not exist between developers and users. I’m thrilled to say that we’ve gone all in with OpenStack. We’re hosted on Launchpad and have a governance structure that allows all parties within the community to have a say in the future of the project.

Preventing Vendor Lock-in

During the Cloud Summit at OSCON, there was a debate titled: “Are Open APIs Enough to Prevent Lock-in?”. Most folks came to the conclusion that the answer is “no,” and I agree. While I feel open APIs are necessary, they are by no means sufficient. Even if a project is open source and allows for open development, it probably will not prevent vendor lock-in. The key is to provide some incentive for vendors to adopt and invest resources within a project. Much like customers don’t want vendor lock-in when choosing a platform, vendors do not want project or feature lock-in when choosing the software to power their business. Each vendor who chooses to participate must have the ability to voice their opinion on the direction of APIs, features, and other project priorities. This is why it is critical that any open source project must take all the steps described above to give the project a chance of being adopted and becoming the de facto standard. There is of course no guarantee that adoption and prevention of vendor lock-in will happen, but I see them as necessary steps.

This is another area where OpenStack has done the correct thing. We are planning on having another developer summit in November, and then once every six months after that time. All design discussions and decision making will happen in public forums such as the mailing list and IRC. We want all participants in the community to have a chance to respond to topics being discussed, and we believe the more we have, the more successful the project will be. Having many voices allows the project to be more applicable to different environments. For example, Rackspace and NASA have different requirements for their compute architectures, but they also share many components as well. Through open participation we can ensure all needs are accounted for. Much like the LAMP stack has powered universities, governments, and competing business, we hope OpenStack can do the same.

Contributor License Agreement (CLA)

During the past couple of weeks a few folks asked what the CLA was all about. When the foundations of OpenStack were forming, the requirement of having a CLA came up from the legal side. Having been involved with open source projects that had very invasive CLAs, initially I had quite a bit of concern. The CLA is actually quite innocuous, and it does NOT require assignment or dual-ownership of copyright. You are the sole owner of code you contribute. For all intents and purposes it is a signed version of the Apache 2.0 license, the CLA just makes these terms more explicit. The CLA is handled through digital signatures, so no papers, pens, or faxing is required.

Get Involved!

Expect to see more posts on my blog related to OpenStack topics. If you would like to get involved, you can join the IRC channel (#openstack on irc.freenode.net), join the mailing list, or start contributing code! There are even jobs around OpenStack popping up already!

MySQL Server Protocol Bug

2010-07-25T00:47:30+00:00

A few months ago I wrote a tool that verified MySQL and Drizzle protocol compatibility, along with testing for all sorts of edge cases. In analyzing protocol command interactions in mysqld, I found that the MySQL server will happily read an infinite amount of data if you exceed the maximum packet size while using a special sequence of protocol packets. The reasoning behind this behavior is so that the server can be polite and flush your data before sending a “max packet exceeded” error message, but perhaps there should be a limit to one’s politeness. What’s more interesting is that you can do this during the client handshake packet without authorization, so anyone could do this to any open MySQL server. The appropriate thing to do here would be to set some maximum limit of data to read and force a connection close when it is reached, otherwise your bandwidth and CPU could be consumed (essentially a DoS attack).

This portion of code was ripped out entirely in Drizzle, so there are no risks there. I submitted this as a bug to MySQL and MariaDB back in February and they both have patches available to fix this as well. You can find the bug here and a patch here. If you have publicly accessible MySQL or MariaDB servers, you probably want to upgrade binaries or patch this.

Bugs vs Blueprints

2010-07-22T13:12:25+00:00

Launchpad has two facilities for filing tasks that need to be done in a source tree, Bugs and Blueprints. I've spoken to folks who think the two should be collapsed in to one thing, and folks who think they need to remain separate. I'm in the latter camp, so I thought I'd weigh in real quick with a quick explanation of what I think the difference between the two is. Whether they are fundamentally the same object in the backend object model is irrelevant to me - I'm talking about developer workflow and thought.

In my view, it's really simple:

A Bug is a record of something that needs to be done - the what
A Blueprint is a record of how something will be done - the how

If you accept that premise, then you can have a task that has both a bug and a blueprint associated with it. Or you could have a bug with no blueprint (sometimes stating the problem implies the solution) It makes less sense to have a blueprint without a bug - but we do this as a shortcut often times because launchpad doesn't have an automatic 'create a bug for this' feature for blueprints. On the other hand, many of the blueprints the Ubuntu project works with during UDS planning aren't really things with bugs - unless you consider them all to be hows of solving bug#1. Or perhaps there is a missing or merely implied definition of the what for many of them, which might mean that the discussion of how meanders for a little bit.

I think the current structure of blueprints backs me up here. It is intended that a blueprint have a wiki page (implying that perhaps the information might be longer in form) and blueprints have a wonderful dependency graphing feature - which is quite useful for project planning ... "to accomplish this (bug#1) I'm going to do this (blueprint#1) which depends on these other things being done first (blueprint#2 and blueprint#3)"

Additionally, blueprints having an associated wiki page implies that anything with a blueprint is something that should turn in to documentation. If you're writing up a description of how you're going to do something, then perhaps that means when you're done you've written a description of how or why you actually did something, which is sort of what internals documentation is, right?

Now - can things be better with blueprints? Of course. There was discussion at the last UDS of having blueprints have a merge-request style discussion/approval process, which I would whole-heartedly approve of. Additionally, the wiki element could be baked in to the blueprint, and bzr branchable.

But I strongly believe that both serve a strong and fundamentally different purpose.

BlitzDB Crash Safety and Auto Recovery

2010-07-22T09:43:14+00:00

Crash Safety is a big deal in the database league. Lack of durability can lead to all sorts of terrible things upon a catastrophic event. Many projects, especially in the so called NoSQL world compromises crash safety in return for higher QPS. The argument there is that the availability of the overall system should be accomplished by replication since a database server can’t be rescued if the physical disk breaks. I happen to agree with this philosophy but I am also aware that this isn’t a correct answer for everyone. So, what will I do with BlitzDB?

Several relational database hackers have pointed out that BlitzDB isn’t any safer than MyISAM since it doesn’t guarantee crash safety. This is currently true but I plan on making BlitzDB much safer than MyISAM by providing following features.

Auto Recovery Routine (startup option)
Tokyo Cabinet’s Transaction API (table-specific option)

The second feature above would actually guarantee BlitzDB to be crash safe (especially combined with auto recovery) but I won’t get into depth in this post since this topic deserves a blog post of it’s own. Let me just state that this feature will be provided in a form like this:

CREATE TABLE t1 (
  a int PRIMARY KEY,
  b varchar(256)
) ENGINE = BLITZDB, CRASH_SAFE;

From here on, I’ll cover how I plan on hacking auto recovery in BlitzDB.

Auto Recovery Challenges

As I blogged a while back, recovering Tokyo Cabinet is relatively simple. However, this is not a sufficient solution in BlitzDB since the data file (hash database that actually holds the rows) and the index file(s) are independent from each other. That is, the likelihood of the data file and the index file(s) to be inconsistent is very high after a crash. So, how can we hack on this? Pretty simple.

Indexes aren’t Important at Recovery Phase

Because BlitzDB logically separates the data file and it’s indexes, index files aren’t that important. If a server crash had occurred, BlitzDB could delete the index file(s) and recompute them from the data file. Needless to say, this process would involve a lot of random access and computation but it would not dominate the time space of the system since it’s a one-time cost. This approach however has one flaw in it such that the index files can’t be recomputed if the data file is broken or is unrecoverable.

Therefore to guarantee crash safety, BlitzDB must ensure that the data file is unbreakable. This is precisely where Tokyo Cabinet’s Transaction API comes in. I’m planning on using it to protect the data file from breaking. If the data file is protected, the table can be rescued. Simple!

So, that’s what I have in mind for making BlitzDB a safer engine. Unfortunately I can’t start hacking on it immediately since I have several bugs to fix first. Nevertheless I’m looking forward to start hacking on it. This challenge should be quite fun to tackle.

A tale of a bug…

2010-07-22T00:10:19+00:00

So I sometimes get asked if we funnel back bug reports or patches back to MySQL from Drizzle. Also, MariaDB adds some interest here as they are a lot closer (and indeed compatible with) to MySQL. With Drizzle, we have deviated really quite heavily from the MySQL codebase. There are still some common areas, but they’re getting rarer (especially to just directly apply a patch).

Back in June 2009, while working on Drizzle at Sun, I found a bug that I knew would affect both. The patch would even directly apply (well… close, but I made one anyway).

So the typical process of me filing a MySQL bug these days is:

Stewart files bug
In the next window of Sveta being awake, it’s verified.

This happened within a really short time.

Unfortunately, what happens next isn’t nearly as awesome.

Namely, nothing. For a year.

So a year later, I filed it in launchpad for MariaDB.

So, MariaDB is gearing up for a release, it’s a relatively low priority bug (but it does have a working, correct and obvious patch), within 2 months, Monty applied it and improved the error checking around it.

So MariaDB bug 588599 is Fix Committed (June 2nd 2010 – July 20th 2010), MySQL Bug 45377 is still Verified (July 20th 2009 – ….).

(and yes, this tends to be a general pattern I find)

But Mark says he gets things through… so yay for him.2

Extending CREATE TABLE Syntax in Drizzle

2010-07-21T17:37:42+00:00

The flexibility to add table-specific options for things like compression, encryption and optimization can be useful to storage engine developers as this flexibility can open up new possibilities. Here’s what I’m talking about:

CREATE TABLE t1 (
  ...
) ENGINE = my_engine, MY_OPTION = your_arg;

Supporting this is relatively easy in Drizzle and this API feature (and a bit more) is available in MariaDB as well. Unfortunately Drizzle’s method to do this isn’t documented in the Wiki yet but it should be added when our Storage Engine API becomes stable (as in, no interface changes).

Implement StorageEngine::doValidateTableOptions()

Here’s the actual interface.

bool StorageEngine::doValidateTableOptions(const std::string &key,
                                           const std::string &state);

This function is called for each table options given at CREATE TABLE syntax execution. The first argument, key is a const reference to a string that represents the option name. The second argument, state represents the argument given for that option.

Therefore, given: COMPRESSION = YES_PLEASE, key would be “COMPRESSION” and state would be “YES_PLEASE”. The objective of this function is to check whether the key/state pair makes sense to your storage engine. If this function returns false, Drizzle will return an error for the CREATE TABLE query. Personally I think this interface can be improved to be a bit more Developer friendly, such as making life easier to validate numeric values without enforcing the developer to play around with the data. Saying that, given the pace that Drizzle is growing, this could be improved before we know it.

Access Options at StorageEngine::doCreateTable()

Here’s the actual interface for doCreateTable().

int doCreateTable(drizzled::Session &session,
                  drizzled::Table &table_arg,
                  const drizzled::TableIdentifier &identifier,
                  drizzled::message::Table &table_proto);

Given that the options were successfully validated, doCreateTable() is called next. In Drizzle, all information regarding a table (including options) is represented in a Google Protocol Buffer message. A reference to that message object is passed to doCreateTable() as the fourth argument so all you need to do is loop through the options list in the message object and extract what you need. Here’s a minimal example that only takes care of one option.

int n_options = table_proto.engine().options_size();
 
for (int i = 0; i < n_options; i++) {
  if (table_proto.engine().options(i).name() == "my_option_name") {
    // Do whatever you like with this stream.
    std::istringstream stream(table_proto.engine().options(i).state());
  }
}

The above example should be simple to extend to handle multiple options. What’s really important in the above example is that the option name can be accessed with the name() accessor and the state (value) of that option with the state() accessor.

So, that’s all I have to cover for now. I hope this feature will help storage engine developers create and provide useful table specific features for their engine.

Happy Hacking.

Progress with Plugins

2010-07-21T12:00:11+00:00

It’s been almost a month since I finished refactoring the entire client so the client is completely independent of my_getopt(Hurray!:-)). For the past one month I ve been working on the plugin side of drizzle. With the help of Monty’s new interface I ‘m slowly bringing boost::program_options into the commandline options of the plugins. I [...]

Drizzle build 1666 tarball has been released

2010-07-20T18:19:16+00:00

Drizzled source tarball based on build 1666 has been released. In this release:

* Continued PBMS work
* Fixes of some crashing optimizer bugs
* We now rely on YES/NO vs. TRUE/FALSE in INFORMATION_SCHEMA / DATA_DICTIONARY per the SQL standard
* Continued work on logging_stats

The Drizzle download file can be found here

At OSCON

2010-07-20T02:24:15+00:00

I’m at OSCON this week. Come say hi and talk Drizzle, Rackspace, cloud, photography, vegan food or brewing.

Drizzle Statistics Scoreboard (logging_stats plugin)

2010-07-16T00:19:34+00:00

Drizzle has a Scoreboard for collecting statistics (logging_stats plugin) I introduced this about 6 months ago with little user documentation unless you are into reading header files which have a descriptive writeup. Below is a detailed explanation of whats included from the user perspective and if your really interested some design details. A good portion of the user level statistics was inspired by the google v2 patch for MySQL.

Tables

drizzle> use data_dictionary;

drizzle> describe CUMULATIVE_USER_STATS;

+---------------------+---------+-------+---------+-----------------+-----------+

| Field               | Type    | Null  | Default | Default_is_NULL | On_Update |

+---------------------+---------+-------+---------+-----------------+-----------+

| USER                | VARCHAR | FALSE |         | FALSE           |           |

| BYTES_RECEIVED      | VARCHAR | FALSE |         | FALSE           |           |

| BYTES_SENT          | VARCHAR | FALSE |         | FALSE           |           |

| DENIED_CONNECTIONS  | VARCHAR | FALSE |         | FALSE           |           |

| LOST_CONNECTIONS    | VARCHAR | FALSE |         | FALSE           |           |

| ACCESS_DENIED       | VARCHAR | FALSE |         | FALSE           |           |

| CONNECTED_TIME_SEC  | VARCHAR | FALSE |         | FALSE           |           |

| EXECUTION_TIME_NSEC | VARCHAR | FALSE |         | FALSE           |           |

| ROWS_FETCHED        | VARCHAR | FALSE |         | FALSE           |           |

| ROWS_UPDATED        | VARCHAR | FALSE |         | FALSE           |           |

| ROWS_DELETED        | VARCHAR | FALSE |         | FALSE           |           |

| ROWS_INSERTED       | VARCHAR | FALSE |         | FALSE           |           |

+---------------------+---------+-------+---------+-----------------+-----------+

drizzle> describe CUMULATIVE_SQL_COMMANDS;

+----------------+---------+-------+---------+-----------------+-----------+

| Field          | Type    | Null  | Default | Default_is_NULL | On_Update |

+----------------+---------+-------+---------+-----------------+-----------+

| USER           | VARCHAR | FALSE |         | FALSE           |           |

| COUNT_SELECT   | BIGINT  | FALSE |         | FALSE           |           |

| COUNT_DELETE   | BIGINT  | FALSE |         | FALSE           |           |

| COUNT_UPDATE   | BIGINT  | FALSE |         | FALSE           |           |

| COUNT_INSERT   | BIGINT  | FALSE |         | FALSE           |           |

| COUNT_ROLLBACK | BIGINT  | FALSE |         | FALSE           |           |

| COUNT_COMMIT   | BIGINT  | FALSE |         | FALSE           |           |

| COUNT_CREATE   | BIGINT  | FALSE |         | FALSE           |           |

| COUNT_ALTER    | BIGINT  | FALSE |         | FALSE           |           |

| COUNT_DROP     | BIGINT  | FALSE |         | FALSE           |           |

| COUNT_ADMIN    | BIGINT  | FALSE |         | FALSE           |           |

+----------------+---------+-------+---------+-----------------+-----------+

In addition the functionality of “show status” and “show global status” resides in the GLOBAL_STATUS, and SESSION_STATUS tables which are constructed from the Scoreboard. The SESSION_STATEMENTS, and GLOBAL_STATEMENTS tables also reside in the plugin.

Desgin Overview

The Scoreboard is a pre-allocated vector of vectors of ScoreboardSlots. It can be thought of as a vector of buckets where each bucket contains pre-allocated ScoreboardSlots. To determine which bucket gets used for recording statistics the modulus operator is used on the session_id. This will result in a bucket to search for a unused ScoreboardSlot.

Startup Options

logging_stats_max_user_count – this is the maximum number of users that statistics will be aggregated for, if this is exceeded new users will not be logged. This should be allocated sufficiently large enough to not reach the maximum, its memory usage is minimal.

logging_stats_bucket_count – the number of buckets to allocate, see Design Overview on bucket description.

logging_stats_scoreboard_size – the number of ScoreboardSlots to pre-allocate, this represents approximately the number of concurrent sessions the system can have. If the system were to exceed this statistics would not be logged for a particular session. It should be allocated to be sufficiently larger then the maximum number of concurrent sessions that is anticipated.

Future Options

This is to be determined based on what users want, the infrastructure is there now to make adding and aggregating new statistics fairly easy.

linux.conf.au 2011 CFP Open!

2010-07-15T00:59:23+00:00

Head on over to http://lca2011.linux.org.au/ and check it out!

You’ve got until August 7th to put in a paper, miniconf, poster or tutorial.

Things I’d like to see come from my kinda world:

topics on running large numbers of machines
latest in large scale web infrastructure
latest going on in the IO space: (SSD, filesystems, SSD as L2 cache)
Applications of above technologies and what it means for application performance
Scalable and massive tcp daemons (i.e. Eric should come talk on scalestack)
exploration of pain points in current technologies and discussion on ways to fix them (from people really in the know)
A Hydra tutorial: starting with stock Ubuntu lucid, and exiting the tutorial with some analysis running on my project.
Something that completely takes me off guard and is awesome.

I’d love to see people from the MySQL, Drizzle and Rackspace worlds have a decent presence. For those who’ve never heard of/been to an LCA before: we reject at least another whole conference worth of papers. It’s the conference on the calendar that everything else moves around.

Drupal 7 with Drizzle

2010-07-12T07:00:00+00:00

I wrote an article on the company blog today about how to configure the latest Drupal 7 alpha release to work with Drizzle as the backend database.

Feel free to check it out if you are interested.

Log Buffer #194, A Carnival of The Vanities for DBAs

2010-07-12T04:10:40+00:00

We’re well into summer and almost at our 200th edition of Log Buffer, a blog of blogs about the database world.

Remember if you find a link or interesting blog post that you think Log Buffer should mention, send a note to the editor at Log Buffer and be sure to include the link, and a short note outlining why you think that particular post would be of value to other DBAs, or what you learned from reading it.

Now on to our weekly reading in Log Buffer #194:

Oracle Exadata is a topic that is getting hotter and hotter. Following Pythian’s announcement of Exadata services Oracle’s Greg Rahn stresses how important it is to “capitalize on the opportunity to re-engineer with Exadata and fully exploit the power of the Oracle Database Machine platform”. Also speaking of Exadata, Kerry Osborne talks on his blog, about the Oracle 11g Release 2 new feature, Parallel Queuing, and how it applies to data warehousing workloads on Exadata. Marc Fielding begins his grand tour of Exadata, writing to share his implementation experience.

Jamie Thomson outlines a number of reasons why a SQL server or BI developer should learn .net. And with more activity on SQLblog.com, Jonathan Kehayias writes about a bug in SQL Server 2008 and why you shouldn’t create large tables in the Model Database.

On In Recovery, Paul Randal posts on Benchmarking: Introducing SSDs (Part 2 – Sequential Inserts), following his first set of tests and analysis of the results. In Part 1 and Part 1b he covers not overloaded log file array.

Willie Favero addresses the demise of private protocol, one last time (he hopes) on his blog Getting the Most out of DB2 for z/OS and System z.

Ronald Bradford posts a quick one on installing MYSQL on Oracle Enterprise Linux and other good advice for optimizing SQL performance using the art of elimination.

Barry Leslie announces PBMS is now in the Drizzle tree.

And to sum things up, Gwen Shapira shares a few of her favorite recent archives, a little older than our weekly news, but valuable nontheless.

Not strictly a database topic, but certainly an important lesson for data architects. Brian Carper shares the difficult life of an edge case. Keep him in mind next time you develop an online shop. The same Brian Carper shares why he migrated from MySQL to NoSQL (Tokyo Cabinet) and why he moved back to SQL (Postgres this time).

Joshua Drake warns the end of life is near for Postgres 7.4, 8.0 and 8.1 and that anyone using these versions should upgrade.

Jonathan Lewis explains the difference between unrecoverable and nologging operations.

Martin Widlake gives an overview and demonstration of a little know view, DBA_TAB_MODIFICATIONS.

And, lastly to close things off on a light note, Iggy Fernandez highlights some of his favorite Dilbert-on-Databases Cartoons.

dedcode

2010-07-11T06:54:41+00:00

It has been a while since I didn’t wrote on my GS0C project “Query Cache”, though many advancements have been done. So far I can say that I have a much clearer idea of the query processing flow and the different data structures used. I have been able to write a first plugin that can actually cache a query into memcached and keep some metadata on a local map object. Another cool option is to use data dictionary views to check int real time the cache content (thanks to the very easy interface). The next step from here is to write the retrieval functionality, this is probably the most difficult part for me cuz I still need to understand the functioning of the client interface.

Before making any further step, here is wiki I have posted on the of drizzle’s wiki page. It is basically an explanation of the query cache plugin interface and the hooks, this has yet to be approved and merged into drizzle trunk.

== Principle of the Query Cache ==
The Query Cache permits to return the results of a query from a cache repository if the query has been already cached; or, to add its results to a cache repository if it’s not cached yet. Therefore the simplest capability that a Query Cache plugin interface needs to provide is the ability to say if the received query is cached or not.

Then:
If the query is cached:
- Retrieve the resultset from the cache and skip the query parsing/processing

If the query is not cached:
- Initiate the Query Cache plugin, where the specific plugin will:
- Check if the query is cacheable
- Reserve a temp area and return a pointer to the session
- Build the hash key
- etc
- Add the results to the current temp area.
- Push the the temp variable and the respective cache key to the cache.

== The Query Cache plugin Interface design ==
In the current design we have introduced the following functions to the interface (drizzled/plugin/query_cache.h)
=== static bool isCached(Session *session); ===
This method is responsible to probe the used cache system for an entry corresponding to the received query.
”Note: The plugin is responsible to check if the query is a Select.”

=== Static bool sendCachedResultset(Session *session); ===
This method is called directly if the query is cached and skip the lexical and query processing.

=== static bool prepareResultset(Session *session); ===
This method is called before the query handling (see: handle_select() method) in order to prepare the resultset that will receive the generated data. An opened session will have its own resultset variable and current query key, the plugin will have to initiate these two variables added to the Class Session.

=== static bool insertRecord(Session *session, List<Item> &item); ===
The result rows generated will ultimately go though the client interface, the query cache system intercept these calls and populate the resultset with the data being send to the client (see: select_send() method)

=== static bool setResultset(Session *session); ===
Once the query processing is terminated this method will send the resultset variable to the cache repository and finalize the environment variables.

The query resultset being stored and populated in the session is of type Google Protobuf:

drizzled/message/resultset.proto

= Plugin Hooks =
The following branch contains the plugin interface hooks in the kernel drizzle:

[lp:~dedzone/drizzle/query-cache-hook/changes]

== The query interception ==

Upon reception of a query, (sql_parse.cc:717) in the mysql_parse method, and before even proceeding to the lexical and query processing, a call to ”isCached(session)” is made, a session pointer is passed for the query text, schema, and may be other parameters (for future use).
If isCached returns ”’true”’ then we’ll try to retrieve the results from the cache with ”sendCachedResultset”. If the retrieving operation succeeds (returns false) then we skip the remaining steps.
In the contrary – ie:
- If fetching the results fails or
- If the query is not cached
Then => proceed normally with the query parsing and processing
<pre>
void mysql_parse(Session *session, const char *inBuf, uint32_t length)
{
lex_start(session);
session->reset_for_next_command();
/* Check if the Query is Cached if and return true if yes
* TODO the plugin has to make sure that the query is cacheble
* by setting the query_safe_cache param to TRUE
*/
bool res= true;
if (plugin::QueryCache::isCached(session))
{
res= plugin::QueryCache::sendCachedResultset(session);
}
if (not res)
{
#if defined(DEBUG)
errmsg_printf(ERRMSG_LVL_DBUG,_(“Results retrieved from cache”));
#endif /* DEBUG */
return;
}
LEX *lex= session->lex;
Lex_input_stream lip(session, inBuf, length);
bool err= parse_sql(session, &lip);
……..
……..
lex->unit.cleanup();
session->set_proc_info(“freeing items”);
session->end_statement();
session->cleanup_after_query();
}
</pre>

== The Cache preparation and reception of the Resulset ==

In our journey executing a query, if the query is a Select will land in the (sql_parse.cc:494) execute_sqlcom_select method, this is basically the entry point of the query processing because it prepares to launch the handle_select method. before making this call will insert our prepareResultset hook, which will instruct the plugin implementation to:
- Prepare a cache to receive the resultset, and link it to the session::resultset pointer.
- create a hash key for our query, and link it to the session::query_cache_key member.
At this point the Session will be ‘aware’ that there is a plugin trying to cache the results of the query.
Immediatly after executing the handle_select, where all the resultset were generated, send to the client and added to the current resultset (As we shall see in the next session), we make a setResultset call, this will:
- Push the resultset to the cache area initiated by the plugin
- Reset the session members, resultset and query_cache_key.
<pre>
bool execute_sqlcom_select(Session *session, TableList *all_tables)
{
LEX *lex= session->lex;
select_result *result=lex->result;
bool res= false;
…..
…..
if (!result && !(result= new select_send()))
return true;
/* Init the Query Cache plugin */
plugin::QueryCache::prepareResultset(session);
res= handle_select(session, lex, result, 0);
/* Send the Resultset to the cache */
plugin::QueryCache::setResultset(session);
if (result != lex->result)
delete result;
}
}
return res;
}
</pre>

== Populating the resultset ==

The idea behind adding data to the resultset is to intercept the send_data() method in (select_send.h:87) When sending a row to the client interface. So the hook will just pass the row (items) to the query cache plugin using insertResultset(session, items).
<pre>
/* Send data to client. Returns 0 if ok */
bool send_data(List<Item> &items)
{
…….
…….
while ((item=li++))
{
if (item->send(session->client, &buffer))
{
my_message(ER_OUT_OF_RESOURCES, ER(ER_OUT_OF_RESOURCES), MYF(0));
break;
}
}
/* Insert this record to the Resultset into the cache */
if (session->query_cache_key != “” && session->getResultsetMessage() != NULL)
plugin::QueryCache::insertRecord(session, items);
session->sent_row_count++;
……
}
</pre>

Testing Drizzle's transaction log

2010-07-09T22:12:48+00:00

One of my first tasks on the Drizzle team was to check out the transaction log, part of Jay Pipes' work on replication. We needed to make sure that the log itself was rock-solid and that any code written to use it would have a worthy foundation. As it turns out, some people are really interested in getting replication working ; )

As can be seen from the blueprint, testing will involve two of our main testing tools, the randgen and our standard test suite (test-run, dtr, etc). So far, our efforts have been concentrated on writing tests for the test suite. These tests have largely consisted of executing queries that should make it into the transaction log, then using the UDF print_transaction_message($log_file,$log_offset) to examine the log's contents. The tests can be found here and executed with ./test-run --suite=transaction_log. (NOTE: The tests themselves are largely complete, but our expected results are not yet finalized / correct due to the bugs noted below). I'd like to extend a special thanks to Joe Daly for his help in writing these tests and finding some bugs in the log.

While we have tested all of the available data types and basic transactions and they are looking quite solid, our efforts have still turned up some interesting bugs:

(this list is maintained at the launchpad blueprint noted above)

Bug #594873: transaction log incorrectly records NULL INSERT into an ENUM as the first permissible value
Bug #594876: transaction log not differentiating NULL values and empty string for char columns
Bug #596972: transaction log logging 0 for NULL INTEGER VALUES
Bug #599582: Transaction log does not appear to be collecting all information for multi-row REPLACE statements
Bug #597910: Transaction log does not appear to be capturing FOREIGN KEY information
Bug #599851: Transaction log not capturing CASCADE'd UPDATE data on a table using a foreign key
Bug #599952: transaction log attributing statements to wrong tables on multi-table, multi-statement transactions
Bug #600032: rollback to savepoint not handled correctly in transaction log
Bug #600795: LOAD DATA INFILE INSERTs not registered by the transaction log

The first 3 bugs relate to how the log captures NULL values for various data types and have been traced to a single root cause. The remaining 6 seem more interesting and serious for replication functionality. To summarize, the transaction log appears to have trouble with trickier (more complex) transactions. These include changes propagated via a foreign key, multi-row REPLACE, and multi-table transactions.

Once these bugs have been addressed, we will continue with test development for the randgen. This will include stress testing the server / log with DDL-generating grammars and validation of the data stored within the log. Initial experiments with high-volume (10+ threads at 250k queries each) have looked promising (no crashes, locks, major slow downs over time). In the meantime, we'll be busy with other tasks (dump+restore)

As always, we welcome any feedback, questions, or bug fixes : )

More information on Drizzle testing can be found here.

Drizzle, WordPress, Amazon EC2

2010-07-09T02:57:24+00:00

After some work I finally got Drizzle, WordPress and Amazon EC2 playing nicely with each other. The backend server for this WordPress site is running Drizzle, and the front end client uses the MySQL libraries for connecting, this is needed as there currently is not a fully functional PHP interface that uses the Drizzle client interface. WordPress required a few changes as Drizzle did not have support for all the same data types.

Why?

Well theres a saying "Eat your own dog food" which sums it up.

Heres a patch for WordPress which contains the schema modifications necessary for running wordpress with Drizzle.

schema.php patch

Ill be working with WordPress to see if they would like these changes incorporated into their source tree.

PBMS in Drizzle

2010-07-08T06:53:45+00:00

Some of you may have noticed that blob streaming has been merged into the main Drizzle tree recently. There are a few hooks inside the Drizzle kernel that PBMS uses, and everything else is just in the plug in.

For those not familiar with PBMS it does two things: provide a place (not in the table) for BLOBs to be stored (locally on disk or even out to S3) and provide a HTTP interface to get and store BLOBs.

This means you can do really neat things such as have your BLOBs replicated, consistent and all those nice databasey things as well as easily access them in a scalable way (everybody knows how to cache HTTP).

This is a great addition to the AlsoSQL arsenal of Drizzle. I’m looking forward to it advancing and being adopted (now much easier that it’s in the main repository)

Drizzle build 1644 tarball has been released

2010-07-07T01:27:31+00:00

Drizzle source tarball based on build 1644 has been released. In this release:

syslog plugin added
PBMS is now a part of trunk

The Drizzle download file can be found here

Drizzle JDBC 0.7

2010-07-04T20:40:40+00:00

Just pushed Drizzle JDBC 0.7 to the maven repository, it contains bug fixes and features both for the MySQL and Drizzle support;

Support user name and password in Properties when connecting.
A lot of datatype fixes thanks to the guys at Continuent and their Connector test suite.
Initial implementation of ParameterMetadata.
Fix umlaut bug reported by Christian Peter.
Fix getTables to work properly after bug report from Christian Peter.
Allow zero-length passwords, bug reported by Christian Peter.
Throw proper exception when connection to server times out.
Implementation of setObject with type conversions.

The binaries are in the central maven repository, http://repo1.maven.org/maven2/org/drizzle/jdbc/drizzle-jdbc/0.7/. Or, if you use maven, add the following to your pom.xml:

<dependency>
<groupId>org.drizzle.jdbc</groupId>
<artifactId>drizzle-jdbc</artifactId>
<version>0.7</version>
</dependency>

The code is in launchpad.