Planet Drizzle

Stewart Smith: 181 branches

Wed, 07 Jan 2009 00:21:12 +0000

That’s what launchpad is reporting r.e. Drizzle. Also, 38 (of the 42 people it lists) have committed in the past month. That is impressive.

Wow… I guess this means our little project is fairly popular… or something like that.

Brian Aker: To Twitter or Not

Mon, 05 Jan 2009 17:30:13 +0000

Almost a year ago I made the post "Twitter..." about my often asked question of the time about whether I should be writing an LJ blog post or just tweeting a thought.

As total number of posts go, Twitter won last year hands down. I comment about my life daily on it.

Want to know what I do with Drizzle day to day? Read twitter.

Want to know about the things which give me a chuckle most days? Read Twitter.

Twitter forces me to boil my thoughts down into simple statements. Fewer characters mean I waste less time getting to the point (talk about rethinking what you learned in college about trying to fill paragraphs in order to write longer papers!).

Unlike my blog which is filled with random photos and lengthy posts on technology, Twitter is more a dump of my mind (though far from all of it). I've not bought into the "you must stick to one topic and create a feed which has the characteristics..." Twitter style guidelines.

It sounds terribly dull.

Dull does nothing for me.

Nor does Saturday Night Live for that matter (but Jon Stewart? Well...)

So I will give absolutely zero promises about blogging more this year. What I will do is make Twitter a bit more ubiquitous in my life so that I can drop more thoughts.

And if this had been a twitter post?

"More Twitter, Less LJ, more thoughts this year."

I would even have the room to add in a URL to the tweet for this post :)

Mark Atword: Tweets for Today

Wed, 31 Dec 2008 00:20:22 +0000

00:26 Drizzle is now on Twitter twitter.com/drizzlebugs #
00:31 @abrichar i've found a handful of facebook apps to be useful. i like fbCal and @harper's Flickr app #
04:19 I'm at Shanghai Garden Restaurant II (524 6th Ave S, Seattle, WA 98104, USA) - bkite.com/03jVU #
04:49 I'm at Home Depot (2701 Utah Ave South, Seattle, WA) - bkite.com/03jXI #
05:25 I'm at AMF Imperial Lanes (2101 22nd Ave S, Seattle, WA 98144, USA) - bkite.com/03jZH #
21:40 Doing more Drizzle errmsg hacking. Making the half-done and overly complex be simple and extensible. #

Automatically shipped by LoudTwitter

Mark Atword: Tweets for Today

Tue, 30 Dec 2008 02:26:02 +0000

17:50 My locksmith has a shop cat #
23:01 @twistedmonk can you convince your staff not on direct deposit to start using it, rendering this issue moot in the future? #
23:29 Johnny Cash's "I Won't Back Down" could be a wonderfully angry and powerful piece, that he sings wrongly, like someone who's already beaten. #
23:53 Glad you liked. Sure! #
00:26 Drizzle is now on Twitter twitter.com/drizzlebugs #
00:31 @abrichar i've found a handful of facebook apps to be useful. i like fbCal and @harper's Flickr app #

Automatically shipped by LoudTwitter

Mark Atword: Follow Drizzle on Twitter

Mon, 29 Dec 2008 23:55:19 +0000

I've created a Twitter for Drizzle at http://twitter.com/drizzlebugs

Follow it for announcements, bugs reports, and other Drizzle related stuff

Monty Taylor: Dutch and French winning, Italian close behind

Wed, 24 Dec 2008 13:01:15 +0000

First of all, I'd like to thank everyone who has helped translating messages for Drizzle so far.

Secondly, congratulations to Dutch and French speakers, since you have translations of all translatable strings! (so far, Mark Atwood is working on an overhaul of the error message system, so I imagine we'll find some more strings that need work)

Italian speakers are close - there are only 76 of 1413 messages untranslated.

After the top three, things fall off a bit. German has 744 untranslated, English (UK) has 905 untranslated (although honestly, there probably aren't that many needing translation). Spanish has 1019 untranslated, Norwegian Bokmal has 1049 and Brazillian Portuguese has 1078. Hindi and Polish round out the top ten with 1243 and 1251 untranslated respectively.

So good job everybody, and thanks for all the hard work! For anyone who wants to pitch in who hasn't, you can just head over to http://translations.launchpad.net/drizzle and jump in

Stewart Smith: Drizzle progress… (testing can be good)

Tue, 23 Dec 2008 04:18:23 +0000

We’ve been working on fixing up the remaining test cases so that they run with Drizzle. We’ve found: bugs in Drizzle, bugs in MySQL (one that seems to have been there for at least 10 years), bugs in the tests, tests that no longer apply and occationally, something like this:

/* Please god, will someone rewrite this to be readable :( */ if (to->pack_length() == from->pack_length() && !(to->flags & UNSIGNED_FLAG && !(from->flags & UNSIGNED_FLAG)) && to->real_type() != DRIZZLE_TYPE_ENUM && (to->real_type() != DRIZZLE_TYPE_NEWDECIMAL || (to->field_length == from->field_length && (((Field_num*)to)->dec == ((Field_num*)from)->dec))) && from->charset() == to->charset() && to->table->s->db_low_byte_first == from->table->s->db_low_byte_first && (!(to->table->in_use->variables.sql_mode & (MODE_NO_ZERO_DATE | MODE_INVALID_DATES)) || (to->type() != DRIZZLE_TYPE_DATE && to->type() != DRIZZLE_TYPE_DATETIME)) && (from->real_type() != DRIZZLE_TYPE_VARCHAR || ((Field_varstring*)from)->length_bytes == ((Field_varstring*)to)->length_bytes)) { // Identical fields /* This may happen if one does 'UPDATE ... SET x=x' */ if (to->ptr != from->ptr) memcpy(to->ptr,from->ptr,to->pack_length()); return 0; }

and no, I haven’t really changed the formatting.

Stephen O'Grady: MySQL: Now and Then

Sun, 21 Dec 2008 16:55:20 +0000

MySQL has never shied from being controversial, from being the iconoclast of the open source world. From working with SCO to embedding non-open source assets to the dual licensing strategy, the vendor has ever been willing to push the envelope and experiment.

And while it’s taken a beating in some quarters for a decision here and there, it remains the de facto open source standard for a relational store. In spite of the fact that there are database people that would vehemently claim that it is anything but a relational store. Even its competitors, however, will generally allow that it is as close to ubiquitous as you get in the database world.

As Jeremy says, however, the times, They Are A-Changin.’ For better or worse, we don’t yet know, but it’s a whole new world.

To explore some of these questions, let’s turn to our old friend the Q&A, as we haven’t done one in a while.

Q: Before we begin, anything to disclose?
A: Yes. MySQL, and MySQL’s parent company Sun are both RedMonk customers, as are market competitors such as EnterpriseDB, IBM and Microsoft.

Q: So what’s been happening with MySQL lately?
A: Quite a bit. In case you haven’t been keeping up with current events, the company got acquired, and since then, a few things have happened. They lost a co-founder, their new parent’s financial fortunes took a tumble (prompting advice), and they (finally) kicked 5.1 out the door. Which Monty wasn’t exactly fired up about.

And those are just a few of the obvious items of note.

Q: We’ll get to the non-obvious in just a bit; for now, what’s changed with MySQL as a business?
A: A lot, and a little. Among other things.

In terms of how business is done, what you hear from acquired MySQL employees is what you’d hear from any other small company acquired by a big company: grumblings about the crush of process, death by meetings and so on balancing praise for the advantages of (even) wider distribution and greater resource availability. Ultimately, little has changed externally. I’ve even spoken to a few users who were unaware that MySQL had been acquired, shocking as that might be to those of us spend our time covering such things. Predictions of a decommitment from Linux, for example, have proven groundless, and clearly the MySQL employees still feel empowered to speak their mind without fear of retribution.

From a business perspective, then, it would seem that the aquisition and the subsequent dip in Sun’s fortunes haven’t had much impact. And also, that the more things change, the more they stay the same.

Q: Has being acquired by Sun changed the business model?
A: That question presumes that there was a fixed, static business model in place - an assertion that I would dispute. While some MySQL revenue models - dual licensing profits and traditional support/service - have been consistent, the firm has historically demonstrated a willingness to adjust and tweak various dials on their business model to try to find an optimal (for the vendor) and fair (for the community) balance between revenue generation and community relations. That has continued, as we see with the MySQL Query Analyzer. So no, I would argue: the acquisition by Sun has not materially changed the business model.

Q: So what has changed with MySQL?
A: The community around the product. It’s come a long way, and there are some interesting questions looming.

Q: Well, let’s start with the progress: how has the MySQL community come a long way?
A: As Jeremy says, “Nowadays MySQL is sort of a universe onto itself.” It’s the closest thing the database has to a ubiquitous platform, and that ubiquity has - predictably - incented a variety of behaviors and developments. Some that offer a potential commercial benefit Sun/MySQL, and some that raise certain questions.

Q: Which are potentially good for Sun/MySQL?
A: Well, the creation of appliance and datawarehousing lines of business for MySQL in Kickfire and Infobright expand the addressable market for the product, introduce it to new audiences and potentially help drive its development by posing new technical challenges in workload and performance requirements.

Q: And how about those that “raise questions?”
A: Well, let’s remember what the dual license mechanism is and how it works. Here’s a basic description I wrote a while back:

A single entity such as MySQL is responsible for the overwhelming majority of all development on a given codebase. Anything they don’t produce themselves, they license. Very often this is practiced in conjunction with the dual-license model; because MySQL is responsible for virtually all of the development of the core code, they own or have licensed appropriately all of the involved IP. As such, they’re free to issue commercial licenses to those who would cannot or choose not to comply with the terms of the open source license - the GPL, in this case.

Generally, this model has served MySQL fairly well. By controlling the intellectual property, they retain the rights to relicense the code, thus protecting a revenue stream. They also were afforded a slightly greater protection from forks versus more collaboratively developed projects like Linux, in that they - theoretically - employed the majority of the people qualified and paid to work on the codebase at the lowest levels. But let’s come back to that.

What’s the catch to the model? In part, it’s that the burden of development is born almost entirely by the MySQL staff, but the more relevant concern here is the inability to consume external contributions - even if they’re excellent - without licensing them.

Stated more simply: as long as MySQL remains committed to the dual licensing model, it will be unable to accept the same patch set that open source only versions of the code can, because they do not share the same licensing concerns. Which is why we’ve seen these spring up, and possibly why the MySQL-derived Drizzle project has taken a different approach from its parent.

Q: Where are these patches being accumulated?
A: In external binaries from the likes of Percona and OurDelta, who add patches aimed at improving performance and so on and release versions of MySQL that MySQL itself cannot. In a manner of speaking, Jeremy is exactly right when he says “you can get a “better” MySQL than the one Sun/MySQL gives you today. For free.” I’d argue it’s more complicated than that, particularly for enterprises that might prioritize support and service over the latest and greatest features, but his argument is certainly reasonable for certain market segments.

Q: Is that legal?
A: Sure. As long as the patches and original source are released under the GPL (and to be applied and distributed, they have to be), it’s all perfectly legal. MySQL itself could even consume the patches if it wished, but it would forfeit, assuming they couldn’t license it all, the ability to exclusively license the database. So it’s a tradeoff: revenue from exclusive database licensing rights vs a more collaboratively developed product.

It will be interesting to see how MySQL evolves its model going forward, because the trend to me seems clear: more and better patches for MySQL are emerging from the community all the time. Which on the one hand is great for the product, but on the other raises questions for the firm.

It’ll be interesting to see MySQL evolve and adapt its models to a changing ecosystem; certainly they’ve shown no hesistancy to experiment in the past.

Q: You mentioned Drizzle; where does it fit in?
A: Well, Drizzle is a very different project from MySQL aimed at a very different audience, though some would say it’s the original MySQL audience. So in spite of the fact that Drizzle is derived from MySQL, it’s being built using a fundamentally different model and has almost entirely different design goals. More on that project here and here, but while it’s early days, it’s definitely one to watch.

Q: Given Drizzle’s MySQL heritage, is it sharing changes and patchsets with its parent, and vice versa?
A: I wondered about that myself after reading Ian’s entry here, but Brian was kind enough to drop by and comment on that, describing the current state thusly:

In general though? There is no bug flow from Drizzle to MySQL. So all of the bug fixes we have made to turn Drizzle into an ACID compliant/OLTP type system? I don’t see those going back (and there are hundreds). On the same token their bug fixes? Other then the bug fixes for the optimizer we just do not see them.

Both projects get fed Innodb, so as far as that goes we are similar (though we have community pushing for Google/Percona patches).

Apples to oranges, then, even on a bug level. Note especially the comment on the Google/Percona patches; that would be possible in Drizzle where it is not in MySQL precisely because of the difference taking in the licensing and development model.

Q: What do you think the future holds for MySQL, generally?
A: MySQL is in good shape, in my view, because they have achieved volume: something that will be increasingly difficult to do, I believe, because of the trend towards diversification. The primary task ahead for them, as it has ever been, is using that volume for better monetization; that would ease any and all of the challenges mentioned above, I believe. As for how they do that, I remain convinced that the clearest path to increased monetization lies with leveraging telemetry. MySQL users are generating petabytes of useful data every day without lifting a finger, but to date, no one is leveraging that. Query Analyzer can provide interesting insights on a customer by customer basis, but what if it had access to telemetry from MySQL users all over the world? A different value altogether. That data would have massive value in aggregate, to a variety of audiences: MySQL, MySQL customers, MySQL partners, and - potentially - hardware designers.

That’s where my focus would be, if I was in their shoes.

Stewart Smith: Speaker: MySQL Conference & Expo 2009 - O’Reilly Conferences, April 20 - 23, 2009, Santa Clara, CA

Fri, 19 Dec 2008 06:52:35 +0000

Yes, I’m speaking at the upcoming MySQL Conference & Expo 2009 - on April 20 - 23 (and yes, it’s in Santa Clara again).

I have three sessions:

MySQL Cluster Tutorial: this time with 6.4 feature goodness. Very hands-on, very interactive.

MySQL Cluster on Windows: (insert witty text about hating operating system freedom here)

Memory Management in MySQL and Drizzle: not magic setting of buffer variables, but memory allocation and management inside the server, a bunch of malloc() discussion and hopefully some interesting numbers.

Monty Taylor: drizzle + xtradb == ?

Thu, 18 Dec 2008 17:31:11 +0000

Vadim announced Percona's XtraDB a couple of days ago, and then followed up with some really fun benchmarks. Of course, the first thought I had was "I need another cup of coffee," which was shortly followed by "I should merge that in to Drizzle!"

First step was merging it in to a branch of the innodb-plugin-upstream branch, making an

lp:~mordred/drizzle/xtradb-upstream

Merged it up to drizzle trunk, and now we have:

lp:~mordred/drizzle/xtradb

But what I really want is for our changes to InnoDB/XtraDB to get merged in. So I also make a branch of XtraDB and applied our stuff to it:

lp:~mordred/percona-xtradb/drizzle

My goal there would be a chunk of code that will compile for either MySQL or Drizzle. I've done what I can for that - with the exception of actually attempting to compile it under MySQL, so it might need a new patch or two.

That has been proposed for merge into the percona-xtradb trunk.

As far as I can tell, that's just about all of the permutations of the code that could possibly be interesting to anyone. Have fun!

Jay Pipes: I Didn't Say "Screw Windows"

nospam@example.com (Jay Pipes) — Thu, 18 Dec 2008 15:31:51 +0000

A number of commenters on my previous entry thought I was basically saying "Screw Windows". Lukas Smith and Bill Karwin, both of whom I respect enormously, noted that Windows is a dominant development platform for MySQL users, and that one of the reasons for MySQL's popularity was that it runs smoothly (for a while now) on Windows.

Bill and Lukas: You are 100% correct.

That said, what I wrote was this:

Forget Windows for now: Use open source, community-maintained, and standardized libraries within the kernel. Don't rewrite libc and various other quality open source libraries because of Not Invented Here syndrome or because Windows lacks these things. Focus on the standards and don't bother with platforms that don't conform to POSIX. If Microsoft wants future MySQL versions to run on its platforms, partner with Microsoft and have them do the port. While you're at it, drop support for old platforms like Netware and other crap that is obselete.

The above is advice to the team in charge of re-architecting the database server. What it boils down to is this: focus first and forement on using standard, community-maintained open source libraries and on creating a neatly-architected, clean and lean kernel. After that, worry about Windows ports.

So, it's not about kicking Windows to the curb. It's about priorities. The priority should be clean code. Because having clean, easy-to-understand code leads to:

A more modular architecture
Exponential increase in community contributors (because many more people can actually understand the code)
A more stable product (because fixing one thing has much less chance of breaking another)

I posit that the above three things would make a Windows port much easier, and cleaner. More developers able to contribute, less time trying to figure out the spaghetti, and less chance the port would break tangential pieces of code.

And thus, I believe a solid Windows port of a core kernel would be easier and faster to complete once a POSIX-compliant and clean kernel is completed.

Jim Winstead: how open is drizzle?

Thu, 18 Dec 2008 02:46:39 +0000

one piece of jay’s advice to mysql got me thinking about something that bugs me about drizzle development. jay said:

Make all decisions open and transparent: For the non-maintenance team, make a policy that all decisions about the kernel design be done in an open forum, with the community able to participate in the discussion. Have stewards that are willing to negotiate the design decisions with the community and do everything in a transparent manner.

since jay is one of the key lieutenants in the drizzle effort, it only seems to fair to put them up against that standard. one thing i have noticed is that there is relatively little discussion on the drizzle mailing list about all the coding that is going on. the latest discussions mostly seem to be flypaper for standards wonkery and taking shots at mysql.

meanwhile, you have big refactorings of code like mark atwood’s plugin-ization of several features that seem to just go into the code without any discussion of the design or implementation.

maybe all the discussion is happening on the irc channel, but that is not a great way to get much of the potential community involved. how much of it is happening off-line? maybe more than the core drizzle developers realize, or would care to admit.

Stewart Smith: VirtualBox 2.1.0 (and OpenSolaris 2008.11)

Thu, 18 Dec 2008 00:05:15 +0000

Upgraded VirtualBox and booted up my OpenSolaris VM. VirtualBox 2.1.0 finally fixes the bug where if 127.0.0.1 was in resolv.conf on the host - no DNS for you in the guest (unless in the guest you were running a DNS server).

Haven’t tried it yet… but OpenGL Accelleration makes at least a checkbox appearance in VirtualBox 2.1…. so that could be rather awesome.

Going a lot better with OpenSolaris 2008.11 than previous releases.. It looks like it might be quite easy to get to the stage of building Drizzle on it.

Just figured out how to change to Dvorak! Yay, I can type again! (Go to Input Methods preference panel and add US/DVORAK as a language, move it to the top, and enable the input method application and do it that way).

Currently installing sunstudioexpress. Why not gcc? I’m pretty sure the version in OpenSolaris is still ancient (so won’t build drizzle) and Sun Studio does produce different warnings (which indicate real bugs in a bunch of cases).

Things I wish were packaged: latest protobufs, latest bzr, gcc 4.x

Brian Aker: Fun with ☃, ☢, and, ☣ :)

Wed, 17 Dec 2008 01:40:23 +0000

Don't try this at home :)

create table ☃ (☢ int, ☣ int, key ☢ (☢), key ☣ (☢,☣)) ENGINE=MYISAM;
insert into ☃ (☢) values (1);
insert into ☃ (☢) values (2);
explain select ☢ from ☃ where ☢=1;
drop table ☃;

And no... this is not for live code, it is in a test case I am working on for Drizzle.

Toru Maesaka: Drizzle’s String Library Diet

Tue, 16 Dec 2008 08:03:54 +0000

Lately I’ve been spending most of my time with Drizzle working towards the Cirrus milestone. Specifically speaking, I’ve been slowly standardizing the codebase by throwing out lots of code in MySQL’s string library and replacing them with appropriate libc and C++ alternatives.

You see, back in the 80s MySQL had reinvented a lot of the string functionalities provided by libc for reasons that I do not know (because it was before my time). Turns out that most of the code is still in use today and I guess there was a good reason back in the day but nowadays this doesn’t seem to make much sense, since:

Despite the criticisms, glibc works darn well.
The priority of optimizing library functions is much higher for standard library developers than it is for you as an application developer.
Using the standard library also helps new Drizzle community developers understand the codebase much faster from seeing functions that they are already familiar with.

Arguably, being returned a pointer to the terminating NULL like most of MySQL functions makes string appending slightly easier but if you ask me, many people (including myself) are not comfortable with this and it makes the codebase look weird, IMHO. An example of this is having to rewind the pointer when passing the string to a third-party function.

Benefits gained from narrowing to UTF-8

Because UTF-8 is the prominent encoding in the areas that we are targeting (web and the cloud), currently Drizzle uses only UTF-8 for its internal representation. So needless to say, support for anything other than UTF-8 were thrown out from the library which helped reduce the size of the library greatly.

Interested in how much slimmer the Drizzle string library is compared to the original one in MySQL 5.1? To illustrate the difference, here are the results from counting the files and lines:

$ wc -l mysql-5.1.30/strings/*.c
...
96798 total
 
$ ll mysql-5.1.30/strings/ | wc -l
78

$ wc -l drizzle/mystrings/*.cc
...
24634 total
 
$ ll drizzle/mystrings/ | wc -l
31

AWESOME.

Stewart Smith: fixing drizzle on linux sparc

Tue, 16 Dec 2008 00:40:05 +0000

Since I got fed up with Solaris the other day, the T1000 is running Debian. This means that “I’ll care about Drizzle on Linux Sparc”.

OMG were things broken in the most “trivial” ways.

A good quick intro to the issues is Memory alignment on SPARC, or a 300x speedup!

It all comes down to memory alignment.

So I pulled the MySQL 6.0 bzr tree onto the box to try it too… I haven’t seen so many compiler warnings in ages (okay… since I last built MySQL.. drizzle is warning-clean and it makes it hard to remember a time before that). I think it works purely by accident.

So I’m gradually getting all of Drizzle working on Linux Sparc (a few things fixed already).

It’d be great if the T1k had faster disk though (make -j30 is fun… but IO isn’t on a single 160GB 7200rpm disk)… anybody wanting to donate an SSD?

Mark Atword: Drizzle is now my job

Mon, 15 Dec 2008 20:02:25 +0000

I've been involved with the Drizzle project since very soon after it began, working on it on nights and weekends.

That has just changed. As of today, I'm no longer a MySQL Professional Services consultant, instead I'm part of a new division of Sun

Much of my time is to be spent working on Drizzle, with a focus on plugin interfaces and making it work well in Extremely Large distributed environments.

I will be blogging heavily about what I am doing. How I sort that blogging out between my personal LiveJournal, my (mostly unused) Sun employee blog, and maybe some other blog system, remains TBD.

This is going to be fun.

Jay Pipes: My advice to MySQL

nospam@example.com (Jay Pipes) — Mon, 15 Dec 2008 19:07:00 +0000

Here is my advice to MySQL. Take it or leave it. Time will tell whether I'm full of shit.

MySQL 5.1 is out the door. Awesome. Great job to all the folks who fixed the thousands of bugs over the last 3 years. MySQL 5.1 should be faster and more stable than 5.0 because of those bug fixes, and features like partitioning are welcome additions to the small percentage of MySQL users who need that functionality. And, even if there are some bugs in partitioning (what feature doesn't have any bugs?), the partitioning feature is as good or better than other competing products. Good job.

However, going forward, here is my advice to MySQL engineering: stop all work on new 6.0 features entirely. Don't scrap the features, just stop development on them now.

Take one month to figure out how to restructure MySQL engineering and priorities with the following steps:

Suggested Steps

Drop the current roadmap: Continuing down the current roadmap without addressing the core problems of a Frankenstein-like core database server kernel will mean that the current roadmap features will take 2-3 times as long to develop. Stop this now. Refocus on re-architecting the kernel to a 21st century, modular design. Tell sales and marketing that you are taking these steps to ensure the long-term viability of the MySQL name and product line.

Make two teams: a maintenance team which maintains server versions = 5.1 and a team which is dedicated entirely to redesigning the MySQL server kernel into a streamlined, black box. Reduce the headcount of the MySQL engineering team if necessary to contain only those engineers who have the ability to design modular, pluggable systems.

Give up on backwards compatibility: To make the changes necessary without making the kernel even more complex than it already is, you will need to relinquish the idea that backwards compatibility is necessary. Guido van Rossum already made this decision for Python 3, recognizing the need for it. MySQL needs to do the same. SQL_MODE? Scrap it. Only do what is correct according to data integrity and SQL standards. Reduce the code complexity and code paths and you will find that features are easier to develop and fixes are easier to identify.

Forget Windows for now: Use open source, community-maintained, and standardized libraries within the kernel. Don't rewrite libc and various other quality open source libraries because of Not Invented Here syndrome or because Windows lacks these things. Focus on the standards and don't bother with platforms that don't conform to POSIX. If Microsoft wants future MySQL versions to run on its platforms, partner with Microsoft and have them do the port. While you're at it, drop support for old platforms like Netware and other crap that is obselete.

Make all decisions open and transparent: For the non-maintenance team, make a policy that all decisions about the kernel design be done in an open forum, with the community able to participate in the discussion. Have stewards that are willing to negotiate the design decisions with the community and do everything in a transparent manner.

Focus all energy towards the APIs: Think of the server kernel as simply a provider of services. Clearly and consistently define these services (as interfaces and plugin APIs) and have the community of engineers vet the design of these APIs. Once these interfaces are clearly defined, document them on public wikis.

Clean up the abysmal messiness of the code base: Refactor, decruft, and standardize the code base to a C99 (minimal), warning-free, environment that uses stdint, stdbool, proper STL templates, and other stuff that is now standard for 5+ years. Clear your heads of the premature optimization syndrome that infects the code base and makes it messy and cluttered. You will find that there are many community resources that would be happy to help in this effort.

Once done, BSD the kernel and turn it over to the open source community: Once the above is done, BSD the kernel code base and let the community support it entirely. Then, focus your energies on creating value-added features as plugins around that community-supported core kernel. Use the resources and expertise in your engineering department to develop niche addons that paying customers want. Package branded versions of the MySQL server (closed or open source) that include a number of these value-added plugins that target a specific industry, such as data-warehousing or security-conscious environments. Sell those packages as Enterprise packages with an Enterprise price point. Provide all support and services for these Enterprise-branded MySQL server packages.

How Long?

Based on what the Drizzle project has been doing, I predict that doing ALL of the above steps would take approximately 12 months to achieve a version 1.0 of a stable, modular kernel. I believe that features could be developed as plugins to that kernel in less time than if the work was not done and the features for 6.x are developed as they currently are.

Predictions

If the above steps are taken, here is what I predict would be the outcome:

Reduction in maintenance costs of the core server by 80%: By turning over maintenance costs to the community, there will be a reduction in maintenance costs. By simplifying the kernel code base, there will be an even bigger reduction in maintenance costs: since one "fix" won't break other things nearly as often as "fixes" do today. This reduction in maintenance costs means that Sun can allocate more of its internal engineering resources to developing value-added plugins which are sold to customers. Because more developer resources are now dedicated to revenue-producing activities, the long-term viability of the database engineering department is ensured.

Sales and marketing efforts become easier: Currently, MySQL sales and marketing are undeniably hindered by two things:

Lack of differentiation between community and enterprise server versions
Much too long release cycle

By following the steps above, these problems are tackled. A simpler and community-supported kernel means a more stable kernel. A more stable kernel means a shorter, more incremental release cycle. The lack of differentiation is solved by MySQL now being able to focus on value-added plugins in branded MySQL packaging. These branded packages are much easier for a sales force to sell, since they represent clear, differentiated value to the customer.

When sales and marketing of a product become easier, only one thing is bound to happen: a strong increase in sales.

MySQL will once-again return to the Open Source community: Much has been made of the inability of community contributors to get contributions into the MySQL server in a reasonable timeframe. By opening up the design and development of the kernel to the community, MySQL would restore much of the trust it has lost in recent years. Instead of being seen as "throwing a bone" to the open source community every once in a while, Sun/MySQL engineering would be seen as an active and trusted partner in open source contributions, stewardship and development.

Conclusion

Or, I am completely full of it and the above is a waste of time.

Stephen O'Grady: links for 2008-12-14

Mon, 15 Dec 2008 01:04:33 +0000

Fire Brand interviews Bill James - Fire Brand of the American League

"Suppose that two players come to the majors at the same time, one of them a .280 hitter and the other a .270 hitter. . .supposing for the purpose of illustration that all baseball skills may be expressed in batting average. The .270 hitter may start out 14-for-40 (.350), earning him an extended opportunity to play. The .280 hitter may start out 5-for-35 (.143), and may never get another clean shot at a job.

Do you suppose that all of the best singers in the world have top dollar recording contracts, or do you suppose that some of them were just never in the right place at the right time? Do you suppose that all of the best actors in the world are in the movies? Same thing. Being in the right place at the right time, having a few things break your way at a crucial moment. . .it's huge."

valhenson: g-speak: The ultimate anti-RSI

"You know Minority Report? The part where Tom Cruise is moving all that stuff around on the computer with little lights on his fingers? Yeah, well, the guy who designed all that stuff for the movie decided to actually implement it."

(tags: valhenson g-speak oblong minorityreport)

Google Code Blog: Native Client: A Technology for Running Native Code on the Web

Eric Day: UC 2009 Proposals

Fri, 12 Dec 2008 20:03:12 +0000

Like a few others I’ve seen this week, I had two proposals for the MySQL Conference & Expo 2009 accepted. I’m very excited for both topics (for different reasons), and will be blogging about each of them in more detail soon. They are:

libdrizzle: A new client library for Drizzle and MySQL - This will be a detailed introduction of the new library, covering important topics like client-side buffering, non-blocking I/O, and concurrent queries. If time permits I’ll also go over features in the new Drizzle protocol.
Map/Reduce and Queues for MySQL using Gearman - I’ll be co-presenting this one with Brian Aker, and we will show how the work we’ve been doing with Gearman can be tied into MySQL (and Drizzle).

Eric Day: Next Project

Fri, 12 Dec 2008 19:45:52 +0000

I spent this past week down in San Jose, CA at my employer’s office for team meetings and to officially kick-off my next big project. The design and architecture was very well received, and I drummed up some excitement with Gearman and working with the OSS community in general (which we’ve not done too much of in the past). We’ll be developing it entirely on Launchpad under GPLv2, and I’ll be writing a number of blog posts covering each component in detail. Why would anyone else find this interesting? It covers many topics of how to write a high-performance application in the cloud. Specific topics will include Gearman, persistent Gearman queues, eventual consistency data models (and related schemas), lightweight Map/Reduce for real-time applications, and how to combine all this with MySQL and/or Drizzle to build an e-mail storage infrastructure in the cloud. While the use case may be specific to e-mail, its concepts (and much of the code) will be easy to translate to a number of other applications. Also, there will be a number of other projects that will get spawned from this, such as a faster Gearman job server written in C (currently it is in Perl), generic persistent queuing system based on Gearman, and Gearman UDFs for MySQL and Drizzle (those are coming very soon).

This was the final result of a long white-boarding session. Now off to code it…

Stewart Smith: Debian about 1234533 times easier to install than Solaris

Fri, 12 Dec 2008 06:33:34 +0000

After what many hours trying to netboot the T1000 to install Solaris Express, I wondered “how hard is it for debian?”.

Easy. get the sparc64 boot.img, put it on TFTP server, add “filename “boot.img”;” or similar to dhcp, boot the T1000 from the service console something like this “bootmode bootscript=”boot net:dhcp”\n restart -c” and install away!

As for Solaris?

Well… dhcpd.conf:

option space SUNW;
option SUNW.root-mount-options code 1 = text;
option SUNW.root-server-ip-address code 2 = ip-address;
option SUNW.root-server-hostname code 3 = text;
option SUNW.root-path-name code 4 = text;
option SUNW.swap-server-ip-address code 5 = ip-address;
option SUNW.swap-file-path code 6 = text;
option SUNW.boot-file-path code 7 = text;
option SUNW.posix-timezone-string code 8 = text;
option SUNW.boot-read-size code 9 = unsigned integer 16;
option SUNW.install-server-ip-address code 10 = ip-address;
option SUNW.install-server-hostname code 11 = text;
option SUNW.install-path code 12 = text;
option SUNW.sysid-config-file-server code 13 = text;
option SUNW.JumpStart-server code 14 = text;
option SUNW.terminal-name code 15 = text;
option SUNW.SbootURI code 16 = text;

host hurricane {
hardware ethernet 0:14:4f:1e:28:e;
fixed-address 192.168.1.19;
option host-name “hurricane”;
filename “sparc64-etch-boot.img”;
#       filename “sol-nv-b103-sparc”;
#       option SUNW.install-server-ip-address 192.168.1.1;
#       option SUNW.install-server-hostname “saturn”;
#       option SUNW.install-path “/mnt/sol-nv-b103-sparc/”;
#       option SUNW.root-server-ip-address 192.168.1.1;
#       option SUNW.root-server-hostname “saturn”;
#       option SUNW.root-path-name “/mnt/sol-nv-b103-sparc/Solaris_11/Tools/Boot”;

}

(obviously changing the comments around) and having the Solaris Express DVD mounted and NFS exported…. it *still* doesn’t work. It goes “unable to mount filesystem” with no further hints (even when tcpdumping the network).

Documentation for doing the simple thing of using $dhcp_server and $nfs_server to network boot a Solaris install on a Sparc box is *COMPLETELY* missing.

Now, I’m a smart guy (and if you don’t believe that, at least believe I’m not stupid). If I can’t get it to boot the installer, what chance do others have?

I’ll try OpenSolaris out when it’s on SPARC (and please oh please oh please just have an easy way to net boot the installer using a linux host). Please take the debian way (just a single file on tftp).

So now it’s goodbye Solaris (I’m not going to have something I can’t re-install, upgrade or security patch) and it’s hello Debian (and sanity).

Yes, this does mean I’ll care about Drizzle on Linux Sparc.

MySQL Performance Blog: 7 Reasons why MySQL Quality will never be the same

Thu, 11 Dec 2008 03:34:17 +0000

I had a call with Monty the other day and I told him why I think MySQL Server Quality will never be the same again. I've been thinking a bit more about it and here is the extended list.

In particular I think MySQL Server will never be able to reach its original quality guidelines (see previous post) and even current release criterias will unlikely be ever reached with any sensible definition of what serious bugs are.

Large Team, Tricky Code During MySQL 3.23 and MySQL 4.0 team the small tight knit team was working on the server and Monty was personally reviewing all the code and knew all the code well to know all the side effects etc. Now MySQL Development is done by different team working on the different components, while MySQL was not designed to be very modular with clear set interfaces and no side effects on the first place to support such development concept easily.

Increasing Complexity Each new MySQL version is getting more and more complicated. MySQL architecture has to be additionally complicated to support modular storage engines with all their interactions and different specifics. Think for example how many DBMS need to do Distributed Transactions (XA) internally just to keep different components in sync. This is multiplied by the fact a lot of MySQL features were implemented in marketing rush being hacks over architecture which was not intended to support it. Remember early MySQL versions knew only about table locks and were not initially designed to support transactions. Most of other databases are designed with these concepts at their core.

Increased User Base I'm not sure if MySQL user base is growing any more but it surely has grown dramatically since MySQL 3.23 or 4.0 times. More users means they naturally find more bugs. This is actually has the good time too. Even though a lot of bugs being found and reported only small fraction of them apply to any given user. The bad thing - in many cases we do not really know which one apply to us

Low Feature Usage Density MySQL 3.23 was simple so Most of its features were used by majority of users. This was even the early Marketing strategy for MySQL. Marten used to frequently say Oracle is Overkill for most users and most users use only 10% of features or less. You know what ? MySQL is becoming overkill for increasingly large amount of users too. A lot of people in the Web world would be as happy with MySQL 4.1 if it would be actively maintained now. And hey this is exactly one if the reasons behind Drizzle project.

Lower usage density means features are less tested by community during release and they also mean they get less attention from developers, because development team (and especially its efficiency) is not growing on the same pace. This also means fewer customers use the same features, and so the revenue associated with feature is not high enough to keep all the bugs reported for this feature. So bugs sometimes stay open for years.

As an example of low usage density features I would mention MySQL 5.1 features. During my talks I often asked people how many of them use 5.1 vs row level replication or events. I think less than 5% are using both features. MySQL 4.1 was out for years and yet I would say no more than 5% of users use prepared statements.

Bug Reporting and Tracking The MySQL 3.23 received quality was at some extent because there were no bugs.mysql.com - bugs were reported to bugs@lists.mysql.com which meant both it was more complicated for users to report a bug - so fewer people did and it was easily to lose the bugs, especially boring and less critical ones. Monty did a very good job keeping track of these but still not as good as when bug tracking system was implemented and so bug would never disappear unless you fix it or call it a feature

MySQL Enterprise Madness For years MySQL QA policy heavily relied on community and I think it is still the case. This means MySQL latest code was instantly downloaded and tested by thousands of users which would find bugs quickly. With MySQL Enterprise it is customers who got the bleeding edge code and for community the binary which is over half a year old was often offered.

Enterprise customers who got the latest and the most bug free/buggy code (less old bugs more new ones) are fractions of percent of MySQL User base and they are also more conservative - few just would grab latest release as soon as it comes out. Many will only upgrade on the schedule or keep on the same release until they run into some issues. This means bugs in Enterprise version have tendency to stay for long time.

The good thing about this item is - it is easily fixable. Just get somebody in his mind to define release policy, and better have him/her understanding engineering not just marketing.

Innodb/Oracle relationship complexities. If the bug happen to affect Innodb things become even more complicated slow. Even though MySQL and Innodb is tight knit system these are two different code bases controlled by different companies. MySQL can't just do changes to Innodb codebase and Innodb can't do same for MySQL. Both require other side approval. This is why I guess all new Innodb Development is going with Innodb Plugin which is separate from Innodb included default MySQL. Why is not this part of stock MySQL 5.1 ? I guess because MySQL 5.1 release was closed for any significant changes for years.

Now how it all affects bugs an not features ? Well if you want to use latest Innodb you will need to use plugin which is not part of standard MySQL distribution so it, just as MySQL Enterprise, gets significantly less testing by community than standard MySQL.

Another problem for Innodb Plugin quality is lack of the full disclosure about bugs. Check this change log for example. You would see some bugs fixed have bug numbers and large number does not. This means there are some bugs which are discovered by Innodb/Oracle team but they are not disclosed via searchable public bugs database.

I should note besides all these process issues Innodb remains remarkably high quality component. The initial version of Innodb Plugin was very good in terms of quality for version having first public release as we tested it back in April.

The Good News in the End

The good news are you really do not need general "quality" or "being free from bugs" to run MySQL in production. MySQL as it is now works for a lot of applications, in particular because most of them use only small portion of MySQL features.

Entry posted by peter | 15 comments

Add to: | | | |

Stewart Smith: OpenSolaris 2008.11 first impressions

Thu, 11 Dec 2008 03:30:34 +0000

Using the wonderful bittorrent, I got the CD image in next to no time (in contrast to the Solaris Express image I’m currently downloading via HTTP that’s taking forever).

Boot time in VirtualBox (off the ISO image) was rather quick, usual questions on keyboard layout and desired language (it’d be neater if these were GUI questions… but anyway). The GNOME desktop loaded up, popup window informed me that it had connected to the network. Awesome.

Package manager: opend quickly, using the opensolaris.org repository and it does seem to have a lot of packages… even MySQL 5.0.67 (and 4.0.24). Not 5.1 though, but it is early days (and it was just released as GA the other day).

At least one unusual thing was SUNWgrub and SUNWgrubS (where the S is for source). I assume this is some packaging oddity as I don’t ese other packages like this.

SUNWii wins the odd package name award.

The Time Slider seems like possibly the most awesome thing ever. It periodically takes ZFS snapshots of your disk and presents you with a time slider in nautilus so you can just view your data how it was in the past (at previous snapshots).

I can’t see how to change the keyboard layout to DVORAK (at least while booted off the CD image).

The getting started guide also shows how to get a development environment going… this is quite promising. Will do proper install shortly and do a step by step “building drizzle on OpenSolaris 2008.11″ post.

Brian Aker: Map/Reduce, MySQL & Drizzle

Fri, 05 Dec 2008 23:17:35 +0000

Something very cool, is coming very soon, for both MySQL and Drizzle :)

mysql> SELECT gman_do("reverse", "does this work?") AS test;
+-----------------+
| test |
+-----------------+
| ?krow siht seod |
+-----------------+

This is part of the huge amounts of efforts we have going on with Drizzle and the wider set of projects that we are working on (and mad props go to Eric Day for the above example UDF work for MySQL).

Monty Taylor: Merging InnoDB Plugin with bzr is fun

Thu, 04 Dec 2008 00:21:10 +0000

As Brian mentioned earlier, I merged in version 1.0.2 of the InnoDB Plugin last night. Took about 2 hours - most of which was compiling and testing on a couple of different machines to make sure I hadn't b0rked anything.

Getting bzr to help me do all of this was actualy quite easy, once I corrected one mistake about how I had merged the code into the tree in the first place. (oops)

In a move that's sure to shock my former boss, I wrote up a quick wiki page on the process, just in case anyone is interested.

Brian Aker: 5.1, Innodb Plugin Release

Wed, 03 Dec 2008 23:56:15 +0000

In MySQL 5.1 related new, Innodb has announced the release of their Innodb Plugin Engine for MySQL.

So why do I blog about it?

Because the Innodb plugin engine is the default engine Drizzle :)

Monty Taylor finished up the work last night to import it into Drizzle, and I just finished my review of the engine and have pushed it into our main branch.

Some of the advantages:

Fast index creation: add or drop indexes without copying the data

Data compression: shrink tables, to significantly reduce storage and i/o

New row format: fully off-page storage of long BLOB, TEXT, and VARCHAR columns

File format management: protects upward and downward compatibility

INFORMATION_SCHEMA tables: information about compression and locking

Stewart Smith: What constitutes an identifier for a table?

Wed, 03 Dec 2008 23:44:52 +0000

Well… there’s:

database
table name

(both of these are quite obvious).

But then you have:

temporary tables

Well… two types of temporary tables:

those created in the course of query execution, typically in /tmp/
those created during ALTER TABLE, typically in the database directory

You may have seen these “#sql-foo.frm” etc files around.

but you can also CREATE TABLE `#sql-foo` (you know, because that’s a good string to use in your app). In fact, you can (and shouldn’t) create a table with the exact same name as the temporary #sql table and use it in your app.

So really the primary key for a table is: string database name, string table name, bool is_tmp. Oh, and the /tmp/ temporary tables just to screw with your braiiiinn.

In drizzle, this is what the storage engine API is ending up looking like. It’s the responsibility of the engine to encode the temporary table names so that they don’t clash with the non-temporary table names. This could be done by putting them in a different set of files, marking them with som eflag, or just passing the is_tmp flag to build_table_filename (or make_table_key) and being done with it.

I think this ends up being the least-hacky approach and it does mean we can remove the bass ackwards if(table_name[0]==’#') checks from some engines.

Mark Atword: Yet another new Drizzle plugin type: replicator

Tue, 02 Dec 2008 10:03:52 +0000

krow just asked me to add another plugin type to Drizzle, for pluggable replication, and so I have, and it's pushing up to Launchpad now to merge with the mainline.

The plugin type is actually called "replicator", to avoid conflicts with the existing source files called "replication.*"

The existing replication system in MySQL 5 is... well. Like the old joke about if you like laws or sausages, you probably shouldn't watch them get made, if you need to think that MySQL 5 replication is rock solid, you probably shouldn't read the source code. Heck, a serious bug was discovered in it at OpenSQLCamp a few weekends ago, just by projecting a random page of code on the wall and having a dozen pairs of eyes read it.

Stephen O'Grady: links for 2008-12-01

Tue, 02 Dec 2008 01:04:07 +0000

Taking control — The Hardball Times

"To close out, let's try to answer our questions about the dartboard of the opening paragraph. Based on the PITCHf/x results it appears that a fastball thrown at a dartboard placed over home plate will hit the bulls-eye with some part of the ball a little less than 1.5 percent of the time. Fully 46 percent of these pitches would miss the dartboard completely. Just think of that when they tell you about somebody's pinpoint control."

(tags: baseball command control hardballtimes via:rob)
Computers Of The World, Unite! - Forbes.com

standards are indeed important. it's just difficult to get buyers to think beyond the present.

(tags: open standards ibm bobsutor interoperability)
Obie Fernandez: Live from Pro Rubyconf '08 / SMACKDOWN

who's using what in the Ruby world

(tags: ruby rails popularity plugins technologies jquery prototype svn git vim emacs postgres mysql)
Jazz Community Site - Jazz Team Blog » Extending Team Concert collaboration to include SharePoint documents

interesting

(tags: sharepoint rational jazz ibm via:bill)
Fire Joe Morgan: The Exit Interview

more FJM

(tags: sports fjm firejoemorgan blog baseball deadspin)
Ian's Blog

questioning the interaction between the Drizzle and the MySQL projects

(tags: ianholsman drizzle mysql opensource)
Live Piracy Map

the obvious conclusion: there are a lot of pirates

(tags: pirates map data via:jesse)
Writers Halt Site Skewering Sportswriters - TV Decoder Blog - NYTimes.com

still bummed about this

(tags: firejoemorgan fjm nytimes)

Stewart Smith: Is your garage internet enabled?

Mon, 01 Dec 2008 23:50:38 +0000

Real noisy fucker. So loud, that if it’s in the garage but the back door is open, I still hear it.

Being used for drizzle dev on Solaris… although a switch to OpenSolaris or Linux is likely imminent. Straight Solaris 10 is just too annoying.

Mark Atword: Me at the Hackfest at OpenSQLCamp

Mon, 01 Dec 2008 20:29:39 +0000

OpenSQLCamp - hackfest
Originally uploaded by datacharmer

I need a haircut, tho not as short as Jay Pipes, seated next to me.

I think at that moment I was either working on more improvements to the errmsg plugin interface to Drizzle, or else I was adding a new plugin type to enable PBXT's blob streaming protocol.

MySQL Performance Blog: MySQL for Hosting Providers - how do they manage ?

Fri, 28 Nov 2008 22:24:29 +0000

Working with number of hosting providers I always wonder how do they manage to keep things up given MySQL gives you so little ways to really restrict how much resources single user can consume. I have written over a year ago about 10+ ways to crash or overload MySQL and since that people have come to me and suggested more ways to do the same.

This is huge hole in MySQL design, thinking little about users isolations and resource quotas and interesting enough I have not seen significant changes in fresh our MySQL 5.1 GA or even something major on the roadmap for future MySQL versions. May be Drizzle will give it a thought ? This surely would help adoption by (especially low end) Hosting Providers and remember this exactly where a lot of kids start to develop their first sites and play with web technologies.

So how do the hosting providers manage to host hundreds of users on single server with single MySQL server ? Well people just seems to be nice and not looking to crash MySQL/DOS server on purpose but rather cause most of the issues unintentionally by running bad queries or installing bad software.

It is good people are nice but it may not be comforting to know you stay up just because nobody wants bring you down rather than because your systems are solid and designed to prevent such abuse.

The systems which I see people implementing are typically focused on the load eliminating - using Google UserStats patches (included in Percona releases) or Log analyzes as well as PROCESSLIST monitoring. This allows you to crack down on users which cause a lot of load which causes a lot of unintentional abusers, but to get MySQL in trouble you do not need a lot of load. You can do this by very light queries which would not show up in the PROCESSLIST or will not take too much combined time in the logs to attract your attention;

For example:

PLAIN TEXT

SQL:

SET @a1:=repeat("a",1000000); SELECT sleep(1);
SET @a2:=repeat("a",1000000); SELECT sleep(1);
SET @a3:=repeat("a",1000000); SELECT sleep(1);
SET @a4:=repeat("a",1000000); SELECT sleep(1);
SET @a5:=repeat("a",1000000); SELECT sleep(1);
SET @a6:=repeat("a",1000000); SELECT sleep(1);
SET @a7:=repeat("a",1000000); SELECT sleep(1);
SET @a8:=repeat("a",1000000); SELECT sleep(1);
SET @a9:=repeat("a",1000000); SELECT sleep(1);
SET @a10:=repeat("a",1000000); SELECT sleep(1);
SET @a11:=repeat("a",1000000); SELECT sleep(1);
SET @a12:=repeat("a",1000000); SELECT sleep(1);
SET @a13:=repeat("a",1000000); SELECT sleep(1);
...

Causes MySQL to "leak" 1MB of memory per second with no apparent good reason - no slow queries or queries in the PROCESS LIST and you can't really track how much memory was allocated for given session (or restrict this number)

Now remove sleep(1) and you will get mysqld quickly running out of memory and being killed by OOM killer or being unusable for any queries. In my test I got OOM though it kept the box stalled for few minutes before that:

Out of memory: Killed process 1081 (mysqld).
automount invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0

Out of memory: Killed process 1081 (mysqld).
automount invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0

Call Trace:
[] out_of_memory+0x8e/0x2f5
[] __alloc_pages+0x245/0x2ce
[] __do_page_cache_readahead+0x95/0x1d9
[] :dm_mod:dm_any_congested+0x38/0x3f
[] filemap_nopage+0x148/0x322
[] __handle_mm_fault+0x1f8/0xe23
[] do_page_fault+0x4cb/0x830
[] error_exit+0x0/0x84

Do I need any particular privileges for this to happen ? Not really - you do not even need to be able to select from the table. Any user with permission to connect to MySQL Server can crash it.

Entry posted by peter | 13 comments

Add to: | | | |

Brian Aker: Time Warp, Solaris

Fri, 28 Nov 2008 22:01:18 +0000

The date is the summer of 1992 and I have a Sun 1 Sparc Station that I need to turn into a working development environment.

What is the process?

I have to ftp a lot of software off of ftp.gnu.org to my local MicroVax. From there I will download all of the software over my 9600 baud modem to the Sparc sitting in the living room of the house I am living in at the time.

It will take days, if not weeks to assemble all of the software and turn the machine into a working system. This was the state of the art at the time.

Today?

Today I take a similar system, install pkg-get and grab packages. What do I discover?

Half of them are out of date. So what do I do?

I could grab one of the package systems that are put together to make my life easier, but they won't play well with the stuff I have downloaded with pkg-get. Could I ditch pkg-get and just install the prepacked stuff I can find elsewhere but they too are incomplete.

So?

I ftp packages off ftp.gnu.org and compile those into /usr/local.

Except unlike 16 years ago, half those packages will not compile any longer on Solaris. Between pkg-get and ftp.gnu.org I do make it work.

None of this will automatically update though, so I am stuck maintaining it. Open Solaris is not an option because it does not run on this system (and Open Solaris comes with a bunch of old compilers/libraries so it is just a different sort of hassle).

One other incredibly annoying thing, except for what I installed in /usr/local, the rest of programs/libraries it takes to compile anything is found across a half dozen directories in /opt and /usr. The entire structure is not GNU like, and GNU like is the model for other systems. There is nothing innovative about picking your own directory structure.

So what are the main differences 16 years later?

I have broadband so the download is a lot faster.

More disk space means I can download everything at once and not be concerned about having enough space to build a working development environment.

gmake -j. This machine has a lot of cores so compiling software is lot faster!

I was asked on Twitter if I would be supporting development for Drizzle on Solaris if I did not work for Sun.

The answer is "yes".

There are still quite a few Solaris users out there and I am very interested in seeing how well we perform on the T5240. It is an interesting piece of hardware because of the number of hardware contexts it provides us.

Now if I had a simple method for installing Ubuntu on it, would I still care about making Solaris work?

I honestly do not know. I am looking forward to finding the time someday to find out if Solaris or Linux runs best on one of the beasts. My T1000 is with Stewart now so I I don't have a way to test what Linux looks like on a working piece of Niagra based hardware that supports Linux currently. The T5240 does not yet run Ubuntu.

If I found that Linux ran better I would be mighty tempted to leave the Solaris porting to others. There are a lot of things that I could be doing to make better use of my time.

Tim commented on "What Sun Should Do". I have not been at Sun long enough to really make any broad comments on the future of Sun.

One thing is very certain in my mind concerning Sun's future. The problem I described above needs to have been fixed a decade ago, and is at the heart of the failure of Solaris. Many point to the failure of Solaris being CDDL and not being GPL. I believe this is an issue, but it is not the center issue.

My Mac has plenty of non-GPL code on it.

It is also a superior development environment to Solaris.

And why is that?

It just works.

And Linux?

It just works.

Solaris?

Someone needs to take a hard look at what is being done today and get it on track.

Stewart Smith: Temporary tables in any engine….

Tue, 25 Nov 2008 18:54:43 +0000

Well… nearly any engine.

I have a plan forming in my head to add some hooks to engines to help with creating temporary tables (the ones created while executing a query, not ones created during ALTER TABLE).

Currently, if you ALTER TABLE and we require a temporary table, it’s still database.table but we generate a table name that’s small, unique and begins with “#sql”.

I’ve changed some of the handler interface to accept two strings (database name, table name) instead of one “path” that may (or may not) end in “.FRM” and may (or may not) begin with “./” and may (or may not) use the “/” separator between database and table name (hint: on win32, it’s sometimes “\”).

So ha_delete_table is now: ha_delete_table(db, table_name). Sanity!

(this has the downside of being a incompatible change that doesn’t break the build as there seems to be no way in C++ to say “derived classes cannot implement a function of this name”).

This is annoying for temporary tables however.

They don’t reside in a database… they’re off in la-la land (otherwise known as opt_drizzle_tmpdir).

Now… in MySQL 5.1 the ability for multiple temporary directories was added, and the MySQL server will cycle through them. The clusterfuck part of this was that a mutex was added… so every time the code goes to get the name of a (or for 99.99999999% of cases, the) temporary directory, it has to grab a mutex. With modern systems being able to have *many* simultaneous IO operations (e.g. create files) this is just dumb. It’s gone.

If you want to use multiple spindles for temporary tables it’s called RAID people (or buy an SSD, mkfs.ext2 it and just wipe it on reboot. simple.)

Anyway… back to temporary tables:

they’re in /tmp or something. So an API that’s foo(const char *db, const char *table_name) doesn’t work as well.

Also, for a bunch of engines, it’s good to know that you’re using a temporary table. You probably want to store these somewhere that never needs fsync() or anything like that. If the server goes away, these tables are *gone*. So optimise for that.

Heck, it may even be good to store temporary tables in your distributed engine (if that engine is memory based) as it’s often faster to access remote memory than local disk (although with SSD this is a whole different ball game… in fact, I’m not sure if it’s even still a ball game.. it’s possibly frisbee)

But currently there’s hardcoded a mi_create (MyISAM create) call in the server and in the 6.0-maria tree, an #ifdef around if it’s mi_create or maria_create.

So not any engine yet… but one can dream. Yes, I dare to dream.

(although why I was dreaming of a small board with a PowerPC 603, 8MB RAM and a mini DVI port the other night is quite beyond me)

Stewart Smith: I don’t read code comments

Tue, 25 Nov 2008 18:25:21 +0000

They are wrong.

Misleading at best.

Reworking parts of Drizzle (which came directly from MySQL) it can get painfully obvious. Things like “afaiu” and “???” appear in more than one place (that is if the comment isn’t just obviously wrong).

A comment merely states what one person thought the code did at some point in the past. It has no relation to what the code actually does now.

Ronald Bradford - 42SQL: Where is the innovation?

Mon, 24 Nov 2008 17:05:24 +0000

The 2009 MySQL Conference has closed it’s submissions for papers. This year the motto is “Innovation Everywhere”.

Last weekend’s Open SQL Camp in Charlottesville, Virginia, we had the chance to talk about the movements in the MySQL ecosystem. I was impressed to get the details of the Percona MySQL Patches, but focus is still in 5.0. (Welcome to the Percona team Tom Basil) Our Delta is attempting now to integrate patches into various MySQL branches. There was an opening keynote by Brian Aker from Drizzle, and Drizzle team Jay Pipes and Stewart Smith on hand. It was also announced that MySQL 5.1.30 will be GA, available in early December.

But these are not innovations that are ground breaking. Last year, it was the announcement of KickFire that I found most intriguing regarding innovation.

What is there this year?. The most interesting thing I read last week was Memcached as a L2 Cache for Innodb - The Waffle Grid Project. This is my kind of innovation. It’s sufficiently MySQL, but just adds another dimension with another companion technology. The patch seems relatively simple in concept and code size, and I’m almost prepared to fire up a few EC2’s to take this one for a spin. I’m doubly impressed because the creators are two friends and colleagues that are not hard core kernel hackers, but professionals on the front line dealing with clients daily. Will it be successful, or viable? That is the question about innovation.

Unfortunately I spend more time these days not seeing innovation in MySQL, but in other alternative database solutions in general. Projects like Clustrix, Inc., LucidDB, and Mongo in the 10gen stack.

Eric Day: Open SQL Camp

Sun, 23 Nov 2008 07:26:56 +0000

Last weekend I attended the OpenSQL Camp in Charlottesville, VA. There was a great turnout, and Baron did an excellent job organizing it! I saw a few folks I met at OSCON over the summer, along with meeting many new people. What a great group - intelligent, fun, and know how to get things done. I had some great conversations, especially with Brian, Stewart, Arjen, Patrick, Mark, and Jay. The food was great too, I was a bit worried about finding vegan food there. Oh, and there was the wine bar, and my new found love for dessert wine. Yum.

All the sessions I attended were great! Postgres MVCC by Greg Sabino Mullane, Sphinx by Peter Zaitsev, MySQL Self Monitoring Replication by Giuseppe Maxia, Postgres Extensions by Kelly McDonald, Google Proto Buffers by Jay Pipes, OurDelta by Arjen Lentz, and Join-Fu by Jay Pipes. The hackathon on Sunday was fun, many more good conversations and project planning for Drizzle and Gearman.

I had planned to give just one presentation on libdrizzle (slides), but ended up giving another with Brian on gearman (slides). Excuse the gearman slides, they’re a bit weak, but in our defense we threw them together 15 minutes before the talk (it was proposed only a few hours before). There were people really enthusiastic about both talks, and I received some great feedback for libdrizzle.

I’m continuing with the Drizzle and Gearman development with all the spare time I can find, and making good progress on both. I’m in the processing of Doxygen-izing both projects, and plan to have some code for people test really soon!