Database War Stories Serialization

Tag:web2.0,database

[database; web2.0] O'Reilly Radar > Web 2.0 and Databases Part 1: Second Life #
As part of the prep for my keynote on Wednesday at the MySQL User Conference, I decided to ask some of my Web 2.0 friends just how they were using databases in their applications. Over the next couple of days, I'm going to post what I heard back. I'm not going to draw any conclusions till the end of the series, but just let people speak for themselves.

[web2.0; database] O'Reilly Radar > Database War Stories #2: bloglines and memeorandum #
Database War Stories #2: bloglines and memeorandum By tim on April 27, 2006 In Monday's installment, Cory Ondrejka of Second Life said 'flat files don't cut it', but Mark Fletcher of bloglines and Gabe Rivera of memeorandum.com apparently don't agree.

[web2.0; database] O'Reilly Radar > Database War Stories #3: Flickr #
Database War Stories #3: Flickr By tim on April 27, 2006 Continuing my series of queries about how 'Web 2.0' companies used databases, I asked Cal Henderson of Flickr to tell me 'how the folksonomy model intersects with the traditional database. How do you manage a tag cloud?'

[web2.0; database] O'Reilly Radar > Database War Stories #4: NASA World Wind #
Database War Stories #4: NASA World Wind By tim on April 27, 2006 Patrick Hogan of NASA World Wind, an open source program that does many of the same things as Google Earth, uses both flat files and SQL databases in his application. Flat files are used for quick response on the client side, while on the server side, SQL databases store both imagery (and soon to come, vector files.) However, he admits that 'using file stores, especially when a large number of files are present (millions) has proven to be fairly inconsistent across multiple OS and hardware platforms.'

[web2.0; database] O'Reilly Radar > Database War Stories #5: craigslist #
Database War Stories #5: craigslist By tim on April 28, 2006 Eric Scheide of craigslist offered me a stream of consciousness summary of the craigslist database setup. At a conference last year, Craig showed a slide (which helped inspire my postings about asymmetric competition [1, 2, 3]) that listed the number of employees at the top ten web sites. Most of them have thousands of employees. Some have tens of thousands. Craigslist, at #7 on the list, has 19.

[web2.0; database] O'Reilly Radar > Database War Stories #6: O'Reilly Research #
Database War Stories #6: O'Reilly Research By tim on May 02, 2006 In building our Research data mart, which includes data on book sales trends, job postings), blog postings, and other data sources, Roger Magoulas has had to deal with a lot of very messy textual data, transforming it into something with enough structure to put it into a database. In this entry, he describes some of the problems, solutions, and the skills that are needed for dealing with unstructured data.

[web2.0; database] O'Reilly Radar > Database War Stories #7: Google File System and BigTable #
Database War Stories #7: Google File System and BigTable By tim on May 03, 2006 Greg Linden of Findory wrote: 'I've been enjoying your series on O'Reilly Radar about database war stories at popular startups. I was thinking that it would be fantastic if you could get Jeff Dean or Adam Bosworth at Google to chat a little bit about their database issues. As you probably know, Jeff Dean was involved designing BigTable and the Google File System. Adam Bosworth wrote a much discussed post about the need for better, large scale, distributed databases.'

[web2.0; database] O'Reilly Radar > Database War Stories #8: Findory and Amazon #
Database War Stories #8: Findory and Amazon By tim on May 04, 2006 Once Greg Linden had pinged me about BigTable (leading to yesterday's entry), it occurred to me to ask Greg for his own war stories, both at Findory and Amazon. We hear a recurrent theme: the use of flat files and custom file systems. Despite the furor that ensued in the comments when Mark Fletcher and Gabe Rivera noted that they didn't use traditional SQL databases, Web 2.0 applications really do seem to have different demands, especially once they get to scale. But history shows us that custom, home-grown solutions created by alpha geeks point the way to new entrepreneurial opportunities...

[web2.0; database] O'Reilly Radar > Database War Stories #9 (finis): Brian Aker of MySQL Responds #
Database War Stories #9 (finis): Brian Aker of MySQL Responds By tim on May 05, 2006 Brian Aker of MySQL sent me a few email comments about this whole 'war stories' thread, which I reproduce here. Highlight -- he says: 'Reading through the comments you got on your blog entry, these users are hitting on the same design patterns. There are very common design patterns for how to scale a database, and few sites really turn out to be all that original. Everyone arrives at certain truths, flat files with multiple dimensions don't scale, you will need to partition your data in some manner, and in the end caching is a requirement.'