00:11 -!- dammit [~midom@84-80-6-088.static.zebra.lt] has joined #rapidxen
00:11 < dammit> hi!
00:11 -!- dammit is now known as domas
00:11 < Diablo-D3> hi domas
00:11 < Diablo-D3> this makes it easier :P
00:11  * werdnus sits back.
00:11 < Diablo-D3> <Diablo-D3> I answered why google uses mysql... its only on small projects that haven't been moved to the big db... they have projects using all sorts of different dbs, but they all
                   eventually get moved to the big db if they become google products
00:11 < Diablo-D3> <Diablo-D3> yahoo probably is the same reason
00:12 < domas> Diablo-D3: google pageads backend is all mysql
00:12 < domas> Diablo-D3:  is the google cash-cow 'small project'?
00:12 < Diablo-D3> thats not true anymore
00:12 < domas> why not true?
00:12 < Diablo-D3> someone said that awhile back, and I asked someone I know who used to work for google
00:12 < Diablo-D3> it started with mysql, and was the last major project of theirs to switch
00:12 < setient> lol
00:12 < domas> they switched to Sybase, then switched back
00:13 < domas> %)
00:13 < Diablo-D3> most of adwords runs on google's db
00:13 < domas> http://xooglers.blogspot.com/2005/12/lets-get-real-database.html
00:13 < domas> most of adwords runs on mysql ;-)
00:13 < Diablo-D3> thats a 3 year old post
00:14 < domas> yeah, but it tells about switching back
00:14 < Diablo-D3> its probably true... they reverted the changes =P
00:14 < domas> so, your knowledge about switching somewhere else is a bit outdated
00:14 < Diablo-D3> you do know google wrote their own db, right?
00:14 < Diablo-D3> bigtable or something like that
00:14 < Diablo-D3> adwords runs on that now
00:14 < domas> hahaha
00:14 < domas> not really
00:15 < domas> they do some export to bigtable
00:15 < domas> for archives
00:15 < domas> but actual OLTP runs all on mysql
00:15 < Diablo-D3> well, said google ex-employee said it does
00:15 < werdnus> newer post http://mysqldatabaseadministration.blogspot.com/2007/04/google-and-mysql.html
00:15 < domas> *shrug*, I probably talk more to google employees than ex-employees
00:16 < domas> I can show you this blog: http://mysqlha.blogspot.com/
00:16 < domas> it mentions me from time to time :)
00:16 < Diablo-D3> btw, even if google does use mysql
00:16 < Diablo-D3> I bet they've rolled a lot of their own stuff to make mysql sane
00:16 < Diablo-D3> which, really, isn't worth it
00:16 < domas> well, I know what they rolled into it :-)
00:16 < domas> most of it is public
00:17 < Diablo-D3> yeah, I know they've submitted patches
00:17 < Diablo-D3> you missed out on an earlier statement
00:17 < Diablo-D3> mysql wants to add transactions to myisam
00:17 < Diablo-D3> if this happens, mysql may, conceivably, not suck
00:17 < domas> big shops use purely innodb anyway
00:18 < domas> for now :)
00:18 < domas> so nobody really cares about whether myisam sucks or not
00:18 < Diablo-D3> innodb is slower than pg, and still has that "runs out of hd space, and your db is corrupted the whole way through" bug
00:18 < Diablo-D3> they never fixed it
00:18 < domas> how does InnoDB corrupt diskspace when runs out of diskspace?
00:18 < domas> ergh, tablespace
00:18 < Diablo-D3> dunno, it just does
00:19 < domas> ghm, not really :)
00:19 < Diablo-D3> I've never had it happen because I don't use mysql on anything serious
00:19 < Diablo-D3> I've only heard about it from 32482340923840834 admins who've had it happen
00:19 < domas> I never had it happen, even though I use mysql in multiple serious places :)
00:19 < domas> thats a big number of admins you know
00:19 < domas> you probably don't use PG on anything serious too
00:19 < domas> if you think that innodb is slower :)
00:19 < Diablo-D3> I do, but not for any website stuff
00:19 < Trystan> a big number of admins who use mysql on something serious to say no less.
00:19 < domas> Trystan: ;-)
00:20 < Diablo-D3> Trystan: hah
00:20 < Trystan> :)
00:20 < Trystan> have fun with your discussion
00:20 < Trystan> im off to bed
00:20 < domas> Diablo-D3: anyway, back to my original questions
00:20 < Diablo-D3> domas: this is not the discussion I really cared about though
00:20 < domas> innodb clusters data by PK
00:20 < Diablo-D3> if you can get mysql to not suck, fine
00:20 < Diablo-D3> a lot of people have managed that
00:20 < Diablo-D3> it usually involves not sucking at sql in the first place, etc etc
00:20 < domas> PG clusters data by... oh wait, it doesn't
00:21 < domas> once you grow your datasets, locality matters
00:21 < domas> (with SSDs it probably won't ;-)
00:21 < Diablo-D3> I've never used the cluster command, btw
00:21 < domas> that just tells how small your datasets are ;-)
00:21 < Diablo-D3> well, yeah
00:21 < domas> or how small the load is
00:21 < Diablo-D3> the largest dataset I've used is only like 3.3 million rows
00:21 < Diablo-D3> wikipedia adds that much a day or something >_>
00:22 < Diablo-D3> but I'm trying to reduce load at the beginning, not the end
00:22 < Diablo-D3> varnish + lighty + fastcgi'ed perl (using memcached where appropriate)
00:22 < domas> and?
00:23 < Diablo-D3> and that should have a large impact
00:23 < domas> who doesn't do that kind of setup?
00:23 < Diablo-D3> apparently wikipedia doesn't
00:23 < domas> we use squid+apache+mod_php
00:23 < domas> how is it worse than your setup?
00:23 < Diablo-D3> squid sucks dick
00:23 < domas> varnish has issues too
00:23 < domas> does varnish support object streaming?
00:23 < Diablo-D3> need more context, whats object streaming?
00:24 < domas> ability to serve the object before backend finished serving it
00:24 < domas> why would lighttpd+fastcgi be much faster than apache/modphp? :)
00:25  * ToreadorVampire apologises for starting this, although the conversation I started had NOTHING to do with mysql or google, or erm ... anything ...
00:25 < domas> hehe
00:25 < Diablo-D3> I think varnish can stream data like that, domas
00:25 < werdnus> ToreadorVampire: hehehe, but it's fun to watch!
00:25 < ToreadorVampire> My original conversation was on C#/Apache2_mod_mono versus Perl
00:25 < ToreadorVampire> Wheeeeeeeeeeeeeeeee
00:25 < domas> see, most of your stuff is "I think"
00:25 < domas> most of my stuff is "I have worked on that"
00:25 < domas> there's a bit of difference there ;-)
00:25 < Diablo-D3> when I say "I think", it means I haven't googled the answer yet
00:25 < Diablo-D3> varnish _does_ have a decent FAQ, you know
00:26  * ToreadorVampire isn't really watching, went back to work a long time ago, I just un-minimise my xchat window from time to time and have a look at the monster I have created ;)
00:26 < domas> Diablo-D3: I know, I've seen varnish code
00:26 < domas> I have commit access to lighty too
00:26 < domas> ghm
00:26 < domas> I've written patches for squid
00:26 < domas> for mysql, for php, for PG, for ..
00:26 < domas> ergh.
00:27 < domas> I don't need to google to support my "I think"
00:27 < Diablo-D3> um, good for you
00:27 < domas> indeed
00:27 < Diablo-D3> varnish still does serve static data faster than squid though =P
00:27 < domas> ToreadorVampire: btw, we had lots of fun with Mono vs GCJ ;-)
00:27 < domas> Diablo-D3: lack of ability to stream is a bit frustrating
00:28 < domas> Diablo-D3: it starts serving data once it is in varnish memory
00:28 < ToreadorVampire> Oh, I don't use mcs/gmcs for Java, only C#
00:29 < Diablo-D3> domas: if it does, that may be by design
00:29 < domas> Diablo-D3: see, not every design decision is good ;-)
00:29 < Diablo-D3> domas: its not entirely an issue unless you're streaming extremely large files
00:29 < Diablo-D3> Im not saying its a bad design decision
00:29 < Diablo-D3> if there is a net gain out of this, then it should be done
00:29 < domas> well, 'extremely large' is different concept once you serve 50k objects a second
00:30 < Diablo-D3> well, when I say extremely large, I mean as in actual media files
00:30 < Diablo-D3> movies, music, hq images, etc etc
00:30 < domas> ToreadorVampire: we were deploying search once, and as JRE was non-free we had a choice to run Lucene either on GCJ or Lucene.NET kind of thing on mono
00:30 < Diablo-D3> that really should be done by a separate cluster running lighty and nothing else, no dynamic content or anything
00:30 < domas> ToreadorVampire: it was fun that .Net was more free than Java at one point in time
00:30 -!- TheKing [~deathking@oooa-02-84-442-0.phlapa.east.verizon.net] has joined #rapidxen
00:31 < domas> Diablo-D3: 100k file is HQ?
00:31 < Diablo-D3> domas: no, but it may be not worth serving
00:31 < Diablo-D3> the only thing you want varnish to serve in that vein is images that every single page has
00:31 < domas> whole internet is not worth serving
00:31 < Diablo-D3> that... may be true
00:31 < domas> anyway
00:32 < domas> I want frontend cache to be way more aggressive than that
00:32 < Diablo-D3> non-text content for the entires in wikipedia still would be handled under a different cluster in my design
00:32 < domas> that needs aggressive evictions too, what varnish didn't support until recently
00:32 < domas> we are always considering varnish, there're pros and cons
00:33 < domas> I'm not telling it is useless shit, I'm just telling, that we know the technology, and it doesn't always do what we need
00:33 < domas> you look at a peak of iceberg and say "yay, I can do this with my brilliant design"
00:33 < Diablo-D3> btw, domas
00:33 < domas> while in reality there's tons of submarine engineering
00:33 < Diablo-D3> Im trying to solve a similar problem
00:33 < Diablo-D3> you know of 4chan, yes?
00:34 < domas> yes ;-)
00:34 < Diablo-D3> you know that 4chan's software completely sucks ass, right?
00:34 < domas> I don't know that
00:34 < Diablo-D3> ahh, well you're in for a treat: it does
00:34 < Diablo-D3> the bulk of it was written a decade ago by a japanese guy who didnt know how to code
00:35 < domas> that can be told about most web software
00:35 < werdnus> haha, including MediaWiki
00:35 < Diablo-D3> moot has gradually made it worse over the years
00:35 < domas> that too
00:35 < TheKing> I can haz cheeseburgur?
00:35 < werdnus> MediaWiki was Magnus' first PHP project, IIRC.
00:35 < Diablo-D3> ichc runs on wordpress
00:35 < Diablo-D3> I HATE WORDPRESS
00:35 < Diablo-D3> I use it on my blog and it makes me sad ;_;
00:36 < Diablo-D3> anyhow, re: 4chan
00:36 < domas> Diablo-D3: once upon a time I made wordpress suck way way less
00:36 < Diablo-D3> moot tried to throw hardware at the problem
00:36  * domas did the 2.1 performance work
00:36 < Diablo-D3> domas: Ive been using wordpress since before 2.x
00:36 < domas> good! me too!
00:36 < domas> we have something in common
00:36 < Diablo-D3> sometime in november marks the start of my 5th year
00:37 < Diablo-D3> anyhow, moot tried to throw hardware at his problem, and he made it much. much. worse.
00:37 < Diablo-D3> I'm attempting to write software that can do imageboarding without being epic fail
00:38 < Diablo-D3> not that moot will ever run it (god forbid moot actually does his job), but at least some chans out there will suck less.
00:38 < Diablo-D3> I'm trying to aim for running a 4chan sized site on a third of the hardware moot has
00:39 < domas> good!
00:39 < Diablo-D3> so, its not like you and I have dissimilar goals
00:40 < domas> meh
00:40 < domas> performance engineering isn't that difficult
00:40 < domas> you just have to understand your costs
00:40 < domas> no magic buzzword bingo will help
00:41 < Diablo-D3> you don't even have to understand them that well... you just have to be willing to benchmark multiple methods
00:41 < domas> meh, I prefer profiling over benchmarking :)
00:42 < domas> benchmarking skews everything
00:43 < Diablo-D3> yeah, but it tells you which methods _don't_ work
00:43 < ToreadorVampire> Note to self:  "Never say anything to Diablo-D3"
00:43 < domas> =)
00:43 < domas> I prefer understanding stuff :)
00:43 < domas> rather than guessing
00:43 < domas> thats why I can tell you why MySQL is better than PG! :-D
00:44 < Diablo-D3> I dunno, I rather like PG
00:44 < domas> I like PG too
00:44 < domas> I know strengths, I know weaknesses
00:44 < domas> actually , I made initial mediawiki support for PG :)
00:45 < Diablo-D3> heh
00:45 < Diablo-D3> domas: convince me to use mysql.
00:45 < werdnus> for what?
00:45 < domas> I don't really care what you use
00:45 < domas> I'm not in Sales/Marketing
00:47 < Diablo-D3> werdnus: the worst thing ever to happen to the internet: imageboard software that works
00:47 < Diablo-D3> domas: well, why would I use mysql
00:47 < werdnus> hey, incidentally, does anybody know if I can get a vhost with rapidxen?
00:47 < Diablo-D3> because I can't seriously see any pros to it
00:47 < werdnus> On, you know, the topic of the room.
00:47 < Diablo-D3> a ... vhost? you mean a vps, right?
00:48 < TheKing> XD
00:48 < werdnus> heh, let me de-IRCify it
00:48 < domas> Diablo-D3: MySQL development was influenced by big shops running big things (and paying money)
00:48 < werdnus> I mean, how can I set the reverse DNS of my VPS?
00:48 < domas> Diablo-D3: it has most of stuff what high-load webshops need
00:48 < Diablo-D3> werdnus: ahh, file a ticket
00:48 < Diablo-D3> werdnus: nenolod or one of his slaves will get to it soon
00:49 < domas> Diablo-D3: anyway, my favorite difference is simply how underlying data is handled
00:49 < domas> nothing at the feature side
00:49 < Diablo-D3> domas: but the only thing I care about is ACID, data integrity outside of the concept of ACID, and performance. In that order.
00:49 < domas> the 'everything is a tree' in InnoDB concept, ability to do covering index reads, etc
00:50 < domas> you get all that with mysql5 strict mode %)
00:50 < domas> anyway
00:50 < domas> if you were after web-scale performance, you'd want to use mysql :)
00:51 < Diablo-D3> if I was performance first, I'd probably use myisam or something
00:52 < domas> not really
00:52 < Diablo-D3> my queries aren't particularly complex
00:52 < domas> at high concurrency innodb is fast
00:53 < Diablo-D3> I have few writes.
00:53 < Diablo-D3> and reads can be blunted away using memcached abusively
00:54 < domas> doesn't work that well with range-reads in scale-out :-)
00:54 < domas> here again, data locality is the king
00:54 < Diablo-D3> well, I don't have 234098230498209480829348023948012840128402350295 items of data
00:54 < Diablo-D3> so, theoretically, I can store everything in memcached
00:54 < domas> then you can store everything in text file too
00:54 < domas> :)
00:54 < TheKing> Highlight D:
00:55 < Diablo-D3> I thought about it, but then thought better of it
00:55 < domas> TheKing: ;-)
00:55 < Diablo-D3> peak /b/ is about 50 posts a minute