Comments for schmichael's blog http://blog.schmichael.com good good study, day day up Wed, 08 Feb 2012 17:41:00 +0000 hourly 1 http://wordpress.org/?v=3.2.1 Comment on Failing with MongoDB by Michael Schurter http://blog.schmichael.com/2011/11/05/failing-with-mongodb/comment-page-1/#comment-138626 Michael Schurter Wed, 08 Feb 2012 17:41:00 +0000 http://blog.schmichael.com/?p=979#comment-138626 Great tip. We didn't adjust the dirty_ratio as far as I know, so that probably caused unneeded IO. That being said I don't remember iowait being an issue; most of our issues were with the global write lock and complexities of sharding. Great tip. We didn’t adjust the dirty_ratio as far as I know, so that probably caused unneeded IO. That being said I don’t remember iowait being an issue; most of our issues were with the global write lock and complexities of sharding.

]]> Comment on Failing with MongoDB by Jonathan Ultis http://blog.schmichael.com/2011/11/05/failing-with-mongodb/comment-page-1/#comment-138625 Jonathan Ultis Wed, 08 Feb 2012 03:04:00 +0000 http://blog.schmichael.com/?p=979#comment-138625 The linux VM flushes the dirty pages in RAM to disk. It starts out flushing lazily. Eventually, if you get too high a percentage of dirty pages, it will start blocking all writes trying to get the deferred IO onto disks - so you don't lose hours of data in the case of a crash. Is it possible that your IO subsystem just couldn't keep up with the sustained write rate? And, you didn't raise your dirty_ratio limit as high as possible to make it act like a pure in-memory DB? http://www.westnet.com/~gsmith/content/linux-pdflush.htm Mongo definitely wouldn't do a good job of putting back pressure on writes without blocking readers, if the IO subsystem is saturated and you hit the dirty_ratio limit. The linux VM flushes the dirty pages in RAM to disk. It starts out flushing lazily. Eventually, if you get too high a percentage of dirty pages, it will start blocking all writes trying to get the deferred IO onto disks – so you don’t lose hours of data in the case of a crash. Is it possible that your IO subsystem just couldn’t keep up with the sustained write rate? And, you didn’t raise your dirty_ratio limit as high as possible to make it act like a pure in-memory DB?

http://www.westnet.com/~gsmith/content/linux-pdflush.htm

Mongo definitely wouldn’t do a good job of putting back pressure on writes without blocking readers, if the IO subsystem is saturated and you hit the dirty_ratio limit.

]]> Comment on Failing with MongoDB by Michael Schurter http://blog.schmichael.com/2011/11/05/failing-with-mongodb/comment-page-1/#comment-138624 Michael Schurter Tue, 07 Feb 2012 00:04:00 +0000 http://blog.schmichael.com/?p=979#comment-138624 4x RAID 0+1 SAS drives for the data files, but I think you're missing the point: our dataset fit in memory, we had syncing turned off, and we didn't use journaling - the disks should never be touched. 4x RAID 0+1 SAS drives for the data files, but I think you’re missing the point: our dataset fit in memory, we had syncing turned off, and we didn’t use journaling – the disks should never be touched.

]]> Comment on Failing with MongoDB by Jonathan Ultis http://blog.schmichael.com/2011/11/05/failing-with-mongodb/comment-page-1/#comment-138623 Jonathan Ultis Mon, 06 Feb 2012 23:50:00 +0000 http://blog.schmichael.com/?p=979#comment-138623 What type of storage subsystem were you using on the Mongo boxes? What type of storage subsystem were you using on the Mongo boxes?

]]> Comment on Failing with MongoDB by Quora http://blog.schmichael.com/2011/11/05/failing-with-mongodb/comment-page-1/#comment-138622 Quora Thu, 26 Jan 2012 20:31:47 +0000 http://blog.schmichael.com/?p=979#comment-138622 <strong>How come many startups used MySQL or PostgreSQL rather than MongoDB when they began?...</strong> > Many of them seem to migrate to MongoDB later - why's that? To be clear I have typically seen "sub-systems" moved to MongoDB. Generally to handle the things where SQL is not great. There are also some people who have moved off MongoDB. Or are very... How come many startups used MySQL or PostgreSQL rather than MongoDB when they began?…

> Many of them seem to migrate to MongoDB later – why’s that? To be clear I have typically seen “sub-systems” moved to MongoDB. Generally to handle the things where SQL is not great. There are also some people who have moved off MongoDB. Or are very…

]]> Comment on Sharing Python data between processes using mmap by A.jhonson http://blog.schmichael.com/2011/05/15/sharing-python-data-between-processes-using-mmap/comment-page-1/#comment-138621 A.jhonson Tue, 17 Jan 2012 14:59:00 +0000 http://blog.schmichael.com/?p=947#comment-138621 please correct me if i'm wrong but you've used the file: mmaptest to exchange the data betwean the 2 scripts "i'm new to python. please correct me if i’m wrong but you’ve used the file: mmaptest to exchange the data betwean the 2 scripts “i’m new to python.

]]>