16:24 < k10> just have nexenta installed.  zfs file system for storage and then running NFS to get to the file system from web hosts.
16:24 < icblenke> I had visions of a block device over NFS used somehow by lofiadm to zfs
16:24 <@rootard> k10: what version are you running?
16:25 < k10> rootard: SunOS nn-file-02 5.11 NexentaOS_20080131 i86pc i386 i86pc Solaris
16:26 <@rootard> hrmm, I think that's b82/elatte-stable?
16:26 < k10> i have been trying to find a crash file in /var/crash/hostname - but nothing....
16:26 <@rootard> does /var/adm/messages have anything interesting?
16:26 < k10> not really.  nothing that sticks out as to why the thing is crashing.
16:27 < k10> i did see some virtual memory error messages yesterday but was unable to write them down.
16:27 <@rootard> ohhh, run a mem check.
16:27 < k10> i can definitely do that, i have 3 machines this is happening on so i have been leaning away from memory problems, but defintiely something i can have 
             run overnight and we can check them out in the morning.
16:28 < k10> just a memtest on each of them be sufficient?
16:29 <@rootard> It's safer to run on all three but if the problem is exactly the same you can probably get away with one.
16:29 <@rootard> If all three boxes are the same kind you might be running into a bug of some variety.
16:30 < k10> one of the machines is production, so i wont be able to run on that - but i can test the others for sure.  we have had the same problem on all 3 of 
             the machines and with earlier version of opensolaris.
16:30 < k10> 2 of the boxes are the same (rackable), but the other one is completely different (supermicro).
16:30 < k10> i am more leaning towards being a retard and having something configured incorrectly.  =)
16:30 <@rootard> What about the drive chipset?
16:31 < k10> not positive on drive chipset, if you point me in the right direction i can get some information about it for ya.  nothing running degraded or failed 
             in the zpool....
16:31 < Bartman007> I remember running into lovely AMD processor errata (defective memory controller under certain conditions)
16:32 < Bartman007> it took RMA'ing 12GB of ram to figure it out. (this is when a 1GB stick was fairly rare)
16:33 < k10> the thing just crashed again, and nothing in /var/adm/messages
16:33 < k10> everything just locks up when it goes down.  no commands work and we have to power the thing off.
16:35 < k10> Could NFS be configured wierd?
16:35 <@rootard> When you are on the console, can you still send keystrokes?
16:35 < k10> yes, can still send keystrokes, but pressing enter does nothing.
16:36 <@rootard> Are you running anything else on this system? It sounds like you might have some kind of run-away process
16:36 < k10> can actually still send keystrokes from ssh connections as well, they just dont do anything either.
16:36 <@rootard> another symptom would be the system is swapping hard...
16:36 < k10> running a vmstat defintely shows that swap is being used.
16:37 < k10> i have 3GB of ram in this thing and ALL it does is NFS for web servers...
16:37 <@rootard> have you watched prstat for processes that take up a bunch of mem/cpu?
16:38 < k10> no, i have not.  though i can.
16:38 < k10> never used prstat before, but looks very similar to top.  i had been checking top to see if anything was wigging out but hadnt found anything really 
             rampant in there.
16:38 < k10> load was getting up around 1.5
16:39 <@rootard> prstat is a little more fine-grained. It also gives correct output where top doesn't (so I hear from #opensolaris)
16:39 < k10> ok, i will use prstat
16:39 < k10> load average is .45 right now and nfsd is .2% cpu
16:40 <@rootard> If nothing else is consuming cpu then the load is probably all disk/network io.
16:40 <@rootard> you can watch io on the pool with "zpool iostat 5"
16:41 < k10> i would assume that is the case.  this machine really doesnt do much at all.  right now i am running a pretty big rsync to move files over from one 
             server to this one.  the rsync is around 3.5% of cpu.  nothing too big.
16:41 < k10> in the spool iostat 5 what should i be looking for?  writes are currenlty about 19M
16:42 < k10> transferring stuff over right now with that rsync...
16:42 < k10> reads are around 300k
16:43 <@rootard> It's just another metric... That sounds reasonable to me given that you are moving data around :)
16:43 < k10> prstat is now showing load of about 1.5
16:43 -!- michael [n=michael@209.31.182.66] has joined ##nexenta
16:43 < k10> rootard, michael is another guy working on this problem with me.  i asked him to join in to fill in any of the stuff i miss (hes the smart one, im the 
             tall one)
16:44 <@rootard> k10: ok
16:44 < michael> good times, what did i miss so far?
16:45 <@rootard> another good way to help figure out what is going on is with dtrace. You can get a bunch of canned scripts by: apt-get install dtrctkit