linux - Hadoop No space left on device erro when there is space available -

-



i have 5 linux machines cluster. there 3 data nodes , 1 master. @ 50% hdfs storage available on each data nodes. run mapreduce job, failed following error

2017-08-21 17:58:47,627 warn org.apache.hadoop.hdfs.dfsclient: error recovery blk_6835454799524976171_3615612 bad datanode[0] 10.11.1.42:50010 2017-08-21 17:58:47,628 warn org.apache.hadoop.hdfs.dfsclient: error recovery block blk_6835454799524976171_3615612 in pipeline 10.11.1.42:50010, 10.11.1.43:50010: bad datanode 10.11.1.42:50010 2017-08-21 17:58:51,785 error org.apache.hadoop.mapred.child: error in synclogs: java.io.ioexception: no space left on device

while on each system df -h gives following information

filesystem               size  used avail use% mounted on devtmpfs                 5.9g     0  5.9g   0% /dev tmpfs                    5.9g   84k  5.9g   1% /dev/shm tmpfs                    5.9g  9.1m  5.9g   1% /run tmpfs                    5.9g     0  5.9g   0% /sys/fs/cgroup /dev/mapper/centos-root   50g  6.8g   44g  14% / /dev/sdb                 1.8t  535g  1.2t  31% /mnt/11fd6fcc-1f87-4f1e-a53c-54cc7117759c /dev/mapper/centos-home  412g  155g  59m  100% /home /dev/sda1                494m  348m  147m  71% /boot tmpfs                    1.2g   16k  1.2g   1% /run/user/42 tmpfs                    1.2g     0  1.2g   0% /run/user/1000

as clear above sdb dicsk (sdd) 31% used centos-home 100%. while hadoop using local file system in mapreduce job when there enough hdfs available? problem? have search @ google , found many such problem no 1 covers situation.

synclogs not use hdfs, writes hadoop.log.dir if you're using mapreduce, value of hadoop.log.dir in /etc/hadoop/conf/taskcontroller.cfg.

if you're using yarn, value of yarn.nodemanager.log-dirs in yarn-site.xml.

one of these should point you're writing logs. once figure out filesystem has problem, can free space there.

another thing remember "no space left on device" if you've exhausted inodes on disk. df -i show this.





wiki

Comments

Post a Comment

Popular posts from this blog

Asterisk AGI Python Script to Dialplan does not work -

-
i'm stuck project have check database of sip extension incoming caller id , dial sip extension. below receive when run script on asterisk cli. -- registered sip '9125' @ 192.168.1.102:50142 > saved useragent "zoiper rv2.8.40" peer 9125 [2017-08-22 13:53:33] notice[5095]: chan_sip.c:24558 handle_response_peerpoke: peer '9125' reachable. (17ms / 2000ms) == using sip rtp tos bits 184 == using sip rtp cos mark 5 -- executing [0112617769@from-trunk:1] answer("sip/obitrunk1-0000000d", "") in new stack > 0x1ca50d0 -- probation passed - setting rtp source address 192.168.1.98:16834 -- executing [0112617769@from-trunk:2] agi("sip/obitrunk1-0000000d", "test.py") in new stack -- launched agi script /var/lib/asterisk/agi-bin/test.py args: ['/var/lib/asterisk/agi-bin/test.py'] env line: agi_request: test.py env line: agi_channel: sip/obitrunk1-0000000d env line: agi_language: en env line: a
Read more

python - Read npy file directly from S3 StreamingBody -

-
i have many .npy (numpy) , .json files saved in s3, need download , load these files. for json files downloading files , loading dictionary using following code: obj = s3.object(bucket, key) response = obj.get() body = response['body'].read().decode('utf-8') data = json.loads(body) that way avoid writing .json file disk , reading it. now want same .npy files, equivalent? currently downloading file, saving temp file on disk , loading np.load avoid if possible. following https://stackoverflow.com/a/28196540/3509999 : obj = s3.object(bucket, key) io.bytesio(obj.get()["body"].read()) f: # rewind file f.seek(0) arr = np.load(f) wiki
Read more

kotlin - Out-projected type in generic interface prohibits the use of metod with generic parameter -

-
i start using kotlin , defined interface this: interface aadapter<vh : recyclerview.viewholder> { fun oncreateaviewholder(parent: viewgroup): vh fun onbindaviewholder(v: vh, position: int) } and when try use on code: class klasa ( private val adapter: aadapter<*> ) { fun dosth(){ //... val vh = this.adapter.oncreateaviewholder(parent) //on below line error adapter.onbindaviewholder(v, position) //... } } i error out-projected type 'aadapter<*>' prohibits use of 'public abstract fun onbindaviewholder(v: t, position: int): unit i tied add "in" or "out" definition i'm confused. how allow it. replace aadapter<*> aadapter<recyclerview.viewholder> . class klasa ( private val adapter: aadapter<recyclerview.viewholder> ) { // ... because have declared aadapter has type parameter vh type subclass of recyclerview.
Read more