r - How to efficiently implement dplyr do call for lmer function? -

-



i have dataset ~400000 rows trying extract lme4 mixed model variance components using dplyr do call in r. function is:

myfunc <- function(dat) {     if (sum(!is.na(dat$value)) > 840) {  # >70% data present             v = data.frame(varcorr(lmer(value ~ 0 + (1|gid) + (1|trial:rep) + (1|trial:rep:block), data=dat)))            data.frame(a=round(v[1,4]/(v[1,4]+(v[4,4]/2)),2), b=round(v[1,4],2), c=round(v[4,4],2), n_obs=nrow(dat), na_obs=sum(is.na(dat$value)))      } else {          data.frame(a=na, b=na, c=na, n_obs= nrow(dat), na_obs=sum(is.na(dat$value)))     } }

this function called dplyr do call after grouping data 4 grouping variables. final dplyr call is:

system.time(out <- tst %>% group_by(iyear,ilocation,trait_id,date) %>%            do(myfunc(.)))

now, when code run on smaller test dataframe of 11000 rows, takes 25 seconds. running on full set of 443k rows takes 8-9 hours finish, awefully slow. seems obvious there part of code pulling down performance can't seem figure out whether lmer part or dplyr causing slow down. have feeling there wrong way function handling vectorization operation not sure. tried initializing 'out' matrix outside function call, didn't improve performance.
unfortunately, don't have smaller reproducible dataset share. hear thoughts on how make code more efficient.

solution: mclapply function parallel package came rescue. @gregor rightly pointed, lmer part slowing things down. ended parallelizing function call:

myfunc <- function(i) {      dat = tst[tst$comb==unique(tst$comb)[i],]  #comb concatenated iyear,ilocation....columns      if (sum(!is.na(dat$value)) > 840) {  # >70% data present per column          v = data.frame(varcorr(lmer(value ~ 0 + rand_factor + nested_random_factor), data=dat)))          data.frame(trait=unique(tst$comb)[i], a=round(v[1,4])/5, b=round(v[1,4],2), c=round(v[4,4],2), n_obs=nrow(dat), na_obs=sum(is.na(dat$value)))       } else {           data.frame(trait=unique(tst$comb)[i], a=na, b=na, c=na, n_obs= nrow(dat), na_obs=sum(is.na(dat$value)))       } }  #initialize empty matrix out <- matrix(na,length(unique(tst$comb)),6)  ## apply function in parallel. output list n_cores = detectcores() - 2 system.time(my.h2 <- mclapply(1:length(unique(tst$comb)),fun = myfunc, mc.cores = n_cores))

a twelve core unix machine took ~2 minutes complete.





wiki

Comments

Post a Comment

Popular posts from this blog

Asterisk AGI Python Script to Dialplan does not work -

-
i'm stuck project have check database of sip extension incoming caller id , dial sip extension. below receive when run script on asterisk cli. -- registered sip '9125' @ 192.168.1.102:50142 > saved useragent "zoiper rv2.8.40" peer 9125 [2017-08-22 13:53:33] notice[5095]: chan_sip.c:24558 handle_response_peerpoke: peer '9125' reachable. (17ms / 2000ms) == using sip rtp tos bits 184 == using sip rtp cos mark 5 -- executing [0112617769@from-trunk:1] answer("sip/obitrunk1-0000000d", "") in new stack > 0x1ca50d0 -- probation passed - setting rtp source address 192.168.1.98:16834 -- executing [0112617769@from-trunk:2] agi("sip/obitrunk1-0000000d", "test.py") in new stack -- launched agi script /var/lib/asterisk/agi-bin/test.py args: ['/var/lib/asterisk/agi-bin/test.py'] env line: agi_request: test.py env line: agi_channel: sip/obitrunk1-0000000d env line: agi_language: en env line: a
Read more

python - Read npy file directly from S3 StreamingBody -

-
i have many .npy (numpy) , .json files saved in s3, need download , load these files. for json files downloading files , loading dictionary using following code: obj = s3.object(bucket, key) response = obj.get() body = response['body'].read().decode('utf-8') data = json.loads(body) that way avoid writing .json file disk , reading it. now want same .npy files, equivalent? currently downloading file, saving temp file on disk , loading np.load avoid if possible. following https://stackoverflow.com/a/28196540/3509999 : obj = s3.object(bucket, key) io.bytesio(obj.get()["body"].read()) f: # rewind file f.seek(0) arr = np.load(f) wiki
Read more

kotlin - Out-projected type in generic interface prohibits the use of metod with generic parameter -

-
i start using kotlin , defined interface this: interface aadapter<vh : recyclerview.viewholder> { fun oncreateaviewholder(parent: viewgroup): vh fun onbindaviewholder(v: vh, position: int) } and when try use on code: class klasa ( private val adapter: aadapter<*> ) { fun dosth(){ //... val vh = this.adapter.oncreateaviewholder(parent) //on below line error adapter.onbindaviewholder(v, position) //... } } i error out-projected type 'aadapter<*>' prohibits use of 'public abstract fun onbindaviewholder(v: t, position: int): unit i tied add "in" or "out" definition i'm confused. how allow it. replace aadapter<*> aadapter<recyclerview.viewholder> . class klasa ( private val adapter: aadapter<recyclerview.viewholder> ) { // ... because have declared aadapter has type parameter vh type subclass of recyclerview.
Read more