java - Joining 2 Spark dataframes, getting result as list -

-



i trying join 2 dataframes, have result list of rows of right dataframe (ddf in example) in column of left dataframe (cdf in example). made work 1 column, having issues adding more columns. 

    seq<string> joincolumns = new set2<>("c1", "c2").toseq();     dataset<row> alldf = cdf.join(ddf, joincolumns, "inner");     alldf.printschema();     alldf.show();      dataset<row> aggdf = alldf.groupby(cdf.col("c1"), cdf.col("c2"))             .agg(collect_list(col("c50")));     aggdf.show();

output:

+--------+-------+---------------------------+ |c1      |c2     |collect_list(c50)          | +--------+-------+---------------------------+ |    3744|1160242|         [6, 5, 4, 3, 2, 1]| |    3739|1150097|                        [1]| |    3780|1159902|            [5, 4, 3, 2, 1]| |     132|1200743|               [4, 3, 2, 1]| |    3778|1183204|                        [1]| |    3766|1132709|                        [1]| |    3835|1146169|                        [1]| +--------+-------+---------------------------+

also, there way like:

    dataset<row> aggdf = alldf.groupby(cdf.col("*"))             .agg(collect_list(col("c50")));

for second part of question, can do:

    string[] fields = cdf.columns();     column[] columns = new column[fields.length];     (int = 0; < fields.length; i++) {         columns[i] = cdf.col(fields[i]);     }     dataset<row> sdf = alldf.groupby(columns).agg(...);




wiki

Comments

Post a Comment

Popular posts from this blog

python - Read npy file directly from S3 StreamingBody -

-
i have many .npy (numpy) , .json files saved in s3, need download , load these files. for json files downloading files , loading dictionary using following code: obj = s3.object(bucket, key) response = obj.get() body = response['body'].read().decode('utf-8') data = json.loads(body) that way avoid writing .json file disk , reading it. now want same .npy files, equivalent? currently downloading file, saving temp file on disk , loading np.load avoid if possible. following https://stackoverflow.com/a/28196540/3509999 : obj = s3.object(bucket, key) io.bytesio(obj.get()["body"].read()) f: # rewind file f.seek(0) arr = np.load(f) wiki
Read more

python - Minimize function with Scipy minimize -

-
trying obtain d value (integer) std_diff objective function minimal, using scipy minimize. my code: def std_diff(d): return std(diff(df['bn'].values,d)); scipy.optimize import minimize b=(3,) res = minimize(std_diff,(1,), method='slsqp', bounds = b) **df['bn'].values** out[72]: array([ 2, 2, 2, 2, 3, 2, 7, 5, 7, 11, 8, 2, 11, 7, 15, 8, 7, 12, 21, 19, 32, 35, 40, 35, 21, 19, 25, 20, 40, 80, 99], dtype=int64) error is"indexerror: many indices array " in case not use bounds : res = minimize(std_diff,(1,), method='slsqp'), error: > in _minimize_slsqp(func, x0, args, jac, bounds, constraints, maxiter, > ftol, iprint, disp, eps, callback, **unknown_options) > 368 fx = float(np.asarray(func(x))) > 369 except: > --> 370 raise valueerror("objective function must return scalar") > 371 # compute constraints > 3
Read more

kotlin - Out-projected type in generic interface prohibits the use of metod with generic parameter -

-
i start using kotlin , defined interface this: interface aadapter<vh : recyclerview.viewholder> { fun oncreateaviewholder(parent: viewgroup): vh fun onbindaviewholder(v: vh, position: int) } and when try use on code: class klasa ( private val adapter: aadapter<*> ) { fun dosth(){ //... val vh = this.adapter.oncreateaviewholder(parent) //on below line error adapter.onbindaviewholder(v, position) //... } } i error out-projected type 'aadapter<*>' prohibits use of 'public abstract fun onbindaviewholder(v: t, position: int): unit i tied add "in" or "out" definition i'm confused. how allow it. replace aadapter<*> aadapter<recyclerview.viewholder> . class klasa ( private val adapter: aadapter<recyclerview.viewholder> ) { // ... because have declared aadapter has type parameter vh type subclass of recyclerview.
Read more