logging - Elasticsearch cluster design for ~200G logs a day -

-



i've created es cluster (version 5.4.1) 4 data nodes, 3 master, 1 client node (kibana).

the data nodes r4.2xlarge aws instance (61g memory, 8vcpu) 30g memory allocated es java.

we're writing around 200g of logs every day , keep last 14 days.

i'm looking recommendations our cluster improve cluster performance, search performance (kibana).

more data nodes? more client nodes? bigger nodes? more replica's? can improve performance option.

is there close design or loads? i'll happy hear other designs , loads.

thanks, moshe

  1. how many shards using? default of 5? pretty number. depending on ask shard should between 10g , 50g; logging use-case rather on 50gb side.
  2. which queries want speed up? target recent data or long time-spans? if interested in recent data, use different node types in hot-warm architecture. more power nodes recent data , less data; bulk of older , less accessed data on less powerful nodes.
  3. generally you'll need find bottleneck. i'd free monitoring plugin , take @ how both kibana , elasticsearch doing.

wild guess: limited on io. prefer local disks on ebs, prefer ssds on spinning disks, , if can, many iops can afford use-case.





wiki

Comments

Post a Comment

Popular posts from this blog

python - Read npy file directly from S3 StreamingBody -

-
i have many .npy (numpy) , .json files saved in s3, need download , load these files. for json files downloading files , loading dictionary using following code: obj = s3.object(bucket, key) response = obj.get() body = response['body'].read().decode('utf-8') data = json.loads(body) that way avoid writing .json file disk , reading it. now want same .npy files, equivalent? currently downloading file, saving temp file on disk , loading np.load avoid if possible. following https://stackoverflow.com/a/28196540/3509999 : obj = s3.object(bucket, key) io.bytesio(obj.get()["body"].read()) f: # rewind file f.seek(0) arr = np.load(f) wiki
Read more

python - Minimize function with Scipy minimize -

-
trying obtain d value (integer) std_diff objective function minimal, using scipy minimize. my code: def std_diff(d): return std(diff(df['bn'].values,d)); scipy.optimize import minimize b=(3,) res = minimize(std_diff,(1,), method='slsqp', bounds = b) **df['bn'].values** out[72]: array([ 2, 2, 2, 2, 3, 2, 7, 5, 7, 11, 8, 2, 11, 7, 15, 8, 7, 12, 21, 19, 32, 35, 40, 35, 21, 19, 25, 20, 40, 80, 99], dtype=int64) error is"indexerror: many indices array " in case not use bounds : res = minimize(std_diff,(1,), method='slsqp'), error: > in _minimize_slsqp(func, x0, args, jac, bounds, constraints, maxiter, > ftol, iprint, disp, eps, callback, **unknown_options) > 368 fx = float(np.asarray(func(x))) > 369 except: > --> 370 raise valueerror("objective function must return scalar") > 371 # compute constraints > 3
Read more

kotlin - Out-projected type in generic interface prohibits the use of metod with generic parameter -

-
i start using kotlin , defined interface this: interface aadapter<vh : recyclerview.viewholder> { fun oncreateaviewholder(parent: viewgroup): vh fun onbindaviewholder(v: vh, position: int) } and when try use on code: class klasa ( private val adapter: aadapter<*> ) { fun dosth(){ //... val vh = this.adapter.oncreateaviewholder(parent) //on below line error adapter.onbindaviewholder(v, position) //... } } i error out-projected type 'aadapter<*>' prohibits use of 'public abstract fun onbindaviewholder(v: t, position: int): unit i tied add "in" or "out" definition i'm confused. how allow it. replace aadapter<*> aadapter<recyclerview.viewholder> . class klasa ( private val adapter: aadapter<recyclerview.viewholder> ) { // ... because have declared aadapter has type parameter vh type subclass of recyclerview.
Read more