java - (Solved) Kafka 0.11.0.0 keeps reseting offset on restart -

-



i have problem kafka 0.11.0.0

when create new topic, put come data , consume java consumer, after restarting kafka 0.11.0.0 offsets consumer group disappear. topic stays , has same data in it, offsets purged. makes consumer download records topics again. weird, 1 topic have old, correct offsets, other offsets deleted, maybe because 1 topic there while.

i commit consumed records commitsync(). offset saved on broker, can restart java consumer , starts correct offset after restarting entire kafka offset consumer groups resets 0. check current commits before consuming after restart kafka-consumer-groups.sh script , it's broker resets them.

i had no problem in kafka 0.10.2.1. experience problem in 0.11.0.0 version.

my consumer has auto.offset.reset set earliest, auto commit set false because i'm committing manually. kafka data stored in non-tmp directory necessary permissions. rest of broker configuration default.

i need 0.11.0.0 version transactions. have no idea problem can be. can cause this? there new config parameter missed somewhere?

@edit topic stays has problems offsets, doesn't entirely purged offset after restarting isn't correct , consumer gets around ~15% of data after every broken restart.

@edit2 not server.log full of:

warn received partitionleaderepoch assignment epoch < latestepoch. implies messages have arrived out of order. new: {epoch:4, offset:1669}, current: {epoch:5, offset1540} partition: __consumer_offsets-26 (kafka.server.epoch.leaderepochfilecache)

it seems it's connected consumer group because of logs:

[2017-08-22 08:59:30,719] info [groupcoordinator 0]: preparing rebalance group scrapperbackup old generation 119 (__consumer_offsets-26) (kafka.coordinator.group.groupcoordinator) [2017-08-22 08:59:30,720] info [groupcoordinator 0]: group scrapperbackup generation 120 empty (__consumer_offsets-26) (kafka.coordinator.group.groupcoordinator)

there logs 1 on restart:

[2017-08-22 09:15:37,948] info partition [__consumer_offsets,26] on broker 0: __consumer_offsets-26 starts @ leader epoch 6 offset 1699. previous leader epoch was: 5 (kafka.cluster.partition)

@edit3 creating new directory kafka/zookeeper data , creating scratch helped. don't know problem, works properly. seems error occurred in data directories of apps.





wiki

Comments

Post a Comment

Popular posts from this blog

elasticsearch - what is the equivalent data type for geo_point in hibernate search? -

-
i using elastic search hibernate search.i want make 1 property data type geo_point ,so can draw tile map using kibana.is there equivalent data type in hibernate search geo_point in elastic search? you can use spatial feature of hibernate search: import org.hibernate.search.annotations.*; @entity @indexed @spatial public class hotel { @latitude double latitude @longitude double longitude // ... } you end geo_point field named location in document. for more information: https://docs.jboss.org/hibernate/stable/search/reference/en-us/html_single/#spatial wiki
Read more

Jenkins: find build number for git commit -

-
each commit git repo triggers build of jenkins pipeline. i want retrieve buildnumber of old build commit hash. how can that? i know information exists because can find in ui. some background why want this: when tags commit want create release (by tagging docker image). each build push image containing build number. want find out build number corresponds commit can find image want release. if understand question correctly, first need create git hook trigger new build. part covered in answer how react new tags in git hooks? ', though if using github, bitbucket or gitlab, there may other ways of going it. then when build initiated, there build number provided variable ' build_number ' ,in jenkins. if want include git tag name, can use in script, there seems fews ways: git parameter plugin git tag message plugin typically these plugins create environment variable can consumed scripts. not providing more concrete examples, since not aware of exact ...
Read more

firebase - How to wait value in Ionic 2 -

-
i'm trying wait value may change after execute "this.bookmarlist.take(1).subscribe(..." my problem when run application, execute second if statement "if(found==false)" before execute subscribe. this code may found thing wrong or need add additions solve problem. let loader = this.loadingctrl.create({ content: "", }); let found=false; loader.present().then(()=>{ this.bookmarklist.take(1).subscribe(data=>{ data.foreach(b=>{ if(b.nid==this.appservice.id && b.url==url){ let alert = this.alertctrl.create({ title: 'already bookmarked', subtitle: 'this note image bookmarked', buttons: ['ok'] }); alert.present(); found=true; } }) }) }).then(()=>{ if(found==false){ this.afd.list('/bookmark/').push({ uemail: th...
Read more