Beautiful Soup fails to recognize UTF-8 encoding on Python 3, IPython 6 console -

-



i trying read xml document using beautiful soup on python 3.6.2, ipython 6.1.0, windows 10, , can't encoding right.

here's test xml, saved file in utf8-encoding:

<?xml version="1.0" encoding="utf-8"?> <root> <info name="愛よ">ÜÜÜÜÜÜÜ</info> <items> <item thing="ÖöÖö">"23Äßßß"</item> </items> </root>

first check xml using elementtree:

import xml.etree.elementtree et  def printxml(xml,indent=''):     print(indent+str(xml.tag)+': '+(xml.text if xml.text not none else '').replace('\n',''))     if len(xml.attrib) > 0:         k,v in xml.attrib.items():             print(indent+'\t'+k+' - '+v)     if xml.getchildren():         child in xml.getchildren():             printxml(child,indent+'\t')  xml0 = et.parse("test.xml").getroot() printxml(xml0)

the output correct:

root:          info: ÜÜÜÜÜÜÜ                 name - 愛よ         items:                  item: "23Äßßß"                         thing - ÖöÖö

now read same file beautiful soup , pretty-print it:

import bs4  open("test.xml") ff:     xml = bs4.beautifulsoup(ff,"html5lib") print(xml.prettify())

output:

<!--?xml version="1.0" encoding="utf-8"?--> <html>  <head>  </head>  <body>   <root>    <info name="愛よ">     ÜÜÜÜÜÜÜ    </info>    <items>     <item thing="ÖöÖö">      "23Äßßß"     </item>    </items>   </root>  </body> </html>

this wrong. doing call explicite encoding specified bs4.beautifulsoup(ff,"html5lib",from_encoding="utf-8") doesn't change result.

doing

print(xml.original_encoding)

outputs

none

so beautiful soup apparently unable detect original encoding though file encoded in utf8 (according notepad++) , header information says utf-8 well, , have chardet installed as doc recommends.

am making mistake here? causing this?

edit: when invoke code without html5lib warning:

userwarning: no parser explicitly specified, i'm using best available html parser system ("html5lib").  isn't problem, if run code on system, or in different virtual environment,  may use different parser , behave differently.  code caused warning on line 241 of file c:\users\my.name\appdata\local\continuum\anaconda2\envs\python3\lib\site-packages\spyder\utils\ipython\start_kernel.py.  rid of warning, change code looks this:   beautifulsoup(your_markup})  this:   beautifulsoup(your_markup, "html5lib")    markup_type=markup_type))

edit 2:

as suggested in comment tried bs4.beautifulsoup(ff,"html.parser"), problem remains.

then installed lxml , tried bs4.beautifulsoup(ff,"lxml-xml"), still same output.

what strikes me odd when specifying encoding bs4.beautifulsoup(ff,"lxml-xml",from_encoding='utf-8') value of xml.original_encoding none contrary written in doc.

edit 3:

i put xml contents string 

xmlstring = "<?xml version=\"1.0\" encoding=\"utf-8\"?><root><info name=\"愛よ\">ÜÜÜÜÜÜÜ</info><items><item thing=\"ÖöÖö\">\"23Äßßß\"</item></items></root>"

and used bs4.beautifulsoup(xmlstring,"lxml-xml"), i'm getting correct output:

<?xml version="1.0" encoding="utf-8"?> <root>  <info name="愛よ">   ÜÜÜÜÜÜÜ  </info>  <items>   <item thing="ÖöÖö">    "23Äßßß"   </item>  </items> </root>

so seems wrong file after all.

found error, have specify encoding when opening file:

with open("test.xml",encoding='utf-8') ff:     xml = bs4.beautifulsoup(ff,"html5lib")

as i'm on python 3 thought value of encoding utf-8 default, turned out it's system-dependent , on system it's cp1252.





wiki

Comments

Post a Comment

Popular posts from this blog

Asterisk AGI Python Script to Dialplan does not work -

-
i'm stuck project have check database of sip extension incoming caller id , dial sip extension. below receive when run script on asterisk cli. -- registered sip '9125' @ 192.168.1.102:50142 > saved useragent "zoiper rv2.8.40" peer 9125 [2017-08-22 13:53:33] notice[5095]: chan_sip.c:24558 handle_response_peerpoke: peer '9125' reachable. (17ms / 2000ms) == using sip rtp tos bits 184 == using sip rtp cos mark 5 -- executing [0112617769@from-trunk:1] answer("sip/obitrunk1-0000000d", "") in new stack > 0x1ca50d0 -- probation passed - setting rtp source address 192.168.1.98:16834 -- executing [0112617769@from-trunk:2] agi("sip/obitrunk1-0000000d", "test.py") in new stack -- launched agi script /var/lib/asterisk/agi-bin/test.py args: ['/var/lib/asterisk/agi-bin/test.py'] env line: agi_request: test.py env line: agi_channel: sip/obitrunk1-0000000d env line: agi_language: en env line: a...
Read more

python - Read npy file directly from S3 StreamingBody -

-
i have many .npy (numpy) , .json files saved in s3, need download , load these files. for json files downloading files , loading dictionary using following code: obj = s3.object(bucket, key) response = obj.get() body = response['body'].read().decode('utf-8') data = json.loads(body) that way avoid writing .json file disk , reading it. now want same .npy files, equivalent? currently downloading file, saving temp file on disk , loading np.load avoid if possible. following https://stackoverflow.com/a/28196540/3509999 : obj = s3.object(bucket, key) io.bytesio(obj.get()["body"].read()) f: # rewind file f.seek(0) arr = np.load(f) wiki
Read more

kotlin - Out-projected type in generic interface prohibits the use of metod with generic parameter -

-
i start using kotlin , defined interface this: interface aadapter<vh : recyclerview.viewholder> { fun oncreateaviewholder(parent: viewgroup): vh fun onbindaviewholder(v: vh, position: int) } and when try use on code: class klasa ( private val adapter: aadapter<*> ) { fun dosth(){ //... val vh = this.adapter.oncreateaviewholder(parent) //on below line error adapter.onbindaviewholder(v, position) //... } } i error out-projected type 'aadapter<*>' prohibits use of 'public abstract fun onbindaviewholder(v: t, position: int): unit i tied add "in" or "out" definition i'm confused. how allow it. replace aadapter<*> aadapter<recyclerview.viewholder> . class klasa ( private val adapter: aadapter<recyclerview.viewholder> ) { // ... because have declared aadapter has type parameter vh type subclass of recyclerview....
Read more