o 7]ha~@sddlmZmZmZddlmZddlmZmZddl Z ddl Z ddl m Z m Z ddlmZddlmZmZmZmZdd lmZdd lmZed d eDZed d eDZedd eDZeeddgBZdZejreddkr{eddks}Je edde!ddZ"ne eZ"hdZ#e dZ$iZ%Gddde&Z'ddZ(Gddde&Z)Gddde)Z*Gdd d e+Z,Gd!d"d"e&Z-Gd#d$d$e&Z.d%d&Z/dS)')absolute_importdivisionunicode_literals) text_type) http_clienturllibN)BytesIOStringIO) webencodings)EOFspaceCharacters asciiLettersasciiUppercase)_ReparseException)_utilscCg|]}|dqSasciiencode.0itemrC/usr/lib/python3/dist-packages/pip/_vendor/html5lib/_inputstream.py rcCrrrrrrrrrcCrrrrrrrrr>   z[ - -/:-@\[-`{-~]c@sHeZdZdZddZddZddZdd Zd d Zd d Z ddZ dS)BufferedStreamzBuffering for streams that do not have buffering of their own The buffer is implemented as a list of chunks on the assumption that joining many strings will be slow since it is O(n**2) cCs||_g|_ddg|_dS)Nr r)streambufferposition)selfrCrrr__init__:szBufferedStream.__init__cCs<d}|jd|jdD]}|t|7}q ||jd7}|SNrr )rDrElen)rFposchunkrrrtell?s zBufferedStream.tellcCsd||ksJ|}d}t|j||kr+|t|j|8}|d7}t|j||ks||g|_dSrH)_bufferedBytesrIrDrE)rFrJoffsetirrrseekFszBufferedStream.seekcCsP|js||S|jdt|jkr#|jdt|jdkr#||S||S)Nrr r )rD _readStreamrErI_readFromBufferrFbytesrrrreadOs    zBufferedStream.readcCstdd|jDS)NcSsg|]}t|qSr)rIrrrrrYsz1BufferedStream._bufferedBytes..)sumrDrFrrrrMXzBufferedStream._bufferedBytescCs<|j|}|j||jdd7<t||jd<|SrH)rCrUrDappendrErI)rFrTdatarrrrQ[s  zBufferedStream._readStreamcCs|}g}|jd}|jd}|t|jkrc|dkrc|dksJ|j|}|t||kr6|}|||g|_nt||}|t|g|_|d7}|||||||8}d}|t|jkrc|dks|rm|||d|S)Nrr )rErIrDrYrQjoin)rFrTremainingBytesrv bufferIndex bufferOffset bufferedData bytesToReadrrrrRbs(      zBufferedStream._readFromBufferN) __name__ __module__ __qualname____doc__rGrLrPrUrMrQrRrrrrrB3s   rBcKst|tjst|tjjrt|jtjrd}nt|dr%t|dt }nt|t }|rCdd|D}|r;t d|t |fi|St |fi|S)NFrUrcSsg|] }|dr|qS) _encoding)endswith)rxrrrrsz#HTMLInputStream..z3Cannot set an encoding with a unicode input, set %r) isinstancer HTTPResponserresponseaddbasefphasattrrUr TypeErrorHTMLUnicodeInputStreamHTMLBinaryInputStream)sourcekwargs isUnicode encodingsrrrHTMLInputStream}s      rwc@speZdZdZdZddZddZddZd d Zd d Z d dZ dddZ ddZ ddZ dddZddZdS)rqProvides a unicode stream of characters to the HTMLTokenizer. This class takes care of character encoding and removing or replacing incorrect byte-sequences and also provides column and line tracking. i(cCsZtjsd|_ntddkr|j|_n|j|_dg|_tddf|_| ||_ | dS)Initialises the HTMLInputStream. HTMLInputStream(source, [encoding]) -> Normalized stream from source for use by html5lib. source can be either a file-object, local filename or a string. The optional encoding parameter must be a string that indicates the encoding. If specified, that encoding will be used, regardless of any BOM or later declaration (such as in a meta element) Nu􏿿r rutf-8certain) rsupports_lone_surrogatesreportCharacterErrorsrIcharacterErrorsUCS4characterErrorsUCS2newLineslookupEncoding charEncoding openStream dataStreamreset)rFrsrrrrGs    zHTMLUnicodeInputStream.__init__cCs.d|_d|_d|_g|_d|_d|_d|_dS)Nr)rK chunkSize chunkOffseterrors prevNumLines prevNumCols_bufferedCharacterrWrrrrs zHTMLUnicodeInputStream.resetcCst|dr |}|St|}|SzvProduces a file object from source. source can be either a file object, local filename or a string. rU)ror rFrsrCrrrrs z!HTMLUnicodeInputStream.openStreamcCsZ|j}|dd|}|j|}|dd|}|dkr#|j|}||fS||d}||fS)N rr r )rKcountrrfindr)rFrNrKnLines positionLine lastLinePospositionColumnrrr _positions   z HTMLUnicodeInputStream._positioncCs||j\}}|d|fS)z:Returns (line, col) of the current position in the stream.r )rr)rFlinecolrrrrEs zHTMLUnicodeInputStream.positioncCs6|j|jkr |s tS|j}|j|}|d|_|S)zo Read one character from the stream or queue if available. Return EOF when EOF is reached. r )rr readChunkr rK)rFrcharrrrrs   zHTMLUnicodeInputStream.charNcCs|dur|j}||j\|_|_d|_d|_d|_|j|}|j r,|j |}d|_ n|s0dSt |dkrWt |d}|dksLd|krJdkrWnn |d|_ |dd}|j r_| || d d }| d d }||_t ||_d S) NrrFr r iz r T)_defaultChunkSizerrrrrKrrrUrrIordr}replace)rFrrZlastvrrrrs0           z HTMLUnicodeInputStream.readChunkcCs(ttt|D]}|jdq dS)Ninvalid-codepoint)rangerIinvalid_unicode_refindallrrY)rFrZ_rrrr~sz*HTMLUnicodeInputStream.characterErrorsUCS4cCsd}t|D]Q}|r qt|}|}t|||dr9t|||d}|tvr6|j dd}q|dkrP|dkrP|t |dkrP|j dqd}|j dqdS)NFrTrir ) rfinditerrgroupstartrisSurrogatePairsurrogatePairToCodepointnon_bmp_invalid_codepointsrrYrI)rFrZskipmatch codepointrJchar_valrrrr#s"  z*HTMLUnicodeInputStream.characterErrorsUCS2Fc Cszt||f}Wn4ty< |D] }t|dksJqddd|D}|s-d|}td|}t||f<Ynwg} ||j|j}|durT|j|j krSn*n| }||j krl| |j|j|||_n| |j|jd| s|nq@d|} | S) z Returns a string of characters from the stream up to but not including any character in 'characters' or EOF. 'characters' must be a container that supports the 'in' method and iteration over its characters. TrcSsg|]}dt|qS)z\x%02x)r)rcrrrrHsz5HTMLUnicodeInputStream.charsUntil..z^%sz[%s]+N) charsUntilRegExKeyErrorrr\recompilerrKrrendrYr) rF charactersoppositecharsrregexr^mrrrrr charsUntil:s:     z!HTMLUnicodeInputStream.charsUntilcCsZ|tur)|jdkr||j|_|jd7_dS|jd8_|j|j|ks+JdSdSrH)r rrKr)rFrrrrungetis   zHTMLUnicodeInputStream.ungetN)F)rcrdrerfrrGrrrrErrr~rrrrrrrrqs   &  /rqc@sReZdZdZ   dddZddZd d Zdd d Zd dZddZ ddZ dS)rrrxN windows-1252TcCsn|||_t||jd|_d|_||_||_||_||_ ||_ | ||_ |j ddus1J| dS)ryidrN)r rawStreamrqrG numBytesMetanumBytesChardetoverride_encodingtransport_encodingsame_origin_parent_encodinglikely_encodingdefault_encodingdetermineEncodingrr)rFrsrrrrr useChardetrrrrGs   zHTMLBinaryInputStream.__init__cCs&|jdj|jd|_t|dS)Nrr)r codec_info streamreaderrrrqrrWrrrrszHTMLBinaryInputStream.resetcCsJt|dr|}nt|}z ||W|Sty$t|}Y|Swr)rorrPrL ExceptionrBrrrrrs  z HTMLBinaryInputStream.openStreamcCs|df}|ddur|St|jdf}|ddur|St|jdf}|ddur,|S|df}|ddur:|St|jdf}|ddurQ|djdsQ|St|jdf}|ddur`|S|rzddl m }Wn t ysYn@wg}|}|j s|j |j}t|tsJ|sn |||||j r||t|jd}|j d|dur|dfSt|jdf}|ddur|StddfS)Nr{r tentativezutf-16)UniversalDetectorencodingr) detectBOMrrrdetectEncodingMetarname startswithr%pip._vendor.chardet.universaldetectorr ImportErrordonerrUrrjrTrYfeedcloseresultrPr)rFchardetrrbuffersdetectorrDrrrrrsV             z'HTMLBinaryInputStream.determineEncodingcCs|jddks Jt|}|durdS|jdvr$td}|dus"JdS||jdkr5|jddf|_dS|jd|df|_|td|jd|f)Nr r{utf-16beutf-16lerzrzEncoding changed from %s to %s)rrrrrPrr)rF newEncodingrrrchangeEncodings   z$HTMLBinaryInputStream.changeEncodingc Cstjdtjdtjdtjdtjdi}|jd}t|t sJ| |dd}d}|s?| |}d}|s?| |dd }d }|rK|j |t |S|j d dS) zAttempts to detect at BOM at the start of the stream. If an encoding can be determined from the BOM return the name of the encoding otherwise return Nonerzrrzutf-32lezutf-32beNrr) codecsBOM_UTF8 BOM_UTF16_LE BOM_UTF16_BE BOM_UTF32_LE BOM_UTF32_BErrUrjrTgetrPr)rFbomDictstringrrPrrrrs&      zHTMLBinaryInputStream.detectBOMcCsV|j|j}t|tsJt|}|jd|}|dur)|jdvr)t d}|S)z9Report the encoding declared by the meta element rNrrz) rrUrrjrTEncodingParserrP getEncodingrr)rFrDparserrrrrr3s z(HTMLBinaryInputStream.detectEncodingMeta)NNNNrT)T) rcrdrerfrGrrrrrrrrrrrrzs * > "rrc@seZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ e e e Z ddZe eZefddZddZddZddZdS) EncodingByteszString-like object with an associated position and various extra methods If the position is ever greater than the string length then an exception is raisedcCst|tsJt||Sr)rjrT__new__lowerrFvaluerrrrFszEncodingBytes.__new__cCs d|_dS)Nr )rrrrrrGJs zEncodingBytes.__init__cCs|SrrrWrrr__iter__NszEncodingBytes.__iter__cCs<|jd}|_|t|krt|dkrt|||dS)Nr rrrI StopIterationrprFprrr__next__Qs  zEncodingBytes.__next__cCs|Sr)rrWrrrnextYszEncodingBytes.nextcCs@|j}|t|kr t|dkrt|d|_}|||dSrHrrrrrprevious]s zEncodingBytes.previouscCs|jt|kr t||_dSrrrIr)rFrErrr setPositionfs zEncodingBytes.setPositioncCs&|jt|kr t|jdkr|jSdS)NrrrWrrr getPositionks  zEncodingBytes.getPositioncCs||j|jdSNr )rErWrrrgetCurrentByteurXzEncodingBytes.getCurrentBytecCsR|j}|t|kr$|||d}||vr||_|S|d7}|t|ks ||_dS)zSkip past a list of charactersr NrErIrrFrrrrrrrzs  zEncodingBytes.skipcCsR|j}|t|kr$|||d}||vr||_|S|d7}|t|ks ||_dSrrrrrr skipUntils  zEncodingBytes.skipUntilcCs(|||j}|r|jt|7_|S)zLook for a sequence of bytes at the start of a string. If the bytes are found return True and advance the position to the byte after the match. Otherwise return False and leave the position alone)rrErI)rFrTr^rrr matchBytesszEncodingBytes.matchBytescCs6z|||jt|d|_WdStytw)zLook for the next sequence of bytes matching a given sequence. If a match is found advance the position to the last byte of the matchr T)indexrErIr ValueErrorrrSrrrjumpTos  zEncodingBytes.jumpToN)rcrdrerfrrGrrrrrrpropertyrEr currentBytespaceCharactersBytesrrr r rrrrrBs"     rc@sXeZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ ddZ dS)rz?Mini parser for detecting character encoding from meta elementscCst||_d|_dS)z3string - the data to work on for encoding detectionN)rrZrrFrZrrrrGs  zEncodingParser.__init__c Csd|jvrdSd|jfd|jfd|jfd|jfd|jfd|jff}|jD]?}d}z|jdWn ty<Y|j Sw|D]\}}|j|r\z|}Wn ty[d}Ynwq?|sc|j Sq$|j S) NsrZr rWrrrrs zEncodingParser.handleCommentcCs|jjtvrdSd}d} |}|durdS|ddkr/|ddk}|r.|dur.||_dSn?|ddkrG|d}t|}|durF||_dSn'|ddkrntt|d}|}|durnt|}|durn|rl||_dS|}q ) NTFrs http-equivr s content-typecharsetscontent) rZrr getAttributerrContentAttrParserrparse)rF hasPragmapendingEncodingattrtentativeEncodingcodec contentParserrrrrs@      zEncodingParser.handleMetacCs |dS)NF)handlePossibleTagrWrrrrs z%EncodingParser.handlePossibleStartTagcCst|j|dS)NT)rrZr&rWrrrrs  z#EncodingParser.handlePossibleEndTagcCsj|j}|jtvr|r||dS|t}|dkr#|dS|}|dur3|}|dus+dS)NTr)rZrasciiLettersBytesrrrspacesAngleBracketsr)rFendTagrZrr"rrrr&s  z EncodingParser.handlePossibleTagcCr)NrrrWrrrrs zEncodingParser.handleOthercCs|j}|ttdgB}|dust|dksJ|dvrdSg}g} |dkr+|r+n0|tvr4|}n'|dvr?d|dfS|tvrK||n |durQdS||t |}q$|dkrj| d|dfSt ||}|d vr|} t |}||krt |d|d|fS|tvr||n||qy|d krd|dfS|tvr||n |durdS|| t |}|t vrd|d|fS|tvr||n |durdS||q) z_Return a name,value pair for the next attribute in the stream, if one is found, or None/Nr )rNT=)r*rr[)'"r) rZrr frozensetrIr\asciiUppercaseBytesrYrrrr()rFrZrattrName attrValue quoteCharrrrrsn      zEncodingParser.getAttributeN) rcrdrerfrGrrrrrr&rrrrrrrs$ rc@seZdZddZddZdS)rcCst|tsJ||_dSr)rjrTrZrrrrrGas zContentAttrParser.__init__cCszy|jd|jjd7_|j|jjdksWdS|jjd7_|j|jjdvrS|jj}|jjd7_|jj}|j|rP|j||jjWSWdS|jj}z|jt|j||jjWWStyy|j|dYWSwtyYdSw)Nrr r+)r-r,)rZr rErrrrr)rF quoteMark oldPositionrrrres2         zContentAttrParser.parseN)rcrdrerGrrrrrr`s rcCs\t|trz|d}Wn tyYdSw|dur,zt|WSty+YdSwdS)z{Return the python codec name corresponding to an encoding or None if the string doesn't correspond to a valid encoding.rN)rjrTdecodeUnicodeDecodeErrorr lookupAttributeError)rrrrrs    r)0 __future__rrrpip._vendor.sixrpip._vendor.six.movesrrrriorr pip._vendorr constantsr r rrrrrr.rr'r/r(invalid_unicode_no_surrogater|rrevalrrascii_punctuation_rerobjectrBrwrqrrrTrrrrrrrrsJ      JgIb= '