o 7]h,@sddlmZmZmZddlmZddlmZm Z ddl m Z ddl m Z ddl mZddl mZmZdd l mZmZmZdd l mZmZdd l mZdd lmZdd lmZeeZe dkraeZne ZGdddeZdS))absolute_importdivisionunicode_literals)unichr)deque OrderedDict) version_info)spaceCharacters)entities) asciiLettersasciiUpper2Lower)digits hexDigitsEOF) tokenTypes tagTokenTypes)replacementCharacters)HTMLInputStream)Trie)csdeZdZdZdfdd ZddZddZdd d Zd d ZddZ ddZ ddZ ddZ ddZ ddZddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1Zd2d3Zd4d5Zd6d7Zd8d9Zd:d;Zdd?Z!d@dAZ"dBdCZ#dDdEZ$dFdGZ%dHdIZ&dJdKZ'dLdMZ(dNdOZ)dPdQZ*dRdSZ+dTdUZ,dVdWZ-dXdYZ.dZd[Z/d\d]Z0d^d_Z1d`daZ2dbdcZ3dddeZ4dfdgZ5dhdiZ6djdkZ7dldmZ8dndoZ9dpdqZ:drdsZ;dtduZdzd{Z?d|d}Z@d~dZAddZBddZCddZDddZEddZFddZGddZHddZIddZJddZKddZLZMS) HTMLTokenizera  This class takes care of tokenizing HTML. * self.currentToken Holds the token that is currently being processed. * self.state Holds a reference to the method to be invoked... XXX * self.stream Points to HTMLInputStream object. Nc sJt|fi||_||_d|_g|_|j|_d|_d|_t t | dSNF) rstreamparser escapeFlag lastFourChars dataStatestateescape currentTokensuperr__init__)selfrrkwargs __class__A/usr/lib/python3/dist-packages/pip/_vendor/html5lib/_tokenizer.pyr#(szHTMLTokenizer.__init__ccsftg|_|r1|jjrtd|jjddV|jjs|jr+|jV|js"|s dSdS)z This is where the magic happens. We do our usually processing through the states and when we have a token to return we yield the token which pauses processing until the next token is requested. ParseErrorrtypedataN)r tokenQueuerrerrorsrpoppopleftr$r(r(r)__iter__7s  zHTMLTokenizer.__iter__c Cst}d}|r t}d}g}|j}||vr+|tur+|||j}||vr+|tustd||}|tvrJt|}|j t ddd|idnd|krTd ksYn|d krjd }|j t ddd|idnfd |krtd ksnd|krdksnd|krdksnd|krdksn|t gdvr|j t ddd|idzt |}Wnt y|d}t d|d?Bt d|d@B}Ynw|dkr|j t ddd|j||S)zThis function returns either U+FFFD or the character based on the decimal or hexadecimal representation. It also discards ";" if present. If not present self.tokenQueue.append({"type": tokenTypes["ParseError"]}) is invoked. r*z$illegal-codepoint-for-numeric-entity charAsIntr,r-datavarsii�r ii)# iiiiiiiiiiiiiiiiiii i i i i i i i i i iiiiir:iii;z numeric-entity-without-semicolonr+)rrrcharrappendintjoinrr.r frozensetchr ValueErrorunget) r$isHexallowedradix charStackcr7rCvr(r(r)consumeNumberEntityGs\          $  z!HTMLTokenizer.consumeNumberEntityFc Csd}|jg}|dtvs!|dtddfvs!|dur+||dkr+|j|dn|ddkrd}||j|ddvrKd}||j|rS|dtvs[|si|dtvri|j|d||}n|j t d d d |j| dd |}n|dturt d |sn||j|dtuszt d |dd}t|}Wn tyd}Ynw|dur|dd kr|j t d dd |dd kr|r||tvs||tvs||dkr|j| dd |}n2t|}|j| |d ||d7}n|j t d dd |j| dd |}|rC|jddd|7<dS|tvrKd}nd}|j t ||d dS)N&r<#F)xXTr*zexpected-numeric-entityr+r6rBznamed-entity-without-semicolon=zexpected-named-entityr-r SpaceCharacters Characters)rrCr rrJrDrrrQr.rr0rF entitiesTriehas_keys_with_prefixlongest_prefixlenKeyErrorr r r!) r$ allowedChar fromAttributeoutputrNhex entityName entityLength tokenTyper(r(r) consumeEntityst                   zHTMLTokenizer.consumeEntitycCs|j|dddS)zIThis method replaces the need for "entityInAttributeValueState". T)r`raN)rg)r$r`r(r(r)processEntityInAttributesz&HTMLTokenizer.processEntityInAttributecCs|j}|dtvr^|dt|d<|dtdkr8|d}t|}t|t|kr4||ddd||d<|dtdkr^|drO|j tdd d |d r^|j tdd d |j ||j |_ dS) zThis method is a generic handler for emitting the tags. It also sets the state to "data" because that's what's needed after a token has been emitted. r,nameStartTagr-NrUEndTagr*zattributes-in-end-tagr+ selfClosingzself-closing-flag-on-end-tag) r!r translater r attributeMapr^updater.rDrr)r$tokenrawr-r(r(r)emitCurrentTokens(     zHTMLTokenizer.emitCurrentTokencCs|j}|dkr|j|_d S|dkr|j|_d S|dkr5|jtddd|jtdddd S|tur;dS|t vrS|jtd ||j t d dd S|j d }|jtd||dd S) NrRrSr*invalid-codepointr+rZFrYTrRrSrs) rrCentityDataStater tagOpenStater.rDrrr charsUntilr$r-charsr(r(r)rs6     zHTMLTokenizer.dataStatecC||j|_dSNT)rgrrr2r(r(r)rvzHTMLTokenizer.entityDataStatecCs|j}|dkr|j|_d S|dkr|j|_d S|tkrdS|dkr;|jtddd|jtdd dd S|t vrS|jtd ||j t d dd S|j d }|jtd||dd S) NrRrSFrsr*rtr+rZr;rYTru) rrCcharacterReferenceInRcdatarrcdataLessThanSignStaterr.rDrr rxryr(r(r) rcdataState"s6     zHTMLTokenizer.rcdataStatecCr{r|)rgrrr2r(r(r)r~?r}z(HTMLTokenizer.characterReferenceInRcdatacC|j}|dkr|j|_d S|dkr+|jtddd|jtdddd S|tkr1dS|jd }|jtd||dd S NrSrsr*rtr+rZr;F)rSrsT) rrCrawtextLessThanSignStaterr.rDrrrxryr(r(r) rawtextStateD&       zHTMLTokenizer.rawtextStatecCrr) rrCscriptDataLessThanSignStaterr.rDrrrxryr(r(r)scriptDataStateVrzHTMLTokenizer.scriptDataStatecCst|j}|tkr dS|dkr'|jtddd|jtddddS|jtd||jdddS) NFrsr*rtr+rZr;T)rrCrr.rDrrxr$r-r(r(r)plaintextStatehs    zHTMLTokenizer.plaintextStatecCs|j}|dkr|j|_dS|dkr|j|_dS|tvr.td|gddd|_|j|_dS|dkrN|j tddd |j td d d |j |_dS|d kri|j tdd d |j ||j |_dS|j tddd |j td dd |j ||j |_dS)N!/rjF)r,rir-rlselfClosingAcknowledged>r*z'expected-tag-name-but-got-right-bracketr+rZz<>?z'expected-tag-name-but-got-question-markzexpected-tag-namerST)rrCmarkupDeclarationOpenStatercloseTagOpenStater rr! tagNameStater.rDrrJbogusCommentStaterr(r(r)rwwsH      zHTMLTokenizer.tagOpenStatecCs|j}|tvrtd|gdd|_|j|_dS|dkr.|jtddd|j |_dS|t urN|jtddd|jtd d d|j |_dS|jtdd d |id |j ||j |_dS)NrkFr,rir-rlrr*z*expected-closing-tag-but-got-right-bracketr+z expected-closing-tag-but-got-eofrZs  z6HTMLTokenizer.scriptDataDoubleEscapedLessThanSignStatecCrr)rrCr rGr.rDrrrrrrr rJrr(r(r)rIrz,HTMLTokenizer.scriptDataDoubleEscapeEndStatecCs:|j}|tvr|jtddS|tvr&|jd|dg|j|_dS|dkr0| dS|dkr:|j |_dS|dvrY|j t ddd |jd|dg|j|_dS|d krx|j t dd d |jdd dg|j|_dS|t ur|j t dd d |j|_dS|jd|dg|j|_dS)NTr-r6rr)'"rXrSr*#invalid-character-in-attribute-namer+rsrtr;z#expected-attribute-name-but-got-eof)rrCr rxr r!rDattributeNameStaterrrrr.rrrrr(r(r)rYsJ     z&HTMLTokenizer.beforeAttributeNameStatecCs|j}d}d}|dkr|j|_n|tvr-|jddd||jtd7<d}nz|dkr4d}ns|tvr=|j|_nj|dkrF|j |_na|d kre|j t d d d |jdddd 7<d}nB|dvr|j t d dd |jddd|7<d}n#|t ur|j t d dd |j|_n|jddd|7<d}|r|jdddt|jddd<|jdddD]\}}|jddd|kr|j t d dd nq|r|dS)NTFrXr-rUrrrrsr*rtr+r;rrrSrzeof-in-attribute-namezduplicate-attribute)rrCbeforeAttributeValueStaterr r!rxr afterAttributeNameStaterr.rDrrrrmr rr)r$r-leavingThisState emitTokenri_r(r(r)rws`          z HTMLTokenizer.attributeNameStatecCsN|j}|tvr|jtddS|dkr|j|_dS|dkr&|dS|tvr:|jd |dg|j |_dS|dkrD|j |_dS|dkrc|j t dd d |jd d dg|j |_dS|d vr|j t dd d |jd |dg|j |_dS|tur|j t ddd |j|_dS|jd |dg|j |_dS)NTrXrr-r6rrsr*rtr+r;rz&invalid-character-after-attribute-namezexpected-end-of-tag-but-got-eof)rrCr rxrrrrr r!rDrrr.rrrrr(r(r)rsP     z%HTMLTokenizer.afterAttributeNameStatecCsn|j}|tvr|jtddS|dkr|j|_dS|dkr,|j|_|j|dS|dkr6|j|_dS|dkrK|j t ddd| dS|d krm|j t dd d|j d d d d7<|j|_dS|dvr|j t ddd|j d d d |7<|j|_dS|tur|j t ddd|j|_dS|j d d d |7<|j|_dS)NTrrRrrr*z.expected-attribute-value-but-got-right-bracketr+rsrtr-rUr r;)rXrS`z"equals-in-unquoted-attribute-valuez$expected-attribute-value-but-got-eof)rrCr rxattributeValueDoubleQuotedStaterattributeValueUnQuotedStaterJattributeValueSingleQuotedStater.rDrrrr!rrrr(r(r)rsV       z'HTMLTokenizer.beforeAttributeValueStatecC|j}|dkr|j|_d S|dkr|dd S|dkr8|jtddd|jddd d 7<d S|t urM|jtdd d|j |_d S|jddd ||j d 7<d S)NrrRrsr*rtr+r-rUr r;z#eof-in-attribute-value-double-quote)rrRrsT rrCafterAttributeValueStaterrhr.rDrr!rrrxrr(r(r)r.      z-HTMLTokenizer.attributeValueDoubleQuotedStatecCr)NrrRrsr*rtr+r-rUr r;z#eof-in-attribute-value-single-quote)rrRrsTrrr(r(r)rrz-HTMLTokenizer.attributeValueSingleQuotedStatecCs|j}|tvr|j|_dS|dkr|ddS|dkr$|dS|dvrB|jt ddd|j ddd |7<dS|d kr`|jt dd d|j ddd d 7<dS|t uru|jt dd d|j |_dS|j ddd ||j tdtB7<dS)NrRr)rrrXrSrr*z0unexpected-character-in-unquoted-attribute-valuer+r-rUr rsrtr;z eof-in-attribute-value-no-quotes)rRrrrrXrSrrsT)rrCr rrrhrrr.rDrr!rrrxrGrr(r(r)rs@        z)HTMLTokenizer.attributeValueUnQuotedStatecCs|j}|tvr|j|_dS|dkr|dS|dkr#|j|_dS|tur>|j t ddd|j ||j |_dS|j t ddd|j ||j|_dS)Nrrr*z$unexpected-EOF-after-attribute-valuer+z*unexpected-character-after-attribute-valueT) rrCr rrrrrrr.rDrrJrrr(r(r)r.s.       z&HTMLTokenizer.afterAttributeValueStatecCs|j}|dkrd|jd<|dS|tur/|jtddd|j||j |_ dS|jtddd|j||j |_ dS)NrTrlr*z#unexpected-EOF-after-solidus-in-tagr+z)unexpected-character-after-solidus-in-tag) rrCr!rrrr.rDrrJrrrrr(r(r)rBs$       z&HTMLTokenizer.selfClosingStartTagStatecCsD|jd}|dd}|jtd|d|j|j|_dS)Nrrsr;Commentr+T) rrxreplacer.rDrrCrrrr(r(r)rTs    zHTMLTokenizer.bogusCommentStatecCsn|jg}|ddkr)||j|ddkr(tddd|_|j|_dSnq|ddvr\d}dD]}||j|d|vrGd }nq3|r[td ddddd |_|j|_dSn>|dd kr|jdur|jj j r|jj j dj |jj j krd}d D]}||j|d|krd }nq}|r|j |_dS|jtddd|r|j||s|j|_dS)NrUrrr6r+T)dD))oOrOCtTyYpPeEFDoctype)r,ripublicIdsystemIdcorrect[)rrArrrr*zexpected-dashes-or-doctype)rrCrDrr!commentStartStater doctypeStatertree openElements namespacedefaultNamespacecdataSectionStater.rJr0r)r$rNmatchedexpectedr(r(r)rcs`         z(HTMLTokenizer.markupDeclarationOpenStatecCs|j}|dkr|j|_d S|dkr)|jtddd|jdd7<d S|dkrE|jtdd d|j|j|j|_d S|t ura|jtdd d|j|j|j|_d S|jd|7<|j |_d S) Nrrsr*rtr+r-r;rincorrect-commenteof-in-commentT) rrCcommentStartDashStaterr.rDrr!rr commentStaterr(r(r)rs6     zHTMLTokenizer.commentStartStatecCs|j}|dkr|j|_d S|dkr)|jtddd|jdd7<d S|dkrE|jtdd d|j|j|j|_d S|t ura|jtdd d|j|j|j|_d S|jdd|7<|j |_d S) Nrrsr*rtr+r--�rrrT) rrCcommentEndStaterr.rDrr!rrrrr(r(r)rs6     z#HTMLTokenizer.commentStartDashStatecCs|j}|dkr|j|_d S|dkr)|jtddd|jdd7<d S|turE|jtddd|j|j|j |_d S|jd||j d 7<d S) Nrrsr*rtr+r-r;r)rrsT) rrCcommentEndDashStaterr.rDrr!rrrxrr(r(r)rs*      zHTMLTokenizer.commentStatecCs|j}|dkr|j|_d S|dkr-|jtddd|jdd7<|j|_d S|t urI|jtddd|j|j|j |_d S|jdd|7<|j|_d S) Nrrsr*rtr+r-rzeof-in-comment-end-dashT) rrCrrr.rDrr!rrrrr(r(r)rs*    z!HTMLTokenizer.commentEndDashStatecCs6|j}|dkr|j|j|j|_dS|dkr4|jtddd|jdd7<|j|_dS|dkrI|jtdd d|j |_dS|d krc|jtdd d|jd|7<dS|t ur|jtdd d|j|j|j|_dS|jtdd d|jdd|7<|j|_dS)Nrrsr*rtr+r-u--�rz,unexpected-bang-after-double-dash-in-commentrz,unexpected-dash-after-double-dash-in-commentzeof-in-comment-double-dashzunexpected-char-in-commentz--T) rrCr.rDr!rrrrcommentEndBangStaterrr(r(r)rsJ       zHTMLTokenizer.commentEndStatecCs|j}|dkr|j|j|j|_d S|dkr)|jdd7<|j|_d S|dkrG|jtddd|jdd 7<|j |_d S|t urc|jtdd d|j|j|j|_d S|jdd|7<|j |_d S) Nrrr-z--!rsr*rtr+u--!�zeof-in-comment-end-bang-stateT) rrCr.rDr!rrrrrrrr(r(r)rs4    z!HTMLTokenizer.commentEndBangStatecCs|j}|tvr|j|_dS|tur0|jtdddd|j d<|j|j |j |_dS|jtddd|j ||j|_dS)Nr*!expected-doctype-name-but-got-eofr+Frzneed-space-after-doctypeT) rrCr beforeDoctypeNameStaterrr.rDrr!rrJrr(r(r)rs$      zHTMLTokenizer.doctypeStatecCs|j}|tvr d S|dkr-|jtdddd|jd<|j|j|j|_d S|dkrG|jtdddd |jd <|j |_d S|t urh|jtdd dd|jd<|j|j|j|_d S||jd <|j |_d S) Nrr*z+expected-doctype-name-but-got-right-bracketr+Frrsrtr;rirT) rrCr r.rDrr!rrdoctypeNameStaterrr(r(r)r*s<         z$HTMLTokenizer.beforeDoctypeNameStatecCs|j}|tvr|jdt|jd<|j|_d S|dkr6|jdt|jd<|j |j|j |_d S|dkrT|j t ddd|jdd7<|j |_d S|t ur|j t dddd |jd <|jdt|jd<|j |j|j |_d S|jd|7<d S) Nrirrsr*rtr+r;zeof-in-doctype-nameFrT)rrCr r!rmr afterDoctypeNameStaterr.rDrrrrrr(r(r)rDs8     zHTMLTokenizer.doctypeNameStatecCsH|j}|tvr dS|dkr|j|j|j|_dS|turDd|jd<|j ||jt ddd|j|j|j|_dS|dvred}d D]}|j}||vr[d}nqL|rd|j |_dSn |d vrd}d D]}|j}||vr|d}nqm|r|j |_dS|j ||jt dd d |idd|jd<|j |_dS)NrFrr*eof-in-doctyper+rT))uU)bB)lL)iIrsS)rr rr)mMz*expected-space-or-right-bracket-in-doctyper-r8)rrCr r.rDr!rrrrJrafterDoctypePublicKeywordStateafterDoctypeSystemKeywordStatebogusDoctypeState)r$r-rrr(r(r)r]s` .+   $     z#HTMLTokenizer.afterDoctypeNameStatecC|j}|tvr|j|_dS|dvr*|jtddd|j||j|_dS|t urK|jtdddd|j d<|j|j |j |_dS|j||j|_dS N)rrr*unexpected-char-in-doctyper+rFrT) rrCr "beforeDoctypePublicIdentifierStaterr.rDrrJrr!rrr(r(r)r,       z,HTMLTokenizer.afterDoctypePublicKeywordStatecCs|j}|tvr d S|dkrd|jd<|j|_d S|dkr*d|jd<|j|_d S|dkrK|jt dddd |jd <|j|j|j |_d S|t url|jt dd dd |jd <|j|j|j |_d S|jt dd dd |jd <|j |_d S)Nrr6rrrr*unexpected-end-of-doctyper+FrrrT) rrCr r!(doctypePublicIdentifierDoubleQuotedStater(doctypePublicIdentifierSingleQuotedStater.rDrrrrrr(r(r)rD          z0HTMLTokenizer.beforeDoctypePublicIdentifierStatecC|j}|dkr|j|_d S|dkr)|jtddd|jdd7<d S|dkrJ|jtdd dd |jd <|j|j|j|_d S|t urk|jtdd dd |jd <|j|j|j|_d S|jd|7<d S)Nrrsr*rtr+rr;rrFrrT rrC!afterDoctypePublicIdentifierStaterr.rDrr!rrrr(r(r)r8       z6HTMLTokenizer.doctypePublicIdentifierDoubleQuotedStatecCr)Nrrsr*rtr+rr;rrFrrTrrr(r(r)rr z6HTMLTokenizer.doctypePublicIdentifierSingleQuotedStatecCs|j}|tvr|j|_d S|dkr |j|j|j|_d S|dkr:|jt dddd|jd<|j |_d S|dkrT|jt dddd|jd<|j |_d S|t uru|jt dd dd |jd <|j|j|j|_d S|jt dddd |jd <|j |_d S) Nrrr*rr+r6rrrFrT)rrCr -betweenDoctypePublicAndSystemIdentifiersStaterr.rDr!rr(doctypeSystemIdentifierDoubleQuotedState(doctypeSystemIdentifierSingleQuotedStaterrrr(r(r)rsH          z/HTMLTokenizer.afterDoctypePublicIdentifierStatecCs|j}|tvr d S|dkr|j|j|j|_d S|dkr,d|jd<|j|_d S|dkr;d|jd<|j |_d S|t kr\|jt dddd |jd <|j|j|j|_d S|jt dd dd |jd <|j |_d S) Nrrr6rrr*rr+FrrT) rrCr r.rDr!rrr"r#rrrrr(r(r)r!s<        z;HTMLTokenizer.betweenDoctypePublicAndSystemIdentifiersStatecCrr) rrCr "beforeDoctypeSystemIdentifierStaterr.rDrrJrr!rrr(r(r)r)rz,HTMLTokenizer.afterDoctypeSystemKeywordStatecCs|j}|tvr d S|dkrd|jd<|j|_d S|dkr*d|jd<|j|_d S|dkrK|jt dddd |jd <|j|j|j |_d S|t url|jt dd dd |jd <|j|j|j |_d S|jt dddd |jd <|j |_d S) Nrr6rrrr*rr+FrrT) rrCr r!r"rr#r.rDrrrrrr(r(r)r$=rz0HTMLTokenizer.beforeDoctypeSystemIdentifierStatecCr)Nrrsr*rtr+rr;rrFrrT rrC!afterDoctypeSystemIdentifierStaterr.rDrr!rrrr(r(r)r"Zr z6HTMLTokenizer.doctypeSystemIdentifierDoubleQuotedStatecCr)Nrrsr*rtr+rr;rrFrrTr%rr(r(r)r#rr z6HTMLTokenizer.doctypeSystemIdentifierSingleQuotedStatecCs|j}|tvr dS|dkr|j|j|j|_dS|tur>|jt dddd|jd<|j|j|j|_dS|jt ddd|j |_dS) Nrr*rr+FrrT) rrCr r.rDr!rrrrrrr(r(r)r&s*     z/HTMLTokenizer.afterDoctypeSystemIdentifierStatecCs`|j}|dkr|j|j|j|_dS|tur-|j||j|j|j|_dS dS)NrT) rrCr.rDr!rrrrJrr(r(r)rs  zHTMLTokenizer.bogusDoctypeStatecCsg} ||jd||jd|j}|tkrn!|dks%J|ddddkr:|ddd|d<n||qd|}|d}|d krft|D] }|jt d d d qR| dd }|rs|jt d|d |j |_ dS)NT]rrUz]]r6rsrr*rtr+r;rZ) rDrrxrCrrFcountranger.rrrr)r$r-rC nullCountrr(r(r)rs6         zHTMLTokenizer.cdataSectionState)Nr)N__name__ __module__ __qualname____doc__r#r3rQrgrhrrrrvrr~rrrrwrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr!rr$r"r#r&rr __classcell__r(r(r&r)rs  HP#         6 "-3rN) __future__rrrpip._vendor.sixrrH collectionsrrsysr constantsr r r r rrrrrr _inputstreamr_trierr[dictrnobjectrr(r(r(r)s"