o {nhN@sVddlmZddlmZddlmZddlmZmZm Z ddl m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZGdddZGd d d eZGd d d eZGd ddeZ GdddeZ!GdddeZ"GdddeZ#GdddeZ$GdddeZ%GdddeZ&eddd/d"d#Z'ed$d &d0d1d,d-Z(d.S)2) annotations) lru_cache) getLogger)COMMON_SAFE_ASCII_CHARACTERSTRACEUNICODE_SECONDARY_RANGE_KEYWORD)is_accentuated is_arabicis_arabic_isolated_formis_case_variableis_cjk is_emoticon is_hangul is_hiragana is_katakanais_latinis_punctuation is_separator is_symbolis_thaiis_unprintable remove_accent unicode_rangeis_cjk_uncommonc@s<eZdZdZdddZdd d Zdd d ZedddZdS)MessDetectorPluginzy Base abstract class used for mess detection plugins. All detectors MUST extend and implement given methods. characterstrreturnboolcCt)z@ Determine if given character should be fed in. NotImplementedErrorselfrr%@/usr/local/lib/python3.10/dist-packages/charset_normalizer/md.pyeligible'zMessDetectorPlugin.eligibleNonecCr )z The main routine to be executed upon character. Insert the logic in witch the text would be considered chaotic. r!r#r%r%r&feed-szMessDetectorPlugin.feedcCr )zB Permit to reset the plugin to the initial state. r!r$r%r%r&reset4r(zMessDetectorPlugin.resetfloatcCr )z Compute the chaos ratio based on what your feed() has seen. Must NOT be lower than 0.; No restriction gt 0. r!r+r%r%r&ratio:szMessDetectorPlugin.ratioNrrrrrrrr)rr)rr-) __name__ __module__ __qualname____doc__r'r*r,propertyr.r%r%r%r&r!s   rc@BeZdZdddZddd Zdd d Zdd d ZedddZdS) TooManySymbolOrPunctuationPluginrr)cCs"d|_d|_d|_d|_d|_dS)NrF)_punctuation_count _symbol_count_character_count_last_printable_char_frenzy_symbol_in_wordr+r%r%r&__init__Ds  z)TooManySymbolOrPunctuationPlugin.__init__rrrcC|SN isprintabler#r%r%r&r'Lz)TooManySymbolOrPunctuationPlugin.eligiblecCsp|jd7_||jkr3|tvr3t|r|jd7_n|dur3t|r3t|dur3|jd7_||_dS)NrF) r<r=rrr:isdigitrrr;r#r%r%r&r*Os    z%TooManySymbolOrPunctuationPlugin.feedcCsd|_d|_d|_dSNr)r:r<r;r+r%r%r&r,a z&TooManySymbolOrPunctuationPlugin.resetr-cCs0|jdkrdS|j|j|j}|dkr|SdS)Nr333333?)r<r:r;)r$ratio_of_punctuationr%r%r&r.fs  z&TooManySymbolOrPunctuationPlugin.ratioNr1r/r0r2 r3r4r5r?r'r*r,r7r.r%r%r%r&r9Cs    r9c@r8)TooManyAccentuatedPluginrr)cCd|_d|_dSrGr<_accentuated_countr+r%r%r&r?s z!TooManyAccentuatedPlugin.__init__rrrcCr@rA)isalphar#r%r%r&r'wrDz!TooManyAccentuatedPlugin.eligiblecC,|jd7_t|r|jd7_dSdSNr)r<r rPr#r%r%r&r*zzTooManyAccentuatedPlugin.feedcCrNrGrOr+r%r%r&r,rQzTooManyAccentuatedPlugin.resetr-cCs*|jdkrdS|j|j}|dkr|SdS)NrIgffffff?rO)r$ratio_of_accentuationr%r%r&r.s  zTooManyAccentuatedPlugin.ratioNr1r/r0r2rLr%r%r%r&rMrs    rMc@r8)UnprintablePluginrr)cCrNrG)_unprintable_countr<r+r%r%r&r?rQzUnprintablePlugin.__init__rrrcCdSNTr%r#r%r%r&r'zUnprintablePlugin.eligiblecCs(t|r |jd7_|jd7_dSrT)rrYr<r#r%r%r&r*szUnprintablePlugin.feedcCs d|_dSrG)rYr+r%r%r&r,s zUnprintablePlugin.resetr-cC|jdkrdS|jd|jS)NrrIrV)r<rYr+r%r%r&r. zUnprintablePlugin.ratioNr1r/r0r2rLr%r%r%r&rXs    rXc@r8)SuspiciousDuplicateAccentPluginrr)cCd|_d|_d|_dSrG_successive_countr<_last_latin_characterr+r%r%r&r?s z(SuspiciousDuplicateAccentPlugin.__init__rrrcCs|ot|SrA)rRrr#r%r%r&r'sz(SuspiciousDuplicateAccentPlugin.eligiblecCst|jd7_|jdur5t|r5t|jr5|r%|jr%|jd7_t|t|jkr5|jd7_||_dSrT)r<rcr isupperrbrr#r%r%r&r*s  z$SuspiciousDuplicateAccentPlugin.feedcCr`rGrar+r%r%r&r,rHz%SuspiciousDuplicateAccentPlugin.resetr-cCr])NrrIrE)r<rbr+r%r%r&r.r^z%SuspiciousDuplicateAccentPlugin.ratioNr1r/r0r2rLr%r%r%r&r_s    r_c@r8)SuspiciousRangerr)cCr`rG)"_suspicious_successive_range_countr<_last_printable_seenr+r%r%r&r?rHzSuspiciousRange.__init__rrrcCr@rArBr#r%r%r&r'rDzSuspiciousRange.eligiblecCsx|jd7_|st|s|tvrd|_dS|jdur"||_dSt|j}t|}t||r7|jd7_||_dSrT)r<isspacerrrgr is_suspiciously_successive_rangerf)r$runicode_range_aunicode_range_br%r%r&r*s     zSuspiciousRange.feedcCr`rG)r<rfrgr+r%r%r&r,rHzSuspiciousRange.resetr-cCs"|jdkrdS|jd|j}|S)N rIrE)r<rf)r$ratio_of_suspicious_range_usager%r%r&r.s zSuspiciousRange.ratioNr1r/r0r2rLr%r%r%r&res    rec@r8)SuperWeirdWordPluginrr)cCs@d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_ dS)NrF) _word_count_bad_word_count_foreign_long_count_is_current_word_bad_foreign_long_watchr<_bad_character_count_buffer_buffer_accent_count_buffer_glyph_countr+r%r%r&r?s zSuperWeirdWordPlugin.__init__rrrcCrZr[r%r#r%r%r&r' r\zSuperWeirdWordPlugin.eligiblecCs|rc|j|7_t|r|jd7_|jdurFt|dus%t|rFt|durFt|durFt|durFt |durFt |durFd|_t|sZt|sZt|sZt |sZt |ra|j d7_ dS|jshdS| sut |sut|r$|jr$|jd7_t|j}|j|7_|dkr|j|dkrd|_n4t|jdr|jdrtdd|jDdur|jd7_d|_n|j dkrd|_|jd7_|d kr|jrd d t|jtd |D}d}|rt||d krd}|s|jd7_d|_|jr|jd7_|jt|j7_d|_d|_d|_d |_d |_ dS|dvrA|durCt|rEd|_|j|7_dSdSdSdS)NrFT?css|]}|VqdSrArd).0_r%r%r& 8sz,SuperWeirdWordPlugin.feed..cSsg|] \}}|r|qSr%r|)r}cir%r%r& @s z-SuperWeirdWordPlugin.feed..rrJro>r~-<=>|~)rRrvr rwrtrr rrrrrxrhrrrplenr<rsrdallrrziprangerqrurFr)r$r buffer_lengthcamel_case_dstprobable_camel_casedr%r%r&r*s            zSuperWeirdWordPlugin.feedcCs4d|_d|_d|_d|_d|_d|_d|_d|_dS)NroFr)rvrsrtrqrpr<rurrr+r%r%r&r,_s zSuperWeirdWordPlugin.resetr-cCs$|jdkr |jdkr dS|j|jS)N rrI)rprrrur<r+r%r%r&r.is zSuperWeirdWordPlugin.ratioNr1r/r0r2rLr%r%r%r&rns    Q rnc@sFeZdZdZdddZdd d Zdd d Zdd dZedddZ dS)CjkUncommonPluginz< Detect messy CJK text that probably means nothing. rr)cCrNrGr<_uncommon_countr+r%r%r&r?vrQzCjkUncommonPlugin.__init__rrrcCt|SrA)r r#r%r%r&r'zrDzCjkUncommonPlugin.eligiblecCrSrT)r<rrr#r%r%r&r*}s zCjkUncommonPlugin.feedcCrNrGrr+r%r%r&r,rQzCjkUncommonPlugin.resetr-cCs.|jdkrdS|j|j}|dkr|dSdS)NrVrIrzrr)r$uncommon_form_usager%r%r&r.s  zCjkUncommonPlugin.ratioNr1r/r0r2) r3r4r5r6r?r'r*r,r7r.r%r%r%r&rqs    rc@r8)ArchaicUpperLowerPluginrr)cCs.d|_d|_d|_d|_d|_d|_d|_dS)NFrT)_buf_character_count_since_last_sep_successive_upper_lower_count#_successive_upper_lower_count_finalr<_last_alpha_seen_current_ascii_onlyr+r%r%r&r?s z ArchaicUpperLowerPlugin.__init__rrrcCrZr[r%r#r%r%r&r'r\z ArchaicUpperLowerPlugin.eligiblecCs$|ot|}|du}|rC|jdkrC|jdkr+|dur+|jdur+|j|j7_d|_d|_d|_d|_|j d7_ d|_dS|jdurQ| durQd|_|jdur| r_|j sh| r||j r||jdurx|jd7_d|_nd|_nd|_|j d7_ |jd7_||_dS)NFr@rTrE) rRr rrFrrrrrr<isasciirdislower)r$r is_concerned chunk_sepr%r%r&r*s@      zArchaicUpperLowerPlugin.feedcCs.d|_d|_d|_d|_d|_d|_d|_dS)NrFT)r<rrrrrrr+r%r%r&r,s zArchaicUpperLowerPlugin.resetr-cCs|jdkrdS|j|jS)NrrI)r<rr+r%r%r&r.s  zArchaicUpperLowerPlugin.ratioNr1r/r0r2rLr%r%r%r&rs   * rc@sBeZdZdddZdddZdd d Zdd d ZedddZdS)ArabicIsolatedFormPluginrr)cCrNrGr<_isolated_form_countr+r%r%r&r?rQz!ArabicIsolatedFormPlugin.__init__cCrNrGrr+r%r%r&r,rQzArabicIsolatedFormPlugin.resetrrrcCrrA)r r#r%r%r&r'rDz!ArabicIsolatedFormPlugin.eligiblecCrSrT)r<r rr#r%r%r&r*rUzArabicIsolatedFormPlugin.feedr-cCs|jdkrdS|j|j}|S)NrVrIr)r$isolated_form_usager%r%r&r.s  zArabicIsolatedFormPlugin.ratioNr1r/r0r2) r3r4r5r?r,r'r*r7r.r%r%r%r&rs    r)maxsizerj str | NonerkrrcCsv|dus|dur dS||krdSd|vrd|vrdSd|vs"d|vr$dSd|vs,d|vr6d|vs4d|vr6dS|d|d}}|D]}|tvrJqC||vrQdSqC|dv|dv}}|s_|rid |vsgd |vridS|ro|rodSd |vswd |vrd |vsd |vrdS|d ks|d krdSd |vsd |vs|d vr|d vrd |vsd |vrdSd|vsd|vrdS|d ks|d krdSdS)za Determine if two Unicode range seen next to each other can be considered as suspicious. NTFLatin Emoticons Combining )HiraganaKatakanaCJKHangulz Basic Latin)rr PunctuationForms)splitr)rjrkkeywords_range_akeywords_range_belrange_a_jp_charsrange_b_jp_charsr%r%r&risZrii皙?Fdecoded_sequencermaximum_thresholdr-debugc CsRddtD}t|d}d}|dkrd}n |dkrd}nd }t|d t|D]2\}}|D] } | |r<| |q0|d krG||d ksM||dkr\td d |D}||kr\nq*|rtd} | t d|d|d|t|dkr| t d|dd| t d|dd|D]} | t | j d| j qt |dS)zw Compute a mess ratio given a decoded bytes sequence. The maximum threshold does stop the computation earlier. cSsg|]}|qSr%r%)r}md_classr%r%r&rNszmess_ratio..rrIi rr rcss|]}|jVqdSrA)r.)r}dtr%r%r&reszmess_ratio..charset_normalizerzIMess-detector extended-analysis start. intermediary_mean_mess_ratio_calc=z mean_mess_ratio=z maximum_threshold=zStarting with: Nz Ending with: iz: )r__subclasses__rrrr'r*sumrlogr __class__r.round) rrr detectorslengthmean_mess_ratio!intermediary_mean_mess_ratio_calcrindexdetectorloggerrr%r%r& mess_ratioFsN      rN)rjrrkrrr)rF)rrrr-rrrr-)) __future__r functoolsrloggingrconstantrrrutilsr r r r r rrrrrrrrrrrrrrr9rMrXr_rernrrrrirr%r%r%r&s(  P"/%1v#L I