o bcsk@sdZdZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z m Z mZddlmZddlmZmZdd lmZd!d d ZGd ddeZddZddZddZddZddZGdddeZGdddeZ GdddeZ!Gdd d e Z"dS)"zRefactoring framework. Used as a main program, this can refactor any number of files and/or recursively descend down directories. Imported as a module, this provides infrastructure to write your own refactoring tool. z#Guido van Rossum N)chain)drivertokenizetoken) find_root)pytreepygram) btm_matcherTcCsTt|ggdg}g}t|jD]\}}}|dr'|r"|dd}||q|S)zEReturn a sorted list of all available fix names in the given package.*fix_N) __import__pkgutil iter_modules__path__ startswithappend) fixer_pkg remove_prefixpkg fix_namesfindernameispkgr'/usr/lib/python3.10/lib2to3/refactor.pyget_all_fix_namess   rc@ eZdZdS) _EveryNodeN__name__ __module__ __qualname__rrrrr+rcCst|tjtjfr|jdurt|jhSt|tjr$|jr"t|jStt|tj rAt }|jD]}|D] }| t|q4q0|St d|)zf Accepts a pytree Pattern Node and returns a set of the pattern types which will match first. Nz$Oh no! I don't understand pattern %s) isinstancer NodePattern LeafPatterntyperNegatedPatterncontent_get_head_typesWildcardPatternsetupdate Exception)patrpxrrrr+/s       r+c Cstt}g}|D];}|jr1zt|j}Wnty#||Yq w|D] }|||q&q |jdur?||j|q ||q tt j j t j j D] }|||qPt|S)z^ Accepts a list of fixers and returns a dictionary of head node type --> fixer list. N) collections defaultdictlistpatternr+rr _accept_typerr python_grammar symbol2numbervaluestokensextenddict) fixer_list head_nodeseveryfixerheads node_typerrr_get_headnode_dictKs(     rEcsfddtdDS)zN Return the fully qualified names for fixers in the package pkg_name. csg|]}d|qS.r).0fix_namepkg_namerr hsz+get_fixers_from_package..F)rrJrrJrget_fixers_from_packageds rMcCs|SNr)objrrr _identityksrPcsjd}tt|jfdd}ttjtjtj h}t }z |\}}||vr*q|tj kr5|r2ncd}n_|tj kr|dkr|\}}|tj ksL|dkrMnR|\}}|tj ks[|dkr\nH|\}}|tj kro|dkro|\}}|tj kr|||\}}|tj ks|d krn |\}}|tj kstnnq Wt|SWt|SWt|SWt|StyYt|Sw) NFcst}|d|dfS)Nrr)next)tokgenrradvancersz(_detect_future_features..advanceTfrom __future__import(,)rgenerate_tokensioStringIOreadline frozensetrNEWLINENLCOMMENTr-STRINGNAMEOPadd StopIteration)sourcehave_docstringrUignorefeaturestpvaluerrSr_detect_future_featuresos\             rnc@seZdZdZdS) FixerErrorzA fixer could not be loaded.N)r!r"r#__doc__rrrrrosroc@seZdZddddZdZdZd4ddZdd Zd d Zd d Z ddZ ddZ d5ddZ d5ddZ ddZd5ddZddZd6ddZddZd d!Z  d7d"d#Zd8d$d%Zd&Zd'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1Zd2d3ZdS)9RefactoringToolF)print_function exec_functionwrite_unchanged_filesFixr NcCsH||_|pg|_|j|_|dur|j|tj|_|jdr)|jj d=n |jdr3|jj d=|j d|_ g|_ t d|_g|_d|_tj|jtj|jd |_|\|_|_g|_t|_g|_g|_t|j|jD]#}|j r~|j!|qr||jvr|j"|qr||jvr|j"|qrt#|j|_$t#|j|_%dS) zInitializer. Args: fixer_names: a list of fixers to import options: a dict with configuration. explicit: a list of fixers to run even if they are explicit. NrrprintrsexecrtrqF)convertlogger)&fixersexplicit_default_optionscopyoptionsr.r r9grammarkeywordsgetrterrorslogging getLoggerry fixer_logwroterDriverrrx get_fixers pre_order post_orderfilesbm BottomMatcherBM bmi_pre_orderbmi_post_orderr BM_compatible add_fixerrrEbmi_pre_order_headsbmi_post_order_heads)self fixer_namesr~r{rBrrr__init__sD              zRefactoringTool.__init__c CsJg}g}|jD]}t|iidg}|ddd}||jr(|t|jd}|d}|jddd |D}zt ||}Wnt yQt d ||fdw||j |j } | jrm|jd urm||jvrm|d |q|d || jdkr~|| q| jdkr|| qt d| jtd} |j| d|j| d||fS)aInspects the options to load the requested patterns and handlers. Returns: (pre_order, post_order), where pre_order is the list of fixers that want a pre-order AST traversal, and post_order is the list that want post-order traversal. r rGrN_cSsg|]}|qSr)title)rHr2rrrrLsz.RefactoringTool.get_fixers..zCan't find %s.%sTzSkipping optional fixer: %szAdding transformation: %sprepostzIllegal fixer order: %r run_orderkey)rzrrsplitr FILE_PREFIXlensplit CLASS_PREFIXjoingetattrAttributeErrorror~rr{ log_message log_debugorderroperator attrgettersort) rpre_order_fixerspost_order_fixers fix_mod_pathmodrIparts class_name fix_classrBkey_funcrrrrs:              zRefactoringTool.get_fixerscOs)zCalled when an error occurs.r)rmsgargskwdsrrr log_errorszRefactoringTool.log_errorcGs|r||}|j|dS)zHook to log a message.N)ryinforrrrrrrszRefactoringTool.log_messagecGs|r||}|j|dSrN)rydebugrrrrr szRefactoringTool.log_debugcCsdS)zTCalled with the old version, new version, and filename of a refactored file.Nr)rold_textnew_textfilenameequalrrr print_outputszRefactoringTool.print_outputcCs8|D]}tj|r||||q||||qdS)z)Refactor a list of files and directories.N)ospathisdir refactor_dir refactor_file)ritemswrite doctests_only dir_or_filerrrrefactors  zRefactoringTool.refactorc Cstjd}t|D]@\}}}|d||||D]}|ds>tj|d|kr>tj||} | | ||qdd|D|dd<q dS)zDescends down a directory and refactor every Python file found. Python files are assumed to have a .py extension. Files and subdirectories starting with '.' are skipped. pyzDescending into %srGrcSsg|] }|ds|qSrF)r)rHdnrrrrL2z0RefactoringTool.refactor_dir..N) rextsepwalkrrrrsplitextrr) rdir_namerrpy_extdirpathdirnames filenamesrfullnamerrrr s   zRefactoringTool.refactor_dirc Cszt|d}Wnty!}z|d||WYd}~dSd}~wwzt|jd}W|n|wtj|d|dd}||fWdS1sPwYdS) zG Do our best to decode a Python source file correctly. rbzCan't open %s: %sNNNrr1rencodingnewline) openOSErrorrrdetect_encodingr^closer\read)rrferrrrrr_read_python_source4s $z#RefactoringTool._read_python_sourcecCs||\}}|dur dS|d7}|r9|d||||}|js&||kr1||||||dS|d|dS|||}|jsG|rX|jrX|jt|dd|||ddS|d|dS)zRefactors a file.N zRefactoring doctests in %szNo doctest changes in %sr)rrzNo changes in %s)rrrefactor_docstringrtprocessed_filerefactor_string was_changedstr)rrrrinputroutputtreerrrrDs     zRefactoringTool.refactor_filec Cst|}d|vr tj|j_z3z|j|}Wn#ty9}z|d||jj |WYd}~W|j|j_dSd}~wwW|j|j_n|j|j_w||_ | d|| |||S)aFRefactor a given input string. Args: data: a string holding the code to be refactored. name: a human-readable name for use in error/log messages. Returns: An AST corresponding to the refactored input stream; None if there were errors during the parse. rrzCan't parse %s: %s: %sNzRefactoring %s) rnr !python_grammar_no_print_statementrr parse_stringr/r __class__r!future_featuresr refactor_tree)rdatarrkrrrrrr[s&      zRefactoringTool.refactor_stringcCstj}|r)|d||d}|js||kr"||d|dS|ddS||d}|js7|rB|jrB|t |d|dS|ddS)NzRefactoring doctests in stdinzzNo doctest changes in stdinzNo changes in stdin) sysstdinrrrrtrrrr)rrrrrrrrrefactor_stdinvs    zRefactoringTool.refactor_stdinc Cst|j|jD]}|||q||j|||j||j| }t | r|jj D]}||vr||r||j tjjdd|jrV||j tjjdt||D]o}|||vrk|||zt|Wn tyzYq\w|jr||jvrq\||}|r|||}|dur|||D]}|jsg|_|j|q|j| }|D]} | |vrg|| <|| || qq\q4t | s0t|j|jD]}|||q|jS)aRefactors a parse tree (modifying the tree in place). For compatible patterns the bottom matcher module is used. Otherwise the tree is traversed node-to-node for matches. Args: tree: a pytree.Node instance representing the root of the tree to be refactored. name: a human-readable name for this tree. Returns: True if the tree was modified, False otherwise. T)rreverserN)rrr start_tree traverse_byrrrrunleavesanyr;rzrrBasedepthkeep_line_order get_linenor6remover ValueErrorfixers_appliedmatch transformreplacerr= finish_treer) rrrrB match_setnoderesultsnew new_matchesfxrrrrrsP          1zRefactoringTool.refactor_treecCsV|sdS|D]"}||jD]}||}|r'|||}|dur'|||}q qdS)aTraverse an AST, applying a set of fixers to each node. This is a helper method for refactor_tree(). Args: fixers: a list of fixer instances. traversal: a generator that yields AST nodes. Returns: None N)r(rrr)rrz traversalr rBr r rrrrs    zRefactoringTool.traverse_bycCs|j||dur||d}|durdS||k}||||||r0|d||js0dS|r<|||||dS|d|dS)zR Called when a file has been refactored and there may be changes. NrzNo changes to %szNot writing changes to %s)rrrrrrt write_file)rrrrrrrrrrrs  zRefactoringTool.processed_filec Csz tj|d|dd}Wnty%}z|d||WYd}~dSd}~ww|)z||WntyI}z |d||WYd}~nd}~wwWdn1sTwY|d|d|_dS) zWrites a string to a file. It first shows a unified diff between the old text and the new text, and then rewrites the file; the latter is only done if the write option is set. wrrzCan't create %s: %sNzCan't write %s: %szWrote changes to %sT)r\rrrrrr)rrrrrfprrrrrs$  zRefactoringTool.write_filez>>> z... c Csg}d}d}d}d}|jddD]d}|d7}||jr?|dur-|||||||}|g}||j} |d| }q|dur\|||jsV|||jdkr\| |q|durk||||||d}d}| |q|dur||||||d |S)aRefactors a docstring, looking for doctests. This returns a modified version of the input string. It looks for doctests, which start with a ">>>" prompt, and may be continued with "..." prompts, as long as the "..." is indented the same as the ">>>". (Unfortunately we can't use the doctest module's parser, since, like most parsers, it is not geared towards preserving the original source.) NrTkeependsrrr) splitlineslstriprPS1r=refactor_doctestfindPS2rstriprr) rrrresultblock block_linenoindentlinenolineirrrrsB        z"RefactoringTool.refactor_docstringc s(z ||}Wn4ty=}z(jtjr&|D] }d|dqd|||j j ||WYd}~Sd}~ww ||rt |j dd}|d|d||dd} }| dg|dksjJ| |ddsy|dd7<j|d g}|r|fd d |D7}|S) zRefactors one doctest. A doctest is given as a block of lines, the first of which starts with ">>>" (possibly indented), while the remaining lines start with "..." (identically indented). z Source: %srz+Can't parse docstring in %s line %s: %s: %sNTrrrrcsg|] }j|qSr)r)rHr!rrrrrL^rz4RefactoringTool.refactor_doctest..) parse_blockr/ry isEnabledForrDEBUGrrrrr!rrrendswithrpop) rrr rrrrr!r clippedrr#rrDs,  "z RefactoringTool.refactor_doctestcCs|jrd}nd}|js|d|n|d||jD]}||q|jr6|d|jD]}||q.|jrdt|jdkrF|dn |dt|j|jD]\}}}|j|g|Ri|qRdSdS) Nwerez need to bezNo files %s modified.zFiles that %s modified:z$Warnings/messages while refactoring:rzThere was 1 error:zThere were %d errors:)rrrrrr)rr*filemessagerrrrrr summarizeas(       zRefactoringTool.summarizecCs"|j||||}t|_|S)zParses a block into a tree. This is necessary to get correct line number / offset information in the parser diagnostics and embedded into the parse tree. )r parse_tokens wrap_toksr_r)rrr rrrrrr$xszRefactoringTool.parse_blockc csft|||j}|D]#\}}\}}\} } } ||d7}| |d7} ||||f| | f| fVq dS)z;Wraps a tokenize stream to systematically modify start/end.rN)rr[ gen_lines__next__) rrr rr<r(rmline0col0line1col1 line_textrrrr/s  zRefactoringTool.wrap_toksccsx||j}||j}|}|D]'}||r |t|dVn||dkr,dVntd||f|}q dVq8)zGenerates lines as expected by tokenize from a list of lines. This strips the first len(indent + self.PS1) characters off each line. Nrzline=%r, prefix=%rTr)rrrrrAssertionError)rrrprefix1prefix2prefixr!rrrr0s   zRefactoringTool.gen_linesr)FF)F)NFNrN)r!r"r#r|rrrrrrrrrrrrrrrrrrrrrrr-r$r/r0rrrrrqsB 7(    O + rqc@r)MultiprocessingUnsupportedNr rrrrr;r$r;csFeZdZfddZ  d fdd ZfddZfd d ZZS) MultiprocessRefactoringToolcs&tt|j|i|d|_d|_dSrN)superr<rqueue output_lockrrkwargsrrrrs z$MultiprocessRefactoringTool.__init__Frc s2|dkrtt|||SzddlWn tytwjdur'td_ _ fddt |D}z8|D]}| q@tt|||Wj t |D]}jdq[|D] }|rp| qfd_dSj t |D]}jdq|D] }|r| qd_w)Nrrz already doing multiple processescsg|] }jjdqS))target)Process_child)rHr"multiprocessingrrrrLsz8MultiprocessRefactoringTool.refactor..)r=r<rrG ImportErrorr;r> RuntimeError JoinableQueueLockr?rangestartrputis_alive)rrrr num_processes processesr2r"rBrFrrsL            z$MultiprocessRefactoringTool.refactorcsf|j}|dur1|\}}ztt|j|i|W|jn|jw|j}|dus dSdSrN)r>rr=r<r task_done)rtaskrrArBrrrEs   z"MultiprocessRefactoringTool._childcs4|jdur|j||fdStt|j|i|SrN)r>rNr=r<rr@rBrrrs  z)MultiprocessRefactoringTool.refactor_file)FFr)r!r"r#rrrEr __classcell__rrrBrr<s   r<)T)#rp __author__r\rrrrrr4 itertoolsrpgen2rrr fixer_utilrrrr r rrr/rr+rErMrPrnroobjectrqr;r<rrrrs:     (