S ?c@sRddlZddlmZmZmZdfdYZeedejdS(iN(tBOLtEOLtEOFtScannercBseZdZdZdZdZdZdZdZ dZ dZ dZ dZ dZdZdZdZdZdZdZddZdZdZdZdZd Zd Zd Zdd Zd ZRS(s A Scanner is used to read tokens from a stream of characters using the token set specified by a Plex.Lexicon. Constructor: Scanner(lexicon, stream, name = '') See the docstring of the __init__ method for details. Methods: See the docstrings of the individual methods for more information. read() --> (value, text) Reads the next lexical token from the stream. position() --> (name, line, col) Returns the position of the last token read using the read() method. begin(state_name) Causes scanner to change state. produce(value [, text]) Causes return of a token value to the caller of the Scanner. tiicCsk||_||_||_g|_d|_|jdd|_d|_d|_ t |_ d|_ dS(sc Scanner(lexicon, stream, name = '') |lexicon| is a Plex.Lexicon instance specifying the lexical tokens to be recognised. |stream| can be a file object or anything which implements a compatible read() method. |name| is optional, and may be the name of the file being scanned or any other identifying string. RiiN( tlexicontstreamtnametqueuetNonet initial_statetbegintnext_postcur_postcur_line_startRtcur_chart input_state(tselfRRR((s7/usr/lib/python2.7/site-packages/Pyrex/Plex/Scanners.pyt__init__?s          cCs|j}xv|s|j\|_}|dkrM|jd|jq |j||j}|dk r |j|q q W|d}|d=|S(s Read the next lexical token from the stream and return a tuple (value, text), where |value| is the value associated with the token as specified by the Lexicon, and |text| is the actual string read from the stream. Returns (None, '') on end of file. iN(Rt scan_a_tokenttextR tproduceteoftperform(RRtactiontvaluetresult((s7/usr/lib/python2.7/site-packages/Pyrex/Plex/Scanners.pytreadXs       cCs|j|_|j|_|j|j|_|j}|r|jrfdG|Gd|j|jfGHn|j}|j |j||j|!}||fS|j|jkr|j t kr|j n|j s|j t krdSntj||jdS(s Read the next input sequence recognised by the machine and return (text, action). Returns ('', None) on end of file. sScanner: read: Performings%d:%dRN(RN(R t start_postcur_linet start_lineRt start_coltrun_machine_inlinedttracet buf_start_postbufferRRt next_charRR tErrorstUnrecognizedInputt state_name(RRtbaseR((s7/usr/lib/python2.7/site-packages/Pyrex/Plex/Scanners.pyRms"        cCs2|j|_d|_x|jr'qW|jS(sA Run the machine until no more transitions are possible. N(R tstateR t backup_statet transitiontback_up(R((s7/usr/lib/python2.7/site-packages/Pyrex/Plex/Scanners.pyt run_machines   cCs|j}|j}|j}|j}|j}|j}|j}|j}|j} t |} d} |j } xZ| rd|d||t |fGn|d} | r| ||||||f} n|}|j |d}|dkr|o|j d}n|r| rd|dGHn|}|dkr|}|| }|| kra||}|d}n|j| }|jjd}|j||}||_| |} | |_t |} ||}|r||}|d}nd }|d krt}d }q|st}d }q|}q|d kr:d }d }q|d kri|d}|}}t}d}q|d krt}d}qd }qo| rdGHn| r| \} }}}}}}nd} Pqo||_||_||_||_||_||_| r| rdG| GHqn| S(s3 Inlined version of run_machine for speed. sState %d, %d/%d:%s -->tnumberRitelsesState %diiRs iiiitblockedtDoingN(R R RRRRR R#R"tlenR R!treprtgetRRRRRR(RR)R RRRRR R#R"tbuf_lenR*R!Rtct new_statet buf_indextdiscardtdata((s7/usr/lib/python2.7/site-packages/Pyrex/Plex/Scanners.pyR s                                             cCsW|j}|jr.dGd Gd||jfGn|dkr|j|_|j}|dkrst|_d|_q)|st|_d|_q)||_n|dkrd|_d |_nn|d kr|jd|_|j|_|_t |_d|_n*|dkr t |_d |_n d |_|jrSd ||jt |jfGHndS(NsScanner: next:t is[%d] %dis iiiiRs--> [%d] %d %ss ( RR!R R t read_charRRRRRRR3(RRR6((s7/usr/lib/python2.7/site-packages/Pyrex/Plex/Scanners.pyR$ s6                      cCs|j|j|jfS(sQ Return a tuple (name, line, col) representing the location of the last token read using the read() method. |name| is the name that was provided to the Scanner constructor; |line| is the line number in the stream (1-based); |col| is the position within the line of the first character of the token (0-based). (RRR(R((s7/usr/lib/python2.7/site-packages/Pyrex/Plex/Scanners.pytpositionRs cCs"|jj||_||_dS(s8Set the current state of the scanner to the named state.N(Rtget_initial_stateR R'(RR'((s7/usr/lib/python2.7/site-packages/Pyrex/Plex/Scanners.pyR ]scCs2|dkr|j}n|jj||fdS(s Called from an action procedure, causes |value| to be returned as the token value from read(). If |text| is supplied, it is returned in place of the scanned text. produce() can be called more than once during a single call to an action procedure, in which case the tokens are queued up and returned one at a time by subsequent calls to read(), until the queue is empty, whereupon scanning resumes. N(R RRtappend(RRR((s7/usr/lib/python2.7/site-packages/Pyrex/Plex/Scanners.pyRcs  cCsdS(sS Override this method if you want something to be done at end of file. N((R((s7/usr/lib/python2.7/site-packages/Pyrex/Plex/Scanners.pyRrsN(t__name__t __module__t__doc__R RRRR#R"R R RRRRRRR R'RR!RRRR-R R$R=R RR(((s7/usr/lib/python2.7/site-packages/Pyrex/Plex/Scanners.pyR s8    2  tyield(R%tRegexpsRRRRtsetattrR(((s7/usr/lib/python2.7/site-packages/Pyrex/Plex/Scanners.pyt s m