XRc@siddlZddlZddlZddlZddlZddljZddljZddl j Z ddl m Z dZ dZdZdZdZdZd Zd Zd d Zd efdYZdefdYZdefdYZdefdYZdefdYZdefdYZdefdYZdefdYZdS(iN(tPKG_FILE_BUFSIZs fast_add.v1sfast_remove.v1s manf_list.v1tfull_fmri_listsmain_dict.ascii.v2stoken_byte_offset.v1sfull_fmri_list.hashsfmri_offsets.v1ic Cs6d}d}tj}x|dkr|tkr|dkrgtj||krgtj|nx|D]}ytjj||j}t |d}|dkrt }n1|rx|D]} | j qWd}d}Pn|j |||j } t| jddjd} |dkr:| }n7|| ksqx|D]}|j qMWd}d}PnWqntk r} | jtjkr|t krx|D]}|j qWd}d}Pnt}qx|D]}|j qWqnXqnWqW|r|dkstdS|dk s.t|SdS(sMOpens all data holders in data_list and ensures that the versions are consistent among all of them. It retries several times in case a race condition between file migration and open is encountered. Note: Do not set timeout to be 0. It will cause an exception to be immediately raised. trbt is N(tNonettimetTruet search_errorstInconsistentIndexExceptiontostpathtjoint get_file_nametopentFalsetclose_file_handletset_file_handletreadlinetinttsplittrstriptIOErrorterrnotENOENTtAssertionError( t data_listt directoryttimeouttmissingt cur_versiont start_timetdtftfhtdlt version_tmpt version_numte((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pytconsistent_open/s^  "      "        tIndexStoreBasecBsheZdZdZdZdZdZdZdZdZ dZ d Z d Z RS( sBase class for all data storage used by the indexer and queryEngine. All members must have a file name and maintain an internal file handle to that file as instructed by external calls. cCsC||_d|_d|_d|_d|_d|_t|_dS(N( t_nameRt _file_handlet _file_patht_sizet_mtimet_inodeRt _have_read(tselft file_name((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyt__init__s      cCs|jS(N(R((R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR scCsz|jrtd|nZ||_||_|jdkrvtj|j}|j|_|j|_ |j |_ ndS(Ns8setting an extant file handle, must close first, fp is: ( R)t RuntimeErrorR*R,RR tstattst_mtimetst_sizeR+tst_inoR-(R/tf_handletf_patht stat_info((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRs      cCs|jS(N(R*(R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyt get_file_pathscCs|j|jS(N(t __class__R((R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyt__copy__scCs&|jr"|jjd|_ndS(sbCloses the file handle and clears it so that it cannot be reused. N(R)tcloseR(R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRs  cCsxd}ttjj||jd}|j|t|dx%|D]}|jt|dqIW|jdS(sWrites the dictionary in the expected format. Note: Only child classes should call this method. s VERSION: twbs N(R R R R R(twritetstrR=(R/R R$titerabletversion_stringt file_handletname((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyt_protected_write_dict_files ! cCsTtj|j}|j|jksH|j|jksH|j|jkrLt S|j S(sThis method uses the modification time and the file size to (heuristically) determine whether the file backing this storage has changed since it was last read. ( R R3R*R-R6R,R4R+R5RR.(R/R9((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyt should_rereads cCs t|_dS(N(RR.(R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pytread_dict_filescCst|g|S(sThis uses consistent open to ensure that the version line processing is done consistently and that only a single function actually opens files stored using this class. (R&(R/R((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR s( t__name__t __module__t__doc__R1R RR:R<RRERFRGR (((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR's    tIndexStoreMainDictcBs}eZdZdddddgZdZdZdZed Zed Z ed Z d Z d Z RS(s8Class for representing the main dictionary file Rt!t@t#t,cCstj||d|_dS(N(R'R1Rt _old_suffix(R/R0((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR1scCstj|||gdS(sThis class relies on external methods to write the file. Making this empty call to protected_write_dict_file allows the file to be set up correctly with the version number stored correctly. N(R'RE(R/R R$((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pytwrite_dict_files cCs|jS(sReturn the file handle. Note that doing anything other than sequential reads or writes to or from this file_handle may result in unexpected behavior. In short, don't use seek. (R)(R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pytget_file_handlescCstj}|jd}|j|d}tj|d}|d}g}xc|D][}|j|d}|d}|d}g} x|D] } | j|d}|d} |d} g} x| D]}|j|d}tj|d}|d}g}xg|D]_}|j|d}t|d}g|dD]}t|^q@}|j||fq W| j||fqW| j| | fqW|j|| fqUW||fS(sUParses one line of a main dictionary file. Changes to this function must be paired with changes to write_main_dict_line below. This should produce the same data structure that _write_main_dict_line in indexer.py creates to write out each line. s iiiii(RKt sep_charsRRturllibtunquoteRtappend(tlinet split_charsttmpttoktatltrestatit action_typetstltat_reststitsubtypetfvltst_restfvit full_valuetpfltfv_restpfit pfmri_indextttoffsets((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pytparse_main_dict_lines>           #cCs2|jd}|jdd}tj|dS(sPulls the token out of a line from a main dictionary file. Changes to this function must be paired with changes to write_main_dict_line below. s Rii(RRRTRU(RWtlst((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pytparse_main_dict_line_for_token)sc Csftj}dtjt|}x9t|D]+\}}|\}}|d|d|f7}xt|D]\}}|\} }|d|d| f7}xt|D]\} } | \} } |d|dtjt| f7}xgt| D]Y\} }|\}}|d|d|f7}x&|D]}|d|d|f7}q,WqWqWqlWq/W|dS( sPaired with parse_main_dict_line above. Transforms a token and its data into the string which can be written to the main dictionary. The "token" parameter is the token whose index line is being generated. The "entries" parameter is a list of lists of lists and so on. It contains information about where and how "token" was seen in manifests. The depth of all lists at each level must be consistent, and must match the length of "sep_chars" and "quote". The details of the contents on entries are described in _write_main_dict_line in indexer.py. s%ss%s%siiiiis (RKRSRTtquoteR@t enumerate(ttokententriesRSR\R]R[R^RaR_RbReRcRfRiRgRjtoffset((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyttransform_main_dict_line4s*         cCsdS(siReturns the number of entries removed during a second phase of indexing. i((R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyt-count_entries_removed_during_partial_indexingZscCs|jdksttjj||j}tjj||j|}tj|||j}|j||_|j |||_|j dk rtj tjj||j n|j||_ dS(s_Moves the existing file with self._name in directory use_dir to a new file named self._name + suffix in directory use_dir. If it has done this previously, it removes the old file it moved. It also opens the newly moved file and uses that as the file for its file handle. N( R)RRR R R R(tportabletrenameR RPtremove(R/tuse_dirtsuffixt orig_pathtnew_pathttmp_name((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyt shift_filebs   "( RHRIRJRSR1RQRRt staticmethodRmRoRuRvR(((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRKs  - & tIndexStoreListDictcBseZdZdddZdZdZdZdZdZd Z d Z d Z d Z d Z dZdZRS(sUsed when both a list and a dictionary are needed to store the information. Used for bidirectional lookup when one item is an int (an id) and the other is not (an entity). It maintains a list of empty spots in the list so that adding entities can take advantage of unused space. It encodes empty space as a blank line in the file format and '' in the internal list. cCs|S(N((tx((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pytscCs|S(N((R((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRscCsStj||g|_i|_d|_g|_||_||_d|_dS(Ni( R'R1t_listt_dictt_next_idt_list_of_emptiest _decode_funct _build_funct _line_cnt(R/R0tbuild_functiontdecode_function((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR1s      cCst|j|jkst|jr| r|jjd}|t|jks[t|t|jkr|jj||jd7_q||j|sN(R'RER(R/R R$((R/s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRQs cCs|jst|jjg|_xt|jD]z\}}|j|jd}|dkrx|jj |n ||j|<|jj ||d|_ |d|_ q5Wt j ||j S(s]Reads in a dictionary previously stored using the above call s i(R)RRtclearRRqRRRRVRRR'RG(R/tiRWRY((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRGs      cCs t|jS(siReturns the number of entries removed during a second phase of indexing. (RR(R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRvs(RHRIRJR1RRRRRRRRRRQRGRv(((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRvs           tIndexStoreDictcBsDeZdZdZdZdZdZdZdZRS(s;Class used when only entity -> id lookup is needed cCs&tj||i|_d|_dS(Ni(R'R1RR(R/R0((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR1s cCs|jS(N(R(R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pytget_dictscCs |j|S(N(R(R/R((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR scCs|jj|S(N(RR(R/R((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR scCsZ|jjx9t|jD](\}}|jd}||j| entity format s N(RRRqR)RR'RG(R/tline_cntRW((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRGs  cCs t|jS(siReturns the number of entries removed during a second phase of indexing. (RR(R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRvs( RHRIRJR1RRRRGRv(((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRs     tIndexStoreDictMutablecBsweZdZdZdZdZdZdZedZ dZ dZ d Z d Z d ZRS( s>Dictionary which allows dynamic update of its storage cCstj||i|_dS(N(R'R1R(R/R0((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR1#scCs|jS(N(R(R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR'scCs|jj|S(N(RR(R/R((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR*scCs |j|S(N(R(R/R((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR-scCs |jjS(N(Rtkeys(R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pytget_keys0scCs)d|krdtj|Sd|SdS(NRt1t0(RTRp(R@((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyt__quote3s cCs|jjxo|jD]d}|jd\}}|ddkrXtj|d}n |d}t|}||j||j||ttjj||jddt|_dS(suOpens the output file for this class and prepares it to be written via write_entity. tabt bufferingN(RQR R R R R(RR)(R/RzR$((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyt open_out_fileIscCsJ|jdk st|jj|jt|dt|ddS(s-Writes the entity out to the file with my_id Rs N(R)RRR?t_IndexStoreDictMutable__quoteR@(R/Rtmy_id((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyt write_entityQs cCstj|||gdS(s Generates an iterable list of string representations of the dictionary that the parent's protected_write_dict_file function can call. N(R'RE(R/R R$((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRQWs cCsdS(siReturns the number of entries removed during a second phase of indexing. i((R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRv_s(RHRIRJR1RRRRRRRGRRRQRv(((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRs         tIndexStoreSetHashcBsGeZdZdZdZdZdZdZdZRS(cCs)tj||tjj|_dS(N(R'R1thashlibtsha1t hexdigestthash_val(R/R0((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR1fscCs|j||_dS(sSet the has value.N(t calc_hashR(R/tvals((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pytset_hashlscCsJt|}|jtj}x|D]}|j|q)W|jS(s7Calculate the hash value of the sorted members of vals.(tlisttsortRRtupdateR(R/Rtvltshasumtv((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRps     cCs tj||||jgdS(s,Write self.hash_val out to a line in a file N(R'RER(R/R R$((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRQ{s cCsw|jj}d}x>t|jD]-\}}|dksCt|j|_q%W|jj|tj||S(sIProcess a dictionary file written using the above method ii( R)ttellRqRRRtseekR'RG(R/tspR\RW((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRGs cCsP|js|jn|j|}|j|krLtj|j|ndS(sbCheck the hash value of vals against the value stored in the file for this object.N(R.RGRRRtIncorrectIndexFileHash(R/Rt incoming_hash((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pytcheck_against_files   cCsdS(sXReturns the number of entries removed during a second phase of indexing.i((R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRvs( RHRIR1RRRQRGRRv(((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRes    t IndexStoreSetcBsheZdZdZdZdZdZdZdZdZ dZ d Z d Z RS( sUsed when only set membership is desired. This is currently designed for exclusive use with storage of fmri.PkgFmris. However, that impact is only seen in the read_and_discard_matching_from_argument method. cCs tj||t|_dS(N(R'R1tsett_set(R/R0((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR1scCs|jS(N(R(R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pytget_setscCs|jjdS(N(RR(R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRscCs|jj|dS(N(Rtadd(R/R((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRscCs|jj|dS(sRemove entity purposfully assumes that entity is already in the set to be removed. This is useful for error checking and debugging. N(RRy(R/R((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRscCs ||jkS(N(R(R/R((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRscCstj||||jdS(s5Write each member of the set out to a line in a file N(R'RER(R/R R$((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRQs cCs|jstd}|jjx^t|jD]M\}}|jd}|t|jksht|j||d}q2Wtj ||S(sIProcess a dictionary file written using the above method is i( R)RRRRqRRRR'RG(R/R\RRW((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRGs   cCs@|jr<x0|jD]"}tj|}|j|qWndS(s`Reads the file and removes all frmis in the file from fmri_set. N(R)tfmritPkgFmritdiscard(R/tfmri_setRWR ((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyt'read_and_discard_matching_from_arguments cCs t|jS(sXReturns the number of entries removed during a second phase of indexing.(RR(R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRvs( RHRIRJR1RRRRRRQRGRRv(((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRs         t InvertedDictcBskeZdZdZdZdZdZedZdZ dZ edZ d Z RS( sClass used to store and process fmri to offset mappings. It does delta compression and deduplication of shared offset sets when writing to a file.cCs/tj||||_i|_i|_dS(sfile_name is the name of the file to write to or read from. p_id_trans is an object which has a get entity method which, when given a package id number returns the PkgFmri object for that id number.N(R'R1t _p_id_transRt _fmri_offsets(R/R0t p_id_trans((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR1s  cCs|j|j|jS(N(R;R(R(R/((s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyR<scCs@y|j|j|Wn!tk r;|g|j|!sN(RR'RE(R/R R$((RR/s8/usr/lib/python2.7/vendor-packages/pkg/search_storage.pyRQs   cCsV|jstx3|jD](}|jd\}}||j|s2      TQ"F6@