͕!Pc@sjddlZddlZddlZddlZddlZddlZy8ddlZeedrejejej nWne k rnXddl j j Z ddlj jjZddlj jjZddljZddlmZddlmZddlmZfZdddfZd efd YZd efd YZ d efdYZ!dS(iNtSIGPIPE(tdeque(tglobal_settings(t DebugValuestftpthttpthttpstTransportEnginecBseZdZRS(sThis is an abstract class. It shouldn't implement any of the methods that it contains. Leave that to transport-specific implementations.(t__name__t __module__t__doc__(((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyR6stCurlTransportEnginecBs[eZdZddZdZeeeeeeeeeeeed ZdZ dZ eedZ eeeeeeeeeed Z eeeeeeeed Z ed Zd Zd Zd ZdZeeeeeeeeeeeeed ZdZedZdZdZdZedZRS(s8Concrete class of TransportEngine for libcurl transport.icCs8||_tj|_g|_d|_||_t|_g|_ g|_ t |_ d|_ d|_i|_d|_|jjtjdxt|jD]|}tj}d|_d|_d|_d|_d|_t|_d|_d|_d|_d|_ |jj!|qW|j|_"dS(Niii(#t_CurlTransportEngine__xporttpycurlt CurlMultit_CurlTransportEngine__mhandlet_CurlTransportEngine__chandlest$_CurlTransportEngine__active_handlest!_CurlTransportEngine__max_handlesRt_CurlTransportEngine__req_qt_CurlTransportEngine__failurest_CurlTransportEngine__successtsett_CurlTransportEngine__orphanst _CurlTransportEngine__file_bufsztNonet _CurlTransportEngine__user_agentt#_CurlTransportEngine__common_headert&_CurlTransportEngine__last_stall_checktsetoptt M_PIPELININGtrangetCurlturltrepourltfobjtr_fobjtfilepathtFalsetsuccesstfileprogtfiletimet starttimetuuidtappendt!_CurlTransportEngine__freehandles(tselft transporttmax_conntiteh((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyt__init__?s8                       cCs;x+|jj\}}|tjkrPqq||_|S(sRAn internal method that invokes the multi-handle's perform method.(RtperformR tE_CALL_MULTI_PERFORMR(R.trettactive_handles((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyt__call_performms  cCsht|d|d|d|d|d|d|d|d| d | d | d | d |  }|jj|d S(sAdd a URL to the transport engine. Caller must supply either a filepath where the file should be downloaded, or a callback to a function that will peform the write. It may also optionally supply header information in a dictionary. If the caller has a ProgressTracker, it should pass the tracker in progtrack. The caller should also supply a class that wraps the tracker in progclass. 'proxy' is the persistent proxy value for this url and is stored as part of the transport stats accounting. 'runtime_proxy' is the actual proxy value that is used by pycurl to retrieve this resource.R%t writefunctheadert progclasst progtracktsslcerttsslkeyR"t compressiblet failonerrortproxyt runtime_proxyN(tTransportRequestRt appendleft(R.R!R%R9R:R;R<R=R>R"R?R@RARBtt((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pytadd_urlys   cCstj}|dkrdStj}g}g}g}g|jD]}||jkrA|^qA}xu|D]m}||j} | dkr||_d} n|jtj|jtj } |j | |j | qiW|t |krt |dkrx|D]v}|j } |j} |j} tj| d| d| }|jj||j||jj ||j |qWn|jj|dS(sqIn some situations, libcurl can get itself tied in a knot, and fail to make progress. Check that the active handles are making progress. If none of the active handles have downloaded any content for the timeout period, reset the transport and generate exceptions for the failed requests.iNR"R+(RtPKG_CLIENT_LOWSPEED_TIMEOUTttimeRR-R*tgetinfoR t SIZE_DOWNLOADt SIZE_UPLOADR,tmintmaxR!R+R"ttxtTransportStallErrorRt remove_handlet%_CurlTransportEngine__teardown_handleRtextend(R.ttimeoutt current_timet time_listt size_listtfailuresthdltq_hdlstht time_elapsedt size_xfrdR!R+turlstemtex((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyt__check_for_stallss<         $      c Cs|jj\}}}|j}|j}g}d }t}d} x|D]\} } } |jj| j| j f} |j | | j | j t j}| j t j}| j t j}| j t j}| j}| j}| j}tj|d}|dkr5|tkr5|| j t j8}n|dkrN||8}n|dkro| j t j}n| j|||dkr|dkr| j|n| j t j}| t jkrd }tj}n| t jt jfkr|t kr| }nd }|t!j"krEt!j"|}||krE||}qEnt!j#|||d|d|d|}| j$d|j%| d7} nS| t j&k}t!j'| || d|d|}| j$d|j%d|| d7} |r|j(r|j)|n|r| r|}n|j)| qLWx |D]} |jj| j| j f} |j | | j | j t j}| j t j}| j t j}| j t j}| j t j*| _+| j}| j}| j}tj|d}|dkr!|tkr!|| j t j8}n|dkr:||8}n|dkrS||}nd}|t,j-kr|t,j.krd}d}n| j|||dkr|dkr| j|n| j t j}|t ks|t/j0krt1| _2| j3|j)|nd }|t!j"krLt!j"|}||krL||}qLnt!j#|||d|d|d|}|dkr| j$d|j%| d7} nY|dkr| j$| d7} d }t!j#|d |d|d|d|}t1|_(n|j(r|j)|n|s |}n|j)| q,Wx;|D]3} |jj4| |j5| |j6j)| q8W||_||_|r|n| dkrd SxH|D]@}|j7}|t,j8kr|j3t!j9|j|qqWd S( sCleanup handles that have finished their request. Return the handles to the freelist. Generate any relevant error information.itreasonR"R+t decayableiRSis$Invalid HTTP status code from serverR!N(:Rt info_readRRRRR tstatsR"RAtaddt record_txRIR RJt TOTAL_TIMEt NUM_CONNECTSt CONNECT_TIMER!R+turlparseturlsplittpipelined_protocolstSTARTTRANSFER_TIMEtrecord_progresstrecord_connectiont RESPONSE_CODEtE_ABORTED_BY_CALLBACKt api_errorstCanceledExceptiontE_HTTP_RETURNED_ERRORtE_FILE_COULDNT_READ_FILEtresponse_protocolsRNtproto_code_maptTransportProtoErrort record_errorRatE_OPERATION_TIMEOUTEDtTransportFrameworkErrort retryableR,t INFO_FILETIMER)Rtpkg_client_lowspeed_limitRGthttplibtOKtTrueR'tclear_consecutive_errorsRPRQR-tconsecutive_errorst PKG_CLIENT_MAX_CONSECUTIVE_ERRORtExcessiveTransientFailure(R.tcounttgoodtbadRWR't done_handlest ex_to_raiset visited_repost errors_seenRZtentemt repostatstnbytestsecondst conn_countt conn_timeR!R+R]tprototrespcodeR^t proto_reasontpmapRSt bytespersecR`trstnumce((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyt__cleanup_requestss                                                                     c Cs |s>|j}|j}g|_g|_|r:||fS|Sg|jD]*}t|drH|j|krH|^qH}x|D]}|jj|qW|sg|_|Sg}x-|jD]"}||kr|j|qqWx|D]}|jj|qW||fS(sdReturn information about retryable failures that occured during the request. This is a list of transport exceptions. Caller may raise these, or process them for failure information. Urllist is an optional argument to return only failures for a specific URLs. Not all callers of check status want to claim the error state of all pending transactions. Transient errors are part of standard control flow. The caller will look at these and decide whether to throw them or not. Permanent failures are raised by the transport engine as soon as they occur. If good_reqs is set to true, then check_stats will return a tuple of lists, the first list contains the transient errors that were encountered, the second list contains successfully transferred urls. Because the list of successfully transferred URLs may be long, it is discarded if not requested by the caller.R!(RRthasattrR!tremoveR,( R.turllistt good_reqstrfRttftfttsts((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyt check_statuss.      *    c Cstj||d|} d} |r3| j} nt|d| jd| jd|d|d|d|d|d | d | jd |d | d | d|  }|jj || S(sInvoke the engine to retrieve a single URL. Callers wishing to obtain multiple URLs at once should use addUrl() and run(). getUrl will return a read-only file object that allows access to the URL's data. 'proxy' is the persistent proxy value for this url and is stored as part of the transport stats accounting. 'runtime_proxy' is the actual proxy value that is used by pycurl to retrieve this resource. 'system' whether the resource is being retrieved on behalf of a system-publisher or directly from the system-repository. tccancelR9thdrfuncR:R=R>R"R?tprogfuncR+R@RARBtsystemN( tfileobjtStreamingFileObjRtget_progress_funcRCtget_write_functget_header_funcR+RRD(R.R!R:R=R>R"R?RR@RARBRR#RRE((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pytget_url s c Cstj||d|} d} |r3| j} nt|d| jd| jd|ddd|d|d |d | d | jd |d |d|  } |jj | | S(sInvoke the engine to retrieve a single URL's headers. getUrlHeader will return a read-only file object that contains no data. 'proxy' is the persistent proxy value for this url and is stored as part of the transport stats accounting. 'runtime_proxy' is the actual proxy value that is used by pycurl to retrieve this resource. RR9RR:t httpmethodtHEADR=R>R"RR+R@RARBN( RRRRRCRRR+RRD( R.R!R:R=R>R"RR@RARBR#RRE((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pytget_url_header.s  cCst|jp|jdkS(sbReturns true if the engine still has outstanding work to perform, false otherwise.i(tboolRR(R.((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pytpendingNscCs|js dS|jdkru|jj}|dkr@d}n|dkrY|d}n|ru|jj|qunx2|jr|jj\}}|j||qxWxW|jr|j r|j j}|jjd}|j |||jj |qW|j |j |jr|j s5|j rtj}||jdkrj||_|jq||jdkr||_|jqndS(sRun the transport engine. This polls the underlying framework to complete any asynchronous I/O. Synchronous operations should have completed when startRequest was invoked.Niig?g@@i(RRRRStselectRtpoptremove_requestR-Rt"_CurlTransportEngine__setup_handlet add_handlet"_CurlTransportEngine__call_performt&_CurlTransportEngine__cleanup_requestsRHRt&_CurlTransportEngine__check_for_stalls(R.RSR!R+RER2t cur_clock((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pytrunUs:             cCs|jj||fdS(soAdd the URL to the list of orphaned requests. Any URL in list will be removed from the transport next time run() is invoked. This is used by the fileobj's __del__ method to prevent unintended modifications to transport state when StreamingFileObjs that aren't close()'d get cleaned up.N(RRd(R.R!R+((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pytorphaned_requestscCs#x|jD]}|j|kr |j|kr ||jkr y|jj|Wntjk rgnX|j||jj |dSq WxIt |j D]8\}}|j|kr|j|kr|j |=dSqWxC|j D]8}|j|kr|j|kr|j j |dSqWdS(s3In order to remove a request, it may be necessary to walk all of the items in the request queue, all of the currently active handles, and the list of any transient failures. This is expensive, so only remove a request if absolutely necessary.N(RR!R+R-RRPR terrorRQR,t enumerateRRR(R.R!R+RZR1RER^((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyRs$  cCsx[|jD]P}||jkr y|jj|Wntjk rInX|j|q q Wd|_|j|_t|_ g|_ g|_ t |_ dS(slReset the state of the transport engine. Do this before performing another type of request.iN(RR-RRPR RRQRRRRRRR(R.tc((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pytresets     c$Cstj||d|}d}|rA| rA| rA|j}nt|d|jd|jd|d|ddd|d |d |d |d |jd |d| d| d| d| d| d|}|jj ||S(sInvoke the engine to retrieve a single URL. This routine sends the data in data, and returns the server's response. Callers wishing to obtain multiple URLs at once should use addUrl() and run(). sendData will return a read-only file object that allows access to the server's response..RR9RR:tdataRtPOSTR=R>R"RR+t read_fobjt read_filepathR@R;R<RARBN( RRRRRCRRR+RRD(R.R!RR:R=R>R"Rt data_fobjtdata_fpR@R;R<RARBR#RRE((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyt send_datas    cCs&|dkrd|_dS||_dS(sIf the downloaded files are being written out by the file() mechanism, and not written using a callback, the I/O is buffered. Set the buffer size using this function. If it's not set, a default of 131072 (128k) is used.ii N(R(R.tsize((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pytset_file_bufszs  cCs |si|_dS||_dS(sSupply a dictionary of name/value pairs in hdrdict. These will be included on all requests issued by the transport engine. To append a specific header to a certain request, supply a dictionary to the header argument of addUrl.N(R(R.thdrdict((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyt set_headers cCs ||_dS(sSupply a string str and the transport engine will use this string as its User-Agent header. This is a header that will be common to all transport requests.N(R(R.tua_str((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pytset_user_agentsc Cs|jtjd|jtjtj|jtjtj|jtjtj |jtj t |jtj tj d |_d |_|jr|jr|j|_|jtj|jq<|jtjdn_|jr)|jdkr|jtj|jn|j|_|j|_n|jtjd|jr^|jtj|jn|jsp|jrg}x:|jjD])\}}d||f}|j|qW|jrx=|jjD])\}}d||f}|j|qWn|jtj|n|jtj|j|j|_|j|_tj|_|j |_ |j!r2yt"|j!d|j#|_$Wntt%k r}|j&t&j'krt(j)|j*n|j&t&j+krt(j,|j*nt-j.d|nX|jtj/|j$|jtj0t |j!|_!nJ|j1rf|jtj2|j1d |_!d |_$nt-j.d|j|j3r|jtj4t n|j5r|j6r|jtj7d |j6|j5|_8|jtj9|j8j:n5|j;r&|jtj7d |jtj9|j;nt<j=|jd }|dkrLd S|j>ryt"|j>d |j#|_?Wqt%k r}|j&t&j'krt(j)|j*nt-j.d|qXn|j@r|jtjAdn|jBr|jtjC|jBn|jDdkr-|jtjEt n|jDdkrR|jtjFt n|jDdkr |jtjGt |jHd k r|jtjI|jHq|j?s|jJr|j?s|jJ|_?n|jtjK|j?|jtjLtMjN|j?jOjPqt-j.d|jn|jDdkr|jtjQt |j?sT|jJr|j?sl|jJ|_?n|jtjK|j?|jtjRtMjN|j?jOjPqt-j.d|jn;|jDdkr|jtjSdnt-j.d|jD|jTr#|jtjU|jTn|jVrE|jtjW|jVn|d kr|jtjXd|jtjYd|jZj[} |jtj\| dt]krt]d} |jtj^| |j_tj\q|j_tj^nd S(sSetup the curl easy handle, hdl, with the parameters specified in the TransportRequest treq. If global parameters are set, apply these to the handle as well.itt-t localhosts%s: %sswb+sUnable to open file: %ssKTransport invocation for URL %s did not specify filepath or write function.iRRNtrbtGETRRs`Transport operation for POST URL %s did not supply data or read_fobj. At least one is required.tPUTsPTransport operation for PUT URL %s did not supply a read_fobj. One is required.tDELETEs#Invalid http method '%s' specified.it ssl_ca_file(shttpshttps(`RR tNOSIGNALtCONNECTTIMEOUTRtPKG_CLIENT_CONNECT_TIMEOUTtLOW_SPEED_LIMITR}tLOW_SPEED_TIMERGtFOLLOWLOCATIONRt MAXREDIRStPKG_CLIENT_MAX_REDIRECTRRARBRtPROXYtNOPROXYRt USERAGENTRR:t iteritemsR,t HTTPHEADERtURLR!R+RHR*R"R%topenRR#tEnvironmentErrorterrnotEACCESRqtPermissionsExceptiontfilenametEROFStReadOnlyFileSystemExceptionRNtTransportOperationErrort WRITEDATAt OPT_FILETIMER9t WRITEFUNCTIONR@t FAILONERRORR<R;t NOPROGRESSR(tPROGRESSFUNCTIONtprogress_callbackRRiRjRR$R?tENCODINGRtHEADERFUNCTIONRtHTTPGETtNOBODYRRt POSTFIELDSRtREADDATAt POSTFIELDSIZEtostfstattfilenotst_sizetUPLOADt INFILESIZEt CUSTOMREQUESTR=tSSLCERTR>tSSLKEYtSSL_VERIFYHOSTtSSL_VERIFYPEERR t get_ca_dirtCAPATHRtCAINFOtunsetopt( R.RXttreqt headerlisttktvt headerstrteRtcadirtcafile((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyt__setup_handle s                                                 cCswx|jD]}|jq Wd|_d|_|jjd|_d|_d|_d|_d|_d|_ dS(s/Shutdown the transport engine, perform cleanup.iN( RtcloseRR-RRRRRR(R.R((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pytshutdowns        cCs|j|jr|jjd|_|js|jrK|jjnytj|j Wqt k r}|j t j krt jd|qqXq|jrtj|j j}|jj|n|j r|jdkr|j}tj|j ||fqn|jr3|jjd|_nd|_d|_t|_d|_ d|_d|_d|_d|_dS(s Cleanup any state that we've associated with this handle. After a handle has been torn down, it should still be valid for use, but should have no previous state. To remove handles from use completely, use __shutdown.sUnable to remove file: %siN(RR#RRR'R(tabortRRR%RRtENOENTRNRtstatRtcommitR)tutimeR$R!R"R&R+R*(RXRtfilesztft((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyt__teardown_handles>                  (RR R R3RRR&RRFRRRRRtpropertyRRRRRRRRRRRt staticmethodRQ(((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyR <s> .     7 ?  #    4 !      RCcBsSeZdZdddddddddddeddddeddedZRS(sA class that contains per-request information for the underlying transport engines. This is used to set per-request options that are used either by the framework, the transport, or both.RcCs||_||_||_||_||_||_||_||_| |_||_ | |_ | |_ | |_ | |_ ||_||_||_||_||_||_||_dS(sCreate a TransportRequest with the following parameters: url - The url that the transport engine should retrieve filepath - If defined, the transport engine will download the file to this path. If not defined, the caller should supply a write function. writefunc - A function, supplied instead of filepath, that reads the bytes supplied by the transport engine and writes them somewhere for processing. This is a callback. hdrfunc - A callback for examining the contents of header data in a response to a transport request. header - A dictionary of key/value pairs to be included in the request's header. compressible - A boolean value that indicates whether the content that is requested is a candidate for transport level compression. data - If the request is sending a data payload, include the data in this argument. failonerror - If the request returns a HTTP code >= 400, terminate the request early, instead of running it to completion. httpmethod - If the request is a HTTP/HTTPS request, this can override the default HTTP method of GET. progtrack - If the transport wants the engine to update the progress of the download, supply a ProgressTracker object in this argument. progclass - If the transport was supplied with a ProgressTracker this must point to a class that knows how to wrap the progress tracking object in way that allows the transport to invoke the proper callbacks. The transport instantiates an object of this class before beginning the request. progfunc - A function to be used as a progress callback. The preferred method is is use progtrack/progclass, but light-weight implementations may use progfunc instead, especially if they don't need per-file updates. read_filepath - If the request is sending a file, include the path here, as this is the most efficient way to send the data. read_fobj - If the request is sending a large payload, this points to a fileobject from which the data may be read. repouri - This is the URL stem that identifies the repo. It's a subset of url. It's also used by the stats system. sslcert - If the request is using SSL, HTTPS for example, provide a path to the SSL certificate here. sslkey - If the request is using SSL, like HTTPS for example, provide a path to the SSL key here. uuid - In order to remove the request from the list of many possible requests, supply a unique identifier in uuid. proxy - If the request should be performed using a proxy, that proxy should be specified here. runtime_proxy - In order to avoid repeated environment lookups we pass the proxy that should be used at runtime, which may differ from the 'proxy' value. system - whether this request is on behalf of a system publisher. Usually this isn't necessary, as the TransportRepoURI will have been configured with correct proxy and runtime_proxy properties. However, for direct access to resources served by the system-repository, we use this to prevent $http_proxy environment variables from being used. A TransportRequest must contain enough information to uniquely identify any pkg.client.publisher.TransportRepoURI - in particular, it must contain all fields used by TransportRepoURI.key() which is currently the (url, proxy) tuple, and is used as the key when recording/retrieving transport statistics.N(R!R%R9RR:RRR;R<RR"R=R>R?R+RRR@RARBR(R.R!R%R9RR:RRR;R<R=R>R"R?RR+RRR@RARBR((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyR3#s*^                    N(RR R RR&R3(((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyRCs    ("RR~RR RHRitsignalRRtSIG_IGNt ImportErrortpkg.client.api_errorstclientRqtpkg.client.transport.exceptionR/t exceptionRNtpkg.client.transport.fileobjRtpkg.misctmisct collectionsRt pkg.clientRtpkg.client.debugvaluesRRkRutobjectRR RC(((sA/usr/lib/python2.7/vendor-packages/pkg/client/transport/engine.pyts4