ó ´2Kc@sjdZddlZdefd„ƒYZdefd„ƒYZdefd„ƒYZd „Zd „ZdS( söobject to map content hashes to file paths The Layout class hierarchy encapsulates bijective mappings between a hash (or file name since those are equivalent in our system) and a relative path that describes where to place that file in the file system. This bijective relation should hold when the union of all layouts is considered as a single set of mappings. In practical terms, this means that only one layout may potentially deposit a hash into any particular location. This is not a difficult requirement to satisfy since each layout may append a unique identifier to the file name or choose to carve out its own namespace at some level of directory hierarchy. The V1Layout places each file into a single layer of 256 directories. A fanout of 256 provides good performance compared to the other layouts tested. It also allows over 8M files to be stored even with filesystems which limit the number of files in a directory to 65k. The V0Layout layout uses two layers of directories; the first has a fanout of 256 while the second has a fanout of 16M. This layout has the problem that for the sizes of images (on the order of 300-500k files) and repos (on the order of 1M files), the second director level usually contains a single file. This imposes a substantial penalty for removing or resyncing the directories because a readdir(3C) must be done for each directory and readdir is two orders of magnitude slower than the open or read ZFS operations, and one order of magnitude slower than ZFS remove. Reducing the number of directories used to hold the downloaded files was a goal for the next layout. To evaluate a layout, it is necessary to measure the insertion time, the removal time, and the time to open a random file. The insertion time affects the publication speed. The removal time effects the time a client may take to clear its download cache. The access time effects how quickly a server can open a file to serve it. File sizes from 1 to 10M were used to asses the scalability of the different layouts.iÿÿÿÿNtLayoutcBs)eZdZd„Zd„Zd„ZRS(stThis class is the parent class to all layouts. It defines the interface which those subclasses must satisfy.cCs t‚dS(s0Return the path to the file with name "hashval".N(tNotImplementedError(tselfthashval((s</usr/lib/python2.7/vendor-packages/pkg/file_layout/layout.pytlookupCscCs t‚dS(s*Return the hash which would map to "path".N(R(Rtpath((s</usr/lib/python2.7/vendor-packages/pkg/file_layout/layout.pyt path_to_hashGscCs|j|ƒ|kS(s_Returns whether this layout would place a file named "file_name" at "rel_path".(R(Rtrel_patht file_name((s</usr/lib/python2.7/vendor-packages/pkg/file_layout/layout.pytcontainsKs(t__name__t __module__t__doc__RRR (((s</usr/lib/python2.7/vendor-packages/pkg/file_layout/layout.pyR?s  tV0LayoutcBs eZdZd„Zd„ZRS(sThis class implements the original layout used. It uses a 256 way split (2 hex digits) followed by a 16.7M way split (6 hex digits).cCs$tjj|dd!|dd!|ƒS(s0Return the path to the file with name "hashval".iii(tosRtjoin(RR((s</usr/lib/python2.7/vendor-packages/pkg/file_layout/layout.pyRUscCstjj|ƒS(s*Return the hash which would map to "path".(RRtbasename(RR((s</usr/lib/python2.7/vendor-packages/pkg/file_layout/layout.pyRYs(R R R RR(((s</usr/lib/python2.7/vendor-packages/pkg/file_layout/layout.pyR Qs tV1LayoutcBs eZdZd„Zd„ZRS(s~This class implements the new layout approach which is a single 256 way fanout using the first two digits of the hash.cCstjj|dd!|ƒS(s0Return the path to the file with name "hashval".ii(RRR(RR((s</usr/lib/python2.7/vendor-packages/pkg/file_layout/layout.pyRbscCstjj|ƒS(s*Return the hash which would map to "path".(RRR(RR((s</usr/lib/python2.7/vendor-packages/pkg/file_layout/layout.pyRfs(R R R RR(((s</usr/lib/python2.7/vendor-packages/pkg/file_layout/layout.pyR^s cCstƒtƒgS(s\This function describes the default order in which to use the layouts defined above.(RR (((s</usr/lib/python2.7/vendor-packages/pkg/file_layout/layout.pytget_default_layoutskscCstƒS(s9This function returns the single preferred layout to use.(R(((s</usr/lib/python2.7/vendor-packages/pkg/file_layout/layout.pytget_preferred_layoutqs(R RtobjectRR RRR(((s</usr/lib/python2.7/vendor-packages/pkg/file_layout/layout.pyt:s