##// END OF EJS Templates
exchange: refactor APIs to obtain bundle data (API)...
exchange: refactor APIs to obtain bundle data (API) Currently, exchange.getbundle() returns either a cg1unpacker or a util.chunkbuffer (in the case of bundle2). This is kinda OK, as both expose a .read() to consumers. However, localpeer.getbundle() has code inferring what the response type is based on arguments and converts the util.chunkbuffer returned in the bundle2 case to a bundle2.unbundle20 instance. This is a sign that the API for exchange.getbundle() is not ideal because it doesn't consistently return an "unbundler" instance. In addition, unbundlers mask the fact that there is an underlying generator of changegroup data. In both cg1 and bundle2, this generator is being fed into a util.chunkbuffer so it can be re-exposed as a file object. util.chunkbuffer is a nice abstraction. However, it should only be used "at the edges." This is because keeping data as a generator is more efficient than converting it to a chunkbuffer, especially if we convert that chunkbuffer back to a generator (as is the case in some code paths currently). This patch refactors exchange.getbundle() into exchange.getbundlechunks(). The new API returns an iterator of chunks instead of a file-like object. Callers of exchange.getbundle() have been updated to use the new API. There is a minor change of behavior in test-getbundle.t. This is because `hg debuggetbundle` isn't defining bundlecaps. As a result, a cg1 data stream and unpacker is being produced. This is getting fed into a new bundle20 instance via bundle2.writebundle(), which uses a backchannel mechanism between changegroup generation to add the "nbchanges" part parameter. I never liked this backchannel mechanism and I plan to remove it someday. `hg bundle` still produces the "nbchanges" part parameter, so there should be no user-visible change of behavior. I consider this "regression" a bug in `hg debuggetbundle`. And that bug is captured by an existing "TODO" in the code to use bundle2 capabilities.

File last commit:

r30133:f6dcda75 default
r30187:3e86261b default
Show More
pycompat.py
196 lines | 5.0 KiB | text/x-python | PythonLexer
timeless
pycompat: add empty and queue to handle py3 divergence...
r28818 # pycompat.py - portability shim for python 3
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
"""Mercurial portability shim for python 3.
This contains aliases to hide python version-specific details from the core.
"""
from __future__ import absolute_import
Pulkit Goyal
pycompat: make pycompat demandimport friendly...
r29584 import sys
Yuya Nishihara
pycompat: provide 'ispy3' constant...
r30030 ispy3 = (sys.version_info[0] >= 3)
if not ispy3:
Pulkit Goyal
py3: conditionalize cPickle import by adding in util...
r29324 import cPickle as pickle
Pulkit Goyal
pycompat: make pycompat demandimport friendly...
r29584 import cStringIO as io
Pulkit Goyal
py3: conditionalize httplib import...
r29455 import httplib
Pulkit Goyal
pycompat: make pycompat demandimport friendly...
r29584 import Queue as _queue
Pulkit Goyal
py3: conditionalize SocketServer import...
r29433 import SocketServer as socketserver
Pulkit Goyal
pycompat: make pycompat demandimport friendly...
r29584 import urlparse
Pulkit Goyal
py3: conditionalize xmlrpclib import...
r29432 import xmlrpclib
Pulkit Goyal
pycompat: make pycompat demandimport friendly...
r29584 else:
import http.client as httplib
import io
import pickle
import queue as _queue
import socketserver
import urllib.parse as urlparse
Pulkit Goyal
py3: conditionalize xmlrpclib import...
r29432 import xmlrpc.client as xmlrpclib
Pulkit Goyal
py3: conditionalize the urlparse import...
r29431
Yuya Nishihara
pycompat: provide 'ispy3' constant...
r30030 if ispy3:
Yuya Nishihara
py3: move xrange alias next to import lines...
r29797 import builtins
Yuya Nishihara
py3: provide (del|get|has|set)attr wrappers that accepts bytes...
r29799 import functools
Martijn Pieters
py3: add an os.fsencode backport to ease path handling
r30119 import os
fsencode = os.fsencode
Yuya Nishihara
py3: move xrange alias next to import lines...
r29797
Yuya Nishihara
pycompat: extract function that converts attribute or encoding name to str...
r30032 def sysstr(s):
"""Return a keyword str to be passed to Python functions such as
getattr() and str.encode()
This never raises UnicodeDecodeError. Non-ascii characters are
considered invalid and mapped to arbitrary but unique code points
such that 'sysstr(a) != sysstr(b)' for all 'a != b'.
"""
if isinstance(s, builtins.str):
return s
return s.decode(u'latin-1')
Yuya Nishihara
py3: provide (del|get|has|set)attr wrappers that accepts bytes...
r29799 def _wrapattrfunc(f):
@functools.wraps(f)
def w(object, name, *args):
Yuya Nishihara
pycompat: extract function that converts attribute or encoding name to str...
r30032 return f(object, sysstr(name), *args)
Yuya Nishihara
py3: provide (del|get|has|set)attr wrappers that accepts bytes...
r29799 return w
Yuya Nishihara
py3: import builtin wrappers automagically by code transformer...
r29800 # these wrappers are automagically imported by hgloader
Yuya Nishihara
py3: provide (del|get|has|set)attr wrappers that accepts bytes...
r29799 delattr = _wrapattrfunc(builtins.delattr)
getattr = _wrapattrfunc(builtins.getattr)
hasattr = _wrapattrfunc(builtins.hasattr)
setattr = _wrapattrfunc(builtins.setattr)
Yuya Nishihara
py3: import builtin wrappers automagically by code transformer...
r29800 xrange = builtins.range
Yuya Nishihara
py3: provide (del|get|has|set)attr wrappers that accepts bytes...
r29799
Yuya Nishihara
pycompat: extract function that converts attribute or encoding name to str...
r30032 else:
def sysstr(s):
return s
Martijn Pieters
pycompat: only accept a bytestring filepath in Python 2
r30133 # Partial backport from os.py in Python 3, which only accepts bytes.
# In Python 2, our paths should only ever be bytes, a unicode path
# indicates a bug.
def fsencode(filename):
if isinstance(filename, str):
return filename
Martijn Pieters
py3: add an os.fsencode backport to ease path handling
r30119 else:
Martijn Pieters
pycompat: only accept a bytestring filepath in Python 2
r30133 raise TypeError(
"expect str, not %s" % type(filename).__name__)
Martijn Pieters
py3: add an os.fsencode backport to ease path handling
r30119
Pulkit Goyal
pycompat: make pycompat demandimport friendly...
r29584 stringio = io.StringIO
timeless
pycompat: add empty and queue to handle py3 divergence...
r28818 empty = _queue.Empty
queue = _queue.Queue
timeless
pycompat: alias xrange to range in py3
r28834
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882 class _pycompatstub(object):
Yuya Nishihara
pycompat: delay loading modules registered to stub...
r29801 def __init__(self):
self._aliases = {}
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882
Yuya Nishihara
pycompat: delay loading modules registered to stub...
r29801 def _registeraliases(self, origin, items):
"""Add items that will be populated at the first access"""
Augie Fackler
pycompat: when setting attrs, ensure we use sysstr...
r30086 items = map(sysstr, items)
self._aliases.update(
(item.replace(sysstr('_'), sysstr('')).lower(), (origin, item))
for item in items)
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882
Yuya Nishihara
pycompat: delay loading modules registered to stub...
r29801 def __getattr__(self, name):
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882 try:
Yuya Nishihara
pycompat: delay loading modules registered to stub...
r29801 origin, item = self._aliases[name]
except KeyError:
raise AttributeError(name)
self.__dict__[name] = obj = getattr(origin, item)
return obj
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882
Pulkit Goyal
py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import...
r29566 httpserver = _pycompatstub()
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882 urlreq = _pycompatstub()
urlerr = _pycompatstub()
Yuya Nishihara
pycompat: provide 'ispy3' constant...
r30030 if not ispy3:
Pulkit Goyal
py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import...
r29566 import BaseHTTPServer
import CGIHTTPServer
import SimpleHTTPServer
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882 import urllib2
import urllib
Yuya Nishihara
pycompat: delay loading modules registered to stub...
r29801 urlreq._registeraliases(urllib, (
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882 "addclosehook",
"addinfourl",
"ftpwrapper",
"pathname2url",
"quote",
"splitattr",
"splitpasswd",
"splitport",
"splituser",
"unquote",
"url2pathname",
"urlencode",
))
Yuya Nishihara
pycompat: delay loading modules registered to stub...
r29801 urlreq._registeraliases(urllib2, (
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882 "AbstractHTTPHandler",
"BaseHandler",
"build_opener",
"FileHandler",
"FTPHandler",
"HTTPBasicAuthHandler",
"HTTPDigestAuthHandler",
"HTTPHandler",
"HTTPPasswordMgrWithDefaultRealm",
"HTTPSHandler",
"install_opener",
"ProxyHandler",
"Request",
"urlopen",
))
Yuya Nishihara
pycompat: delay loading modules registered to stub...
r29801 urlerr._registeraliases(urllib2, (
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882 "HTTPError",
"URLError",
))
Yuya Nishihara
pycompat: delay loading modules registered to stub...
r29801 httpserver._registeraliases(BaseHTTPServer, (
Pulkit Goyal
py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import...
r29566 "HTTPServer",
"BaseHTTPRequestHandler",
))
Yuya Nishihara
pycompat: delay loading modules registered to stub...
r29801 httpserver._registeraliases(SimpleHTTPServer, (
Pulkit Goyal
py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import...
r29566 "SimpleHTTPRequestHandler",
))
Yuya Nishihara
pycompat: delay loading modules registered to stub...
r29801 httpserver._registeraliases(CGIHTTPServer, (
Pulkit Goyal
py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import...
r29566 "CGIHTTPRequestHandler",
))
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882
Yuya Nishihara
pycompat: delay loading modules registered to stub...
r29801 else:
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882 import urllib.request
Yuya Nishihara
pycompat: delay loading modules registered to stub...
r29801 urlreq._registeraliases(urllib.request, (
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882 "AbstractHTTPHandler",
"addclosehook",
"addinfourl",
"BaseHandler",
"build_opener",
"FileHandler",
"FTPHandler",
"ftpwrapper",
"HTTPHandler",
"HTTPSHandler",
"install_opener",
"pathname2url",
"HTTPBasicAuthHandler",
"HTTPDigestAuthHandler",
Gregory Szorc
pycompat: add HTTPPasswordMgrWithDefaultRealm to Python 3 block...
r29414 "HTTPPasswordMgrWithDefaultRealm",
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882 "ProxyHandler",
"quote",
"Request",
"splitattr",
"splitpasswd",
"splitport",
"splituser",
"unquote",
"url2pathname",
"urlopen",
))
import urllib.error
Yuya Nishihara
pycompat: delay loading modules registered to stub...
r29801 urlerr._registeraliases(urllib.error, (
timeless
pycompat: add util.urlerr util.urlreq classes for py3 compat...
r28882 "HTTPError",
"URLError",
))
Pulkit Goyal
py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import...
r29566 import http.server
Yuya Nishihara
pycompat: delay loading modules registered to stub...
r29801 httpserver._registeraliases(http.server, (
Pulkit Goyal
py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import...
r29566 "HTTPServer",
"BaseHTTPRequestHandler",
"SimpleHTTPRequestHandler",
"CGIHTTPRequestHandler",
))