##// END OF EJS Templates
wireprotov2: implement commands as a generator of objects...
wireprotov2: implement commands as a generator of objects Previously, wire protocol version 2 inherited version 1's model of having separate types to represent the results of different wire protocol commands. As I implemented more powerful commands in future commits, I found I was using a common pattern of returning a special type to hold a generator. This meant the command function required a closure to do most of the work. That made logic flow more difficult to follow. I also noticed that many commands were effectively a sequence of objects to be CBOR encoded. I think it makes sense to define version 2 commands as generators. This way, commands can simply emit the data structures they wish to send to the client. This eliminates the need for a closure in command functions and removes encoding from the bodies of commands. As part of this commit, the handling of response objects has been moved into the serverreactor class. This puts the reactor in the driver's seat with regards to CBOR encoding and error handling. Having error handling in the function that emits frames is particularly important because exceptions in that function can lead to things getting in a bad state: I'm fairly certain that uncaught exceptions in the frame generator were causing deadlocks. I also introduced a dedicated error type for explicit error reporting in command handlers. This will be used in subsequent commits. There's still a bit of work to be done here, especially around formalizing the error handling "protocol." I've added yet another TODO to track this so we don't forget. Test output changed because we're using generators and no longer know we are at the end of the data until we hit the end of the generator. This means we can't emit the end-of-stream flag until we've exhausted the generator. Hence the introduction of 0-sized end-of-stream frames. Differential Revision: https://phab.mercurial-scm.org/D4472

File last commit:

r39595:07b58266 default
r39595:07b58266 default
Show More
test-wireproto-command-capabilities.t
431 lines | 15.4 KiB | text/troff | Tads3Lexer
/ tests / test-wireproto-command-capabilities.t
Kyle Lippincott
tests: mark tests that fail when using chg as #require no-chg...
r38041 #require no-chg
Gregory Szorc
wireproto: implement capabilities for wire protocol v2...
r37551 $ . $TESTDIR/wireprotohelpers.sh
$ hg init server
Gregory Szorc
tests: explicitly define compression engines for tests...
r37840
zstd isn't present in plain builds. Make tests easier by removing
zstd from the equation.
$ cat >> server/.hg/hgrc << EOF
> [server]
> compressionengines = zlib
> EOF
Gregory Szorc
wireproto: define and implement HTTP handshake to upgrade protocol...
r37575 $ hg -R server serve -p $HGPORT -d --pid-file hg.pid -E error.log
$ cat hg.pid > $DAEMON_PIDS
A normal capabilities request is serviced for version 1
$ sendhttpraw << EOF
> httprequest GET ?cmd=capabilities
> user-agent: test
> EOF
using raw connection to peer
s> GET /?cmd=capabilities HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 200 Script output follows\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-0.1\r\n
Augie Fackler
tests: glob away content-length changes relating to missing zstd bindings...
r37838 s> Content-Length: *\r\n (glob)
Gregory Szorc
wireproto: define and implement HTTP handshake to upgrade protocol...
r37575 s> \r\n
s> batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
A proper request without the API server enabled returns the legacy response
$ sendhttpraw << EOF
> httprequest GET ?cmd=capabilities
> user-agent: test
> x-hgupgrade-1: foo
> x-hgproto-1: cbor
> EOF
using raw connection to peer
s> GET /?cmd=capabilities HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> x-hgproto-1: cbor\r\n
s> x-hgupgrade-1: foo\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 200 Script output follows\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-0.1\r\n
Augie Fackler
tests: glob away content-length changes relating to missing zstd bindings...
r37838 s> Content-Length: *\r\n (glob)
Gregory Szorc
wireproto: define and implement HTTP handshake to upgrade protocol...
r37575 s> \r\n
s> batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
Restart with just API server enabled. This enables serving the new format.
$ killdaemons.py
$ cat error.log
$ cat >> server/.hg/hgrc << EOF
> [experimental]
> web.apiserver = true
> EOF
Gregory Szorc
wireproto: implement capabilities for wire protocol v2...
r37551 $ hg -R server serve -p $HGPORT -d --pid-file hg.pid -E error.log
$ cat hg.pid > $DAEMON_PIDS
Gregory Szorc
wireproto: define and implement HTTP handshake to upgrade protocol...
r37575 X-HgUpgrade-<N> without CBOR advertisement uses legacy response
$ sendhttpraw << EOF
> httprequest GET ?cmd=capabilities
> user-agent: test
> x-hgupgrade-1: foo bar
> EOF
using raw connection to peer
s> GET /?cmd=capabilities HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> x-hgupgrade-1: foo bar\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 200 Script output follows\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-0.1\r\n
Augie Fackler
tests: glob away content-length changes relating to missing zstd bindings...
r37838 s> Content-Length: *\r\n (glob)
Gregory Szorc
wireproto: define and implement HTTP handshake to upgrade protocol...
r37575 s> \r\n
s> batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
X-HgUpgrade-<N> without known serialization in X-HgProto-<N> uses legacy response
$ sendhttpraw << EOF
> httprequest GET ?cmd=capabilities
> user-agent: test
> x-hgupgrade-1: foo bar
> x-hgproto-1: some value
> EOF
using raw connection to peer
s> GET /?cmd=capabilities HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> x-hgproto-1: some value\r\n
s> x-hgupgrade-1: foo bar\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 200 Script output follows\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-0.1\r\n
Augie Fackler
tests: glob away content-length changes relating to missing zstd bindings...
r37838 s> Content-Length: *\r\n (glob)
Gregory Szorc
wireproto: define and implement HTTP handshake to upgrade protocol...
r37575 s> \r\n
s> batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
X-HgUpgrade-<N> + X-HgProto-<N> headers trigger new response format
$ sendhttpraw << EOF
> httprequest GET ?cmd=capabilities
> user-agent: test
> x-hgupgrade-1: foo bar
> x-hgproto-1: cbor
> EOF
using raw connection to peer
s> GET /?cmd=capabilities HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> x-hgproto-1: cbor\r\n
s> x-hgupgrade-1: foo bar\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-cbor\r\n
Augie Fackler
tests: glob away content-length changes relating to missing zstd bindings...
r37838 s> Content-Length: *\r\n (glob)
Gregory Szorc
wireproto: define and implement HTTP handshake to upgrade protocol...
r37575 s> \r\n
Gregory Szorc
wireprotoserver: use our CBOR encoder...
r39478 s> \xa3GapibaseDapi/Dapis\xa0Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
Gregory Szorc
stringutil: teach pprint() to indent...
r39414 cbor> {
b'apibase': b'api/',
b'apis': {},
b'v1capabilities': b'batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash'
}
Gregory Szorc
wireproto: define and implement HTTP handshake to upgrade protocol...
r37575
Restart server to enable HTTPv2
$ killdaemons.py
$ enablehttpv2 server
$ hg -R server serve -p $HGPORT -d --pid-file hg.pid -E error.log
Matt Harbison
tests: arrange for a server in wireproto-command-capabilities.t to be killed...
r37779 $ cat hg.pid > $DAEMON_PIDS
Gregory Szorc
wireproto: define and implement HTTP handshake to upgrade protocol...
r37575
Only requested API services are returned
$ sendhttpraw << EOF
> httprequest GET ?cmd=capabilities
> user-agent: test
> x-hgupgrade-1: foo bar
> x-hgproto-1: cbor
> EOF
using raw connection to peer
s> GET /?cmd=capabilities HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> x-hgproto-1: cbor\r\n
s> x-hgupgrade-1: foo bar\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-cbor\r\n
Augie Fackler
tests: glob away content-length changes relating to missing zstd bindings...
r37838 s> Content-Length: *\r\n (glob)
Gregory Szorc
wireproto: define and implement HTTP handshake to upgrade protocol...
r37575 s> \r\n
Gregory Szorc
wireprotoserver: use our CBOR encoder...
r39478 s> \xa3GapibaseDapi/Dapis\xa0Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
Gregory Szorc
stringutil: teach pprint() to indent...
r39414 cbor> {
b'apibase': b'api/',
b'apis': {},
b'v1capabilities': b'batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash'
}
Gregory Szorc
wireproto: define and implement HTTP handshake to upgrade protocol...
r37575
Request for HTTPv2 service returns information about it
$ sendhttpraw << EOF
> httprequest GET ?cmd=capabilities
> user-agent: test
> x-hgupgrade-1: exp-http-v2-0001 foo bar
> x-hgproto-1: cbor
> EOF
using raw connection to peer
s> GET /?cmd=capabilities HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> x-hgproto-1: cbor\r\n
s> x-hgupgrade-1: exp-http-v2-0001 foo bar\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-cbor\r\n
Gregory Szorc
debugcommands: perform handshake when obtaining httpv2 peer...
r37663 s> Content-Length: *\r\n (glob)
Gregory Szorc
wireproto: define and implement HTTP handshake to upgrade protocol...
r37575 s> \r\n
Gregory Szorc
wireprotoserver: use our CBOR encoder...
r39478 s> \xa3GapibaseDapi/Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xa7Ibranchmap\xa2Dargs\xa0Kpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyInamespaceBnsCnewCnewColdColdKpermissions\x81DpushKcompression\x81\xa1DnameDzlibQframingmediatypes\x81X&application/mercurial-exp-framing-0005Nrawrepoformats\x82LgeneraldeltaHrevlogv1Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
Gregory Szorc
stringutil: teach pprint() to indent...
r39414 cbor> {
b'apibase': b'api/',
b'apis': {
b'exp-http-v2-0001': {
b'commands': {
b'branchmap': {
b'args': {},
b'permissions': [
b'pull'
]
},
b'capabilities': {
b'args': {},
b'permissions': [
b'pull'
]
},
b'heads': {
b'args': {
b'publiconly': False
},
b'permissions': [
b'pull'
]
},
b'known': {
b'args': {
b'nodes': [
b'deadbeef'
]
},
b'permissions': [
b'pull'
]
},
b'listkeys': {
b'args': {
b'namespace': b'ns'
},
b'permissions': [
b'pull'
]
},
b'lookup': {
b'args': {
b'key': b'foo'
},
b'permissions': [
b'pull'
]
},
b'pushkey': {
b'args': {
b'key': b'key',
b'namespace': b'ns',
b'new': b'new',
b'old': b'old'
},
b'permissions': [
b'push'
]
}
},
b'compression': [
{
b'name': b'zlib'
}
],
b'framingmediatypes': [
b'application/mercurial-exp-framing-0005'
],
b'rawrepoformats': [
b'generaldelta',
b'revlogv1'
]
}
},
b'v1capabilities': b'batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash'
}
Gregory Szorc
wireproto: define and implement HTTP handshake to upgrade protocol...
r37575
capabilities command returns expected info
Gregory Szorc
wireproto: implement capabilities for wire protocol v2...
r37551
Gregory Szorc
debugcommands: ability to suppress logging of handshake...
r37736 $ sendhttpv2peerhandshake << EOF
Gregory Szorc
wireproto: implement capabilities for wire protocol v2...
r37551 > command capabilities
> EOF
creating http peer for wire protocol version 2
Gregory Szorc
debugcommands: ability to suppress logging of handshake...
r37736 s> GET /?cmd=capabilities HTTP/1.1\r\n
Gregory Szorc
debugcommands: perform handshake when obtaining httpv2 peer...
r37663 s> Accept-Encoding: identity\r\n
s> vary: X-HgProto-1,X-HgUpgrade-1\r\n
s> x-hgproto-1: cbor\r\n
s> x-hgupgrade-1: exp-http-v2-0001\r\n
s> accept: application/mercurial-0.1\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: Mercurial debugwireproto\r\n
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-cbor\r\n
s> Content-Length: *\r\n (glob)
s> \r\n
Gregory Szorc
wireprotoserver: use our CBOR encoder...
r39478 s> \xa3GapibaseDapi/Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xa7Ibranchmap\xa2Dargs\xa0Kpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyInamespaceBnsCnewCnewColdColdKpermissions\x81DpushKcompression\x81\xa1DnameDzlibQframingmediatypes\x81X&application/mercurial-exp-framing-0005Nrawrepoformats\x82LgeneraldeltaHrevlogv1Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
Gregory Szorc
wireproto: implement capabilities for wire protocol v2...
r37551 sending capabilities command
s> POST /api/exp-http-v2-0001/ro/capabilities HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
Gregory Szorc
wireprotov2: define response data as CBOR...
r37740 s> *\r\n (glob)
Gregory Szorc
wireprotov2: change command response protocol to include a leading map...
r37743 s> content-type: application/mercurial-exp-framing-0005\r\n
Gregory Szorc
wireproto: implement capabilities for wire protocol v2...
r37551 s> content-length: 27\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: Mercurial debugwireproto\r\n
s> \r\n
s> \x13\x00\x00\x01\x00\x01\x01\x11\xa1DnameLcapabilities
s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
Gregory Szorc
wireprotov2: change command response protocol to include a leading map...
r37743 s> Content-Type: application/mercurial-exp-framing-0005\r\n
Gregory Szorc
wireproto: implement capabilities for wire protocol v2...
r37551 s> Transfer-Encoding: chunked\r\n
s> \r\n
Gregory Szorc
wireprotov2: implement commands as a generator of objects...
r39595 s> 13\r\n
s> \x0b\x00\x00\x01\x00\x02\x011
s> \xa1FstatusBok
Gregory Szorc
wireproto: implement capabilities for wire protocol v2...
r37551 s> \r\n
Gregory Szorc
wireprotov2: implement commands as a generator of objects...
r39595 received frame(size=11; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=continuation)
s> 1cc\r\n
s> \xc4\x01\x00\x01\x00\x02\x001
s> \xa4Hcommands\xa7Ibranchmap\xa2Dargs\xa0Kpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyInamespaceBnsCnewCnewColdColdKpermissions\x81DpushKcompression\x81\xa1DnameDzlibQframingmediatypes\x81X&application/mercurial-exp-framing-0005Nrawrepoformats\x82LgeneraldeltaHrevlogv1
s> \r\n
received frame(size=452; request=1; stream=2; streamflags=; type=command-response; flags=continuation)
s> 8\r\n
s> \x00\x00\x00\x01\x00\x02\x002
s> \r\n
Gregory Szorc
wireproto: implement capabilities for wire protocol v2...
r37551 s> 0\r\n
s> \r\n
Gregory Szorc
wireprotov2: implement commands as a generator of objects...
r39595 received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
Gregory Szorc
stringutil: teach pprint() to indent...
r39414 response: [
{
b'status': b'ok'
},
{
b'commands': {
b'branchmap': {
b'args': {},
b'permissions': [
b'pull'
]
},
b'capabilities': {
b'args': {},
b'permissions': [
b'pull'
]
},
b'heads': {
b'args': {
b'publiconly': False
},
b'permissions': [
b'pull'
]
},
b'known': {
b'args': {
b'nodes': [
b'deadbeef'
]
},
b'permissions': [
b'pull'
]
},
b'listkeys': {
b'args': {
b'namespace': b'ns'
},
b'permissions': [
b'pull'
]
},
b'lookup': {
b'args': {
b'key': b'foo'
},
b'permissions': [
b'pull'
]
},
b'pushkey': {
b'args': {
b'key': b'key',
b'namespace': b'ns',
b'new': b'new',
b'old': b'old'
},
b'permissions': [
b'push'
]
}
},
b'compression': [
{
b'name': b'zlib'
}
],
b'framingmediatypes': [
b'application/mercurial-exp-framing-0005'
],
b'rawrepoformats': [
b'generaldelta',
b'revlogv1'
]
}
]
Gregory Szorc
wireproto: implement capabilities for wire protocol v2...
r37551
$ cat error.log