0 Title
DIP-1 Common protocol description
1 Intro
This document describes the Data Interchange Protocol (DIP), designed to
exchange filtered data that can be stored as a graph structure between
network nodes.
2 Purpose
This protocol will be used to share the models (originally imageboard posts)
across multiple servers. The main differnce of this protocol is that the node
can specify what models it wants to get and from whom. The nodes can get
models from a specific server, or from all except some specific servers. Also
the models can be filtered by timestamps or tags.
3 Protocol description
The node requests other node's changes list since some time (since epoch if
this is the start). The other node sends a list of post ids or posts in the
XML format.
Protocol version is the version of the sync api. Model version is the version
of data models. If at least one of them is different, the sync cannot be
performed.
The node signs the data with its keys. The receiving node saves the key at the
first sync and checks it every time. If the key has changed, the info won't be
saved from the node (or the node id must be changed). A model can be signed
with several keys but at least one of them must be the same as in the global
ID to verify the sender.
Each node can have several keys. Nodes can have shared keys to serve as a pool
(several nodes with the same key).
Each post has an ID in the unique format: key-type::key::local-id
All requests pass a request type, protocol and model versions, and a list of
optional arguments used for filtering.
Each request has its own version. Version consists of 2 numbers: first is
incompatible version (1.3 and 2.0 are not compatible and must not be in sync)
and the second one is minor and compatible (for example, new optional field
is added which will be igroned by those who don't support it yet).
Post edits and reflinks are not saved to the sync model. The replied post ID
can be got from the post text, and reflinks can be computed when loading
posts. The edit time is not saved because a foreign post can be 'edited' (new
replies are added) but the signature must not change (so we can't update the
content). The inner posts can be edited, and the signature will change then
but the local-id won't, so the other node can detect that and replace the post
instead of adding a new one.
3.1 Requests
There is no constraint on how the server should calculate the request. The
server can return any information by any filter and the requesting node is
responsible for validating it.
The server is required to return the status of request. See 3.2 for details.
3.1.1 pull
"pull" request gets the desired model id list by the given filter (e.g. thread, tags,
author)
Sample request is as follows:
<?xml version="1.1" encoding="UTF-8" ?>
<request version="1.0" type="pull">
<model version="1.0" name="post">
<timestamp_from>0</timestamp_from>
<timestamp_to>0</timestamp_to>
<tags>
<tag>tag1</tag>
</tags>
<sender>
<allow>
<key>abcehy3h9t</key>
<key>ehoehyoe</key>
</allow>
<!-- There can be only allow block (all other are denied) or deny block (all other are allowed) -->
</sender>
</model>
</request>
Under the <model> tag there are filters. Filters for the "post" model can
be found in DIP-2.
Sample response:
<?xml version="1.1" encoding="UTF-8" ?>
<response>
<status>success</status>
<models>
<id key="id1" type="ecdsa" local-id="1" />
<id key="id1" type="ecdsa" local-id="2" />
<id key="id2" type="ecdsa" local-id="1" />
<id key="id2" type="ecdsa" local-id="5" />
</models>
</response>
3.1.2 get
"get" gets models by id list.
Sample request:
<?xml version="1.1" encoding="UTF-8" ?>
<request version="1.0" type="get">
<model version="1.0" name="post">
<id key="id1" type="ecdsa" local-id="1" />
<id key="id1" type="ecdsa" local-id="2" />
</model>
</request>
Id consists of a key, key type and local id. This key is used for signing and
validating of data in the model content.
Sample response:
<?xml version="1.1" encoding="UTF-8" ?>
<response>
<!--
Valid statuses are 'success' and 'error'.
-->
<status>success</status>
<models>
<model name="post">
<!--
Content tag is the data that is signed by signatures and must
not be changed for the post from other node.
-->
<content>
<id key="id1" type="ecdsa" local-id="1" />
<title>13</title>
<text>Thirteen</text>
<thread><id key="id1" type="ecdsa" local-id="2" /></thread>
<pub-time>12</pub-time>
<!--
Images are saved as attachments and included in the
signature.
-->
<attachments>
<attachment mimetype="image/png">TWFuIGlzIGRpc3Rpbmd1aXNoZWQsIG5vdCBvbmx5IGJ5I</attachment>
</attachments>
</content>
<!--
There can be several signatures for one model. At least one
signature must be made with the key used in global ID.
-->
<signatures>
<signature key="id1" type="ecdsa" value="dhefhtreh" />
<signature key="id45" type="ecdsa" value="dsgfgdhefhtreh" />
</signatures>
<attachment-refs>
<attachment-ref ref="TWFuIGlzIGRpc3Rpbmd1aXNoZWQsIG5vdCBvbmx5IGJ5IGhpcyByZWFzb24sIGJ1dCBieSB0"
url="/media/images/12345.png" />
</attachment-refs>
</model>
<model name="post">
<content>
<id key="id1" type="ecdsa" local-id="id2" />
<title>13</title>
<text>Thirteen</text>
<pub-time>12</pub-time>
<edit-time>13</edit-time>
<tags>
<tag>tag1</tag>
</tags>
</content>
<signatures>
<signature key="id2" type="ecdsa" value="dehdfh" />
</signatures>
</model>
</models>
</response>
3.1.3 put
"put" gives a model to the given node (you have no guarantee the node takes
it, consider you are just advising the node to take your post). This request
type is useful in pool where all the nodes try to duplicate all of their data
across the pool.
3.2 Responses
3.2.1 "not supported"
If the request if completely not supported, a "not supported" status will be
returned.
3.2.2 "success"
"success" status means the request was processed and the result is returned.
3.2.3 "error"
If the server knows for sure that the operation cannot be processed, it sends
the "error" status. Additional tags describing the error may be <description>
and <stack>.