wiki:BoxBackupStoreProtocol

Version 2 (modified by chris, 8 months ago) (diff)

more protocol docs, formatting

Store Protocol

This is the protocol used by bbackupd to communicate with bbstored, as described in the application for a registered IANA port number (which was granted: TCP port 4186). However it is normally run on the unregistered port 2201 instead.

The protocol is wrapped in TLS, which must be negotiated first. The server verifies the CA that signed the client certificate, and the common name of the client certificate specifies the account number which it is allowed to connect to.

The protocol is a binary stream of messages (wrapped in TLS). After the handshake is complete, the client sends a request message to the server, and receives a response message in reply. Although the client could in theory send more requests without waiting for replies, it currently does not.

Code Generator

The protocol is defined by a machine-readable and human-readable text file, BackupProtocol.txt. You will likely find additional commands described in that file, which have been added since this document was written. The protocol description is parsed by makeprotocol.pl to generate the code for the client and server classes.

Messages

Each message is defined by a name, a number of flags, a list of fields (with names and types), and optional constants. For example, the Login message is defined as:

Login		2	Command(LoginConfirmed)
	int32		ClientID
	int32		Flags
	CONSTANT	Flags_ReadOnly	1

Its message ID (on the wire) is 2. It is a Command message, which means that the client will wait for a reply after sending it, which must be a LoginConfirmed? message, otherwise the client will throw an UnexpectedReply? exception. It has two fields which are 32-bit integers (ClientID and Flags) and defines a single constant, Flags_ReadOnly, with value 1.

Message types

Flags that can be listed on the first line include:

  • Command(x): the client waits for a reply after sending this, and checks that the type of the reply message is x.
  • Reply: sent by the server to the client. A message can be used as both a Command and a Reply (e.g. Finished). The same message may be a Reply for multiple Commands (e.g. Success).
  • EndsConversation?: the server stops its read-reply loop after handling such a message.
  • IsError?(Type,SubType?): always a reply. The client handles this by raising an exception in CheckReply?().
  • StreamWithCommand?: causes the server to call the three-argument form of DoCommand? (with the filtered stream as the third argument).

Message classes

The generated code includes a unique message class for each message type, listing all of its fields and their types. For example the Login message generates a BackupProtocolLogin? message class, which has two private fields (mClientID and mFlags), is constructed with values for them (by the sender), and these values can be retrieved (by the receiver). This message happens to be a Command/Request? (a client to server message), so the sender is the client, and the receiver is the server.

class BackupProtocolLogin : public BackupProtocolMessage, public BackupProtocolRequest
{
public:
        BackupProtocolLogin();
        BackupProtocolLogin(const BackupProtocolLogin &rToCopy);
        ~BackupProtocolLogin();
        int GetType() const;
        enum
        {
                TypeID = 2
        };
        enum
        {
                Flags_ReadOnly = 1
        };
        std::auto_ptr<BackupProtocolMessage> DoCommand(BackupProtocolReplyable &rProtocol,
                BackupStoreContext &rContext) const; // IMPLEMENT THIS

        std::auto_ptr<BackupProtocolMessage> DoCommand(BackupProtocolReplyable &rProtocol,
                BackupStoreContext &rContext, IOStream& rDataStream) const
        {
                THROW_EXCEPTION_MESSAGE(CommonException, NotSupported,
                        "This command requires no stream parameter");
        }
        bool HasStreamWithCommand() const { return 0; }
        void SetPropertiesFromStreamData(Protocol &rProtocol);
        int32_t GetClientID() {return mClientID;}
        int32_t GetFlags() {return mFlags;}
        BackupProtocolLogin(int32_t ClientID, int32_t Flags);
        void WritePropertiesToStreamData(Protocol &rProtocol) const;
        void SetClientID(int32_t ClientID) {mClientID = ClientID;}
        void SetFlags(int32_t Flags) {mFlags = Flags;}
        virtual void LogSysLog(const char *Action) const;
        virtual void LogFile(const char *Action, FILE *file) const;
        virtual std::string ToString() const;
private:
        int32_t mClientID;
        int32_t mFlags;
};

Client usage

The client uses the protocol by constructing a SocketStreamTLS (to connect to the server) and then creating a BackupProtocolClient? on that stream. The stream should no longer be used directly after that (at least not to read or write bytes).

    std::auto_ptr<SocketStream> apConn(new SocketStream);
    apConn->Open(Socket::TypeINET, "localhost", 2003);
    BackupProtocolClient protocol(apConn);

The client then calls Query* methods on the BackupProtocolClient? object, and gets a std::auto_ptr<message class> in reply:

    std::auto_ptr<TestProtocolSimpleReply> reply(protocol.QuerySimple(41));
    TEST_THAT(reply->GetValuePlusOne() == 42);

Messages to check the protocol version, login and logout (finish/close the connection) are in no way special, and are defined in exactly the same way as all other messages, except that a command which is defined as EndsConversation? will cause the receiver to send a reply (if it is the server) and then disconnect automatically. For example:

Finished	4	Command(Finished)	Reply	EndsConversation

Server usage

The server side, on handling a new incoming connection, constructs a BackupProtocolServer? object, and calls its DoServer? method:

void BackupProtocolServer::Connection(std::auto_ptr<SocketStream> apStream)
{
        BackupProtocolServer server(apStream);
        BackupStoreContext context;
        server.DoServer(context);
}

It also constructs a BackupStoreContext? object, which is passed to every command handler method, and can be used to store global state, such as the identity of the logged-in user and a reference to a BackupFileSystem?.

The server must also implement the DoCommand? method of each message class, for example:

std::auto_ptr<BackupProtocolMessage> BackupProtocolLogin::DoCommand(BackupProtocolReplyable &rProtocol, BackupStoreContext &rContext) const
{
        CHECK_PHASE(Phase_Login)

        // Check given client ID against the ID in the certificate certificate
        // and that the client actually has an account on this machine
        if(mClientID != rContext.GetClientID())
        {
                BOX_WARNING("Failed login from client ID " <<
                        BOX_FORMAT_ACCOUNT(mClientID) << ": "
                        "wrong certificate for this account");
                return PROTOCOL_ERROR(Err_BadLogin);
        }
...
        // Return success
        return std::auto_ptr<BackupProtocolMessage>(new BackupProtocolLoginConfirmed(clientStoreMarker, blocksUsed, blocksSoftLimit, blocksHardLimit));
}

There are two forms of DoCommand?, one for commands that take a stream after their properties (those that send variable length data, such as uploading and downloading files) and those that do not. Only one can be implemented for each command. Whether or not each command takes a stream is defined in BackupProtocol?.txt, by the presence of StreamWithCommand? on the first line of the command definition. However this is not defined for reply messages (which are not commands). It is documented as a comment on the command. The client must always and only read the stream (by calling ReceiveStream?) if the server has sent one (with SendStreamAfterCommand?).

Handshake message

Both sides send a handshake immediately on connection, and then read the other side's handshake. The message format is just a 32-byte character field:

char mIdent[32];

The ident used by both sides is currently "Box-Backup:v=C". If either side does not recognise the ident sent by the other, it will disconnect immediately.

Protocol messages

After handshaking, all messages are protocol messages, of the form:

Object header:

u_int32_t mObjSize; u_int32_t mObjType; (followed by mObjSize bytes of data) (followed by a stream, but only if the command's flags include StreamWithCommand?)

Message Types : handshake, command, reply, error, stream.

Since each command has its own sequence of parameters, given above, you could regard each command as a message type. Message opcodes : Operation codes are given in brackets after each message description above. Message Sequences : After handshake, the client sends Command messages to the server, and receives Reply messages (one of which is Error).

Commands (requests) and replies

Any command (except Handshake) may receive an Error reply. Otherwise, the expected reply type for each message is defined in BackupProtocol?.txt:

client message expected response from server
Handshake Handshake
Version Version
Login LoginConfirmed?
Finished Finished
SetClientStoreMarker? Success
GetObject? Success (followed by stream)
MoveObject? Success
GetObjectName? ObjectName?
CreateDirectory? Success
ListDirectory? Success + stream
ChangeDirAttributes? Success
DeleteDirectory? Success
UndeleteDirectory? Success
StoreFile? + stream Success
GetFile? Success + stream
SetReplacementFileAttributes? + stream Success
DeleteFile? Success
GetBlockIndexByID Success + stream
GetBlockIndexByName? Success + stream
GetAccountUsage? AccountUsage?
GetIsAlive? IsAlive?

Error

Error message (mObjType = 0):

int32 Type int32 SubType?

where Type is always 1000 and SubType? is one of:

CONSTANT ErrorType? 1000 CONSTANT Err_WrongVersion 1 CONSTANT Err_NotInRightProtocolPhase 2 CONSTANT Err_BadLogin 3 CONSTANT Err_CannotLockStoreForWriting 4 CONSTANT Err_SessionReadOnly 5 CONSTANT Err_FileDoesNotVerify 6 CONSTANT Err_DoesNotExist 7 CONSTANT Err_DirectoryAlreadyExists 8 CONSTANT Err_CannotDeleteRoot 9 CONSTANT Err_TargetNameExists 10 CONSTANT Err_StorageLimitExceeded 11 CONSTANT Err_DiffFromFileDoesNotExist 12 CONSTANT Err_DoesNotExistInDirectory 13 CONSTANT Err_PatchConsistencyError 14

Version

Version message (mObjType = 1):

int32 Version

Login

Login Message (mObjType = 2):

int32 ClientID int32 Flags CONSTANT Flags_ReadOnly 1

The successful reply is a LoginConfirmed? message.

LoginConfirmed?

LoginConfirmed? message (mObjType = 3):

int64 ClientStoreMarker? int64 BlocksUsed? int64 BlocksSoftLimit? int64 BlocksHardLimit?

Finished

Finished message (mObjType = 4):

no data fields

The client sends this as the last command. The server replies with one of the same, and then closes the connection.

Success

Success message (mObjType = 5):

int64 ObjectID

The successful response to most commands. The ObjectID field is not always used, and its meaning depends on the command that it's replying to.

SetClientStoreMarker?

SetClientStoreMarker? message (mObjType = 6):

int64 ClientStoreMarker?

GetObject?

GetObject? (mObjType = 10):

int64 ObjectID CONSTANT NoObject? 0 # reply has stream following, if ObjectID != NoObject?

MoveObject?

MoveObject? (mObjType = 11):

int64 ObjectID int64 MoveFromDirectory? int64 MoveToDirectory? int32 Flags Filename NewFilename?

CONSTANT Flags_MoveAllWithSameName 1 CONSTANT Flags_AllowMoveOverDeletedObject 2

GetObjectName?

GetObjectName? (mObjType = 12):

int64 ObjectID int64 ContainingDirectoryID CONSTANT ObjectID_DirectoryOnly 0

# set ObjectID to ObjectID_DirectoryOnly to only get info on the directory

ObjectName?

ObjectName? (mObjType = 13):

int32 NumNameElements? int64 ModificationTime? int64 AttributesHash? int16 Flags # NumNameElements? is zero if the object doesn't exist CONSTANT NumNameElements_ObjectDoesntExist 0 # a stream of Filename objects follows, if and only if NumNameElements? > 0

CreateDirectory?

CreateDirectory? (mObjType = 20):

int64 ContainingDirectoryID int64 AttributesModTime? Filename DirectoryName? # stream following containing attributes

ListDirectory?

ListDirectory? (mObjType = 21):

int64 ObjectID int16 FlagsMustBeSet? int16 FlagsNotToBeSet? bool SendAttributes? CONSTANT Flags_INCLUDE_EVERYTHING -1 CONSTANT Flags_EXCLUDE_NOTHING 0 CONSTANT Flags_EXCLUDE_EVERYTHING 15 CONSTANT Flags_File 1 CONSTANT Flags_Dir 2 CONSTANT Flags_Deleted 4 CONSTANT Flags_OldVersion 8 CONSTANT RootDirectory? 1

ChangeDirAttributes?

ChangeDirAttributes? (mObjType = 22):

int64 ObjectID int64 AttributesModTime? # stream following containing attributes

DeleteDirectory?

DeleteDirectory? (mObjType = 23):

int64 ObjectID

UndeleteDirectory?

UndeleteDirectory? (mObjType = 24):

int64 ObjectID

StoreFile?

StoreFile? (mObjType = 30):

int64 DirectoryObjectID int64 ModificationTime? int64 AttributesHash? int64 DiffFromFileID # 0 if the file is not a diff Filename Filename # then send a stream containing the encoded file

GetFile?

GetFile? (mObjType = 31):

int64 InDirectory? int64 ObjectID # error returned if not a file, or does not exist # reply has stream following, containing an encoded file IN STREAM ORDER # (use GetObject? to get it in file order)

SetReplacementFileAttributes?

SetReplacementFileAttributes? (mObjType = 32):

int64 InDirectory? int64 AttributesHash? Filename Filename # stream follows containing attributes

DeleteFile?

DeleteFile? (mObjType = 33):

int64 InDirectory? Filename Filename # will return 0 if the object couldn't be found in the specified directory

GetBlockIndexByID

GetBlockIndexByID (mObjType = 34):

int64 ObjectID

GetBlockIndexByName?

GetBlockIndexByName? (mObjType = 35):

int64 InDirectory?

GetAccountUsage?

GetAccountUsage? (mObjType = 40):

# no data members

AccountUsage?

AccountUsage? (mObjType = 41):

int64 BlocksUsed? int64 BlocksInOldFiles? int64 BlocksInDeletedFiles? int64 BlocksInDirectories? int64 BlocksSoftLimit? int64 BlocksHardLimit? int32 BlockSize?

GetIsAlive?

GetIsAlive? (mObjType = 42):

# no data members

A ping/keepalive request.

IsAlive?

IsAlive? (mObjType = 43):

# no data members

A ping/keepalive response.