Synchronization

Introduction

XPipe can synchronize all connection configuration data across multiple installations. It uses your own remote Git repository for that purpose. You can synchronize with this repository in all XPipe application instances. Every change you make in one instance will be reflected in the repository. This approach means that you have full control over your data, as you can freely choose how and where to host this repository.

Setup

To start off, you need to create an empty git repository with your git provider of choice. This repository should be private:

Next, copy and paste the URL into the XPipe remote repository setting:

You can check if the configuration works by clicking on the Test button.

Ensure that your repository visibility is set to private to prevent data leakage.

You don't have to link a remote repository. Just enabling git sync without one will create and maintain a local repository. This can be useful if you're just looking for having a local history.

You also need to have a locally installed git client ready on your local machine. You can try running git --help in a local terminal to check. If you don't have one, you can visit https://git-scm.com to install git.

There are multiple ways to authenticate. Most repositories use the git over HTTPS scheme where you have to specify a username and password / token. Some providers also support the SSH protocol, which is supported as well by XPipe.

Password / token auth

For the usual password / token authentication when using an HTTPS git URL, you will need to provide credentials. If no credentials are configured, XPipe will ask you for your credentials when they are required. If you have already configured your git CLI to store your credentials, then XPipe should pick them up automatically. If you haven't done that yet, you probably don't want to enter your credentials every single time in XPipe.

To automate the authentication process in the git CLI in general, there are multiple ways to store your git credentials and handle authentication. Many special tools like the GitHub CLI handle the whole authentication automatically for you when installed. Some newer git client versions can also authenticate via special web services where you just have to log in into your account in your browser. These approaches will store your credentials for your git client using a credential helper. For more information, see:

Alternatively, if no credentials are configured for your local git client yet, you have the option to specify them in XPipe itself without making them available to the entire git CLI client:

Nowadays, most providers require a personal access token (PAT) to authenticate from the command-line instead of traditional passwords. You can find common PAT pages here:

GitHub: Personal access tokens (classic)
GitLab: Personal access token
BitBucket: Personal access token
Gitea: Settings -> Applications -> Manage Access Tokens section Set the token permission for repository to Read and Write. The rest of the token permissions can be set as Read. XPipe just needs enough permissions to commit changes to the private repository.

SSH auth

As an alternative to this password / token authentication, git also supports the SSH protocol. With such a setup, you can authenticate to your remote git repository using an SSH key. For this, you can use the full functionality of XPipe's key handling, including advanced key retrieval methods such as agents. XPipe will automatically detect if you enter an SSH git URL and show additional options for authentication:

You can find common SSH docs pages here:

GitHub: Adding a new SSH key to your GitHub account
GitLab: Use SSH keys to communicate with GitLab
BitBucket: Provide Bitbucket Cloud with your public key

First commits

If everything works out, XPipe should push an initial commit to your remote repository once you restart XPipe with a valid git configuration.

By default, no connection categories are set to sync so that you have explicit control on what connections to sync. So at the start, your remote repository will be empty, which is intentional. You can start syncing connections and categories by marking them explicitly to sync.

Syncing categories

To have your connections of a category put inside your git repository, you first need to change its sync configuration. In your Connections tab under the category overview on the left side, you can open the category configuration menu either by right-clicking the category or click on the ⚙️ icon when hovering over the category, and then clicking on the 🔧 configure button.

category-config

Then, set the Sync with git repository value to Yes to sync the category and connections to your git repository. This will add all syncable connections in that category to the git repository. The sync settings for a category are inherited by default from its parent if not explicitly set.

sync-add

Adding files to git

When everything is set up, you have the option to add any additional files such as SSH keys, identities, and various other data files to git as well. Next to every file chooser is a git button that will add the file to the git repository:

sync-file

These files are then added to the git vault data directory and committed in encrypted form. This means that any files you add to git, such as keys, are only committed in encrypted form just as other secrets.

Local connections

Any connection located under the local machine cannot be shared as it refers to connections and data that are only available on the local system.

Certain connections that are based on a local file, for example, SSH configs, can be shared via git if the underlying data, in this case the file, have been added to the git repository as well.

GPG signing

If you have enabled GPG signing of your git commits, then this should work as expected as XPipe use this the same configuration as your git CLI that you normally use.

The only more complicated case is on Windows where you can have potentially two gpg executables with the git bash gpg and the native Gpg4win executable. By default, XPipe will choose the gpg client which has keys configured in it, under the assumption that you only use one of the two clients. If you switch between using both gpg clients, make sure that your git GPG key is available in gpg4win, as this client will be used if both clients have keys added.

Local repositories

If you don't want to host your remote git repository on some server somewhere and instead prefer a local option, for example, a flash drive, to sync data between machines, you can do that as well.

Git and XPipe support a file:// URL as well in addition to the more commonly used https and SSH URLs. You can create a local sync repository as follows:

cd <external repo dir>
git init --bare

Note that the --bare option is important, without it, git can't push to the repository.

Then, you can configure a file URL in XPipe:

Note that the syntax for a file URL is file://<absolute path> where the absolute path on Windows has to be adjusted to use forward slashes and start with one as well. The path /home/user/xpipe_repo on Linux would become file:///home/user/xpipe_repo and the path C:\Users\User\xpipe_repo on Windows would become file:///C:/Users/User/xpipe_repo.

Reverting to a previous commit

In case anything goes wrong, or you just want to restore the vault at an earlier date, you can do so easily using the normal git reset functionality. The first step is to find the SHA of the commit you want to revert to. This will look like this: 42e2e5af9d49de268cd1fda3587788da4ace418a. You can find the commit SHA in any web interface of git your provider or on the command line with git log.

Once you have obtained the SHA, you can run the following commands:

cd "<User Home>/.xpipe/storage"
git reset --hard <SHA>
git push --force -u remote-storage HEAD

The force push is required to overwrite the remote repository. Without the push, XPipe would just pull the changes from the remote repository again.

Note that if multiple instances are synced to this repository, you will have to perform the reset on all instance repositories. Otherwise, starting another XPipe instance where the local reset was not performed, will push the reverted changes again.

Team vaults

The git synchronization also supports multiple different users or groups for when you are working with your team. It allows each team member to share access to systems that are designated to be available to all users, while also allowing each user or group to keep selected systems and authentication options private.

Team vaults are supported starting from the Professional plan.

Vault users and groups

To start off, each team member first has to sync to the same git repository. With no team vault settings configured yet, all members who sync with that repository will have full access to the same connections with this configuration.

You can then configure the vault authentication in the Vault settings menu. First, you can choose between user-based and group-based authentication.

User-based authentication

Each member has their own passphrase to either remember or stored in a password manager. With this passphrase, each team member can choose to encrypt certain connections and identities to only be available for themselves. This is suitable for smaller teams and requires minimal setup.

Group-based authentication

Each member can be assigned a group, which requires a group secret to unlock. With vault groups, this secret can be queried automatically, e.g. via commands, HTTP requests, password manager, and more. It is essentially the right type of authentication for larger teams where an organization administrator can configure the group assignments from a central location.

It is essentially what many people refer to as role-based access control (RBAC), however, that name was not chosen, to distinguish it from that as each member can only be a member of one group. There is not a flexible system of multiple roles that can be assigned to each user.

That administrator can, for example, hook up the vault authentication to an Azure Key Vault and manage access via Azure Active Directory permissions and the Azure CLI.

Configuring access

Once the vault authentication is configured, you can choose the scope when creating synced identities.

These identities can be configured to either be global or personal / group-only identities:

Then, configuring a connection to use this personal identity will automatically make XPipe also encrypt the connection itself and all subconnections with your personal key as well. This will result in only your user having access to the identity and all associated connections. Any kind of key files associated with personal identities are also added to the repository using your personal key for encryption.

Other vault users can still see that changes are made to all your personal connections and identities. However, they can't decrypt the changes itself and will only see binary diffs in the repository.

Changing authentication secrets

If, at a later date, you want to change the passphrase / group secrets of a vault user / group, you can do so at the edit button for the user / group entry. This action will re-encrypt the entire vault data and restart. This operation will take a while as this is a compute-intensive operation.

You can also delete a user / group. This will perform the same operation with the main difference that all personal / group-only connections will be encrypted with the generic vault key again, meaning that they will be accessible to everyone.

Additional considerations

A team vault is essentially just a git repository consisting out of files that are encrypted with various keys. There is no database or other service involved. This introduces a few limitations that it is good to be aware of.

Trust

This authentication system for encryption and decryption is not the most complex system you will find when compared to other solutions. It is designed to be relatively simple and robust. Each team member has full control over all connection entries they have been granted access for, meaning that they can modify them, delete them, and inspect the used passwords and keys for them. There is no fine-grained permission system akin to an administrator only granting something like read-only permissions to certain connections. Each team member has to be trusted with the connection access that they are given. If you are looking to minimize possible leaks of secrets in this scenario, using security keys like Yubikeys for server authentication is the best option. This will prevent the possibility of any used private keys being leaked when an individual user's system is compromised.

Scaling limits

Furthermore, this vault sync system is not designed for an arbitrary number of users. The maximum amount of users who can share a vault is often limited by git itself. If you have something like 50 users working on one shared vault with the same connections, there will probably be a lot of continuous pushing and pulling going on, including a lot of merge conflicts at some point. The git workflow does not scale well when working on the same data, this is also true for other normal repositories.

Multiple vaults

A workspace is a completely separate data storage location, independent of others. It lives in its own directory on your local machine and has its own connections, settings, and vault. They can be created in the settings menu:

For fine-grained access control, like a few team members having access to a few specific connections outside of the normal group borders, you can try using multiple workspaces. In many cases, a separate workspace with a separate sync repository configured is the best option to solve these cases. Separate workspaces also work well when reducing the aforementioned increasing load on git repositories when more and more users start syncing to it.

Merge conflicts

When working in a team, an occasional merge conflict might occur when another user pushes changes while your local XPipe instance is running. XPipe will automatically check in the background if someone else pushed something to the remote and will attempt to perform a fast-forward merge automatically. You will see this notification when this happens:

In most cases, you won't have to do anything as it can be handled automatically. However, if you and another team member modify the same connection configuration at the same time, you might receive a merge conflict error that can't be solved automatically. In this case, you can choose between either overriding your changes or the external changes.

Example setups

This is how each variant of the team vault authentication could look like for group-based authentication.

Password manager

Each group member must have access to a supported password manager. Assuming that the secrets in that password manager are managed by an organization administrator, they can make the group secret available to group members in the password manager. If a password manager is configured in the XPipe settings, you can then retrieve that group secret to unlock the vault with the necessary name or key of the secret in the password manager.

For details on how to use password managers in XPipe, see the password manager page.

Command

If your secret is not stored in a password manager, and instead at the central location with built-in access control like an Azure Key Vault, the command option is probably the right one for you. It will simply execute a command and read the secret from stdout. Assuming that the necessary CLI tools are available on your system, this gives you a lot of flexibility on how and where to retrieve the secret from.

For example, you could follow this guide to create an Azure Key Vault and retrieve the command with the Azure CLI. Many CLI tools like the Azure CLI return json output. You can extract the raw values using a tool like jq. For example, if you create a new key vault called xpipe-auth-vault and the secret called my-group-secret, you can then pass the secret to XPipe by specifying the following command with the jq extraction for the raw value:

az keyvault secret show --name "my-group-secret" --vault-name "xpipe-auth-vault" | jq -r .value

The access control is built in as the Azure CLI requires you to authenticate to your Azure Active Directory setup, which automatically gives you the permissions you were assigned to. You just need to be assigned permissions to read the associated secret vault, and this command will work.

File-based secret

If your organization administrator can manage the file system of devices, creating a simple file containing the group secret is also an option. Although this is not as secure as the other ones. This file can then be used to obtain the groups secret. The file path in XPipe also supports ~ on all operating systems to allow you to refer to files in your home directory regardless of which operating system you're using.

HTTP request

You can also run a small HTTP server to handle the group secret authentication for you. Essentially, you can specify an HTTP URL in XPipe that will get called with a POST request. XPipe expects the group secret to be written in the body of the response. The access control can be implemented for example by matching the request IPs against a fixed table, for example, in an office set up with fixed device IPs. This approach requires a bit of manual scripting to create this HTTP server, but can be a good solution for certain setups.

If the configured authentication method fails, or you don't have one configured yet, you can also just log in to the group manually in the login dialog.

On this page