Migrate from self-hosted to PostHog Cloud

If you're attempting this migration, feel free to ask questions and provide feedback via the PostHog Communty Slack workspace or a GitHub issue. You can also raise a support ticket in the app.

Requirements

• An existing PostHog instance running at least 1.30.0 (For upgrade instructions, take a look at this guide)
• A new project on PostHog Cloud

Approach

This migration has two parts:

1. Migrate your events, this will also create the necessary person, person distinct ID, and related records.
2. Migrate your metadata (projects, dashboards, insights, actions, cohorts, feature flags, experiments, annotations).

Note: before migrating events, make sure to deactivate the GeoIP app on your destination PostHog instance.

Option 1 for events migration: self-hosted migration tool (alpha)

To migrate your events, you can read data directly from your ClickHouse cluster and ingest the data with the Python library using our self-hosted migration tool.

First, clone the repo and install the requirements.

git clone https://github.com/PostHog/posthog-migration-tools
cd posthog-migration-tools
pip3 install -r requirements.txt

Next, run the migration script with your ClickHouse details, PostHog details, start date, end date, and fetch limit.

python3 ./migrate.py \
   --clickhouse-url https://some.clickhouse.cluster:8443 \
   --clickhouse-user default \
   --clickhouse-password some-password \
   --clickhouse-database posthog \
   --team-id 1234 \
   --posthog-url https://app.posthog.com \
   --posthog-api-token "abx123" \
   --start-date 2023-06-18T13:00:00Z \
   --end-date 2023-06-18T13:10:00 \
   --fetch-limit 10000

This script prints a "cursor" in the case the migration fails. It can be used to resume from where it got to with by adding the --cursor argument to the command above.

python3 ./migrate.py \
   --clickhouse-url https://some.clickhouse.cluster:8443 \
   --clickhouse-user default \
   --clickhouse-password some-password \
   --clickhouse-database posthog \
   --team-id 1234 \
   --posthog-url https://app.posthog.com \
   --posthog-api-token "abx123" \
   --start-date 2023-06-18T13:00:00Z \
   --end-date 2023-06-18T13:10:00 \
   --fetch-limit 10000 \
   --cursor the-cursor-value-from-the-output

Note: this script adds a $lib property of posthog-python, overriding any $lib property already set.

Option 2 for events migration: the replicator

To migrate your events, we can use the PostHog Replicator app, which enables us to stream both incoming and historical events to PostHog Cloud

Installing the Replicator

Start by logging in to your existing self-hosted instance and navigating to the 'Apps' tab. Next, search for the 'Replicator' app and install it if it isn't already.

For the configuration:

• Host: either app.posthog.com for US or eu.posthog.com for EU
• Project API Key: the API key for the project in Cloud

After clicking save and activating it (toggle on the left side), the Replicator will begin forwarding incoming events to Cloud. Note that it may take around 15-20 minutes for the first events to arrive.

Exporting historical events

Now that the Replicator is sending new events, we'll set up a batch job to export past events. First, go back to the configuration menu and click the gear next to "Export historical events" under "Jobs", which will open the following popup.

Select a start and end date on the calendar for the range you would like to export. Once you've chosen a date range, click "Run job now" to start the export.

Depending on the amount of data you have stored, this may take a fair amount of time. You can monitor the progress by clicking the "App metrics" button (graph) and going to the "Historical exports" tab.

Switching tracking in you product

Now that we've migrated our events, the next step is to switch over tracking within your product to direct any new events to your new PostHog cloud instance.

First, if you are using the Replicator, ensure that it is still running on your self-hosted instance. Any events sent to your old instance will be forwarded as long as it is running.

Next, begin swapping out the API key and host within all the areas of your product that you track. Once done, events using the new API key will go directly to your Cloud instance.

Once this is complete and you've double-checked that no more events are coming to the old instance and that all historical exports have finished, you can deactivate the Replicator app.

Migrate your metadata

Follow the instructions on the PostHog migrate metadata repo:

Note This process has the following caveats:

1. Project API Key. You'll need to replace the API key in your code with the new API key.
2. "created by" information. Every object will appear as if it was created by you.
3. "created at" information. Every object will appear as if it was created on the time you ran this script.

1. Clone the repo and cd into it
   Terminal

   git clone https://github.com/PostHog/posthog-migrate-meta
   cd posthog-migrate-meta

2. Install the dependencies by running yarn
3. Run the script
   Terminal

   ts-node --source [posthog instance you want to migrate from] --sourcekey [personal api key for that instance] --destination [posthog instance you want to migrate to.] --destinationkey [personal api key for destination instance]

For more information on the options see the repo's readme

Migrating your custom apps

If the app was previously extracting events using the scheduled tasks api:

1. (fastest option) you could move this logic from the app into scheduled tasks on your server potentially using a cron job
2. If you can make your app generalizable enough that others can benefit then submit your app to the store.
   • To make it generalizable you'll want to convert anything specific to your configuration into a plugin.json config value

If the app was transforming events before ingestion:

1. (fastest option) you could move this logic from the app into your client before you send the event
2. If you can make your app generalizable enough that others can benefit then submit your app to the store.
   • To make it generalizable you'll want to convert anything specific to your configuration into a plugin.json config value

If the app was used to send events to a custom destination:

1. (fastest option) Convert your app to work as a webhook. We are soon releasing a webhook destination. You can subscribe for updates on the roadmap

2. If you can make your app generalizable enough that others can benefit then submit your app to the store.

   • To make it generalizable you'll want to convert anything specific to your configuration into a plugin.json config value

   If the options above don't work and you were previously paying a substantial amount self-hosting then email us at hey@posthog.com with a link to the public github repo and we can see if it's appropriate as a private cloud app.