Upgrading self-hosted live chat without losing a conversation

Self-hosting your website's live chat is the right call if you care about who holds your customer conversations. It also means the upgrades are yours. Chatwoot moves quickly, and a chat platform that falls a long way behind becomes a project to upgrade instead of a routine. This is the method we use to keep a self-hosted Chatwoot current without losing a single conversation, and without a widget that looks fine to your monitoring while it fails for your visitors.

Know what you are upgrading

A typical container deployment of Chatwoot has five moving parts:

• the web container (the Rails application and the API),
• the worker container (Sidekiq, which sends notifications and runs background jobs),
• PostgreSQL (conversations, contacts, messages, configuration),
• Redis (queues and live state),
• a reverse proxy such as nginx in front, terminating TLS and passing WebSocket traffic.

On our platforms these run as Podman containers, started by a systemd unit so the stack comes back in order after a reboot. Write down, before you begin, the image tags of every container, the database schema version and the operating system's package state. You will need them for the rollback.

Step one: take a baseline you can compare against

Before you touch anything, record what the database holds:

podman exec chatwoot-postgres psql -U chatwoot -d chatwoot_production -At -c "
  select 'conversations', count(*) from conversations union all
  select 'contacts', count(*) from contacts union all
  select 'users', count(*) from users union all
  select 'inboxes', count(*) from inboxes union all
  select 'messages', count(*) from messages;"

Save the output with a timestamp. After the upgrade and again after a reboot, run the same query. Conversations, users, inboxes and messages should be identical; contacts may grow slightly, because live visitors keep arriving. Anything else is a reason to stop.

Step two: quiesce, then dump

A dump taken while workers are writing is a dump of a moving target. Stop the web and worker containers, leave the database running, then dump:

podman stop chatwoot-web chatwoot-worker
podman exec chatwoot-postgres pg_dump -U chatwoot -Fc chatwoot_production > chatwoot_production.dump

Copy alongside it the environment file, the compose file, the proxy configuration and the systemd unit, into a dated directory. Keep the previous container images pulled; do not let a cleanup job remove them until the new version has run for a while.

Step three: write the rollback before you need it

Write the exact commands to return to the previous version, now, while you are calm. For Chatwoot the rollback is: stop the stack, restore the previous compose and environment files, start only the database and Redis, drop and recreate the database, restore the dump, start the stack, and confirm the API reports the old version.

podman exec chatwoot-postgres dropdb -U chatwoot chatwoot_production
podman exec chatwoot-postgres createdb -U chatwoot chatwoot_production
podman exec -i chatwoot-postgres pg_restore -U chatwoot -d chatwoot_production --no-owner --no-privileges < chatwoot_production.dump

Note that the database must go back with the application. Chatwoot runs schema migrations on upgrade, and an old application against a migrated schema is not a supported state.

Step four: upgrade in one deliberate move

Change the image tag for web and worker together, pull, and run the database preparation step before the application starts serving:

podman-compose pull
podman-compose run --rm web bundle exec rails db:chatwoot_prepare
podman-compose up -d

If you are several releases behind, read the release notes for every release you are crossing, not only the newest one.

This is also a good moment to bring the host's operating system packages current and reboot, so the platform and its foundation stay in step. Do it in the same window, after the application is verified once, so you know which change caused what.

Step five: check from the visitor's side

Most monitoring checks that a URL returns a successful status. A chat widget can pass that and still be broken for every visitor. Check the things a browser actually does:

1. The API answers and reports the new version, with its queue and data services healthy: curl -s https://chat.example.com/api.
2. The login page and the super admin sign-in return successfully.
3. The widget script loads: curl -sI https://chat.example.com/packs/js/sdk.js.
4. The widget page loads with a real browser user agent. Recent Chatwoot releases skip requests that look like crawlers, so a bare curl may get an empty response that is not a fault. Test with a browser, or send a browser user agent.
5. The live connection upgrades: a WebSocket request to /cable should answer with 101 Switching Protocols. If your proxy does not pass the upgrade headers, chat looks loaded but messages never arrive in real time.
6. Attachments and avatars load, which proves file storage is mounted and readable.
7. The embedding headers are still right. The Content-Security-Policy frame-ancestors directive must still list the websites that embed the widget.

Then run the record counts again, reboot, and run everything once more. A stack that only works until the next reboot has not been upgraded.

One container detail that saves hours

Shared volumes and SELinux. If two containers share a storage volume (the web and worker containers commonly share the file storage used for attachments), mount it with the shared label :z, not the private label :Z. A private label is applied for one container at a time, and the other container is then denied access to files it needs, which shows up as server errors on images and attachments rather than as an obvious permission problem. Keep SELinux enforcing; label the volume correctly instead.

After an upgrade, send a test notification and confirm it arrives.

What to check on your own chat platform

• Do you know which release you run, and how far behind it is?
• Do you have a dump from before your last upgrade, and the images to go with it?
• Is your rollback written down, or only in someone's head?
• Does your monitoring load the widget like a browser, including the WebSocket?
• Do your record counts match before and after each change?

Run through that list once and the next upgrade becomes a routine evening task instead of a risk.

If you want your live chat run this way, on your own instance, our engineers can help. Talk to an engineer

Related: Managed live chat on Chatwoot, your own instance · Platform currency and upgrade discipline