Troubleshooting

The 15 most likely problems, each with the commands to find the cause.

Before you start

• Log in to the main server as root.
• Find the live API container. After the first update its name is xtreamgo-api-blue or xtreamgo-api-green. The commands xtreamgo logs api, xtreamgo shell and xtreamgo health follow the live container by themselves. The plain docker commands in this document use the variable $API.

API=$(docker ps --filter name=xtreamgo-api --format '{{.Names}}' | head -1); echo "$API"

• Run the built-in diagnostic and keep its output for XtreamGo support:

xtreamgo status
xtreamgo health
xtreamgo quality

• Do not run install.sh again to repair a panel. On a panel that is in use the installer stops before it changes anything. See Install the main server.

Where logs live

To judge the state of your panel, use xtreamgo quality. Do not rely on /var/log/xtreamgo-health.log. Several checks of the daily check do not apply to a customer server and report a failure on a healthy panel.

1. Containers are not healthy after the installation

Symptoms: Services started (some still initializing) or WARNING: some containers did not start at the end of the installer. Containers in the state Created.

xtreamgo status
docker ps -a --filter name=xtreamgo- --format 'table {{.Names}}\t{{.Status}}'
xtreamgo health
docker logs --tail 50 xtreamgo-api

• The first start applies all database changes and can take up to 10 minutes. frontend, portal, website and nginx wait for a healthy API and stay in Created until then.
• When the API is healthy and other containers are still in Created, start them. Use this command on a panel that has not received an update yet:

cd /home/xtreamgo && docker compose -f docker-compose.yml --env-file .env.production up -d

• When the API never becomes healthy, continue with item 2.

2. The API container keeps restarting

docker ps -a --filter name=xtreamgo-api --format 'table {{.Names}}\t{{.Status}}'
docker logs --tail 100 "$API" 2>&1 | grep -iE 'fatal|panic|error|refus' | tail -20
grep -cE '^(JWT_SECRET|CUSTOMER_JWT_SECRET|PORTAL_JWT_SECRET|DB_PASS)=.+' /home/xtreamgo/.env.production
docker inspect --format='{{.State.Health.Status}}' xtreamgo-db

• The count of the grep command must be 4. The API refuses to start in production mode when CUSTOMER_JWT_SECRET or PORTAL_JWT_SECRET is missing.
• A database error in the log means that the database is not ready. Continue with item 4.
• [LicenseChecker] lines are not a start failure. See item 7.

3. Port conflicts

Symptoms: xtreamgo-nginx does not start. Docker reports address already in use or port is already allocated.

ss -ltnp | grep -E ':(80|443|8080)\s'
docker logs --tail 20 xtreamgo-nginx
grep -E '^(HTTP_PORT|HTTPS_PORT|XC_PORT)=' /home/xtreamgo/.env.production

• Stop and disable the other service that holds the port, for example a web server installed by the hosting image (systemctl disable --now apache2 or nginx).
• The compose file publishes ${HTTP_PORT:-80}, ${HTTPS_PORT:-443} and ${XC_PORT:-8080}. The installer writes XC_PORT only. The installer does not support moving the panel to other ports after the installation. Contact XtreamGo support before you edit these values.

4. The database does not start

docker ps -a --filter name=xtreamgo-db --format '{{.Names}}\t{{.Status}}'
docker logs --tail 60 xtreamgo-db
free -h
df -h /var/lib/docker
dmesg | grep -i 'out of memory' | tail -5

• Memory: the database starts with a 2 GB buffer pool. On a small server set a lower value, for example DB_BUFFER_POOL=512M, in /home/xtreamgo/.env.production and recreate only the database container:

cd /home/xtreamgo && docker compose -f docker-compose.yml --env-file .env.production up -d --no-deps db

• Disk: a full disk stops the database. Continue with item 11.
• Access denied in the API log after .env.production changed: the passwords in the file must match the ones the database volume was created with. If you edited the file, put the original values from CREDENTIALS.txt back. If the values in CREDENTIALS.txt do not work either, contact XtreamGo support.

5. The browser shows 502 Bad Gateway

nginx cannot reach the API. The usual cause is that the API container was restarted and nginx still holds its old address. Two helpers repair this by themselves: a cron job every minute (/usr/local/bin/nginx-selfheal.sh) and the service xtreamgo-nginx-selfheal-daemon.

docker ps --filter name=xtreamgo-api --format '{{.Names}}\t{{.Status}}'
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1/api/v1/build/info
sed -n '/BG_UPSTREAM_BEGIN/,/BG_UPSTREAM_END/p' /home/xtreamgo/nginx/nginx.conf
systemctl is-active xtreamgo-nginx-selfheal-daemon
cat /etc/cron.d/xtreamgo-nginx-selfheal

• When an API container runs and is healthy, test and reload nginx:

docker exec xtreamgo-nginx nginx -t && docker exec xtreamgo-nginx nginx -s reload

• The server line in the upstream block must name a running container (xtreamgo-api-blue or xtreamgo-api-green). On a panel that has not received an update yet, xtreamgo-api-blue is a network name of the container xtreamgo-api. When the line names a container that does not run, contact XtreamGo support and do not edit the file by hand.
• When no API container runs, continue with item 2.

6. Certificate problems

ls -l /home/xtreamgo/ssl/ /home/xtreamgo/nginx/conf.d/
echo | openssl s_client -connect <domain>:443 -servername <domain> 2>/dev/null | openssl x509 -noout -subject -dates
docker exec xtreamgo-nginx nginx -t
crontab -l | grep xtreamgo-ssl-renew
getent ahostsv4 <domain> | head -1; curl -s -4 ifconfig.me; echo
curl -sI http://<domain>/.well-known/acme-challenge/test | head -1

• The request fails: the A record must point at this server and port 80 must be reachable from outside. The last command must return an HTTP answer from your server (404 is fine).
• The certificate is expired: run bash /home/xtreamgo/ssl-renew.sh; echo $?. Exit code 1 means that renewal or the copy failed. Check port 80 and the cron line.
• nginx rejected the TLS configuration; previous state restored: send the output of nginx -t to XtreamGo support.
• HTTPS works for the domain but not for other host names: ssl-setup.sh writes a server block for one domain only.

Details are in HTTPS and domain.

7. The admin panel answers 403 "license_invalid"

curl -s http://127.0.0.1/api/v1/license/status
docker logs "$API" 2>&1 | grep LicenseChecker | tail -20
curl -sS -o /dev/null -w '%{http_code}\n' https://api.xtreamgo.io/install/SHA256SUMS
grep -c '^LICENSE_KEY=.\+' /home/xtreamgo/.env.production

The message field of the first command names the reason. The reasons and actions are listed in License. During a lock subscribers stay connected on the paths that License lists as still available. Other subscriber paths, for example the MAG portal address /c/, are blocked.

8. The installer rejects the license

9. A stream does not start

docker exec "$API" ffprobe -v error -show_entries format=format_name:stream=codec_name -of default=nw=1 "<source-url>"
docker logs --tail 200 "$API" 2>&1 | grep -i '<stream name or id>' | tail -20
grep '^MAX_CONCURRENT_STREAMS=' /home/xtreamgo/.env.production
curl -sI "http://127.0.0.1/live/<username>/<password>/<stream-id>.ts" | head -5
df -h /var/lib/docker

• ffprobe fails: the source is not reachable from your server or needs other credentials. The panel cannot start a stream from a source it cannot read.
• The limit MAX_CONCURRENT_STREAMS is 100 after installation.
• The curl test returns 302 with a Location on a load balancer: the stream runs there. Check the agent on that server as in item 10.
• The disk that holds /var/lib/docker is full: HLS segments cannot be written. Continue with item 11.
• If the stream runs on a load balancer, read journalctl -u xtreamgo-agent -n 100 on that server.

10. A load balancer shows offline

On the load balancer:

systemctl is-active xtreamgo-agent
journalctl -u xtreamgo-agent --no-pager -n 40
grep '^DB_HOST=' /opt/xtreamgo-agent/.env
timeout 3 bash -c '</dev/tcp/<main-ip>/3306' && echo "3306 open" || echo "3306 closed"

On the main server:

systemctl is-active xtreamgo-db-access-sync.timer xtreamgo-db-forward
journalctl -u xtreamgo-db-access-sync --no-pager -n 5
iptables -S XTREAMGO_DBACCESS
ss -ltnp | grep ':3306'

• [Agent] Failed to connect to database or Failed to ping database: port 3306 on the main server is closed for this load balancer. The sync log must show [db-access-sync] ok with the IP of the load balancer. Check the provider firewall.
• The forwarder listens on the first address that hostname -I prints on the main server. If the main server has several addresses and DB_HOST on the load balancer names a different one, contact XtreamGo support.
• [Agent] AGENT_SERVER_ID is required and must be > 0 or [Agent] AGENT_API_KEY is required: the agent was set up with install-lb.sh. Reinstall it from the panel. See Add a load balancer server.
• [db-access-sync] WARN: LB query failed: the database container on the main server has a problem. Continue with item 4.

11. The disk is full

df -h
docker system df
du -sh /var/lib/docker/volumes/xtreamgo_*/ 2>/dev/null
du -sh /var/lib/docker/volumes/xtreamgo_api_data/_data/* 2>/dev/null | sort -h | tail
du -sh /home/xtreamgo/backups /home/xtreamgo/.xg-update-*.tar.gz 2>/dev/null
docker images | grep xtreamgo
du -sh /var/lib/docker/containers 2>/dev/null

Typical causes:

• Backups. backup.sh keeps 7 days in /home/xtreamgo/backups. Backups made in the panel are in xtreamgo_api_data/_data/backups and follow the retention set on the Backups page (30 days by default for automatic backups). Copy old backups to another machine, then delete them on the Backups page.
• VOD files, recordings and catch-up in xtreamgo_api_data. Remove content in the panel, not in the file system.
• Leftover update files (/home/xtreamgo/.xg-update-<component>.tar.gz) after an interrupted update. Ask XtreamGo support before you remove them.
• Images. A successful update removes old panel images by itself. It keeps the images of the running containers, of the stopped previous API container and the rollback-pre-update tags. Do not remove these by hand. They are the rollback of the last update.
• Container logs are limited to 5 files of 50 MB per service (250 MB) and are rarely the cause.

Do not delete files inside /var/lib/docker/volumes/xtreamgo_db_data.

12. Forgotten admin password

Option A: ask XtreamGo support to reset the admin password for your license. The panel applies the reset at its next license check, within about one minute.

Option B: reset it on the server. The API converts a plain text password in the user table into a hash each time it starts.

xtreamgo db

UPDATE reg_users SET password='<temporary-password>' WHERE username='admin';
exit

API=$(docker ps --filter name=xtreamgo-api --format '{{.Names}}' | head -1)
docker restart "$API"

The restart interrupts the API and running requests for a short time. Use a temporary password shorter than 60 characters. Log in with it and change the password at once. Option B does not need the old password. The password reset by e-mail on the login page only works when SMTP is configured and the admin account has your e-mail address.

13. No SSH access after the installation

The installer enables ufw. Before that it allows port 22 and every port that the SSH service listens on at that moment. A lockout is still possible when the SSH port was changed after the installation, or when a provider firewall blocks the port.

1. Open the console of your hosting provider.
2. Run ufw status and ss -tlnp | grep sshd to compare the open ports with the SSH port.
3. Run ufw allow <ssh-port>/tcp.

14. An update failed

curl -s http://127.0.0.1/api/v1/build/info
docker ps -a --filter name=xtreamgo-api --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'
ls -la /home/xtreamgo/.xg-update-*.tar.gz 2>/dev/null
cat /home/xtreamgo/nginx/.bg-rollback.meta 2>/dev/null
docker images | grep -E 'xtreamgo-(api|frontend|portal)'
df -h /var/lib/docker /home

• A failed update leaves the previous version in service, or restores it. The Updates page says which of the two happened. Check the version in the first command.
• The Updates page shows a message that starts with Could not reach CMS: the license server did not accept the update request. See the section SSH access for updates in Updates and rollback.
• Low disk space is a possible cause. The new images are loaded next to the running ones.
• The reason ends with Restoring the previous version failed too; check the panel now: contact XtreamGo support at once.
• Send the output of the commands above to XtreamGo support. Do not start or remove API containers by hand.

15. Two API containers run at the same time

Symptoms: after a reboot, or after xtreamgo start, xtreamgo restart or xtreamgo update without a service name, the list shows xtreamgo-api and also xtreamgo-api-blue or xtreamgo-api-green.

docker ps --filter name=xtreamgo-api --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.CreatedAt}}'
sed -n '/BG_UPSTREAM_BEGIN/,/BG_UPSTREAM_END/p' /home/xtreamgo/nginx/nginx.conf
curl -s http://127.0.0.1/api/v1/build/info

Why it can happen: after the first update the API runs in a container that Docker Compose does not manage. The systemd unit xtreamgo.service runs docker compose up -d at boot, and xtreamgo start, xtreamgo restart and xtreamgo update without a service name act on all Docker Compose services. These can start the original container xtreamgo-api again next to the live one. xtreamgo restart api, xtreamgo update api and ssl-setup.sh do not.

• The container named in the upstream block is the one that serves the panel.
• The container xtreamgo-api without a suffix is the extra one when a container with a suffix is in the upstream block. Stop it with docker stop xtreamgo-api, then check the panel.
• Report the case to XtreamGo support, so that the cause can be fixed on your server.

How to check it worked

After any fix:

xtreamgo health
curl -s http://127.0.0.1/health
curl -s http://127.0.0.1/api/v1/license/status

Expected results: every container healthy (the API line of xtreamgo health shows the live API container), "status":"healthy" and "licensed":true. Then log in and play one stream.

If something goes wrong

When a step in this document does not help, write to XtreamGo support at support@xtreamgo.io and attach:

• The output of xtreamgo status and xtreamgo quality.
• The last 100 lines of the API log: docker logs --tail 100 "$API".
• The exact time of the problem and the text of the error message.

Remove passwords and your license key from the output before you send it.