VPS Debugging Guide

VPS Debugging Guide for Obaki

This guide is a practical reference for debugging a .NET API running on an Ubuntu VPS.

Target setup:

Cloudflare
  -> Caddy
  -> Obaki .NET API running through systemd
  -> database / external APIs / background workers

Obaki service name:

obaki

API local port:

127.0.0.1:5000

Public API domain example:

api.jjosh.dev

1. Debugging mindset

On a PaaS, logs and deployment status are usually built in.

On a VPS, debug by layers:

DNS / Cloudflare
  -> Caddy
  -> systemd
  -> .NET app
  -> database / external APIs
  -> background workers

Core habit:

1. Check systemd.
2. Check journal logs.
3. Test local port.
4. Test Caddy/domain.
5. Check config/env/runtime.
6. Only then blame code.

2. Check Obaki service status

sudo systemctl status obaki --no-pager

Important statuses:

active (running)
failed
activating (auto-restart)

If it keeps restarting:

journalctl -u obaki -n 100 --no-pager

3. Live-tail logs

Live-tail Obaki logs:

journalctl -u obaki -n 100 -f

With timestamps:

journalctl -u obaki -n 100 -f -o short-iso

Only current boot:

journalctl -u obaki -b -f

Stop live tail:

Ctrl + C

4. Check recent logs

Last 100 lines:

journalctl -u obaki -n 100 --no-pager

Last 200 lines with timestamps:

journalctl -u obaki -n 200 --no-pager -o short-iso

Logs from current boot:

journalctl -u obaki -b --no-pager

Logs since today:

journalctl -u obaki --since today --no-pager

Logs from the last 30 minutes:

journalctl -u obaki --since "30 minutes ago" --no-pager

Search for errors:

journalctl -u obaki --no-pager | grep -i error

5. Restart and watch logs

sudo systemctl restart obaki
journalctl -u obaki -n 100 -f

If the app crashes on startup, the error usually appears immediately.

6. Check Caddy logs

Caddy live logs:

journalctl -u caddy -n 100 -f

Recent Caddy logs:

journalctl -u caddy -n 100 --no-pager

Check Caddy status:

sudo systemctl status caddy --no-pager

Reload Caddy:

sudo systemctl reload caddy

Validate Caddy config:

sudo caddy validate --config /etc/caddy/Caddyfile

7. Check ntfy logs

If ntfy is also running on the same VPS:

journalctl -u ntfy -n 100 -f

Recent ntfy logs:

journalctl -u ntfy -n 100 --no-pager

Check ntfy status:

sudo systemctl status ntfy --no-pager

8. Check if Obaki is listening

Show listening ports:

sudo ss -tulpn

Check port 5000:

sudo ss -tulpn | grep 5000

Expected:

127.0.0.1:5000

If nothing listens on port 5000, Obaki is not running or is using a different port.

If it listens on 0.0.0.0:5000, it may be publicly exposed. Prefer 127.0.0.1:5000.

9. Test Obaki locally from the VPS

Test root endpoint:

curl http://127.0.0.1:5000

Test health endpoint:

curl http://127.0.0.1:5000/health

If this fails, Cloudflare and Caddy are not the issue.

10. Test through public domain

Test headers:

curl -I https://api.jjosh.dev

Verbose test:

curl -v https://api.jjosh.dev

Health endpoint through Caddy/domain:

curl -v https://api.jjosh.dev/health

Interpretation:

Local 127.0.0.1:5000 works, but domain fails
-> Caddy / Cloudflare / DNS / SSL / firewall issue

Local 127.0.0.1:5000 fails
-> Obaki app issue

11. Check DNS

From your PC or VPS:

nslookup api.jjosh.dev

or:

dig api.jjosh.dev

If Cloudflare DNS is set to DNS only:

api.jjosh.dev -> your VPS public IP

If Cloudflare proxy is enabled:

api.jjosh.dev -> Cloudflare IPs

That is normal.

12. Check firewall

Check UFW status:

sudo ufw status verbose

Usually allowed:

22/tcp
80/tcp
443/tcp

Usually not exposed directly:

5000
8080

Expected flow:

Public internet
  -> 80/443
  -> Caddy
  -> 127.0.0.1:5000 for Obaki
  -> 127.0.0.1:8080 for ntfy

13. Check VPS resources

RAM:

free -h

Disk:

df -h

CPU/process view:

top

Better process viewer:

htop

Install htop:

sudo apt install -y htop

Check dotnet processes:

ps aux | grep dotnet

Check Obaki process:

ps aux | grep obaki

14. Check deployment folder

List Obaki folder:

ls -lah /opt/obaki

List current release:

ls -lah /opt/obaki/current

List all releases:

ls -lah /opt/obaki/releases

Check where current points:

readlink -f /opt/obaki/current

Check DLL files:

ls -lah /opt/obaki/current/*.dll

If systemd uses:

ExecStart=/usr/bin/dotnet /opt/obaki/current/Obaki.Api.dll

then this file must exist:

/opt/obaki/current/Obaki.Api.dll

15. Run the app manually

Stop the service first:

sudo systemctl stop obaki

Run the app as the Obaki Linux user:

sudo -u obaki /usr/bin/dotnet /opt/obaki/current/Obaki.Api.dll

This often shows clearer startup errors.

Start the service again:

sudo systemctl start obaki

16. Check systemd service config

Show the full service definition:

sudo systemctl cat obaki

Recommended shape:

[Unit]
Description=Obaki .NET API
After=network.target

[Service]
WorkingDirectory=/opt/obaki/current
ExecStart=/usr/bin/dotnet /opt/obaki/current/Obaki.Api.dll
Restart=always
RestartSec=5
KillSignal=SIGINT
SyslogIdentifier=obaki
User=obaki

EnvironmentFile=/etc/obaki/obaki.env
Environment=DOTNET_GCHeapHardLimitPercent=40

[Install]
WantedBy=multi-user.target

Do not use this unless the app explicitly supports systemd notify:

Type=notify

17. Check environment file

Show env file permissions:

sudo ls -lah /etc/obaki/obaki.env

Expected:

-rw-r----- root obaki /etc/obaki/obaki.env

Fix ownership and permissions:

sudo chown root:obaki /etc/obaki/obaki.env
sudo chmod 640 /etc/obaki/obaki.env

Environment file example:

ASPNETCORE_ENVIRONMENT=Production
ASPNETCORE_URLS=http://127.0.0.1:5000

ConnectionStrings__DefaultConnection=Host=localhost;Port=5432;Database=obaki;Username=obaki;Password=change-this

Resend__ApiKey=re_xxxxxxxxxxxxxxxxx
[email protected]

ExternalProvider__BaseUrl=https://example.com/api
ExternalProvider__ApiKey=change-this

Do not log secrets.

Do not commit production .env files.

18. Check .NET runtime

dotnet --info

List runtimes:

dotnet --list-runtimes

For ASP.NET Core API, you need:

Microsoft.AspNetCore.App
Microsoft.NETCore.App

If logs show:

You must install or update .NET to run this application.
Framework: 'Microsoft.AspNetCore.App'
No frameworks were found.

install the ASP.NET Core runtime or publish self-contained.

Example:

sudo apt update
sudo apt install -y aspnetcore-runtime-10.0

Then verify:

dotnet --list-runtimes

Restart:

sudo systemctl restart obaki

19. Add a health endpoint

Add a simple health endpoint in your .NET API:

app.MapGet("/health", () => Results.Ok(new
{
    Status = "OK",
    Time = DateTimeOffset.UtcNow
}));

Test locally:

curl http://127.0.0.1:5000/health

Test publicly:

curl https://api.jjosh.dev/health

This tells you:

App alive locally
Caddy routing works
Domain works
SSL works

20. Add startup logs

At startup, log safe values:

logger.LogInformation("Obaki starting");
logger.LogInformation("Environment: {Environment}", app.Environment.EnvironmentName);
logger.LogInformation("Content root: {ContentRoot}", app.Environment.ContentRootPath);

Never log:

connection strings
API keys
JWT secrets
SMTP passwords
database passwords

21. Add background worker logs

For the outbox worker:

logger.LogInformation("Outbox worker tick started.");
logger.LogInformation("Outbox worker found {Count} pending messages.", count);
logger.LogInformation("Outbox worker completed.");

For failures:

logger.LogError(ex, "Outbox worker failed.");

For endpoint checker:

logger.LogInformation("Endpoint checker started.");
logger.LogInformation("Endpoint checker found new data: {HasNewData}", hasNewData);

Since the workers run every 2 minutes, this level of logging is acceptable.

Avoid logging every second.

Avoid dumping full payloads unless needed.

22. Common failure map

Domain does not resolve

Problem:
Cloudflare DNS

Check:
nslookup api.jjosh.dev
dig api.jjosh.dev

HTTPS fails

Problem:
Caddy / SSL / Cloudflare SSL mode

Check:
journalctl -u caddy -n 100 --no-pager
sudo caddy validate --config /etc/caddy/Caddyfile

Domain returns 502

Problem:
Caddy is reachable but cannot reach Obaki

Check:
curl http://127.0.0.1:5000/health
sudo ss -tulpn | grep 5000
journalctl -u obaki -n 100 --no-pager

Local curl fails

Problem:
Obaki is not running or wrong port

Check:
sudo systemctl status obaki --no-pager
journalctl -u obaki -n 100 --no-pager

Service shows activating auto-restart

Problem:
App is crashing on startup

Check:
journalctl -u obaki -n 100 --no-pager
sudo -u obaki /usr/bin/dotnet /opt/obaki/current/Obaki.Api.dll

Missing framework

Problem:
.NET runtime missing

Fix:
sudo apt install -y aspnetcore-runtime-10.0

Permission denied

Problem:
Wrong file ownership, env permissions, app user permissions, or deploy user permissions

Check:
ls -lah /opt/obaki/current
sudo ls -lah /etc/obaki/obaki.env
sudo systemctl cat obaki

GitHub deploy succeeds but old app still runs

Problem:
Wrong symlink, wrong ExecStart path, or service did not restart

Check:
readlink -f /opt/obaki/current
sudo systemctl status obaki --no-pager
journalctl -u obaki -n 100 --no-pager

GitHub deploy fails with host key verification

Problem:
The workflow could not verify the SSH host key. Usually `VPS_HOST` or `VPS_PORT` is wrong, SSH is blocked by the firewall, or `VPS_SSH_KNOWN_HOSTS` has the wrong host key.

Fix:
Check `VPS_HOST` and `VPS_PORT`. If you set `VPS_SSH_KNOWN_HOSTS`, make sure it contains a trusted known_hosts line that matches both values.

For port 22:

ssh-keyscan -p 22 YOUR_VPS_HOST

For a custom SSH port, the output should start with:

[YOUR_VPS_HOST]:YOUR_PORT

Caddy works but app behavior is wrong

Problem:
App config, database, external API, or background worker logic

Check:
journalctl -u obaki -n 200 --no-pager
curl http://127.0.0.1:5000/health

23. Daily VPS debugging cheat sheet

sudo systemctl status obaki --no-pager
journalctl -u obaki -n 100 --no-pager
journalctl -u obaki -n 100 -f
sudo systemctl restart obaki

sudo systemctl status caddy --no-pager
journalctl -u caddy -n 100 --no-pager
sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl reload caddy

curl http://127.0.0.1:5000/health
curl -v https://api.jjosh.dev/health

sudo ss -tulpn
free -h
df -h
readlink -f /opt/obaki/current
ls -lah /opt/obaki/current
dotnet --list-runtimes

24. Recommended minimum observability for Obaki

Start with:

systemd journal
Caddy logs
health endpoint
startup logs
worker lifecycle logs

Later, consider:

Seq
Grafana Loki
OpenTelemetry
Sentry
Better Stack

Do not overbuild observability early.

For the current VPS setup, journalctl plus a health endpoint is enough.