Setting up a new VPS properly is crucial for security, stability, and performance. This guide covers 5 Phase of our VPS setup, including SSH access, firewall configuration, and security enhancements.

Summary of All Phases

Our VPS setup is divided into multiple phases for a structured and efficient deployment:

  • Phase 1: Initial VPS setup – SSH access, security hardening, and firewall configuration.
  • Phase 2: Install and configure HestiaCP for website, database, and email management.
  • Phase 3: Set up Docker and Portainer for containerized applications.
  • Phase 4: Deploy applications step by step, including FOSSBilling, ERPNext, Nextcloud, FileRun, Bitwarden, Vikunja, Joplin Server, OpenProject, Mailcow, and more.
  • Phase 5: Backup and monitoring – Implement Restic with Google Drive and install server monitoring tools.

Each phase ensures a smooth and secure deployment for self-hosted applications and services.

Step 1: install Ubuntu 24.04 LTS

Before starting, ensure your VPS provider allows you to install the OS. Choose Ubuntu 24.04 LTS as your operating system.

After installation, you’ll need to access your VPS via SSH.

Step 2: Accessing SSH with PuTTY

To access your VPS, follow these steps:

  1. Download and Install PuTTY:
  3. Connect to Your VPS:
    • Enter your VPS IP address.
    • Default SSH port: 22 (we will change it later for security).
    • Click Open and log in with your credentials.
  4. Change Your SSH Port (Optional but Recommended):
    • Open SSH config file:
      sudo nano /etc/ssh/sshd_config
    • Find #Port 22, remove the #, and change it to a new port, e.g., 222:
      Port 222
    • Save and exit (Ctrl + X, then Y, then Enter).
    • Restart SSH:
      sudo systemctl restart ssh
    • Update your firewall to allow the new port:
      sudo ufw allow 222/tcp
    • Now, reconnect using PuTTY with the new port (222).

Step 3: Create a New User (Non-Root)

For security, avoid using root for daily tasks.

  1. Create a new user:
    sudo adduser yourusername

    Follow the prompts to set a password.

  2. Grant sudo access:
    sudo usermod -aG sudo yourusername
  3. Switch to the new user:
    su - yourusername

Step 4: Set Up SSH Key-Based Authentication

Using SSH keys instead of passwords enhances security.

  1. Generate an SSH key on your local machine:
    ssh-keygen -t rsa -b 4096

    Press Enter to save in the default location.

  2. Copy the public key to your VPS:
    ssh-copy-id -p 222 yourusername@your-vps-ip
  3. Disable Password Authentication (Extra Security Step):
    • Edit SSH config:
      sudo nano /etc/ssh/sshd_config
    • Set the following:
      PasswordAuthentication no
PermitRootLogin no
    • Restart SSH:
      sudo systemctl restart ssh

Now, you can only log in with your SSH key, making your VPS much more secure.

Step 5: Configure UFW Firewall

To prevent unauthorized access, configure UFW (Uncomplicated Firewall).

  1. Enable UFW and allow essential ports:
    sudo ufw allow 222/tcp  # (Your SSH port)
sudo ufw allow 80/tcp    # (HTTP for websites)
sudo ufw allow 443/tcp   # (HTTPS for websites)
sudo ufw enable
  2. Check firewall status:
    sudo ufw status

This ensures that only necessary services are accessible.

Step 6: Verify System Security and Running Services

Check if SSH is running on the correct port:

sudo ss -tulnp | grep ssh

Expected output should show SSH listening on your custom port (222).

Check firewall status:

sudo ufw status

Ensure only the allowed ports are open.

Conclusion

At this stage, your VPS is: ✅ Running Ubuntu 24.04 LTS ✅ Secured with SSH key authentication ✅ Configured with a non-root user ✅ Protected by a firewall (UFW) ✅ Ready for further installations

Next, we move to Phase 2: Installing HestiaCP to manage websites, databases, and services efficiently.

Step 1: Download the Installation Script

Log in as root via SSH and run:

bash

wget https://raw.githubusercontent.com/hestiacp/hestiacp/release/install/hst-install.sh

 

Ensure the file has downloaded properly:

bash

ls -l hst-install.sh

 

Step 2: Run the Installation Script

Run the script as root:

bash

bash hst-install.sh

 

This will launch an interactive installation where you can configure different services.

Step 3: Customize Installation (Optional)

If you want to customize the installation (selecting services manually), use:

bash

bash hst-install.sh --interactive yes

 

You’ll be prompted to select:

  • Hostname (e.g., panel.yourwebsite.com)
  • Admin username (Default: admin)
  • Email address
  • Password (or auto-generate one)
  • Web server options (Apache, Nginx, PHP-FPM)
  • Database options (MariaDB, PostgreSQL, or MySQL 8)
  • Mail services (Exim, Dovecot, SpamAssassin, ClamAV)
  • Firewall options (iptables, Fail2Ban)
  • Other settings (File manager, API, web terminal, quotas, etc.)

Recommended HestiaCP Installation Choices

ComponentRecommendedNotes
NGINX (Proxy Server)✅ YesBest for performance; handles static content efficiently.
Apache (Backend Server)✅ YesRequired for PHP websites; works well with Nginx.
PHP-FPM (Application Server)✅ YesFaster than traditional PHP processing.
Bind DNS Server❌ Noif using Cloudflare for DNS, so this is unnecessary.
Exim Mail Server❌ NoRecomended install Mailcow for email later. or yes if simple
ClamAV (Antivirus for Mail)❌ NoOnly needed if using Exim/Dovecot. if above choose yes then yes
SpamAssassin (Spam Filtering)❌ NoNot needed since you’re skipping Exim. if above choose yes then yes
Dovecot (IMAP/POP3 Server)❌ NoYou don’t need this because of Mailcow.if above choose yes then yes
MariaDB (Database Server)✅ YesBetter performance than MySQL 8 for your setup.
MySQL 8❌ NoMariaDB is a better choice unless you specifically need MySQL 8.
PostgreSQL❌ NoOnly needed if you’re using PostgreSQL-based apps (ERPNext, etc.).
Vsftpd (FTP Server)❌ NoUse SFTP instead (more secure).
Firewall (iptables + Fail2Ban)✅ YesEssential for security.

Installation Command

To install HestiaCP with these options, use:

bash

bash hst-install.sh --nginx yes --apache yes --phpfpm yes --named no --exim no --clamav no --spamassassin no --dovecot no --mariadb yes --postgresql no --vsftpd no --iptables yes --fail2ban yes

 

Summary of Choices

✅ Best setup for performance and security
✅ Compatible with your Cloudflare DNS and Mailcow plans
✅ Uses MariaDB (better than MySQL 8 for most cases)
✅ No unnecessary mail services since you’ll install Mailcow separately

Here’s the updated installation command with a note for troubleshooting:

Installation Command

Run the following command to install HestiaCP with optimized settings:

bash

bash hst-install.sh --nginx yes --apache yes --phpfpm yes --named no --exim no --clamav no --spamassassin no --dovecot no --mariadb yes --postgresql no --vsftpd no --iptables yes --fail2ban yes

 

Important Note

If you encounter any errors during installation, refer to the official HestiaCP Installation Guide for troubleshooting steps and alternative installation methods.

Step 4: Reboot VPS

Once installation is complete, reboot your VPS:

bash

reboot

 

Step 5: Access HestiaCP Panel

After rebooting, access the HestiaCP web interface:

arduino

https://your-vps-ip:8083

 

or, if you set a custom hostname:

https://panel.gethostx.com:8083

 

Log in with:

  • Username: admin
  • Password: (Generated or set during installation)

Step 6: Allow Port 8083 in UFW (If Needed)

If you can’t access the panel, ensure port 8083 is allowed in your firewall:

bash

sudo ufw allow 8083/tcp

sudo ufw reload

 

Step 7: Secure HestiaCP

  • Change the admin password immediately.
  • Set up automatic backups for your settings and websites.
  • Install an SSL certificate (Let’s Encrypt) for panel.gethostx.com.

Next Step: Configuring HestiaCP for Websites and Services

Once HestiaCP is installed, you can create websites, databases, and email accounts.

 

Great! Now that HestiaCP is installed, we can proceed with the next steps. Since MariaDB is already installed with HestiaCP, we’ll move forward with FOSSBilling setup.

Next Steps:

✅ Verify MariaDB Installation & Secure It
✅ Install & Configure FOSSBilling (billing.gethostpro.com)
✅ Set Up SSL & Reverse Proxy (if needed)

Here’s a MariaDB Verification & Security Checklist before proceeding with FOSSBilling installation:

Step 1: Verify MariaDB Installation

MariaDB is installed with HestiaCP by default, but let’s confirm it’s running properly.

  1. Check MariaDB Service Status:

    bash

    sudo systemctl status mariadb

     

    • If it’s running, you should see “active (running)”.
    • If not, start it manually:
      bash

      sudo systemctl start mariadb

       

    • Enable it to start on boot:
      bash

      sudo systemctl enable mariadb

       

  2. Check MariaDB Version:

    bash

    mysql -V

     

    • Expected output: MariaDB x.x.x

  3. Log in to MariaDB:

    bash

    sudo mysql -u root -p

     

    • If prompted, enter your MariaDB root password (HestiaCP might have set this already).

Step 2: Secure MariaDB Installation

Run the security script to remove weak settings:

bash

sudo mysql_secure_installation

 

Follow the prompts: ✅ Set a strong root password (if not already set)
✅ Remove anonymous users
✅ Disallow remote root login
✅ Remove test databases
✅ Reload privilege tables

Step 3: Create a Database for FOSSBilling

We’ll create a dedicated database and user for FOSSBilling.

  1. Log into MariaDB:
    bash

    sudo mysql -u root -p

     

  2. Create the database (replace fossbilling_db with your preferred name):
    sql

    CREATE DATABASE fossbilling_db;

     

  3. Create a new user (replace fossuser and yourpassword):
    sql

    CREATE USER 'fossuser'@'localhost' IDENTIFIED BY 'yourpassword';

     

  4. Grant permissions to the user:
    sql

    GRANT ALL PRIVILEGES ON fossbilling_db.* TO 'fossuser'@'localhost';

     

  5. Apply changes and exit:
    sql

    FLUSH PRIVILEGES;

    EXIT;

     

Step 4: Verify Firewall & Port Settings

HestiaCP should have already configured the firewall, but confirm MariaDB is not accessible publicly.

  1. Check if MariaDB is listening only on localhost (127.0.0.1):

    bash

    sudo ss -tulnp | grep mysql

     

    • Expected output: 127.0.0.1:3306

  2. If MariaDB is publicly accessible (0.0.0.0:3306), restrict it:

    • Open MariaDB config:
      bash

      sudo nano /etc/mysql/mariadb.conf.d/50-server.cnf

       

    • Find bind-address = 0.0.0.0 and change it to:
      ini

      bind-address = 127.0.0.1

       

    • Save and restart MariaDB:
      bash

      sudo systemctl restart mariadb

       

Next Step: Install FOSSBilling

With MariaDB secured and a database ready, we can now install FOSSBilling on billing.gethostpro.com.

Let me know if you need any modifications before proceeding! 🚀

GUI Method to create database

Since you’ve installed HestiaCP, you can easily create a database using the GUI. Here’s how:

Step 1: Log in to HestiaCP

  1. Open your browser and go to your HestiaCP panel:
    arduino

    https://panel.yourip:8083

     

  2. Enter your HestiaCP admin username and password, then click Login.

Step 2: Create a New Database

  1. In the HestiaCP Dashboard, click on “DB” (Databases) from the top menu.
  2. Click the “+” (Add Database)” button.
  3. Fill in the required details:
    • Database Name → Enter a name (e.g., fossbilling_db).
    • Database User → Choose a username (e.g., fossuser).
    • Password → Generate or set a strong password.
  4. Click “Save” or “Add”.

Step 3: Verify Database Creation

  • You will see the newly created database listed under Databases.
  • Note down the Database Name, Username, and Password—you’ll need these for FOSSBilling installation.

Step 4: Enable Remote Access (If Needed)

  • If FOSSBilling is on the same VPS, no remote access is needed.
  • If needed, click “Edit” on the database and allow access for a specific IP.

Now let’s move forward with the FOSSBilling installation. 🚀

Next Steps:

  1. Download & Extract FOSSBilling
  2. Set Up Web Server & Permissions
  3. Run the Web Installer