As host OS with bhyve

Install needed packages

pkg info: Check which packages are installed. Needed are:; vm-bhyve — Management system for bhyve virtual machines; bhyve-firmware — Collection of Firmware for bhyve
kldstat: Kernel modules; vmm.ko — virtualzation driver; zfs.ko — for the file system
sysrc vm_enable="YES": enable vm
sysrc vm_dir="zfs:vmzfs/vm": set root directory for the vm; after the reboot the templates will show up; alternatively vm init
vm init: set templates and config files in the vm_dir directory
vm switch create public: create the public bridge which is used by the virtual machine for the taptun virtual divices. the bridge is called vm-public but in the config files of the virtual machines the name is only public
vm switch add public em0: add a physical intervace em0 that is used by the bridge to access he internet. Note that if you have multiple physical devices you have to add the device that has internet access. For example em1 may be the device that has access to the internet in which case you have to change the above command accordingly. You can check that the bridge has the correct information by issuing ifconfig vm-public or just ifconfig if the bridge name is not vm-public and check under members to see which interfaces are in the bridge. The correct emX should be listed
vi /path/to/vm_dir/.config/system.conf: if there are two physical ethernet devices say em0 and em1 the ports_public="emX" lists which the interface the switch public is used. Note also in this file there is switch_list="public" which is used in the config files in each virtual machine

Working with vm

This the the management system with which we will start, stop update and maintain the guests

vm start jupyter: start a virtual machine with name jupyter
vm list: list all virtual machine
vm info: Check information about running virtual machines (guests)
vm stop jupyter: stop gracefully the virtual machine with name jupyter
vm poweroff jupyter: stop forcefully the virtual machine with name jupyter
vm console jupyter: connect to the virtual machine console and log on
exit: It is a good idea to logoff the host before exiting the vm console; Press ~ (tilde) then ^D (Ctrl+D) to terminate the console; you may need to press Shift to enter the tilde character; only the righthand shift works, pressing left shift with tilde simply supplies tilde to the guest (console)
vm snapshot jupyter@snapshotname: create a snapshot of a virtual machine. It is (should be) stopped at the time of taking the snapshot
vm info jupyter: will give information about the virtual machine include the list of available snapshots of that machine
vm clone existing new: Will create a copy of a virtual machine - for example create a single base system and to test various things simply clone the virtual machine and work with it. Note that you can modify the above command to clone a snapshot of a virtual machine

Managing virtual machines

Most options of a virtual machine can be modifed in the config file.

Disk size

Increase the disk size of a virtual machine, first increase the size of the virtual drive from the host, then apply the changes in the guest.

truncate -s +4G DiskFile

Enlarge the file which corresponds to the virtual drive

gpart show

find the name of the disk for example vtbd0


=>      40  16777136  vtbd0  GPT  (8.0G) [CORRUPT]
        40      1024      1  freebsd-boot  (512K)
      1064       984         - free -  (492K)
      2048   4194304      2  freebsd-swap  (2.0G)
   4196352   4190208      3  freebsd-zfs  (2.0G)
   8386560   8390616         - free -  (4.0G)

gpart recover vtbd0

After the file is enlarged it seen as Corrupt; fix it


=>      40  16777136  vtbd0  GPT  (8.0G)
        40      1024      1  freebsd-boot  (512K)
      1064       984         - free -  (492K)
      2048   4194304      2  freebsd-swap  (2.0G)
   4196352   4190208      3  freebsd-zfs  (2.0G)
   8386560   8390616         - free -  (4.0G)

swapoff -a

disables the swap: /dev/vtbd0p2 — if the swap is at the second partition

gpart resize -i 3 /dev/vtbd0

grow the root partition

zpool list

check which zfs pool is used (we use zfs partitions)


NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot  1.88G  1.27G   616M        -         -    18%    67%  1.00x    ONLINE  -

zpool online -e zroot vtbd0p3

expand the pool to the whole disk

Bibliography

As a `bhyve` guest OS

Set a FreeBSD template

vm iso

see available iso and img files for installation

/path/to/vmdir/.templates where files are stored

vm iso https://path.to.location/file

download installation media

vi /path/to/vm/.templates/freebsd.conf

edit the config file

loader="bhyveload"
cpu=2
cpu_cores=2
cpu_sockets=1
memory=8G
network0_mac="E4:3E:7E:E3:E4:84"
network0_type="virtio-net"
network0_switch="public"
disk0_name="FreeBSD"
disk0_type="virtio-blk"
disk0_dev="sparse-vol"
zfs_zvol_opts="volblocksize=8k compress=on"
graphics="no"

Change the name of the disk0_name to something suitable

vm create -t freebsd -s 8G FreeBSD

create the virtual machine

vm install FreeBSD FreeBSD-LATEST-RELEASE.iso

start the installation of FreeBSD operating system with whichever iso file you decided to use

vm console FreeBSD

get to the console and follow installation instructions

the above config file sets a zfs volume, one of the questions asked during installing is which filesystem you want: ufs or zfs. With ufs after installation there was kernel panic, with zfs no such issues were encountered

after installation powerdown the system

vm start FreeBSD

start the virtual machine

vm console FreeBSD

get to the console and install necessary tools

pkg install socat

first installs the pkg system if not already installed

installs socat

ZFS on FreeBSD

zpool info

  pool: vmzfs
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
config:

	NAME           STATE     READ WRITE CKSUM
	vmzfs          DEGRADED     0     0     0
	  mirror-0     DEGRADED     0     0     0
	    gpt/disk1  ONLINE       0     0     0
	    gpt/disk2  ONLINE       0     0     0
	    gpt/disk3  ONLINE       2     0     0
	    gpt/disk4  REMOVED      0     0     0

errors: No known data errors

The above issue appeared on my server.

Bibliography

ZFS Guide on FreeBSD

As a KVM guest OS

Test a ready made image

Get a machine: Get a ready made virtual machine image
Run the machine: following change the bios part; qemu-system-x86_64 -m 4096 -smp 4 -serial mon:stdio -nographic -drive file=FreeBSD-14.0-RELEASE-amd64.qcow2 -enable-kvm
Close the machine from within freebsd: poweroff

Prepare a ready made image

Download say FreeBSD-15.0-RELEASE-amd64-BASIC-CLOUDINIT-ufs.qcow2.xz: Get a machine
cp FreeBSD-15.0-RELEASE-amd64-BASIC-CLOUDINIT-ufs.qcow2.xz vm00.qcow2.xz: Make a copy of the file to avoid future downloads
uzxz vm00.qcow2.zx: Expand the image
qemu-img resize vm00.qcow2 +4G: Resize the hard disk of the work copy to a desired size in our case increase by 4G
/usr/bin/qemu-system-x86_64 \ -enable-kvm -m 8192 -cpu host -smp 4 \ -drive file=./vm00.qcow2 \ -monitor unix:/run/vm00.sock,server,nowait \ -nic tap,mac=b4:e1:ad:2b:d3:00 \ -serial mon:stdio -nographic \ -pidfile /run/user/1001/vm00.pid \ -name vm00,process=vm00: run the machine with access to serial console. For this FreeBSD image some scripts will be run for you on the virtual machine. Subsequently the virtual machine will reboot and you will be greeted with a login for a root user. Just type root to access FreeBSD are root. From within the root console of the virtual machine get some things ready, such as remote access
passwd: set a root password and reboot the virtual machine
vi /etc/rc.config: set the hostname to a meaningful name
vi /etc/ssh/sshd_config: modify sshd to allow root login
reboot: after setting up a root password reboot the machine. Send your public key file to the guest with ssh-copy-id root@192.168.1.XXX or ssh-copy-id root@guest_hostname where guest_hostname resolves to the ip address of the guest. You should be able to log in the guest with ssh root@192.168.1.XXX or ssh root@guest_hostname. Log in with your new root password either from the serial console or remotely.
vi /etc/ssh/sshd_config: modify sshd to allow root login but only with public key authentication.
service sshd restart: make sure the sshd daemon uses the new configuration - there are alternative ways to do it of course
: perform any tasks you want on your virtual machine that runs FreeBSD. One good idea would be to add a new user with wheel privileges instead of using root login. Install any software that you need
pkg update: It is a good idea to add the various ports for easier installation of software. Be where of pkg update issue
poweroff: power down your virtual machine if not in use

Expand disk after installation

After creating the guest virtual machine at one point your drive may be full in which case you may need to increase the drive space (Ram and the like are changes to qemu-system-x86_64) options. From the host do

echo system_powerdown | socat - unix-connect:/path/to/work.sock: stop the virtual machine
qemu-img resize work.qcow2 +4G: expand the drive with the desired amount
qemu-system-x86_64 -drive file=work.qcow2: start the virtual machine. Include all options that you regularly use
ssh root@work: access the root console in the guest
gpart show: see the names of the dist as viewed by gpart. In my case ada0
gpart recover ada0: if there is a corruption fix it (possible after expanding the disk with qemu-img
gpart resize -i 5 ada0: here -i 5 points to (the last) partition of the disk where the root partition is stored
df -h: Check where root partition is mounted on /dev mine was different from /dev/ada0p5. It was on /dev/gpt/rootfs
growfs /dev/gpt/rootfs: grow the partition (this is assuming you used ufs type filesystem)
reboot: reboot and log on again to verify the changes

Apache on FreeBSD

Install and configure Apache

pkg install apache24

installs Apache with output

apachectl -M

list loaded modules

vi /etc/rc.conf

add apache24_enable="yes" to run apache at start up, the alternative is to issue which adds the above line to the same file

sysrc apache24_enable="yes"

cd /usr/local/etc/apache24/

go to the apache24 config directory and create necessary certificate keys

openssl ecparam -name prime256v1 -out p256-apache24.pem

To use https create necessary keys for apache, to use self signed certificates. Start with creating a key

openssl req -x509 -newkey ec:p256-apache24.pem -keyout apache24.key -out apache24.crt -days 365 -nodes

create the certificate and the corresponding key. The files apach24.key and apache24.crt are the needed files.

vi /usr/local/etc/apache24/httpd.conf

and comment out enable

LoadModule ssl_module libexec/apache24/mod_ssl.so LoadModule socache_shmcb_module libexec/apache24/mod_socache_shmcb.so LoadModule include_module libexec/apache24/mod_include.so: allow https connections and server side includes
Include etc/apache24/extra/httpd-ssl.conf: within this file there are calls to SSLcertificates and keys. Those certificates must be present - they may not come with the package. Create your own if necessary.
Include etc/apache24/sites-enabled/*.conf: create director /usr/local/etc/apache24/sites-enabled and with the above line place all virtual hosts in that directory. All files ending with .conf will be served by apache; For basic authentication the authentication file must be in the root directory /usr/local/www

vi /usr/local/etc/apache24/extra/httpd-ssl.conf

Verify the correct place of the certificates that were created

SSLCertificateFile "/usr/local/etc/apache24/apache24.crt"
SSLCertificateKeyFile "/usr/local/etc/apache24/apache24.key"

service apache24 start

Start apache24

Php on FreeBSD

install and configure

pkg install php84: or with the script ./bsd_php_mods.sh which in addition installs apache24 and a bunch of useful extensions and reports
php -m: check available modules it appears json , openssl , pcre and spl (SPL) are installed by default
cp /usr/local/etc/php.ini-developement /usr/local/etc/php.ini: create a php.ini file

Note that for php82 also worked.

PostgreSQL on FreeBSD

Here the operating system runs on a virtual machine. Access its terminal as root

Installation

pkg update: update to the latest package management in FreeBSD
pkg search postgresql: find the latest versions of PostgreSQL available for you
pkg install postgresql18-server postgresql18-client: make sure you have enough space and install the software. postgresql18-client gives you psql command that lest you manage the databases from within the virtual machine.
service postgresql initdb: initiate the database; files such as /var/db/postgress/data18/postgresql.conf are created
vi /etc/rc.conf: add postgresql_enabled="yes" to start the database on boot

Configuration for local access

Edit /var/db/postgres/data18/postgresql.conf (this is for PostgeSQL v18) and set


	listen_address = 'localhost,192.168.1.88'
	password_encryption = scram-sha-256

where 192.168.1.88 is the IP address of the machine that host PostgeSQL. Make sure the passwords are stored in the specified format before setting passwords of any users.

Next edit /var/db/postgres/data18/pg_hba.conf and set


	host    all             all             127.0.0.1/32            trust
	host    all             all             192.168.68.0/24 scram-sha-256

Accessing psql interactive terminal

su postgres: as root to access postgres user
# psql: Now you have terminal access to postgresql. You will be greeted by terminal
# psql -U username -h host dbname: login a particular user to manage the user's databases you will be greeted with terminal
\l: lists all databases
SELECT * from pg_users: lists all users
\du: lists all users and their privileges
CREATE USER username WITH ENCRYPTED PASSWORD 'strongpassword';: create a user with the specified password
DROP USER username;: deletes the user username
CREATE DATABASE dbname WITH OWNER=user;: create a database
DROP DATABASE dbname;: delete database
CREATE TABLE tablename (field TYPE, ..., field TYPE);: when login as user username create table with the necessary specifications in the database the user owns
\dt: list all tables in the current database
\d+ tablename: list all columns and their types available in tablename

Export databases

# pg_dump -U username -h host dbname > filename.sql

export command. You may need to edit pg_hpa file to allow trust from the unix socket that pg_dump uses by setting

local all all trust

in /var/db/postgres/data18/pg_hba.conf (the Address information is empty) so as to avoid any authorization errors

Import databases

CREATE USER username WITH ENCRYPTED PASSWORD 'strongpassword';: create the user if necessary, note the single quotes
CREATE DATABASE dbname WITH OWNER=user;: create the database with the owner
exit: drop to bash
# psql -U username -d dbname -f /path/to/backup.sql

Useful BSD tools

doas

allows you to run code as a different user; sudo alternative

pkg install doas: installs doas
cp /usr/local/etc/doas.conf.example /usr/local/etc/doas.conf: create doas's config file
vi /usr/local/etc/doas.conf: make changes that suit you

OpenSSL

This library comes pre-installed with BSD and Linux variants. The use case here is to create client keys so as to avoid password authentication.

ssh-keygen -t ed25519: create curve ed25519 key pairs, which are stored in your $HOME/.ssh directory
ssh-keygen -t ecdsa: create curve elliptic curve DSA signing key pair, which are stored in your $HOME/.ssh directory
ssh-copy-id user@hostname: copy the created keys to the authorized_keys file on the remote server. From now on you can use publickey authentication instead of password authentication
pkg install fusefs-sshfs: on freebsd
apt-get install sshfs: on Linux variants
vi /etc/fstab: sshfs#user@hostname:/path/to/share /path/to/mountpoint fuse noauto,rw,uid=username,gid=groudname,user,reconnect,IdentityFile=/path/to/home/.ssh/id_ecdsa 0 0
mount /path/to/mountpoint: the above command can be executed as a user. No need for root access

socat

A multi purpose relay, which had multiple use cases

You have a personal computer and various firewall rules on your work place or anywhere else are prohibiting you from using ssh to log onto your computer you can hide the ssh trafic as https trafic
You manage a virtual machines and need to access the unix socket where the virtual machine monitor (e.g., qemu monitor) accepts connections.

pkg install socat

installs socat on freebsd

openssl req -x509 -newkey rsa:2048 -keyout client.key -out client.crt -days 300 -nodes

create certificate for the client

mv client.* $HOME/.ssh/

place the client certificates in your home ssh directory

openssl req -x509 -newkey rsa:2048 -keyout tunnel.key -out tunnel.crt -days 300 -nodes

create certificate for the server tunnel

avoid fancy parameters both at the server end (where socat runs) and the client end (your laptop from which you connect to your home ps that runs socat) as it makes it more likely for the firewall to filter your traffic, hence the RSA choice

mv tunnel.* /root/

move the tunnel certificate and key to the root home

/path/to/socat -L /run/Socat_21022_22 OPENSSL-LISTEN:21022,fork,reuseaddr,certificate=/root/tunnel.crt,key=/root/tunnel.key,verify=0 tcp:127.0.0.1:22 &

by running the above code at boot (e.g. within /etc/rc.local you can hide your ssh connections within https. The tunnel listens to incoming port 21022 and redirects it to local port 22. Since this is only for hiding traffic no keys are verified (verify=0).

vi $HOME/.ssh/config/

tell ssh to wrap your traffic into https connection by setting

Host vicktricks.net
	ProxyCommand =/usr/bin/socat STDIO OPENSSL-CONNECT:%h:21022,verify=0,cert=/path/to/home/.ssh/client.crt,key=/path/to/home/.ssh/client.key

router

with the above specifications make sure the router you have at home and connects you to the wider internet forwards port 21022 to the IP address associated with your computer (where socat tunnel.crt/tunnel.key are run

As host OS with bhyve

Disk size

As a bhyve guest OS

Set a FreeBSD template

ZFS on FreeBSD

Bibliography

As a KVM guest OS

Test a ready made image

Prepare a ready made image

Expand disk after installation

Apache on FreeBSD

Install and configure Apache

Php on FreeBSD

install and configure

PostgreSQL on FreeBSD

Installation

Configuration for local access

Accessing psql interactive terminal

Export databases

Import databases

Useful BSD tools

OpenSSL

socat

As a `bhyve` guest OS