Failing DHCP Reservation - OSMC, Raspberry Pi 3 & OpenBSD

I recently installed OSMC (Open Source Media Center) on a Raspberry Pi 3.  Initially I simply let it connect via DHCP and completed the installation and configuration.  This went as expected with no obvious issues.  Another post will detail out the project, install and lessons learned.

My project required that the Raspberry Pi be setup for DHCP since it will likely be used in multiple locations.  However while in locations I control, I wanted to have a consistent IP and so I setup a DHCP reservation within the DHCPD.conf (OpenBSD).   

Disclaimer:  The issue I outline was not related to OSMC or the Raspberry Pi, but since this was the project I was working on when it happened it all gets some attention. 

DHCP Reservation - I got lazy and did something I typically wouldn't or knew I shouldn't.  Since I was in the middle of configuring and using the IP address that DHCP assigned, I did the unthinkable and setup the reservation with the same IP.  My intent was to complete the setup and then find a suitable IP for it.  

Setup was completed, device was running for a week and then because of another issue, DHCP was restarted and at some point the Pi was restarted.  The PI wouldn't reconnected to the wireless after the restart.  I won't go through every step of troubleshooting but multiple steps were taken like removing the Wifi setup and reconfiguring, connecting to a guest network etc.  

On first review of the DHCP logs, I saw nothing that stood out.  The PI would only get an IP if I removed the reservation.  I don't troubleshoot DHCP much these days, it has just worked so well unless there has been a syntax error in the configuration.  After a little more digging, I found an error in /var/log/messages stating that I either needed to remove the reservation or fix the .leases file (/var/db/dhcpd.leases).  This struck me as I had eventually changed the reservation to an IP not in the DHCP range.

More investigation revealed that the IP I had chosen for DHCP reservation, actually had been used before (at some point in time) and there was a lease still in the DHCP.leases file with the same IP.  I'm not sure the odds on this but strangely this occurred in two locations during my testing.

Had I taken the steps that I knew or once knew to do without fail, this issue would never have occurred.  I documented this as a reminder to myself and hopefully to help someone else who might make a similar mistake.

The moment I removed the offending line in the DHCP.leases file, I saw the PI connect to the IP stated in the reservation.

OpenBSD - Compact Flash Media Installation w/Read-Only Configuration

The core "Installation Guide" (coming soon) is compliments of Lin.  He was able to compile the information from various sources and has proven it out multiple times through various installations.  

I recently had reason to install a Alix2D3 and followed his tutorial.   I strayed slightly from the instructions and will note it at the beginning along with a few oddities that I encountered. 


If you choose not to configure the CF Card as read-only be advised that Flash memory, regardless of format, is limited to a finite number of erase/write cycles for any "block," before that block can no longer be written to successfully.  In other words, the life expectancy of your system could be very short.  

I had initially setup a device in a read-write state intending to return at a later date to configure it as read-only.  This proved catastrophic as I exceeded the write cycles of the CF card in a rather short period of time.  You have been sufficiently warned...better to learn from others miss fortune.

Component List:

• VMware Fusion - MacBook Pro
• OpenBSD 4.8 ISO 
• 2G CF Card
• ALIX2D3 - PCEngines mainboard & Case
• NULL Modem Cable 
• USB -> Serial cable
• Compact Flash Card Reader

Helpful Hints based on my most recent installation:

Each time I partitioned and formatted the CF card without a reboot, I encountered write errors on the CF card and the installation failed.  If you experience this same issue, I found that partitioning followed by an immediate restart of the installation was the answer.  This may simply be a problem related to Fusion or the method I am using for installation.

To minimize the installation footprint, I did not install any "x" packages.  If you do find something missing, it is a trivial matter to add at a later date.

I opted to use a 9600 baud rate throughout instead of 19200.

While using both "screen /dev/tty.usbserial 9600" and putty under windows to console to the Alix2D3 device, I experienced times where the output to the console was clean but it wouldn't accept any input from the keyboard.  This led me on a wild goose chase for a time but I was able to determine that this happens on both a windows and Mac OSX platform.  Resetting the terminal would not resolve the issue, it required that I move the usb -> serial cable to a different port.  No reset of the device was necessary.

I also utilized another OpenBSD virtual machine to mount the CF Card after the installation to make the minor modifications necessary in fstab.  If you choose to make additional changes or symbolic links while mounted in this way, please use extreme caution as any mistake could lead to an unusable installation.

If setting up as a firewall remember the following:

• /etc/mygate
• rc.conf (pf=YES)
• sysctl.conf (ip forwarding)
• pf.conf requires syntax changes for ver. 4.7 and later. 

-----------------------------------------------

Installing OpenBSD on CompactFlash

----------------------------------------------

First you need to get a compactflash or other media installed with OpenBSD

Download latest .iso image and save it on a box with vmware or other vm technology

Create a virtual machine with no disk drives, and point the cdrom to the OpenBSD install .iso file.

Boot from cd on the vm, thus entering the OpenBSD install process.

When you get to the (I)nstall, (U)pgrade or (S)hell? prompt, connect/insert the usb device holding your intended media to the usb hub of the virtual machine.  OpenBSD should recognize the media correctly if you insert it after boot.  Otherwise you might get errors asking about the geometry of the disk.

Proceed with normal install on sd0 or other appropriate device.

Make sure you include the ramdisk kernel in the install bsd.rd or whatever.

-----------------------------------------

Booting the New OpenBSD install

-----------------------------------------

After install insert your media (compact flash) into a running OpenBSD machine.  Mount the sd0a (if that is /) partition for example and enter the /etc dir.

Create a boot file (/etc/boot.conf) with the following, unless you answered yes to the questions regarding the console output when prompted during installation:

stty com0 19200

set tty com0

Now when you boot from the new install, the serial port will be used for the console, so you can connect from another PC via a serial cable using screen thus:

sudo screen /dev/ttyS0 19200

My Mac required syntax of

screen /dev/tty.usbserial 19200

A regular boot from the new install will probably fail as your media is now the boot disk and will be referenced differently (not sd0) by the OS, while the /etc/fstab file probably still says sd0a or whatever.

In order to fix this and other problems, you actually want to boot the ramdisk first.  To do so, use the following as the boot line in the above boot.conf to begin with.

boot wd0a:/bsd.rd 

or type

wd0a:/bsd.rd

at the boot prompt

When you get to the prompt, choose "S" for shell.

Now you can work with things to prepare for a real boot.

-------------------------------------------

Get into a Sane Working Environment

-------------------------------------------

Mount your partitions so that /mnt is the root of the newly installed system, and everything is mounted correctly under that.

Next, we chroot to /mnt thus:

# /mnt/usr/sbin/chroot /mnt

Set some environmental variables thus:

# export TERM=vt220

# export HOME=/

vi should work for you now, but if not it's located at /usr/bin/vi

Update the /etc/fstab file to reflect how your OS views your install media now.

/dev/wd0a / ffs rw 1 1 

Note: that the sd0a from the usb device specified to install OpenBSD on originally has been changed to wd0a

Find out what your network interfaces are by running ifconfig

Now copy /etc/hostname.xxx to create files for your interfaces such as

/etc/hostname.dc0

/etc/hostname.vr0

/etc/hostname.vr1

or whatever they are - you can even edit them to assign IP's etc if you like.

My notes indicate the following is important in the /etc/ttys file (baud rate):

tty00   "/usr/libexec/getty std.19200"  vt220    on secure"

YOUR SYSTEM SHOULD BOOT NORMALLY NOW, AND YOU SHOULD BE ABLE TO GET TO IT FROM A SERIAL INTERFACE LIKE HYPERTERMINAL, MINICOM, OR SCREEN. (But only if you edit the /etc/boot.conf file to boot regularly, and not to the ramdisk)

To do so, remove the boot line - default is to boot normally after 5 seconds

Note: While this should be true, I found that I had to add the boot parameter to the last line of my boot.conf file in order for my system to boot without manually typing return at the >boot prompt.

---------------------------------------------------

Tweak OpenBSD to boot from read only disk

---------------------------------------------------

Edit the /etc/fstab as follows to provide a 50MB ram disk /mfs

Note: that wd0a has been changed to read only (ro)

/dev/wd0a / ffs ro,softdep 1 1

#/dev/wd0a / ffs rw,noatime 1 1

swap /mfs mfs rw,nosuid,-P=/skel,-s=102400 0 0

Make the dir for the ram disk above

# mkdir /mfs

Now, we have written in fstab that the content of /skel (-P=/skel)

Directory must be copied in /mfs at boot.

So, now, all the files will be stored in /skel. Start by making the directories:

# mkdir /skel

# mkdir /skel/dev

Now we can make the rights link, but first we make a symbolic link to
the real path:

# cd /mfs

# ln -s /skel/dev/ .

---------------------

Create the devices

---------------------

# mv /dev/ptyp* /skel/dev/

# mv /dev/ttyp* /skel/dev/

# cd /dev/

# for i in /mfs/dev/* ; do ln -s $i . ; done

# rm -rf /mfs/dev

------------------------------

Important RW Directories

------------------------------

In the / (root) dir:

# mv /root/ /skel/

# mv /tmp/ /skel/

# cd /

# ln -s /mfs/root .

# ln -s /mfs/tmp .

In the /var dir:

# mkdir /skel/var

# mv /var/backups/ /skel/var/

# mv /var/cron/ /skel/var/

# mv /var/db/ /skel/var/

# mv /var/empty/ /skel/var/

# mv /var/log/ /skel/var/

# mv /var/mail/ /skel/var/

# mv /var/msgs/ /skel/var/

# mv /var/named/ /skel/var/

# mv /var/run/ /skel/var/

# mv /var/spool /skel/var/

# mv /var/tmp/ /skel/var/

# ln -s /skel/var/ /mfs/var

# cd /var/

Note:  The next line did not work under ksh.  I had to create the symbolic links manually.

# for i in /mfs/var/* ; do ln -s $i . ; done

# rm -rf /mfs/var/

That should pretty much do it.  You will still need to boot in read-write mode to create your ssh keys and a few other things might have to be done in rw mode...

Create lock and unlock files thus

mkdir /usr/local/conf

echo "mount -o rw,noatime /dev/wd0a /" > /usr/local/conf/unlock.txt

echo "mount -o ro,softdep /dev/wd0a /" > /usr/local/conf/lock.txt

now you can lock and unlock when needed as follows:

`cat /usr/local/conf/unlock.txt`

or

`cat /usr/local/conf/lock.txt`

You might want to leave the following line in fstab, just commented out

#/dev/wd0a / ffs rw,noatime 1 1

So you can easily adjust if you need to reboot in RW mode.

----------------------------------------

Examples of what stuff should look like

----------------------------------------

# ls -lhF /var/

total 40

# ls -lhF /skel

total 16

drwxr-xr-x   2 root  wheel   2.0K Jul  9 22:16 dev/

drwx------   2 root  wheel   512B Jul  9 22:24 root/

drwxrwxrwt   2 root  wheel   512B Jul  9 22:25 tmp/

drwxr-xr-x  13 root  wheel   512B Jul  9 22:29 var/

# ls -lh /mfs/

#

drwxr-xr-x   2 root  wheel      512B Jul  9 21:01 account/

drwxrws---   2 root  wheel      512B Jul  9 21:01 audit/

drwxrwx---   2 root  authpf     512B Jul  9 21:01 authpf/

lrwxr-xr-x   1 root  wheel       16B Jul  9 22:37 backups@ -> /mfs/var/backups

drwxr-xr-x   3 root  wheel      512B Jul  1 17:54 cache/

drwxrwx---   2 root  wheel      512B Feb 23 08:26 crash/

lrwxr-xr-x   1 root  wheel       13B Jul  9 22:37 cron@ -> /mfs/var/cron

lrwxr-xr-x   1 root  wheel       11B Jul  9 22:37 db@ -> /mfs/var/db

lrwxr-xr-x   1 root  wheel       14B Jul  9 22:37 empty@ -> /mfs/var/empty

drwxrwxr-x   5 root  games      512B Feb 23 08:26 games/

lrwxr-xr-x   1 root  wheel       12B Jul  9 22:37 log@ -> /mfs/var/log

lrwxr-xr-x   1 root  wheel       13B Jul  9 22:37 mail@ -> /mfs/var/mail

lrwxr-xr-x   1 root  wheel       13B Jul  9 22:37 msgs@ -> /mfs/var/msgs

lrwxr-xr-x   1 root  wheel       14B Jul  9 22:37 named@ -> /mfs/var/named

drwxr-x---   2 root  operator   512B Jul  9 21:01 quotas/

lrwxr-xr-x   1 root  wheel       12B Jul  9 22:37 run@ -> /mfs/var/run

drwxr-xr-x   2 root  wheel      512B Jul  9 21:01 rwho/

lrwxr-xr-x   1 root  wheel       14B Jul  9 22:37 spool@ -> /mfs/var/spool

lrwxr-xr-x   1 root  wheel       12B Jul  9 22:37 tmp@ -> /mfs/var/tmp

drwxr-xr-x  10 root  daemon     512B Jul  9 21:01 www/

drwxr-xr-x   2 root  wheel      512B Jul  9 21:09 yp/

 

Pf Tips

Some basic notes and commands to aid in diagnosing PF.

pfctl -d	Disable the packet filter
pfctl -e	Enable the packet filter
pfctl -Fa -f /etc/pf.conf	Flush all (nat, filter, queue, state, info, table) rules and reload from the file /etc/pf.conf
pfctl -s rules	Report on the currently loaded filter ruleset.
pfctl -s nat	Report on the currently loaded nat ruleset.
pfctl -s state	Report on the currently running state table (very useful).
pfctl -v -n -f /etc/pf.conf	This does not actually load any rules, but allows you to check for errors in the file before you do load the ruleset. This is obviously good for testing.

pflogd logs in tcpdump binary format

To watch only packets that match a certain interface use the following: 

tcpdump -n -e -ttt -i pflog0 host 192.168.0.1

To watch packets that match a specific port:

tcpdump -n -e -ttt -i pflog0 port 443

To determine a specific rule as identified from tcpdump.  For example rule #100 below.  Note:  This is not the same as simply editing the pf.conf file and going to line #100.  PF expands rules due to macros etc...

pfctl -vvs rules|grep \@100

PXE Boot | OpenBSD | Soekris

SET UP A PXE BOOT SERVER ON THE NETWORK

• Install OpenBSD on a box with more than one network interface. Set the server up on the network with the primary interface active and configured to talk on the network.

• Configure the 2nd interface on a private subnet for dhcp and tftp use by the new PXE box.

SET UP ACCESS TO THE BOOT SERVER

• edit /etc/sysctl.conf /etc/rc.conf and enable packet filtering and forwarding

• edit pf.conf to resemble the following simple, limited access structure

ext_if="sis0"
int_if="sis1"
set skip on lo
scrub in all
nat on $ext_if from !($ext_if) -> ($ext_if:0)
block in
pass out
pass quick on $int_if no state
pass in quick on $ext_if proto tcp from 159.212.48.138 to ($ext_if) port ssh
antispoof quick for { lo $int_if }


GET THE PXE FILES

• This box needs to download or acquire the latest PXE boot files from the OpenBSD project.

• If you need to go through a proxy, you should set env variable like following example.

export http_proxy="http://proxy.blah.com:8080"

• You can check this with the env command.

• Then use wget specifying the --proxy-user=username and --proxy-password=password as needed. An example follows:

wget --proxy-user=someuser --proxy-password=p4ssw0rd http://openbsd.mirrors.hoobly.com/4.3/i386/pxeboot

• You will also need to get the ramdisk kernel bsd.rd
• The Files can also be transfered from another server where they were downloaded. scp is recommended.
  

SET UP DHCP TO BOOT THE PXE BOX

• add something like the following to your dhcp.conf file, note the filename "pxeboot"; directive - this should be set to the name of your PXE boot file.

shared-network LOCAL-NET {
option domain-name "boot.net";
option domain-name-servers 159.212.72.72, 159.212.99.99;

subnet 10.0.10.0 netmask 255.255.255.0 {
option routers 10.0.10.1;
filename "pxeboot";
range 10.0.10.100 10.0.10 150;
default-lease-time 3600;
max-lease-time 10800;
}
}

• make sure to set your hostname.* file for the 2nd interface to the corresponding IP address of the option routers setting, in this example 10.0.10.1

• add the name of your 2nd interface to the dhcp.interfaces file

• start dhcpd manually with the following (user your own device name)

dhcpd sis1

• check for dhcpd running with

ps aux | grep dhcpd

• if it's not running, check for errors with

cat /var/log/messages | grep -i dhcp


SET UP TFTP TO SERVE THE FILES

• uncomment (remove the #) the line in /etc/inetd.conf as follows:

#tftp dgram udp wait root /usr/libexec/tftpd tftpd -s /tftpboot

• create the directory in the above line /tftpboot

• place the pxeboot and bsd.rd files in the /tftpboot directory

• you may want to create an etc subdir and put a boot.conf file in it.

• sample /tftpboot/etc/boot.conf for a soekris install would looke like this:

stty com0 19200
set tty com0

• either use a line in the boot.conf to specify the bsd.rd or rename the file bsd.rd to simply bsd

• restart inetd (not sure if this is needed, but I did it)

pkill inetd
inetd

SOEKRIS INSTALL

• Plug a crossover cable in to the 2nd interface of your INSTALL SERVER and the primary interface of the server you wish to install on.

• Boot the new server and enter the boot BIOS by typing CTL-P on a Soekris box.

• Give the PXE boot command

boot f0

• Its a regular OpenBSD install from here.

OpenBSD - Howto create bootable media (thumb drive)

1. Boot from CD or start your OpenBSD VM.
   
2. At the "Install, Upgrade or Shell" prompt, select the shell.
3. Insert the pen drive. You will see kernel messages about the insertion, including the virtual SCSI drive # assigned to the drive. For this example, I will use "sd0" but it may be another number.
4. Put an MBR on the drive, reserve all space as a single MBR partition for OpenBSD:
   # fdisk -iy sd0
5. Put an OpenBSD disklabel on the drive, assign all available space as a single "a" partition on sd0:
   # disklabel -E sd0
   Use "a a" to add partition a, take the defaults for all prompts, once the partition is created, use the "q" command to exit the disklabel program. Answer "y" to write the disklabel.
6. Format the "a" partition on sd0 as an OpenBSD FFS partition:
   # newfs sd0a
7. Mount it:
   # mount /dev/sd0a /mnt
8. Copy the second stage boot loader and the ramdisk kernel to it:
   # cp /bsd.rd /mnt
   # cp /usr/mdec/boot /mnt
9. Create an /etc/boot.conf on the drive so you do not have to request the ramdisk kernel manually at boot time:
   # mkdir /mnt/etc
   # echo set image /bsd.rd > /mnt/etc/boot.conf
10. Install the PBR:
    # /usr/mdec/installboot -v /mnt/boot /usr/mdec/biosboot sd0
11. Unmount the pen drive:
    # umount /mnt