home :: FreeBSD

ZFS and USB disks

Having played with ZFS and USB disks over the past few weeks on FreeBSD 9.0, I have come to the conclusion that multiple USB disks + ZFS raid-z is not an excellent combination... USB disks can appear or disappear dynamically, and ZFS does not deal with this beautifully.

ZFS does deal with devices appearing and disappearing, but with an understanding that devices are mostly stable. So, I will use ZFS on internal disks, and for USB disks, I will limit myself to single-disk volumes (which seem to work fine).

I would be interested in a filesystem that would deal with disks being plugged in (or plugged out) dynamically (USB disks or otherwise), but I am not sure that this exists. I will hunt around for something like this.

/FreeBSD | Posted at 17:13 | permanent link

Content Addressable Storage

Because I am backing up a whole bunch of machines that may have identical files on them (many copies of Windows, old backups, etc), I am interested in finding ways of having n copies of a file not occupy n times the amount of storage.

I see three ways of doing this:

1. Using ZFS file depuplication on my backup server. This sounds good, but all of the ZFS configuration guides I've seen recommend having lots of RAM when you use dedup, and my little backup servers have only 2 to 4 GB of RAM, so this seems like a non-starter.
2. Using a manual dedup script that uses hardlinks to free up diskspace on a Unix volume... Something like dedup from Roderick Schertler (documented here), trimstrees at CPAN, harlink from Julian Andres Klode, opendedup, or fileuniq on sourceforge (there is a list here) actually. That sounds attractive, but I am not sure how long it would take on 1 TB of data, and I am not sure hardlinks will work well on files owned by multiple users.
3. Then, there is the possibility of Content Addressable Storage. EMC does this on its Centera product line, and it is quite useful in the Corporate space... in the Open Source world, I have been interested by Poul-Henning Kamp's stow package for a while (see here for the old version or here for the new version that phk is working on this year).

Stow looks interesting... I'll try to find some time to dig further into it.

/FreeBSD | Posted at 16:48 | permanent link

Going for simple FreeBSD instead of FreeNAS

So, I have been playing with my small soekris net6501-70-based NAS device for home.

Lots of things have happened since I last blogged about this:

1. I decided not to use FreeNAS 8 on my net6501, but rather to just use FreeBSD 9 with ZFS, NFS, Samba, and rsync.
2. I have copied my olde NSLU2 shared space onto my net6501 under ZFS... it took over 24 hours to copy over the network, the NSLU2 being fairly slow.
3. I have set up a script to take snapshots of my ZFS storage on a regular basis using a script I found on the internet: zfs-snapshot.sh... this is almost as nice as have the Mac OS X time machine on my little NAS server (ok, not as nice, but still pretty good)
4. I have had some difficulty with an old Maxstor 1TB USB drive, which seems to appear/disappear... it has caused corruption in my main ZFS pool, which I needed to fix manually, and then scrub the ZFS pool (which took almost 7 hours).
5. Given all of this, I have ordered a small HP microserver. External USB drives are handled strangely by server O/Ses as they appear/disappear, change device names, etc. Let see how well the microserver with internal drives will do.
6. By the way, when dealing with External USB disks, never use device names (such as /dev/da0 or /dev/da1s1a...); these names can change dynamically as drives are plugged in, or powered off, etc. Always label your filesystems (e.g. for UFS: tunefs -L disk1 /dev/da0s1a, which then appears as /dev/ufs/disk1; or for GPT disks: /dev/gptid/760ae9e2-5988-12f1-a136-000054cf2648) and use these names in ZFS or fstab... this will avoid a LOT of grief when reboting, or disconnecting or reconnecting a device, or indeed when a device fails.

So, lots is happening with my small NAS server.

More on this next weekend.

/FreeBSD | Posted at 01:01 | permanent link

Upgrading FreeBSD from 8.2 to 9.0

I have just finished upgraing one of my FreeBSD machines from 8.2 to 9.0, rebuilding from source as explained in the FreeBSD handbook here.

One of the painful steps in upgrading a FreeBSD machine, is the mergemaster step, as it forces you to go through a whole bunch of files, just to accept CVS tag changes... or so I thought !!! Turns out that there are optional flags to mergemaster, and running "mergemaster -FiU" does pretty much what I've been hoping mergemaster to do. Yay!

/FreeBSD | Posted at 00:41 | permanent link

FreeNAS, ZFS, and external USB disks

I have set up my Soekris net6501 with the latest beta of FreeNAS 8, FreeNAS-8.2.1-ALPHA-r10110M-x64. I installed 4 external 1 TB external USB disks on the machine, and set them up as a RAID-Z ZFS volume in FreeNAS and set about exporting CIFS and NFS partitions to the rest of my household. FreeNAS 8 is going to be an excellent product when it releases v.8.2.1... I look forward to seeing it in use in the coming months.

However, I have received the following FreeNAS alert twice now: The ZFS volume status is UNKNOWN: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'.

Interestingly enough, both times the error was corrected without any problem and the ZFS pool was back in HEALTHY state when I looked into it, but still that message sounds fairly omnious. Could it simply come from the fact that the drives go to sleep and are sometimes not ready when ZFS tries to access them (hence they somehow timeout)??? Or is it something more serious and one of my (new) drives is about to go south?

I wonder whether there are any recommendations against setting up a RAID-Z zpool across external USB drives that can go to sleep (and hence take 30-60 seconds to wake up). Hmmmm, I should investigate ZFS best practices and see if USB drives are usable this way. Else, I'll just stick with UFS with rsync for replication/backup.

More next weekend :-)

/FreeBSD | Posted at 23:34 | permanent link

Identity services

A quick follow-up to my recent blog entry about identity servers on small home networks:

• I have been playing with FreeRadius (available on FreeBSD as part of the ported packages -- /usr/ports/net/freeradius2, I believe) to see if I could use that as an identity server for my home network... to be investigated further in the coming weeks.
• On a loosely related topic, I find the myOpenID services interesting, especially the "myOpenID for domains" stuff. It seems like I could use this to enable OpenIDs like http://openid.rax.org/my-name to work? It looks like it might be free if I have 10 IDs or less on my domain. Interesting. However, I need to see what level of independence I would need to lose on my domain (including the ability to change providers in the future) if I did that.
• More interesting stuff to investigate on future weekends :-)

/FreeBSD | Posted at 00:13 | permanent link

Moving forward with FreeNAS

FreeNAS 8 is a nanobsd configuration, based on FreeBSD 8.2. In order to boot FreeNAS 8 on my Soekris net6501 under the amd64 image of FreeNAS, one needs to rebuild the FreeBSD kernel that is at the heart of FreeNAS with the line "device mptable" added to the kernel configuration file, as described on the unofficial Soekris wiki here.

Therefore, I used the instructions from the FreeNAS Forums here to set up a VMware virtual machine running FreeBSD to setup the FreeNAS sources, adjusted the kernel configuration file in "/usr/local/freenas/trunk/nanobsd/FREENAS.amd64" to add the device mptable line as described above, and then built a FreeNAS-8.2.1-ALPHA-r10006M-x64 image that I put onto a USB key and booted my Soekris net6501 with it.

FreeNAS 8 seems to look good on the net6501 and I am able to set it up with 2 external 2 TB USB disks partitioned with ZFS. For some strange reason, the FreeNAS 8 website seems to call for at least 6 GB of RAM to run it with ZFS, but my 2 GB net6501 seems to do just fine so far.

So, I just need to complete the set up of my FreeNAS box, export one of the disks through CIFS, NFS, and rsync, and set up a regular backup from disk1 to disk2. Once that runs fine, I can probably connect a 3rd 2 TB drive to that net6501, and set up some form of RAID-5 between the 3 drives (with snapshots, perhaps I don't need the weekly backup from disk1 to disk2, as I normally do on my NSLU2).

FreeNAS seems like a winner so far. More info later.

Miscellaneous comments & ramblings:

• I still need Soekris to fix the USB boot code on the net6501 before I consider this completely operational: the net6501 will freeze on boot when my two 2 TB external USB drives are connected at boot time. Works fine when I connect them after the machine has booted... if at least Soekris could add an option in the BIOS to not scan certain USB ports for bootable devices, I could move forward.
• FreeNAS 8 is based on FreeBSD 8.2, which is unfortunate as it only has ZFS v15, as opposed to FreeBSD 9.0 which comes with ZFS v28 (and perhaps the upcoming FreeBSD 8.3 also?)
• There appears to be a certain amount of tension between the "new" FreeNAS 8 team, and the "legacy" FreeNAS 7 team. The latter has based FreeNAS 7.5 on FreeBSD 9.0 to get the newest FreeBSD stuff (including ZFS v28 as stated above) and is starting to talk about calling its branch FreeNAS 9, which is not going over too well with the FreeNAS 8 people (who actually own the FreeNAS trademark). Seems like FreeNAS 7 will either die or else it will need to fork off with another name (see here).
• I don't have a strong opinion on this at this point, except for the fact that I am disappointed that FreeNAS 8 specifies minimum RAM requirements which seem to me to be unreasonably high (4 GB minimum, and 6 GB minimum if one wants ZFS) for some of the small machines that one could want to run FreeNAS upon. I like the FreeNAS 7 community effort, but I fear that it will end up as a dead-end effort once FreeNAS 8 is solid & feature-complete.
• Anyway, I decided to set up my NAS server as a FreeNAS 8 server, to see if it ran OK in 2 GB of RAM (which, frankly, I don't see why it couldn't).
• Seems like the net6501 will work as a home NAS device anyway, so finally I don't need the HP Proliant Microserver I was referring to the other day.

Anyway, enough for tonight... good night to all.

/FreeBSD | Posted at 23:28 | permanent link

What are people using for small identity servers at home?

So, on my home network I have:

• a DHCP server
• a web server
• a CIF file server
• an NFS file server
• an rsync backup server
• a network printer
• a mail server

and various other odds and ends.

However, it strikes me that I do not have a small and simple identity server, where I could define user accounts for all of my various devices and OSes (Windows, Mac OS X, FreeBSD, and probably some Linux also).

What do people use? NIS yellow pages? that seems too unixoid and will probably not help with the Windows machines. Some sort of Microsoft Active Directory? Through Samba? that feels too microsofty perhaps. Is there something that bridges the gap? Going beyond my little home network, is there something that would let me be accepted as an OpenId authenticator? Also, something I could use to allow logins into my machines hosted outside of the home network.

Hmmm. Over the years, we have gotten pretty good at doing IP networks, file sharing, printer sharing, web serving, and even web services and the like... but we are still struggling with having some form of simple authentication mechanism standardized. Probably because it is hard to have trust in authentication mechanisms, and especially the quality of the data they contain(*).

Anyway, that's another interesting weekend project.

(*) Perhaps we are setting the bar too high? I do not need strong authentication with non-repudiation and all that jazz... I would like a simple userid/password mechanism that would return something like the old unix uid/gid, but in 64-bit range, and I would like it to be easily integrated with Windows, Unix, and Mac OS X. Hmmm... fuzzy requirements... it's always more complicated than it looks like. What are people using out there? Any solid and usable public domain reference implementations I could look at and implement? Would Radius fit the bill?

/FreeBSD | Posted at 02:28 | permanent link

Setting up a new NAS device at home

For the last few years, I have been using a couple of Linksys NSLU2 NAS devices coupled with external USB disks for shared storage across my home network. I find that this works quite well to store files from my Mac, Windows, and even FreeBSD machines. One of the NSLU2 is unslung and runs an rsync daemon that I use to save files from my various FreeBSD machines.

The NSLU2 is a nice device, but it had become fairly old (no gigabit ethernet, limited expandability of the software, etc). So I have been meaning to set up a replacement. I have decided to use a Soekris net6501 to build my own little NAS box, using a couple of external 2 TB external drives.

Now, the question is: what software will I run on the net6501? I have been quite tempted by FreeNAS, but FreeNAS 8 has a stated minimum RAM size requirement of 4 GB (with 6 GB listed as the minimum for ZFS usage), and the net6501 is limited to 2 GB RAM (actually, I think that's even a limit of the Intel Atom CPU that it is built upon). FreeNAS 0.7 has a significantly lower RAM requirement, but I am afraid that it will be orphaned, as the main development path for FreeNAS is now version 8.

Of course, I could just set up FreeBSD on the net6501 and be done with it.

Anyway, I still have a few weeks to decide, as I am waiting for a new BIOS release for the net6501, as I am having some issues with the new USB boot code of the Soekris device: if I plug in 2 large (2 TB) external USB disks, in addition to the small internal 8 GB USB key that holds the FreeBSD image, then the device hangs on boot while probing the USB devices. If I just connect the external USB drives after the machine is booted, it works perfectly. I am hoping that the USB probe code will be fixed in the next BIOS release.

Fun for the weekend :-)

/FreeBSD | Posted at 02:06 | permanent link

Setting up FreeBSD 9.0 with a ZFS root filesystem: Success!

So, I investigated last week's problem with ZFS and FreeBSD 9.0 and I discovered that the problem was very simple to fix. I needed to add:

    # echo 'zfs_enable="YES"'>>  /mnt/etc/rc.conf

to the procedure described last week (on January 15th) so that the zfs mountpoint options would be executed. So now, things are working fine... I am going to play with this configuration a bit to see if the fact that I haven't configured a swap space causes problem with a 2GB memory configuration when I try to compile world.

More later.

Late addition to this post: turns out that the author of the initial page at http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/9.0-RELEASE had also found the error and fixed it on the wiki, but I had not gone back to check. In the end, it's OK, this has allowed me to explore ZFS enough so that I now understand it better.

/FreeBSD | Posted at 19:49 | permanent link

Setting up FreeBSD 9.0 with a ZFS root filesystem

With the new version of FreeBSD being just out, I decided to try out the ZFS filesystem. Notably, I would like to find out whether it works correctly with 2GB of memory, or whether more is really needed (I read that it doesn't really work with less than 1GB of RAM, but then again, there are conflicting reports that it needs lots more, though sometimes that's qualified with 'if you want prefetch' or 'if you are trying to do dedup'). Actually, I am building a small NAS server I am building around a Soekris net6501-70 which comes with 2GB RAM, and I would like to find out if I can use ZFS with it.

So, I looked up http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/9.0-RELEASE and tried to set up a vmware virtual machine with ZFS only.

I did not bother setting up swap space, nor a mirrored disk configuration, so the commands I used were:

• boot a vmware virtual machine from the amd64 FreeBSD9.0-RELEASE DVD
• choose the Shell option at the partitioning dialog in bsdinstall
• run the following commands:

    # gpart create -s gpt da0
    # gpart add -s 64K -t freebsd-boot -l boot0 da0
    # gpart add -t freebsd-zfs -l disk0 da0
    # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
    # zpool create -o altroot=/mnt zroot /dev/gpt/disk0
    # zfs create zroot/tmp
    # chmod 1777 /mnt/tmp
    # zfs create zroot/usr
    # zfs create zroot/var
    # zfs create zroot/home
    # exit

• And then, after installation:
  (update: one must add # echo 'zfs_enable="YES"'>> /mnt/etc/rc.conf here as per the post on 21 January)

    # echo 'zfs_load="YES"'>>  /mnt/boot/loader.conf
    # echo 'vfs.root.mountfrom="zfs:zroot"'>>  /mnt/boot/loader.conf
    # zfs unmount -a
    # zpool export zroot
    # zpool import -o cachefile=/tmp/zpool.cache -o altroot=/mnt zroot
    # zfs set mountpoint=/ zroot
    # cp /tmp/zpool.cache /mnt/boot/zfs/
    # zfs unmount -a
    # zpool set bootfs=zroot zroot
    # zpool set cachefile='' zroot
    # zfs set mountpoint=legacy zroot
    # zfs set mountpoint=/tmp zroot/tmp
    # zfs set mountpoint=/usr zroot/usr
    # zfs set mountpoint=/var zroot/var
    # zfs set mountpoint=/home zroot/home

The last 4 commands (i.e. after 'zfs set mountpoint=legacy zroot') give error messages, but I think that's OK. However, once I rebooted the virtual machine, the filesystems (except for /) were not mounted and therefore FreeBSD did not come up properly.

So, this was a first try... More on this next weekend :-)

/FreeBSD | Posted at 22:57 | permanent link

FreeBSD 9.0 is out

Yay! FreeBSD 9.0 is out!

The announcement says:

The FreeBSD Release Engineering Team is pleased to announce the availability
of FreeBSD 9.0-RELEASE.  Some of the highlights:
    - ZFS updated to version 28
    - NFS subsystem updated, new implementation supports NFSv4 in addition to NFSv3 and NFSv2

Please see http://www.FreeBSD.org for more info.

Note also that FreeBSD 9.0-RELEASE is dedicated to the memory of Dennis M. Ritchie, one of the fathers of UNIX.

/FreeBSD | Posted at 21:19 | permanent link

FreeBSD 9.0-RELEASE ISO are out

Found the ISO images for FreeBSD 9.0-RELEASE on ftp.freebsd.org and ftp4.fr.freebsd.org.

I've already got the amd64 version installed on a vmware player virtual image successfully from the ISO -- which works beautifully with FreeBSD under Windows 7; check it out!

I'm off to Zurich for a couple of days... more FreeBSD over the weekend.

/FreeBSD | Posted at 03:05 | permanent link

Waiting for the next FreeBSD release

I have been setting up a small computer -- a Soekris net6501-70 -- to use as a NAS (Network Attached Storage) device on my home network. The idea is to replace the small, reliable, Linksys NSLU2 (slugs) that I have been using for shared network usage in my home, with a new device with more disk space and Gigabit Ethernet ports since my home network (at least the wired part) is Gigabit Ethernet.

Ideally, I would have liked to run FreeNAS on the device and to run ZFS as a filesystem, but it didn't turn out that way, as the net6501 has 2 GB RAM, and FreeNAS requires a lot more than that.

Therefore I have installed FreeBSD 9.0-RC3 on my net6501-70 and I even had the pleasure to discover that I could run FreeBSD in amd64 mode (i.e. 64-bits mode).

Now, I cannot wait for FreeBSD 9.0 to be finally and officially released, and for the next iteration of the ne6501 combios to be effectively release.

Can't wait!

/FreeBSD | Posted at 00:29 | permanent link