I've been waiting for this ever since it was announced - and deduplication is now available in ZFS! (hence thishastily written blog...)If you're running OpenSolaris 2009.06 and are pointing to the hot-off-the-presses repository you can upgrade by using System -> Package Manager -> File -> Updates, or use the CLI example shown in Roman Ivanov's blog.Some painless downloading and a reboot, and you have access to the bits that contain ZFS dedup.$ uname -aSunOS suzhou 5.11 snv_128a i86pc i386 i86pc Solaris$ cat /etc/release OpenSolaris Development snv_128a X86 Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Use is subject to license terms. Assembled 23 November 2009
Now, I'm a rather paranoid guy, and the machine in question is my main desktop, so I'm goingto experiment on a ZFS pool in a ramdisk, rather than the pool I keep all my data. At least for today!Here, I create a 100MB ramdisk, put a ZFS pool in it, andturn on dedup for its top-level ZFS dataset. I then create a ZFS dataset in it, and hand it tomy normal, non-root userid.
Note: the error message below is because you don't turn on 'dedup' at the pool level - you turn it on at the ZFS dataset level. This is despite the (possibly confusing fact) that candidate blocks for deduplication are acrossthe entire ZFS pool (not the dataset), and the space savings ratio due to deduplication are reported at the pool level.So, in this case, the zpool command thinks I'm looking for the deduplication ratio, which is a read-only attribute.After that error, I turn dedup on for the ZFS dataset at the top of the pool. Out of habit, I turn on compression too.# ramdiskadm -a dimmdisk 100m/dev/ramdisk/dimmdisk# zpool create dimmpool /dev/ramdisk/dimmdisk# zfs create dimmpool/fast# zpool set dedup=on dimmpoolcannot set property for 'dimmpool': 'dedup' is readonly# zfs set dedup=on dimmpool# zfs set compression=on dimmpool# chown savit /dimmpool/fastNow, I'll populate that ZFS dataset with a few directories and copy some files into the first of them:$ mkdir /dimmpool/fast/v1$ mkdir /dimmpool/fast/v2$ mkdir /dimmpool/fast/v3$ mkdir /dimmpool/fast/v4$ mkdir /dimmpool/fast/v5$ cp *gz /dimmpool/fast/v1$ zpool list dimmpoolNAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOTdimmpool 95.5M 5.32M 90.2M 5% 1.00x ONLINE -No duplicate data so far, which isn't a surprise,now for the real test. Let's copy the same data into different directoriesand see what happens:$ cp *gz /dimmpool/fast/v2$ cp *gz /dimmpool/fast/v3$ cp *gz /dimmpool/fast/v4$ cp *gz /dimmpool/fast/v5$ zpool list dimmpoolNAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOTdimmpool 95.5M 6.55M 89.0M 6% 5.00x ONLINE -
Wow! I have 5 copies of the same data and ZFS trimmed away the excess copies when storing on disk.A bigger test

Well, that's probably a "best case scenario". Let's try something bigger, and use a real disk.I'll use a spare ZFS pool called 'tpool'.Since this is a ZFS pool I've had for a while, I have to upgrade the pool to the latest on-disk format, which I show here.Note which level (21, in listing below) provides deduplication.For a change of pace, I'll do this via pfexec from my userid, instead of being root.$ zpool list tpoolNAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOTtpool 736G 87.1G 649G 11% 1.00x ONLINE --$ pfexec zpool status tpool pool: tpool state: ONLINEstatus: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable.action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on older software versions. scrub: none requestedconfig: NAME STATE READ WRITE CKSUM tpool ONLINE 0 0 0 c18t0d0p2 ONLINE 0 0 0errors: No known data errors$ zpool upgrade -vThis system is currently running ZFS pool version 22.The following versions are supported:VER DESCRIPTION--- -------------------------------------------------------- 1 Initial ZFS version 2 Ditto blocks (replicated metadata) 3 Hot spares and double parity RAID-Z 4 zpool history 5 Compression using the gzip algorithm 6 bootfs pool property 7 Separate intent log devices 8 Delegated administration 9 refquota and refreservation properties 10 Cache devices 11 Improved scrub performance 12 Snapshot properties 13 snapused property 14 passthrough-x aclinherit 15 user/group space accounting 16 stmf property support 17 Triple-parity RAID-Z 18 Snapshot user holds 19 Log device removal 20 Compression using zle (zero-length encoding) 21 Deduplication 22 Received propertiesFor more information on a particular version, including supported releases, see:http://www.opensolaris.org/os/commun...version/NWhere 'N' is the version number.$ pfexec zpool upgrade tpoolThis system is currently running ZFS pool version 22.Successfully upgraded 'tpool' from version 16 to version 22Now I can turn dedup on. As I mentioned before, you turn it on at the ZFS dataset level.Here, I want all ZFS datasets in this pool to inherit this property, so I set it at the topmost level.$ pfexec zfs set dedup=on tpool$ zfs get dedup tpoolNAME PROPERTY VALUE SOURCEtpool dedup on local$ zpool get dedupratio tpoolNAME PROPERTY VALUE SOURCEtpool dedupratio 1.00x -Now I'll copy about 28GB worth of data - note how the free disk space is only reduced by 5GB!$ zpool list tpoolNAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOTtpool 736G 206G 530G 28% 1.02x ONLINE -$ cp -rp /usbpool/Music /tpool/$ zfs list tpool/MusicNAME USED AVAIL REFER MOUNTPOINTtpool/Music 27.9G 513G 27.9G /tpool/Music$ zpool list tpoolNAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOTtpool 736G 211G 525G 28% 1.20x ONLINE -Conclusions


This is a powerful new feature for ZFS - providing deduplication as a new native feature of the most advanced file system.This currently is a feature of OpenSolaris, not Solaris 10, as consistent with Sun's practice of introducingnew features in OpenSolaris before placing them in Solaris 10. Plus - we leverage OpenSolaris in our OpenSolaris-based series of storage appliances - the Sun Storage 7000 Unified Storage Systems -(a bit wordy, that title) another example of leveraging our intellectual property to provideunique solutions for the systems and storage markets.

More...