hammer2 - live dedup, cleanup

Operating Systems / DragonFlyBSD - Matthew Dillon [apollo.backplane.com] - 30 August 2015 01:39 UTC

- First attempt at a live dedup. The H2 strategy code now caches {data_off, crc} info to track recently accessed data blocks. The cache is checked in the strategy_write code after device-level block encoding. If we get a cache hit, the disk block is compared against the write data and reused if it matches.

- This 'live' dedup should catch most typical 'cp' or 'cpdup' style commands. There will also be a bulk dedup capable of catching everything.

- Note that 'df' output might be a bit confusing because the 'Used' field represents the topology and does not take into account dedups. 'Avail' is calculated from the actual freemap. To make things look right the total disk size is adjusted upward so it matches Used+Avail. This mechanism will likely change.

Here is an example with one copy of /usr/src and 13 copies of /usr/src. The first copy eats around 872MB, and a 'du' will show each copy eating about the same. But because of dedup each subsequent copy actually only eats around 160MB as you can see from the 'Avail' field:

test40# df -h /mnt Filesystem Size Used Avail Capacity /dev/serno/WD-WX51A82J2299.s1f@LOCAL 99G 934M 99G 1% Filesystem Size Used Avail Capacity /dev/serno/WD-WX51A82J2299.s1f@LOCAL 106G 8.5G 97G 8%

- Rename hammer2_bulkscan.c to hammer2_bulkfree.c since that is basically all it does.

- Move the synchronization code to its own file, hammer2_synchro.c. (note: This code is currently in rip-up mode and will not operate properly).

3f01eba hammer2 - live dedup, cleanup
sys/vfs/hammer2/Makefile | 3 +-
sys/vfs/hammer2/hammer2.h | 79 ++--
sys/vfs/hammer2/hammer2_bulkfree.c | 754 ++++++++++++++++++++++++++++++++
sys/vfs/hammer2/hammer2_bulkscan.c | 754 --------------------------------
sys/vfs/hammer2/hammer2_chain.c | 62 ++-
sys/vfs/hammer2/hammer2_flush.c | 2 +-
sys/vfs/hammer2/hammer2_freemap.c | 16 +-
sys/vfs/hammer2/hammer2_inode.c | 10 +-
sys/vfs/hammer2/hammer2_io.c | 203 +++++++--
sys/vfs/hammer2/hammer2_ioctl.c | 2 +-
sys/vfs/hammer2/hammer2_strategy.c | 199 ++++++++-
sys/vfs/hammer2/hammer2_synchro.c | 849 ++++++++++++++++++++++++++++++++++++
sys/vfs/hammer2/hammer2_thread.c | 807 +---------------------------------
sys/vfs/hammer2/hammer2_vfsops.c | 9 +
sys/vfs/hammer2/hammer2_vnops.c | 3 +
sys/vfs/hammer2/hammer2_xops.c | 12 +-
16 files changed, 2085 insertions(+), 1679 deletions(-)

Upstream: gitweb.dragonflybsd.org

  • Share