Browsed by
Month: June 2019

Sonarr and the “Use Hardlinks instead of Copy” conundrum with Unraid

Sonarr and the “Use Hardlinks instead of Copy” conundrum with Unraid

Sooo I haven’t updated my blog in quite a while.. In fact there hasn’t been a new post in about 18 months.. What can I say? I’ve been slacking! Today I decided to tackle a problem that has been causing me heartache with my server for quite some time, and I thought it was the ideal topic to talk about.. Sonarr gives me the option of using a “hard link” when moving files rather than copying the file and duplicating space.. But why isn’t it working?

 

Wait, what’s a hard link?

Let’s start with the basics. A hard link essentially allows you to link two (or more) files that both point to the same location on disk. In other words I can create a “link” between ‘File1‘ and ‘File2‘ such that opening either File will open the same source data from disk. Even though both File1 and File2 exist on the Filesystem, there is actually only one real copy of the data on disk. The benefit of hard links is that it allows a file to ‘exist’ in two or more locations but not take up any additional disk space.

When File1 and File2 have a hard link designated between them it doesn’t matter if I go to open File1 or File2, I am actually opening the same data.
root@Tower:/mnt/disk3/Documents# touch File1
root@Tower:/mnt/disk3/Documents# echo "Adding some text to File1" > File1
root@Tower:/mnt/disk3/Documents# ln File1 File2
root@Tower:/mnt/disk3/Documents# cat File2
Adding some text to File1

I could go ahead and delete File1 from the Filesystem, but it wouldn’t matter because I still have File2. The fact that File1 is the ‘original’ file is irrelevant once a hard link has been established. They are now both considered equal.
root@Tower:/mnt/disk3/Documents# rm File1
root@Tower:/mnt/disk3/Documents# cat File2
Adding some text to File1

In order for me to explain the problem I’m having with Sonarr it’s important that we first understand how this works. To do that we need to take a step back and talk about how files are stored in Unix like file systems.

 

Files and inodes

When we look at a file in Linux it encapsulates two distinct but linked concepts:

  1. File Content and structure
  2. Metadata and properties

The file content is written out to disk as raw blocks of data. The File will act as a reference point to the location of these blocks so we know where the File begins and ends. This information is stored using the concept of an “Index Node” or ‘inode’.

The inode is a data structure in Linux systems that describes the File; It will point to the location on disk as well as providing all of the Files associated metadata and permissions. Every File on Linux is associated with an inode number which is stored in an inode table. From the inode number the kernel’s file system driver can access the inode contents, including the location of the file on disk.

Finding the inode number

You can get the inode number for a file using the ‘ls -i‘ or ‘ls --inode‘ command:
root@Tower:/mnt/disk3/Documents# ls -i
414 File1

Above we can see that File1 has an inode number  of 414. So this means whenever a user attempts to read from File1 the Kernel will open the data at the disk location specified for this inode. To demonstrate the concept of a hard link more clearly, let’s create a link again and confirm the inode number for the new File matches:
root@Tower:/mnt/disk3/Documents# ln File1 File2
root@Tower:/mnt/disk3/Documents# ls -i
414 File1
414 File2

So now we can see that regardless of whether we try to open or edit File1 or File2 it does not matter as it refers to the same inode, and therefore the same disk location.

You can also find how many reference links (Or more simply put how many hardlinks) a file has by running “ls -l” or “ls -li” to include the inode number. You can see below each File has 2 links:
root@Tower:/mnt/disk3/Documents# ls -l
-rw-rw-rw- 2 root root 0 Jun 29 17:36 File1
-rw-rw-rw- 2 root root 0 Jun 29 17:36 File2

We can check where these references are using the inode number:
root@Tower:/mnt/disk3/Documents# find . -inum 414
./File1
./File2

 

So what’s the problem?

I have the Sonarr configurable option “Use Hardlinks instead of Copy” set to enabled. This option can be revealed by toggling “Advanced Settings” to “Shown” in the top right:

Advanced Options

You can find the “Use Hardlinks instead of Copy” option under “Importing”:

Use hardlinks instead of copy option

Theoretically this should allow for any new media file to exist simultaneously in the downloads folder and my Plex library. In an ideal world it would mean everything is fully automated for my downloads, however in practice the hardlink option does not seem to work. The files I download are being copied rather than linked therefore each file consumes twice as much space. Unfortunately this means frequent manual intervention by deleting the contents of the download directory to free up space.

After performing some initial checks I was immediately able to confirm that there is no hard link for new downloads. Using an example file that was just downloaded I could see there was only 1 reference:
root@Tower:/mnt/user/Media/downloads# ls -li
 71244 -rw-rw-r-- 1 nobody users 1346128584 Jun 29 15:58 FileName

The inode number also only returns one item:
root@Tower:/mnt/user/Media/downloads# find /mnt/user/Media -inum 71244
/mnt/user/Media/downloads/FileName

After some further research, this initial “proof” that hardlinks weren’t working turned out to be a quirk of the Unraid system..

 

Unraid and the FUSE filesystem

My research lead me to a post on the Unraid forums with a user stating the hard links do not work in Unraid. This comment from limetech himself set me straight:

“Your ‘ls -li’ command is listing “pseudo” inode numbers in the fuse file system.  The two names indeed point to same file.”

If we refer to the Unraid User guide we’ll find this little tidbit which expands on the above statement:

“User Shares are implemented using proprietary code which builds a composite directory hierarchy of all the data disks. This is created on a tmpfs file system mounted on /mnt/tmp. User Shares are exported using a proprietary FUSE pseudo-file system called ‘shfs’ which is mounted on /mnt/users.”

There is a number of things we can discern from the above conversation which will help our investigation:

#1: If I make a hardlink on a user share it will also make a hard link on the actual disk. Proof:
root@Tower:/mnt/user/Documents# touch UserShareFile
root@Tower:/mnt/user/Documents# ln UserShareFile UserShareHardLink
root@Tower:/mnt/user/Documents# ls -li /mnt/disk3/Documents
413 -rw-rw-rw- 2 root root 0 Jun 29 19:00 UserShareFile
413 -rw-rw-rw- 2 root root 0 Jun 29 19:00 UserShareHardLink 

#2: The shfs FUSE file system generates “pseudo” inode values and therefore there is no way to determine if a file in a user share has a hard link. The “ls -l” output will indicate that there is two links but we have no way of determining where they are. Proof:
root@Tower:/mnt/user/Documents# ls -li
321717 -rw-rw-rw- 2 root root 0 Jun 29 19:00 UserShareFile
321719 -rw-rw-rw- 2 root root 0 Jun 29 19:00 UserShareHardLink
root@Tower:/mnt/user/Documents# find . -samefile UserShareFile
./UserShareFile
root@Tower:/mnt/user/Documents# find /mnt/user -samefile UserShareFile
/mnt/user/Documents/UserShareFile

 

Re-starting the investigation

Now let’s go back and check out that file I downloaded in the previous section armed with the knowledge we learned from the forum gurus. As a reminder this is the file we’re dealing with:
root@Tower:/mnt/user/Media/downloads# ls -li
 71244 -rw-rw-r-- 1 nobody users 1346128584 Jun 29 15:58 FileName

This file actually exists on Disk3:
root@Tower:/mnt/disk3/Media/downloads# ls -li
7156367 -rw-rw-r-- 1 nobody users 1346128584 Jun 29 15:58 FileName

From the command output we can see that there is no additional links to this file, but let’s double check with the inode value:
root@Tower:/mnt/disk3/Media/downloads# find /mnt -inum 7156367
/mnt/disk3/Media/downloads/FileName

So we’re back to square one – The hardlink functionality is definitely not working, and this time we have proof!

After enabling debug level logging in Sonarr I finally found the culprit:
19-6-29 15:38:44.2|Debug|EpisodeFileMovingService|Hardlinking episode file: /downloads/FileName to /series/Series Name/Series Season X/FileName
19-6-29 15:38:44.2|Debug|DiskTransferService|HardLinkOrCopy [/downloads/FileName] > [/series/Series Name/Series Season X/FileName]
19-6-29 15:38:44.2|Debug|DiskProvider|Hardlink '/downloads/FileName' to '/series/Series Name/Series Season X/FileName' failed.
[v2.0.0.5322] Mono.Unix.UnixIOException: Invalid cross-device link [EXDEV]

 

The Resolution

You cannot have a hard link which spans across different mount points. All shares in Unraid are attached to the system via the /mnt/user mount point. However each Docker application can also define it’s own internal mappings for mount points. In this scenario I had provided Sonarr with a few different mounts at the root of the container:
/downloads => /mnt/user/Media/downloads
/series => /mnt/user/Media/series

Even though these are on the same system level mount point, the container sees them as two different mount points. To resolve this I simply need to map them to the same mount point within the container. I updated the config for Sonarr to put the two directories under a single mount point named media:
/media/ => /mnt/user/Media

I then had to go and apply the same mapping to Transmission. This unfortunately had a negative consequence of pausing all of my torrents as the source data could no longer be found under /series.  After painstakingly updating each torrent one-by-one and setting the new location, then verifying the source data location, I was back in action. Sonarr thankfully allows you to easily edit the root folder for series in bulk so that was a much more simple fix.

After fixing the container to only have a single mount point the hard links are now being created sucessfully:
19-6-29 22:44:20.7|Debug|EpisodeFileMovingService|Hardlinking episode file: /media/downloads/FileName to /media/series/SeriesName/SeasonName/FileName
19-6-29 22:44:20.7|Debug|DiskTransferService|HardLinkOrCopy [/media/downloads/FileName] > [/media/series/SeriesName/SeasonName/FileName]
19-6-29 22:44:20.7|Debug|DiskProvider|Setting permissions: 0644 on /media/series/SeriesName/SeasonName/FileName

I am also able to confirm this using the inum of the file:
root@Tower:/mnt/disk1/Media/downloads# find /mnt -inum 7280027077
/mnt/disk1/Media/series/SeriesName/SeasonName/FileName
/mnt/disk1/Media/downloads/FileName