Browsed by
Tag: sonarr

Telegram notifications with unRAID and Docker applications

Telegram notifications with unRAID and Docker applications

Telegram has been a dream come true. I recently discovered the simplicity of setting up a Telegram bot for getting notifications from my server. One of the grievances I’ve had since starting my server journey was the fact that I needed to constantly monitor things like available storage, cache disk usage, new downloads, etc.. I often found out too late when something was going wrong and had to spend some time restoring functionality to the server. I’ve finally found a super simple solution to this problem; A Telegram Bot to send automated notifications.

I’m not going to talk too much about creating or setting up a Telegram bot because I really didn’t spend any time playing with that functionality. What I am going to cover in this post is more about using the Telegram bot API for receiving notifications from unRAID and the various applications I have running like Sonarr, Radarr, and Ombi as well as some general system level stuff like available storage space.

 

Telegram Setup

There are a few things you need to do with Telegram before you get started with setting up notifications:

  1. Create a Telegram bot
  2. Optional: Create a channel where you’d like to have these notifications sent to. I say this is optional because you can have the messages sent directly to you instead.
  3. Get your channel ID

Create a Telegram bot

To create a bot you’ll need to start a conversation with @BotFather on Telegram. When you start a conversation with this bot you’ll be given a number of options, one of which is /newbot. You just send this command to the bot, follow the prompts on screen, and you’re done! It’s literally that easy to get started with your first Telegram bot. As proof here’s a screenshot of my conversation:

The end of the last message (which I’ve hidden) will provide you with your private bot API token. You will need to record this long string of characters for use later on.

Create your Telegram channel

As mentioned above this step is optional, but I would recommend it if you want to add additional users for notifications, or additional bots for other purposes. In my scenario I created a private channel and created separate bots for each application, then added them all to one channel. The reason I have separate bots for each application is because you can enable the “Sign Messages” option in Telegram. This option asks the bot to sign the messages so I know which bot sent which message. This allows me to know what application the message came from.

Get your Telegram channel chat ID

This part was less than obvious at first, but I discovered a super simple solution after some brief research. You can add the bot @get_id_bot to your channel. When the bot receives a message in this channel it will output the channel chat ID. If you want to be directly messaged instead of creating a new channel, you can just message the bot directly to get your personal chat ID. If you have added @get_id_bot to your channel you should remove it once you have your ID as it will continuously respond to every new message.

That’s it!

You’re now ready to get started! Below I’ll run through some simple notifications I have from Sonarr, Radarr and Ombi.

 

Sonarr & Radarr

I’ve used Sonarr as the example here, but this also applies to Radarr as they are basically the same application. You can find Telegram as one of the options found under Settings > Connect. Sonarr will allow you to send notifications whenever a new episode is grabbed, downloaded, upgraded or renamed. You just need to provide your Bot Token and Chat ID that we already recorded in the previous steps:

Options when setting up Telegram notifications in Sonarr

Unfortunately there doesn’t seem to be any (obvious) way of customising the notifications that are sent. I don’t have any problems with these notifications, but it would be nice to have the ability to customise the notification content.

 

Ombi

Ombi on the other hand does allow plenty of notification customisation using the list of defined variables, and various categories that can be enabled or disabled. You also have the option of using complete Telegram markdown or raw HTML formatting for each notification content:

There are links within this page in Ombi that explains the Telegram markdown, and possible variables that can be used. As an example I customised the issue notification to provide more detail:

{Alias} has reported a new issue for the title {Title}
*Category:* {IssueCategory}

*Subject*: {IssueSubject}

*Description:* {IssueDescription}

 

This gives me a nice overview of the problem without needing to log into Ombi to view the details.

 

System level unRAID alerts

The last area that I wanted to monitor was available storage space. From my last post I addressed hard links in Sonarr not working for me and storage being filled up due to duplicate files. While that issue was ongoing my server kept crashing because the storage kept being filled with hundreds of GBs of useless duplicates. I want to avoid something like that happening again and Telegram seems like an obvious choice here as it’s instantaneous so I can be proactive rather than reactive. However, unRAID doesn’t have any capabilities for defining notifications like this.. Until I discovered the “CA User Scripts” application. This incredibly flexible application allows you to write your own bash scripts and schedule them for execution.

I wrote a bash script that will calculate the amount of free space on the array and cache disk, and send a notification to my Telegram channel using the Telegram API:

#!/bin/bash
TOKEN=XXX
CHAT_ID=XXX
URL="https://api.telegram.org/bot$TOKEN/sendMessage"
LIMIT=95
CACHE_LIMIT=60

#Checks how much space% is consumed on the array
ARRAY_USED_PERCENT=`df -h /mnt/user | awk '{print $5}' | sed -ne 2p | cut -d'%' -f1`
#Checks how much available space there is in GB
ARRAY_AVAIL_SPACE=`df -h /mnt/user | awk '{print $4}' | sed -ne 2p`
#Checks how much space% is consumed on the cache drive
CACHE_USED=`df -h /mnt/cache | awk '{print $5}' | sed -ne 2p | cut -d'%' -f1`
#Checks how much available space there is in GB
CACHE_AVAIL_SPACE=`df -h /mnt/cache | awk '{print $4}' | sed -ne 2p`

if [ $ARRAY_USED_PERCENT -ge $LIMIT ] 
then
    MESSAGE="WARNING: Free Array Space is at `expr 100 - $ARRAY_USED_PERCENT`%. There is $ARRAY_AVAIL_SPACE available."
    curl -s -X POST $URL -d chat_id=$CHAT_ID -d text="$MESSAGE"
fi

if [ $CACHE_USED -ge $CACHE_LIMIT ] 
then
    MESSAGE="WARNING: Free Cache space has fallen below `expr 100 - $CACHE_USED`%. There is $CACHE_AVAIL_SPACE available."
    curl -s -X POST $URL -d chat_id=$CHAT_ID -d text="$MESSAGE"
fi

This script is scheduled to run every hour using the “CA User Scripts” app.

I definitely plan on making more use of these notifications.. but this been such an awesome ease-of-life improvement for me.

Sonarr and the “Use Hardlinks instead of Copy” conundrum with Unraid

Sonarr and the “Use Hardlinks instead of Copy” conundrum with Unraid

Sooo I haven’t updated my blog in quite a while.. In fact there hasn’t been a new post in about 18 months.. What can I say? I’ve been slacking! Today I decided to tackle a problem that has been causing me heartache with my server for quite some time, and I thought it was the ideal topic to talk about.. Sonarr gives me the option of using a “hard link” when moving files rather than copying the file and duplicating space.. But why isn’t it working?

 

Wait, what’s a hard link?

Let’s start with the basics. A hard link essentially allows you to link two (or more) files that both point to the same location on disk. In other words I can create a “link” between ‘File1‘ and ‘File2‘ such that opening either File will open the same source data from disk. Even though both File1 and File2 exist on the Filesystem, there is actually only one real copy of the data on disk. The benefit of hard links is that it allows a file to ‘exist’ in two or more locations but not take up any additional disk space.

When File1 and File2 have a hard link designated between them it doesn’t matter if I go to open File1 or File2, I am actually opening the same data.
root@Tower:/mnt/disk3/Documents# touch File1
root@Tower:/mnt/disk3/Documents# echo "Adding some text to File1" > File1
root@Tower:/mnt/disk3/Documents# ln File1 File2
root@Tower:/mnt/disk3/Documents# cat File2
Adding some text to File1

I could go ahead and delete File1 from the Filesystem, but it wouldn’t matter because I still have File2. The fact that File1 is the ‘original’ file is irrelevant once a hard link has been established. They are now both considered equal.
root@Tower:/mnt/disk3/Documents# rm File1
root@Tower:/mnt/disk3/Documents# cat File2
Adding some text to File1

In order for me to explain the problem I’m having with Sonarr it’s important that we first understand how this works. To do that we need to take a step back and talk about how files are stored in Unix like file systems.

 

Files and inodes

When we look at a file in Linux it encapsulates two distinct but linked concepts:

  1. File Content and structure
  2. Metadata and properties

The file content is written out to disk as raw blocks of data. The File will act as a reference point to the location of these blocks so we know where the File begins and ends. This information is stored using the concept of an “Index Node” or ‘inode’.

The inode is a data structure in Linux systems that describes the File; It will point to the location on disk as well as providing all of the Files associated metadata and permissions. Every File on Linux is associated with an inode number which is stored in an inode table. From the inode number the kernel’s file system driver can access the inode contents, including the location of the file on disk.

Finding the inode number

You can get the inode number for a file using the ‘ls -i‘ or ‘ls --inode‘ command:
root@Tower:/mnt/disk3/Documents# ls -i
414 File1

Above we can see that File1 has an inode number  of 414. So this means whenever a user attempts to read from File1 the Kernel will open the data at the disk location specified for this inode. To demonstrate the concept of a hard link more clearly, let’s create a link again and confirm the inode number for the new File matches:
root@Tower:/mnt/disk3/Documents# ln File1 File2
root@Tower:/mnt/disk3/Documents# ls -i
414 File1
414 File2

So now we can see that regardless of whether we try to open or edit File1 or File2 it does not matter as it refers to the same inode, and therefore the same disk location.

You can also find how many reference links (Or more simply put how many hardlinks) a file has by running “ls -l” or “ls -li” to include the inode number. You can see below each File has 2 links:
root@Tower:/mnt/disk3/Documents# ls -l
-rw-rw-rw- 2 root root 0 Jun 29 17:36 File1
-rw-rw-rw- 2 root root 0 Jun 29 17:36 File2

We can check where these references are using the inode number:
root@Tower:/mnt/disk3/Documents# find . -inum 414
./File1
./File2

 

So what’s the problem?

I have the Sonarr configurable option “Use Hardlinks instead of Copy” set to enabled. This option can be revealed by toggling “Advanced Settings” to “Shown” in the top right:

Advanced Options

You can find the “Use Hardlinks instead of Copy” option under “Importing”:

Use hardlinks instead of copy option

Theoretically this should allow for any new media file to exist simultaneously in the downloads folder and my Plex library. In an ideal world it would mean everything is fully automated for my downloads, however in practice the hardlink option does not seem to work. The files I download are being copied rather than linked therefore each file consumes twice as much space. Unfortunately this means frequent manual intervention by deleting the contents of the download directory to free up space.

After performing some initial checks I was immediately able to confirm that there is no hard link for new downloads. Using an example file that was just downloaded I could see there was only 1 reference:
root@Tower:/mnt/user/Media/downloads# ls -li
 71244 -rw-rw-r-- 1 nobody users 1346128584 Jun 29 15:58 FileName

The inode number also only returns one item:
root@Tower:/mnt/user/Media/downloads# find /mnt/user/Media -inum 71244
/mnt/user/Media/downloads/FileName

After some further research, this initial “proof” that hardlinks weren’t working turned out to be a quirk of the Unraid system..

 

Unraid and the FUSE filesystem

My research lead me to a post on the Unraid forums with a user stating the hard links do not work in Unraid. This comment from limetech himself set me straight:

“Your ‘ls -li’ command is listing “pseudo” inode numbers in the fuse file system.  The two names indeed point to same file.”

If we refer to the Unraid User guide we’ll find this little tidbit which expands on the above statement:

“User Shares are implemented using proprietary code which builds a composite directory hierarchy of all the data disks. This is created on a tmpfs file system mounted on /mnt/tmp. User Shares are exported using a proprietary FUSE pseudo-file system called ‘shfs’ which is mounted on /mnt/users.”

There is a number of things we can discern from the above conversation which will help our investigation:

#1: If I make a hardlink on a user share it will also make a hard link on the actual disk. Proof:
root@Tower:/mnt/user/Documents# touch UserShareFile
root@Tower:/mnt/user/Documents# ln UserShareFile UserShareHardLink
root@Tower:/mnt/user/Documents# ls -li /mnt/disk3/Documents
413 -rw-rw-rw- 2 root root 0 Jun 29 19:00 UserShareFile
413 -rw-rw-rw- 2 root root 0 Jun 29 19:00 UserShareHardLink 

#2: The shfs FUSE file system generates “pseudo” inode values and therefore there is no way to determine if a file in a user share has a hard link. The “ls -l” output will indicate that there is two links but we have no way of determining where they are. Proof:
root@Tower:/mnt/user/Documents# ls -li
321717 -rw-rw-rw- 2 root root 0 Jun 29 19:00 UserShareFile
321719 -rw-rw-rw- 2 root root 0 Jun 29 19:00 UserShareHardLink
root@Tower:/mnt/user/Documents# find . -samefile UserShareFile
./UserShareFile
root@Tower:/mnt/user/Documents# find /mnt/user -samefile UserShareFile
/mnt/user/Documents/UserShareFile

 

Re-starting the investigation

Now let’s go back and check out that file I downloaded in the previous section armed with the knowledge we learned from the forum gurus. As a reminder this is the file we’re dealing with:
root@Tower:/mnt/user/Media/downloads# ls -li
 71244 -rw-rw-r-- 1 nobody users 1346128584 Jun 29 15:58 FileName

This file actually exists on Disk3:
root@Tower:/mnt/disk3/Media/downloads# ls -li
7156367 -rw-rw-r-- 1 nobody users 1346128584 Jun 29 15:58 FileName

From the command output we can see that there is no additional links to this file, but let’s double check with the inode value:
root@Tower:/mnt/disk3/Media/downloads# find /mnt -inum 7156367
/mnt/disk3/Media/downloads/FileName

So we’re back to square one – The hardlink functionality is definitely not working, and this time we have proof!

After enabling debug level logging in Sonarr I finally found the culprit:
19-6-29 15:38:44.2|Debug|EpisodeFileMovingService|Hardlinking episode file: /downloads/FileName to /series/Series Name/Series Season X/FileName
19-6-29 15:38:44.2|Debug|DiskTransferService|HardLinkOrCopy [/downloads/FileName] > [/series/Series Name/Series Season X/FileName]
19-6-29 15:38:44.2|Debug|DiskProvider|Hardlink '/downloads/FileName' to '/series/Series Name/Series Season X/FileName' failed.
[v2.0.0.5322] Mono.Unix.UnixIOException: Invalid cross-device link [EXDEV]

 

The Resolution

You cannot have a hard link which spans across different mount points. All shares in Unraid are attached to the system via the /mnt/user mount point. However each Docker application can also define it’s own internal mappings for mount points. In this scenario I had provided Sonarr with a few different mounts at the root of the container:
/downloads => /mnt/user/Media/downloads
/series => /mnt/user/Media/series

Even though these are on the same system level mount point, the container sees them as two different mount points. To resolve this I simply need to map them to the same mount point within the container. I updated the config for Sonarr to put the two directories under a single mount point named media:
/media/ => /mnt/user/Media

I then had to go and apply the same mapping to Transmission. This unfortunately had a negative consequence of pausing all of my torrents as the source data could no longer be found under /series.  After painstakingly updating each torrent one-by-one and setting the new location, then verifying the source data location, I was back in action. Sonarr thankfully allows you to easily edit the root folder for series in bulk so that was a much more simple fix.

After fixing the container to only have a single mount point the hard links are now being created sucessfully:
19-6-29 22:44:20.7|Debug|EpisodeFileMovingService|Hardlinking episode file: /media/downloads/FileName to /media/series/SeriesName/SeasonName/FileName
19-6-29 22:44:20.7|Debug|DiskTransferService|HardLinkOrCopy [/media/downloads/FileName] > [/media/series/SeriesName/SeasonName/FileName]
19-6-29 22:44:20.7|Debug|DiskProvider|Setting permissions: 0644 on /media/series/SeriesName/SeasonName/FileName

I am also able to confirm this using the inum of the file:
root@Tower:/mnt/disk1/Media/downloads# find /mnt -inum 7280027077
/mnt/disk1/Media/series/SeriesName/SeasonName/FileName
/mnt/disk1/Media/downloads/FileName