Browsed by
Category: Desktop

Recovering the partition table of a corrupted USB stick using TestDisk

Recovering the partition table of a corrupted USB stick using TestDisk

Recovering the partition table of a corrupted USB

Yesterday I came across an extremely useful utility called TestDisk. I managed to rescue my Dads micro-SD card when all hope looked lost. Somehow the micro-SD card partition table became corrupted after his phone got wet. I have never heard corruption-by-water before but I guess that’s a thing!

First signs of partition table corruption

The first problem we noticed after drying the phone out was that most of the apps on his android device were missing. After removing the memory card and re-inserting it again a message popped up saying that he needed to format the card before it could be used. Obviously this doesn’t sound good and first my thought jumped straight to water damage. However despite this I powered on to see if I could avoid formatting the device. After attaching it to my desktop a similar message appeared: You need to format the disk in drive G: before you can use it. 

windows partition table formatIn disk management I could see the device as a RAW filesystem so the first thing I did was open up DISKPART and see if that could manipulate the partition. With DISKPART I encountered a weird a problem in that there was only one partition (Partition 1) and it was marked as active (Denoted by a *) however all commands were failing with an error stating that a partition must be selected. This also meant that CHKDSK would not work either. Still refusing to give up I moved onto the tried-and-trusted GParted ISO VM, but this only gave me the option of creating a new partition table which in turn would require formatting the device.

TestDisk – My saviour

It was at this point I happened to stumble across a utility called TestDisk while trying to Google a solution for my corruption issue. Their website states that TestDisk is a free and open source data recovery software tool designed to recover lost partition and unerase deleted files. Recovering a lost partition sounded like exactly what I needed so I figured I would give it a go.. and it worked! TestDisk managed to rescue the partition table and restore partition 1 as the active partition. We plugged the memory card back into my Dad’s phone and voila all his apps were back to normal.

How to use TestDisk to recover a partition table

While I’m here I figure I may as well show you how to use TestDisk and the procedure I followed. The first prompt you receive after launching the TestDisk executable is whether or not you wish to create a log file of the completed actions. If you choose to create the text file, testdisk.log , it will contain TestDisk options, technical information and various
outputs; including any folder/file names TestDisk was used to find and list onscreen. I went with ‘No Log’:

TestDisk log partition table

Now we are good to go with finding the partition..

  1. First step is to choose your media device. Mine was listed as ‘Disk /dev/sdd’ at the time.  You can choose your appropriate device via your arrow keys on your keyboard.
  2. Next you choose your partition table type from the list. For this step I went with ‘EFI GPT’
  3. Lastly you choose the ‘Analyse’ option. At this point I was given a message stating that there were no partitions found, but after another scan it found the primary partition.
  4. Use to arrow keys to navigate to the partition and press the return key (Enter)
  5. Now you will be given the option to ‘Write’ the partition table

This managed to successfully recover the partition table without formatting the sdCard and therefore retaining all of the data. I am so happy I stumbled across TestDisk and I know I’m going to end up needing it again in the future.

SSD benchmark – My desktop SSD

SSD benchmark – My desktop SSD

SSD benchmark – My desktop SSD

Time for a SSD benchmark test! My desktop currently has a 120GB M500 Crucial SSD installed for booting my OS and related applications. It’s not exactly a top-of-the-line drive by any means but it’s more than enough for every day use. The spec sheet for this model claims I should be getting up to 500 MB/second reads and 400MB/s writes. So when I saw the results of my CrystalDiskMark SSD benchmark test I was underwhelmed to say the least. It’s worth nothing that this drive was 75% full at the time of the test so I understand this has the potential to affect speeds – But not this much!

desktop ssd benchmark

56MB/s sequential writes?? Time to figure out what’s going on here..

Troubleshooting steps

First off I needed to ensure that AHCI was enabled. AHCI stand for Advance Host Controller Interface. This is a hardware mechanism that allows software to communicate with Serial ATA (SATA) devices such as SSDs. Windows supports AHCI out-of-the-box by default but just to be sure I went into device manager to confirm the AHCI controller was enabled and running.

ssd device manager

So it’s enabled – Great! Now I just need to confirm that the SSD is being managed by this controller. Right click the controller, choose properties, then navigate to the ‘Details’ tab. In this section you are greeted with a drop down menu where I chose ‘Children’ and could see my SSD listed so AHCI is definitely enabled.

ssd controller properties

Then I needed to confirm that ‘TRIM’ was enabled. TRIM support is essential for an SSD to run the way it should to avoid slow writing times. This command allows an operating system to inform a solid-state drive (SSD) which blocks of data are no longer considered in use and can be wiped internally. To test if this is enabled you can run the command “fsutil behavior query DisableDeleteNotify”. If this returns as 0 then TRIM is enabled.

The next step was to make sure I was at the latest revision of the firmware. The firmware download for my SSD comes as an ISO package so I stuck it onto a USB using unetbootin and made a backup of my current SSD before proceeding – I’ve had enough bad experiences to warrant backups! Thankfully the installation completed successfully without any issues and I was upgraded from MU03 to the MU05 revision. Rebooted and went through another SSD benchmark with CrystalDiskMark. No improvement whatsoever.

ssd desktop benchmark

So considering there was little to no change in the speeds I figured I would confirm the results with another benchmark tool Atto disk benchmark. Now I’m seeing more along the lines of the expected speeds! At my maximum I reached over 500MB/s reads and maxed out about 140MB/s writes. I’m still a little disappointed with the write speeds though but this is a start at least.

atto disk benchmark
As this was an SSD I was recommended to test out ‘AS SSD Benchmark’ which also confirmed that AHCI was enabled and my alignment was okay. This reported speeds were pretty much along the same lines as what I’ve seen so far with write speeds being reported between 113MB/s and 134MB/s. Still disappointing.

ssd benchmark

I went ahead and asked someone else to run the same test on their Crucial M4, which is an older model than mine. He was clocking double my write speeds with similar read speeds using the same test config as me. Only other difference between his test and mine was that his drive has higher total capacity with about 50% free whereas I only had 25% free. I went about reducing the space consumed on my drive and then re-ran the test only to receive roughly the same numbers again..

The conclusion

Sorry to get your hopes up folks! After a bit more digging I realised this was down to false advertising rather than anything being wrong with my drive / configuration. The M500 series drives are capable of up to 400MB/s write speed at the higher capacity spectrum of drives. My specific model – CT120M500SSD1 – I found is only capable of about 130MB/s which more or less matches up with the speeds I was recording in some of the benchmarks. Here is a screenshot from the product page on newegg:

newegg specifications ssd

If you found this interesting, take a look at my unRAID benchmark tests also!

 

Giving up on Proxmox, working with Hyper-V

Giving up on Proxmox, working with Hyper-V

I have abandoned Proxmox!

The installation of Proxmox should have been straightforward but unfortunately it just was not to be. I attempted the installation process from scratch this time and came out the other end with the same result.. Debian refusing to boot with the pve kernel. This left me with a few options..

  • Install Proxmox on a different flavour of Linux
  • Install the bare metal ISO of Proxmox
  • Go with a different hypervisor

proxmox logo

After weighing up my options and a bit more research I decided to just ditch Proxmox altogether. The first attempt at installation didn’t leave me with much confidence and I was tired of messing about with bootable USBs. I decided to go completely against my original decision and try out something different..

hyper-v over proxmox

Client Hyper-V over Promox

Client Hyper-V is the name for the virtualisation technology that was introduced in Windows 8. It is not enabled by default so many people probably don’t even realise they have it. It needs to be enabled via the control panel. Client Hyper-V is more or less a slightly more limited version of the Server implementation of Hyper-V. From reading the technet article regarding these limitations I do not think they are going to affect me, but I did read somewhere that the free version can only run a small number of VMs. I haven’t really looked into this to be sure, but I don’t see this being a major issue yet. The only requirements for enabling Hyper-V are:

  • Your desktop must have at least 4GB of RAM. Mine has 16GB so I have more than enough for multiple VMs
  • Your CPU must support SLAT technology (Second Level Address Translation). My AMD 8350 supports SLAT so this also not an issue.

Ok, good to go!

Does my CPU support SLAT?

Microsoft has a handy little utility called coreinfo that allows you to check for this. Once you have downloaded it you will need to extract it to some directory of your choice. Then you will want to open a command prompt (Admin) in this directory by pressing Win + x and choosing “Command Prompt (Admin)”. Now navigate to the directory where you stored coreinfo. Now run the command ‘coreinfo -v’. On an AMD if your processor supports SLAT it will have an asterix in the NP row as below:

SLAT check for hyper-v instead of proxmox

Enabling Hyper-V

Because Hyper-V is an optional feature it will need to be enabled via the control panel. Open the Control Panel, click Programs, and then click Programs and Features. Click Turn Windows features on or off. Find Hyper-V in this list and enable it, click OK, and then it will request to reboot your machine.

enable hyper-v instead of proxmox

Easy right?

The fun begins..

Guess what!?! More issues! yay! For some reason I am just not allowed to implement any sort of virtualisation technology outside of VirtualBox. Damn you Oracle, what you have done! During my first attempt at enabling Hyper-V it reached about 90% progress after restarting so I just figured I had missed a prerequisite or something on those lines.. So I tried again and got the same result. The message appearing on my screen was “We couldn’t complete the features”. Interesting, but at least I didn’t turn my PC into a paperweight this time. Some furious google-fu brought me all kinds of results but the primary answer seemed to be related to virus guards, but I do not have one installed other than Windows Defender or Microsoft Security Essentials – Whatever the built in option is called for Windows 8. So I am working under the assumption that this is not responsible.

A second common answer for this was a backlog of Windows Updates preventing the hyper-v installation from completing. Apparently my computer just hates me because even something as simple as getting windows updates to install was proving difficult. Windows continuously refused to connect to the update server every time I tried checking. I kept getting Windows Update Error 0x80243003 in return so I needed to run a utility from Microsoft for repairing Windows update. This worked… after running it countless times as each time I ran it there was a new problem! Eventually after getting all the updates installed I attempted to enable Hyper-V once more and this time reached about 93%! On top of that the error changed to “We couldn’t complete the updates” which I guess can be considered progress.. ?

hyper-v error while replacing proxmox

I tried reviewing the event logs for my installation attempt but this did not shed any light for me as to what happened. Moving on the final relatively common resolution was to enable bit locker on the drive where hyper-v is installed. Was worth a try.. I mean, what harm could it do?

Bitlocker – Windows encryption

BitLocker lets you encrypt the hard drive(s) on your Windows based system. It’s basically there to protect your data on the off chance that someone robs your physical computer so they will not be able to boot or access your data without the password. So went through the process of enabling BitLocker..

enabling bitlocker while replacing promox
To cut a long story short, it worked but my computer didn’t play nice with it (surprise surprise). Following a reboot after enabling bitlocker I was greeted with an orange and white striped screen like below:

orange screen after enabling bitlocker
Not quite paperweight material this time as I quickly realised I could still type in the password that I had set despite having nothing to look at. I just typed and hoped and thankfully it worked out. I found this rather annoying and potentially a dodgy situation since I can’t see what or where I’m typing.. I did not test to see if Hyper-V worked as I did not have the patience to let the process completely encrypt the SSD before disabling it..

So.. What now?

I never did get Hyper-V enabled and I do not think I will any time soon. So for now I am going to decide on how to proceed from here.. Surely I’ll get something working eventually..

hyper-v is not going to work instead of proxmox

Proxmox – Tried installing it and it didn’t go so well..

Proxmox – Tried installing it and it didn’t go so well..

Proxmox refused to install

When I finally got my PC back working as expected and was successfully able to boot into both Linux and Windows I figured it was about time to actually install my hypervisor. As I mentioned in my previous post I had decided to install Proxmox as a level two hypervisor on Debian. The instructions for this seemed pretty straightforward, but alas as per my luck so far it was not…

proxmox logo

I used unetbootin to create the bootable USB for Proxmox. If you have not heard of unetbootin then I highly recommend it.

Getting started with Proxmox

The installation steps listed for Debian on the Promox site seemed relatively straightforward. The first step involved checking my hosts file to ensure that my hostname is resolvable:

luke@debian:~$ cat /etc/hosts
127.0.0.1 localhost

127.0.1.1 debian.mydomain.com debian
Next I needed to edit my sources list to add the Proxmox VE repository. If you have ever used Linux before you have more than likely come across the command apt-get before. Well Apt uses a file that lists the ‘sources’ from which packages can be obtained. This file is /etc/apt/sources.list. The below three entries needed to be added to this list:

luke@debian:~$ nano /etc/apt/sources.list
deb http://ftp.at.debian.org/debian wheezy main contrib
deb http://download.proxmox.com/debian wheezy pve
deb http://security.debian.org/ wheezy/updates main contrib

Add the Proxmox VE repository key:
luke@debian:~$ wget -O- "http://download.proxmox.com/debian/key.asc" | apt-key add -

Update your repository and system by running
luke@debian:~$ apt-get update && apt-get dist-upgrade

Install the kernel and kernel headers
luke@debian:~$ apt-get install pve-firmware pve-kernel-2.6.32-26-pve
luke@debian:~$ apt-get install pve-headers-2.6.32-26-pve

At this point the next step is to restart and select the pve kernel. It’s a surprisingly straightforward process. I honestly thought it would be more complicated. However, as is the usual paradigm so far it’s just one step forward two steps back with everything…

Problems begin..

Once again I am having some issues with actually booting the system. After rebooting into GRUB and choosing the appropriate kernel, this appears on my screen..

Loading, please wait...
Usb 5-1: device descriptor read/64, error - 62
Fsck from util-linux 2.25.2
/dev/mapper/debian--vg-root: clean, 166811/7012352 files, 1550982/28045312 blocks

Left it at that for well over an hour before just turning it off and have not gone back to it since. Proxmox has just straight out refused to work with me so far and at this point I’m just about ready to give up and continue using VirtualBox.. We’ll see..

Installing Debian, write-protected USB, and Windows MBR..

Installing Debian, write-protected USB, and Windows MBR..

I tried installing Debian and somehow broke Windows

My attempt to install Proxmox didn’t go quite as smoothly as I wanted – quite the opposite in fact. I ended up spending hours fixing the utter mess I made of my computer. My Windows PC had more or less become an expensive paper weight for a brief period of time. All of this happened when I tried install Debian.. Read on for more..

Creating a bootable USB for Debian

When building my PC I made the decision to not purchase a DVD drive as I figured I never use it. This means when installing an operating system I need to create what is known as a bootable USB – Basically the equivalent of inserting your Windows installation DVD when installing Windows. I have done this numerous times in the past and usually it’s a straight forward process. Not this time though. For some reason my USB stick had become write protected meaning that I could not write any new files to it. This also meant I couldn’t format it either so it was pretty much useless. Great start!

windows write-protected usb

The disk is write-protected

Remove the write-protection or use another disk

I brought up DISKPART in the hopes that I could just remove the attribute and then try cleaning it. Nothing is straight forward though and this too just spat back an error at me stating that the disk is write protected. “DiskPart has encountered an error: The media is write protected“.

windows diskpart
So at this point I can’t write to it. I can’t format it. I can’t clean it using DiskPart. I was about to give up and just get another USB but I figured trusty Ubuntu might be able to help. I have an Ubuntu VM that I use as a development environment with Oracle VirtualBox but I don’t consider that an ideal solution, hence me looking for a decent hypervisor.. Anyway back on topic! I booted up my Ubuntu VM and for some reason managed to resolve the issue using fdisk! Here’s what I did:
The following command will list all detected hard disks:

# fdisk -l | grep '^Disk'

Disk /dev/sda: 251.0 GB, 251000193024 bytes

Disk /dev/sdb: 251.0 GB, 251000193024 bytes

You want to find the appropriate entry for your disk and then run fdisk against that disk
# fdisk /dev/sdb

After successfully completing the required tasks I created a new ext3 filesystem on the USB
# mkfs.ext3 /dev/sdb1

And it worked! The read only attribute was removed but obviously the USB was not visible in Windows due to the unsupported filesystem. So back into DISKPART I went, ensured the attribute had been removed and then ran the clean command. Brought up the Windows DiskManagement GUI and formatted the USB with NTFS. Success!

All that effort just to make the USB work? What now?

Remember that I haven’t even gotten around to getting the ISO onto the USB yet! In my previous post I mentioned that Proxmox could be used as either a type 1 or type 2 hypervisor so I felt that my life might be easier if I installed it as a type 2 on Debian. So I loaded the Debian ISO onto the USB using a piece of software called unetbootin. This is probably the one and only thing that went smoothly throughout this whole process. Rebooted my computer, chose the USB from the boot menu and followed the installation process successfully to my spare SSD. No issues so far and everything appeared to be working properly until I restarted and realised that Windows wasn’t appearing in the GRUB boot menu. Everything on the SSD was still accessible and I could mount the drive in Debian no problem. A quick google lead me to believe this was an easy fix:
# os-prober
# update-grub

Alas Windows failed to appear in the menu after these two commands. Some research into this was pointing me towards a BIOS option called ‘Secure Boot’ which might not be allowing the Windows UEFI to interact with GRUB. So in my attempt to resolve this I set my BIOS to SecureBoot UEFI only and then disabled an option called ‘Compatibility Support Module’. I am not a smart man. After saving these settings and restarted the computer I was now facing a black screen. No BIOS splash screen, no GRUB, nothing. I went from a fully functional windows desktop, to a semi-functional debian desktop, right down to an expensive paperweight all in the course of about one hour. Impressive stuff.

My first step was to consult my motherboard documentation in which I learned I have a button on my motherboard that allows me to boot straight into the BIOS which is quite a handy feature to have. Unfortunately this button was no use to me and I still couldn’t see anything other than a black screen. Some further digging made me realise at this point my only option was to short the motherboard CMOS in an attempt to reset it. Thankfully this allowed me back into the BIOS and from there I was able to get back into Debian. Phew!

Windows Startup Repair isn’t very useful

To rule out any Debian / GRUB involvement in the Windows issue I disconnected the Debian SSD to try and force a Windows boot. “Reboot and Select proper Boot device or Insert Boot Media in Selected Boot device and press a key” appeared on my screen. To cut a long story short it turns out I somehow managed to destroy my Windows Master Boot Record (MBR) while installing Debian. I still have not figured out how and I’m not sure I ever will. So I install gparted on Debian, reformat the USB and install Windows using unetbootin. With this I attempted to complete a startup repair but this just told me the repair failed. I brought up the command prompt and ran the command ‘bootrec /fixmbr’ against the Windows SSD and rebooted – This time I’m no longer getting the ‘Reboot and Select proper Boot device or Insert Boot Media in Selected Boot device and press a key’. Instead I’m getting a blinking cursor so I don’t really know if I’ve made progress or gone backwards..

At this point I’m really thinking anything that can go wrong will go wrong so I’m just waiting for the whole thing to burst into flames. Fixing the MBR was quite a painful process in which I learned if you are ever running startup repair from the windows installation disk, run it 3 times. Don’t ask why, just do it. So back into startup repair I went and ran it 3 times despite it telling me it failed each time. Then:
bootrec /fixmbr
bootrec /fixboot
bootrec /scanos
bootrec /redbuildbcd

At this point the rebuildbcd failed to add the windows entry to the MBR. Went back into DISKPART, set the partition as active again then restarted and got a new error about failing to read the boot configuration table. Went through another 3 startup repairs for good measure and volia, Windows booted. Easy as that. Only took about 6 hours of work..

Now to actually try and install Proxmox..

Choosing a hypervisor

Choosing a hypervisor

The confusion of choosing a hypervisor

As I discussed in my previous post there is a wealth of information out around the web on virtualisation, and when I started researching what hypervisor to choose it was no different. In fact the more I looked the more indecisive I found myself being. However I recalled my first post on this topic and the article I had mentioned reading about setting up gaming machines and utilising what is known as PCI pass-through. I quickly learned that this doesn’t work well with Nvidia graphics cards and this turned out to be deciding factor for me in the end.

Hypervisor PCI pass-through; What is it and why does it matter?

If you have ever tried playing a game on a virtual machine you would quickly realise that this is not a viable solution at all for a gaming desktop. This is because normally the GPU is emulated by the hypervisor and a resource manager carves up the resources and passes them to the individual machines. This is where PCI pass-through comes into play – Rather than sharing the resources of the GPU among the multiple machines, you can have the hypervisor assign the whole card to the machine so it has independent ownership of it. There is no virtualisation layer in between the GPU and the VM managing resources so this allows for near native performance. In theory you should not even know that you are using a VM!

Many months ago I decided to get two GTX 970’s for my gaming desktop rather than opting for an AMD alternative. I am living to regret that decision somewhat as I am now learning that Nvidia does not allow their consumer grade GPU’s to utilise this pass-through technology. For this privilege you need to upgrade to their Quadro series which from what I can tell offer no other benefits other than allowing pass-through. Did I mention they’re also much more expensive and far inferior when compared to their GTX counterparts? Nice one Nvidia! Since I don’t plan on replacing my GPUs any time soon so this has more or less ruled out ESXi for me but I learned that it is possible (with a lot of effort) to implement this on a Linux based hypervisor such as KVM / Proxmox.

And the winner is..

proxmox hypervisor

I narrowed my choices down to KVM and Proxmox (Which is based on KVM) as the only two viable options. In the end I decided I would proceed with Proxmox as my hypervisor for the simple reason that it has built-in web GUI for management and it has the option of either a type 1 or 2 hypervisor. This leaves me with plenty of flexibility and simple management.

The hypervisor

The hypervisor

The hypervisor

I never realised until today just how many hypervisor options are out there. Not only how many but that there are different types as well. Obviously I had heard of the industry standards ESXi and the Microsoft alternative Hyper-V but little did I realise that is only scratching the surface. You’ve also got XenServer, Proxmox and KVM just to name some of the more popular hypervisor options. In the last few days I have managed to go from having a good idea of what I wanted to implement, to landing myself in a vast sea of information that just seems limitless. Each hypervisor has it’s benefits and limitations when compared to it’s competitors so I have a lot of research to do before making my final decision and sticking with it. So let’s start with the basics:

What is a hypervisor?

A hypervisor is a piece of software that can create and run virtual machines. The computer or server that this software runs on is known as the host, and the virtual machines are known as guests. A certain defined portion of the resources from the host machine such as memory and disk space are allocated to a guest machine to use. These guest machines have no idea that they do not own these resources – As far as the VM (Virtual Machine) is concerned it is a single independent entity. The hypervisor is actually controlling the resources of the host and distributing them as required.

There are in fact two types of hypervisor – Type 1 and Type 2. What type you choose really doesn’t matter as they serve the same purpose, but for educations sake the distinction is there. A type 1 hypervisor is commonly known as a ‘bare metal hypervisor’ because it runs directly on the hardware of the host. In other words there is no operating system or any other software in between the hardware layer and hypervisor.

hypervisor type 1
Img Source: https://www.flexiant.com

The other type is a Type 2 hypervisor, commonly referred to as a hosted hypervisor. This is software that runs on an operating system. One of the more common examples of this would be Oracle Virtualbox or VMware Workstation. I guess the main disadvantage of a hosted hypervisor is that you are adding an unnecessary extra layer to the environment. However this has the benefit of making management somewhat more intuitive in my opinion.

hypervisor type 2
Img Source: https://www.flexiant.com

Which is the best hypervisor option?

If you find out please let me know! Generally people are quick to recommend ESXi due to the fact it has more or less become the industry standard and having experience in such a widely used product would be beneficial. This was part of my original reasoning behind ESXi because I do encounter it frequently in work but I don’t have a whole lot of experience working with it. It makes ESXi very hard to ignore for this reason, but on the flip side the possibilities are somewhat limited without purchasing a license. I’m not going into the differences between the free and licensed options in this post, but it is definitely something I am going to do in the future.

Then of course you have Hyper-V, another very commonly implemented option from Microsoft. This has the massive advantage of being free and comes bundled with Windows Server as an add-on feature. If you are running Windows 8 at home you can also enable Client Hyper-v which isn’t as feature-filled as the Server alternative but offers the same purpose. Finally you have your Linux alternatives which are also mostly free if you are not interested in receiving support. These options are much less widely used but are certainly growing in popularity.

I’ve got a lot of work to do….