Safely removing SATA hard drive from a running system

I have a Debian-based system with a SATA controller running in AHCI mode. I sometimes need to plug and unplug a disk from an internal SATA bay. The bay is nothing fancy – you just slide a disk cradle into it and the SATA and power sockets make contact.

Wyzard on Unix & Linux Stack Exchange
has a good advice on how to do a proper hot unplug. The following is based on
that advice:

First you need to unmount the partitions. Then, deactivate any LVM groups with vgchange -an – I don’t use LVM, so I haven’t tried that. Finally, it is a very good idea to unregister device from the kernel and spin it down by

echo 1 > /sys/block/(device name)/device/delete

Then I can hear the disk spin down, wait for it to stop completely and just pull it out. When I put in back in, it registers properly. Please note, that I am using AHCI mode in the controller.

How to spin down USB hard drive for safe removal

Let’s assume you are running a Debian-based system and have a hard drive at /dev/sdc with a partition /dev/sdc1
mounted. To safely remove the hard drive, first you need to unmount all its
partitions. In this case you may do:

udisks --umount /dev/sdc1

and then spin the disk down with:

udisks --detach /dev/sdc

You should the notice (usually, hear) the disk spin down.

Please note, that we are referring here to the device sdc not its partition
sdc1.

Thanks, user76204.

Remove all .pyc files from a git repository

When working on Python project you may sometimes forget to .gitignore your
*.pyc files and end up having them tracked by the repo.

To fix this you need use git rm:

find . -name "*.pyc" -exec git rm -f {} \;

Then, add a line:

*.pyc

to the .gitignore file in the repository root to have them permanently
ignored.

Credit goes to Yuji.

htdebug, a script for tracing HTTP redirects

Recently, I have written a simple Python script that helps trace HTTP redirects
along with information on rel=canonical links. I called it htdebug.

Here it is:

#!/usr/bin/env python

import urllib
import urllib2
import sys
import requests

import lxml
from lxml import etree
from cStringIO import StringIO

def get_canonical(html_content):
    parser = etree.HTMLParser()
    tree = etree.parse(StringIO(html_content), parser)
    canonical_hrefs = tree.xpath("/html/head/link[@rel='canonical']/@href")
    return canonical_hrefs

if __name__ == "__main__":
    url = sys.argv[1]
    print "Probing {} for redirect path.".format(url)
    response = requests.get(url)
    for resp in response.history+[response]:
        print
        print "[{}] {}".format(resp.status_code, resp.url)
        print "canonical link: {}".format(get_canonical(resp.content))

After placing it in a executable file named htdebug in your $PATH, you can
use it like so:

$ htdebug http://yahoo.com
Probing http://yahoo.com for redirect path.

[301] http://yahoo.com/
canonical link: []

[200] https://www.yahoo.com/
canonical link: ['https://www.yahoo.com/']

HTTP status codes are shown in square brackets.

BuiltWith lookup script

BuiltWith is a web service that lets you check what web technologies are used on a given website. This simple script lets you do that directly from command line.

#!/usr/bin/env python

import urlparse
import sys
import requests

import lxml
from lxml import etree
from cStringIO import StringIO

if __name__ == "__main__":
    domain = sys.argv[1]
    base_url = "http://builtwith.com/"
    url = urlparse.urljoin(base_url, domain)
    print "Visiting {} ...\n".format(url)
    response = requests.get(url)

    parser = etree.HTMLParser()
    tree = etree.parse(StringIO(response.content), parser)

    tech_items = tree.xpath("//div[contains(@class, 'techItem')]//h3")
    for item in tech_items:
        print "".join(item.itertext()).strip()

After placing it in a executable file named builtwith in your $PATH, you can
use it like so:

$ builtwith 37signals.com
Visiting http://builtwith.com/37signals.com ...

GeoTrust SSL
RapidSSL
Dyn DNS
Google Apps for Business
Postmark
Campaign Monitor
DKIM
SPF
Shockwave Flash Embed
Google Conversion Tracking
Clicky
html5shiv
Prototype
YouTube
Amazon S3
IPhone / Mobile Compatible
Device Width