// Internet Duct Tape

Duct tape is a great tool because it is so shiny and sticky. You can use it to glue so many things together, even if they end up looking like Frankenstein by the end of it. All you need is a camera, a cellphone, an MP3 player and a piece of duct tape to get yourself the only mobile convergence device worth having. The programming language Perl has quite often been called “duct tape for the internet” because it lets you easily transform text and interact with web sites.

(photo by philgarlic)

Here is a simple Perl script that downloads an RSS feed and bookmarks each entry to del.icio.us:

my $delicious = Net::Delicious->new(
				    {'user'=>$user,
				     'pswd'=>$password},
				    'updates'=>'.',
				    'xml_parser'=>'simple',
				    'debug'=>1);
my $feed = XML::FeedPP::RSS->new($rss_url);
$feed->normalize();         # Sort by pubDate and remove non-unique
foreach my $item ($feed->get_item()) {
  my $description = $item->description();
  $description =~ s/<.*?>//g; # remove HTML
  my %args = ('url'=>$item->link(),
	      'description'=>$item->title(),
	      'extended'=>$description,
	      'tags'=>"from_feed",
	      'replace'=>'no');
  my $retval = $delicious->add_post(%args);
}

The problem with Perl is that you have to either run it on your own machine, or buy web hosting that lets you run your own Perl scripts (or Python/Ruby). This is a real pain in the butt.

Enter Yahoo Pipes

Yahoo created one of the most innovative web tools I’ve ever seen. Yahoo Pipes lets you do all kinds of conversions and filtering on the web without requiring a web host to host your programs. If you want to convert XML/RSS data to other XML/RSS then look no further.

There’s still room for improvement:

• Scraping web sites that do not have information in XML/RSS. There are other companies that let you do this, but they’re even harder to use than Pipes (IE: dapper.net).
• Notifying you when your pipes don’t work. That makes finding existing pipes and mashing up multiple pipes tricky as best. It’s hard to use a tool when things constantly change underneath you.
• It would also be nice if allowed HTML in the Pipes descriptions as it is hard to describe how to use them sometimes.
• Better debugging messages when developing your own Pipes

Yahoo Pipes is targeted towards programmers, not casual users, but there is still a million and one things you can do with it. Here are some of my pipes that are free for other people to use.

Simple

• Switch title and description in any RSS feed
• Remove older results from a feed
• Remove del.icio.us daily link post from any blog feed

Social Sites

• Get your Technorati Rank as an RSS feed with a link to Add to Favorites
  • You will need your Technorati API key
• Keep track of when your site is submitted to Digg or Reddit
  • Displays number of Diggs in the title, and links to the Reddit voting/comment page instead of directly to the article
• Convert any Reddit feed to link to the comments page instead of the article it’s voting on
• Get a list of post titles and links (no description) from a Google Reader shared items feed

Personal

• Posts from my blog in the past 10 days
• Technorati Rank for engtech.wordpress.com

Got Pipes?

Are there any specific RSS feeds mashups you’re looking for but don’t have the Yahoo Pipes expertise to create? Leave a comment on this post and I’ll see what I can come up with.

See the full list of free software I have created.

You can get frequent updates about all of my new software, tools or blog themes by subscribing to IDT Labs by RSS or by email. Or you could just subscribe to my main blog, Internet Duct Tape.