The Data Current

September 25, 2008

New Pipes (Digg Through API and Sourceforge SVN)

I use Yahoo Pipes a lot. Many of my pipes are not published because they are just aggregators with filters, specific to my interests. Lately, I have been trying to create more generic pipes that might be useful to other users, either stand-alone or as sub-pipes. The following are just two new pipes that I believe are very usable and ready to be published. I am working on and testing several other pipes, so expect more announcements in the future.

The Digg data through API pipe uses the Digg API to produce a Digg container feed that includes additional information not provided in the standard Digg feeds currently. This additional information includes:

Image enclosure
Original source domain
Original link
Submission date
Container name
A content:encoded element containing some of the additional information from other elements

The resulting feeds should be element compatible with the official Digg feeds and should be usable as substitutes of the official feeds.

The Sourceforge SVN pipe creates a feed that lists the last 5 revisions of a Sourceforge project with a SVN repository. This pipe is specific to projects with an SVN repository and it will not work with a CVS repository.

Note: The pipe currently does not check to see if the repository has less than 5 revisions and may not work for such repositories.

July 30, 2008

Removing Flash-based content from feeds with Yahoo Pipes

Labels: Communications

I have not written about Yahoo Pipes in detail on this blog yet, but I thought this was interesting so I am posting about it. To quickly summarize, Yahoo Pipes allows you to produce feeds or widgets from web-accessible sources in several formats. I find it very convenient to aggregate and filter feeds.

Recently, I wanted to remove some Flash-based content from a feed that I follow since I am unable to see it in my reader anyway. I was trying to use the Regex module to match the tags for the content and replace them with a note stating that they were removed. Surprisingly, many of my regular expressions I was entering were not working and the Flash-based content remained in the feed. After testing quite a few expressions I finally found one that worked. Here it is:

[<][^<]*application/x-shockwave-flash[^>]*[>][<]/[a-z]*[>]

I still do not understand what was wrong with the other expressions I tried. Most of them were simpler and more specific than this one, and they should have matched. Anyway, this works for now.

UPDATE: Wired recently published an introduction to Pipes article