Advanced RSS Feed Processing
Yahoo Pipes is a powerful composition tool to aggregate, manipulate, and mashup content from around the web. Simple commands can be combined together to create output that meets your needs.
For example, you can combine many RSS feeds into one, then sort and filter it. The resulting feed can then be used to provide dynamic content in your lens.
This lens will give you an introduction to Pipes, and how to create your own.
New: How to change the URLs of your pipes.
Where to Begin?
The first thing you should do is search for some existing Pipes, because you never know - someone may have already created the Pipe you need. Just enter in some keywords, and a whole bunch of Pipes will be presented to you. For example "amazon" or "squidoo".
Searching for Pipes is a great way to find ideas for your pipes. You can view the source of any pipe you find, and see how it was put together. This is a great way to learn how to create your own.
When you're ready to take the plunge, click on the large "Create a pipe" button. You'll be shown a blank sheet, on to which you can drag and drop the modules you need from the menu on the left. Modules are connected to each other by click-n-dragging a link between them.
At the bottom of the window is a debug window, which will show you the intermediate output of which ever module you have selected. Click on Refresh if it doesn't seem to have updated yet.
Below are a series of tutorials that take you step by step to creating your own Pipe, followed by some more advanced tutorials. At the end these is a Guestbook for your feedback and comments, as well as Plexo for your best Pipes.
Simple User Input
To add one to your Pipe, expand the "User inputs" in the left hand menu, and select the type you want. Here, I've picked the "String Input" field.This field has 5 parameters: the 1st is just a name, so it doesn't matter what you call it; the 2nd is the prompt that the user will see when they run the Pipe; 3rd is for the order in which you want the inputs to appear, if you have more than one; 4th is the default value that the user will see; and the 5th is the value used for debugging.
In the debug window at the bottom of the screen, you should see that debug value appear.
Creating a URL from User Input
Add the "Url Builder" from the "Url" sub-menu. In this example, I've set the Base to be "http://squidutils.com/lensmaster". I then linked the previous Text Input field by clicking on the blue dot at the bottom of the field, and dragging it to the dot on the right of the Path Element.
Add another Path Element by clicking on the (+) icon, and enter the remainder of the URL "lenses.rss". Note that you do not put in any additional "/" characters.
In the debug window, you should see "http://squidutils/lensmaster/thefluffanutta/lenses.rss", depending on what default value you specified in the Text Input.
Fetching an RSS Feed
You can enter a fixed URL, or connect it to the "URL Builder" module in the previous example.The debug window should show the first few items in the feed, and you can expand them by clicking on them.
Filtering a Feed
The filter is configured to only show items where the "category" element contains whatever string the user has entered in the "Text Input" field.
The debug window should show only the items which match. Note the special "Pipe Output" box at the bottom, which serves as the end point for your Pipe, and is from where your resulting output will emerge.
This Pipe has been published as Squidoo Lenses by Lensmaster and Category, and you can see the complete source code here.
Dealing with Dates...
You'll see that in this example, I've used "item.y:published.utime", which has been automatically generated from "item.pubDate". This allows me to do a greater than comparision against the result from the "Date Builder" module.
Remapping Elements in a Feed
This quick example shows how you can take a feed, and replace the description for each item with its category tag.
How To Customise Your Pipe URLs
The default URLs for pipes are pretty unwieldy, but Yahoo has provided the ability for you to change them into something more manageable. Here's how:
The first step is to customise the URL of your profile. Start by going to the "My Pipes" page (assuming you're logged in), and look for your profile URL near the top. There should be an (edit) link at the end.
Once you've done this, click on the "View Results" of any of your pipes. Just below the title and description, there should be a link similar to before:
Important Note: If you want to seperate your keywords in the URL, you must use underscores; dashes will get filtered out when you save it.
Once you have your new addresses, you can bookmark them, show them to your friends, etc. If you want to get the RSS output, then simply add ?_render=rss to the end of your URL.
What to do with your Pipes?
If you think that other people would benefit from your creation, then you may Publish it. Make sure you add a good description and some tags to help people find it.
You can also add your pipe to this list below...
Some Example Pipes
If you have a Yahoo Pipe you'd like to show off, then feel free to add it here:
Pipes from thefluffanutta
All of the Pipes that I have created - go on, take a look...3 points
Pipes: Ones - five random friends
A pipe made to display the top lens from each of 5 lensmasters feeds2 points
Pipes: Squidoo Lenses by Lensmaster and Category
Enter your lensmaster name and a category. You will then have a list of all of your lenses in that category.1 point
Pipes: Squidoo Lenses by Lensmaster and Keyword
Enter your lensmaster name and a keyword. You will then have a list of all of your lenses that contain that keyword in the title.1 point
Pipes: Amazon: Show Similar Items (US and UK)
This will generate an RSS Feed containing Amazon items that are similar to a specified ASIN or ISBN.0 points
Pipes: eBay Multiple Country Price Search
This Pipe uses 10 of Ebay's RSS feeds from all over the world and US. The feeds are form UK, Canada, France, Indian, Spain, Italy, Singapore, Netherla...0 points
Pipes: eBay Simple Price Search
This Pipe was designed to search for a price range in eBay.com. It will allow you to add your search terms and Minimum amount and Maximum amounts. Qui...0 points
Learn More about Yahoo Pipes
- Pipes - About the Editor
- The Pipes editor is a JavaScript authoring tool that lets you create and edit Pipes in an intuitive visual interface.
- Pipes - Module Reference
- The modules in the Pipes Editor have been grouped into categories based on their functionality.
- Pipes - Tutorials
- Many excellent tutorials for Pipes have been written by users...
- Pipes - Frequently Asked Questions
- Why can't I connect some modules to others?
Why can't I put some modules inside a ForEach container?
My Pipe is broken, what do I do? - Yahoo Pipes And How To Merge, Filter, And Sort Multiple RSS Feeds
- Today's video is the next video in our RSS series, which shows you how to use Yahoo Pipes, a free service to manipulate multiple RSS Feeds to create another unique RSS Feed. This allows you to drive more backlinks to your website among other benefits.
- Working with Yahoo! Pipes, No Programming Required by Tony Loton (Book)
- In this book you will learn how to:
* Create Pipes by selecting modules from the library, and connecting them together graphically on the Pipe Editor's canvas; no programming required.
* Publish your Pipes for the benefit of others.
* Discover and take advantage of already-published Pipes.
* Embed Pipe results directly in your own web pages.
* Combine Yahoo! Pipes with Microsoft Popfly to create more innovative mashups
Latest Blog Posts from Yahoo Pipes
- JOB: Principal Software Engineer - Pipes Team
- Passionate about Pipes? Want to be part of a great team and be involved in the evolution of Pipes and YQL? Got Java skills? For more information see http://careers.yahoo.com/jdescription.php?frm=jsres&oid=25394.
- Pipes Blog » Blog Archive » Term Extractor Module stays!
- Please see this posting: http://developer.yahoo.net/blog/archives/2009/08/term_extraction_stays.html. The Term Extractor Module will be moved out of the deprecated folder back into the String folder shortly. ...
- Pipes Blog » Blog Archive » Term Extractor module to be ...
- Attention Pipe Developers and Users. The Term Extractor module listed in the String folder in the Pipes editor will be discontinued and cease to work after August 31st. It will be moved to the Deprecated folder shortly. ...
- Pipes Blog » Blog Archive » Microsoft PopFly and Google Mashup ...
- As many of you might have seen PopFly (and previously Google Mashup Editor) are in the process (or have already) shut down. Its a great shame since as we know *a lot* of the applications created on these visual mashup editors used Pipes ...
Mashup Case Studies with Yahoo! Pipes
Featured Author
Pipes developers can look at the web pages, examining the underlying HTML code, and then following through to the Pipes themselves.
"Mashup Case Studies with Yahoo! Pipes", available as paperback or PDF.
Here are the links to some of them. They're not meant to look nice, but to demonstrate various techniques:
- Pipe JSON
- Shows items for sale on eBay.
- Trading Cockpit
- Displays the output from two pipes. To the left is a list of stock sector charts, and to the right is related news for whichever sector link you click.
- LOTONtech Finance Portal
- Shows how a Yahoo! Pipe can screen-scrape information from a third-party web site, to be displayed in your own portal.
- Product Download Form
- It shows how a HTML form can capture user input to be used as a Pipe's parameter(s). Enter a username to see the downloads available to the fictional user.
- Google Maps
- Displays a Google Map showing locations sourced from a Yahoo! Pipe.
Reader Feedback
Kate-Phizackerley wrote...
Nice lens. I've just recent a recent-lenses lens using Yahoo Pipes and it was great to have your lens I could feature for anybody who wanted to know more about the techie side and how Yahoo Pipes can be used.
Kate
Posted April 06, 2009
lakeerieartists wrote...
Thanks for this tutorial. I went to pipes and was totally confused, plus couldn't seem to run the videos. This will really help. :)
Posted December 01, 2008
vbright105 wrote...
Okay...I HAVE to come back to this. I want to utilize this one! Let's just hope I remember to come back.
Posted October 09, 2008
CleanerLife wrote...
Amazing how many cool Yahoo things I find outside of Yahoo -- I use Yahoo every day, and never heard of this! Good thing you made this Lens!
Posted September 08, 2008
badmsm wrote...
Fluff, you're the man! So grateful to you for sharing your expertise with us. :)
Posted August 29, 2008
triathlontraining wrote...
I'd only just heard about Yahoo Pipes (from adez7!) and now it's one of my next things to learn. Your tutorial is perfect! :)
Posted May 19, 2008
Chadrew wrote...
Great tutorial. I love Yahoo Pipes, I was so amazed that the tool I was searching for already existed. Plus the interface is very nice.
Posted March 22, 2008
Tony Loton wrote
Anyone interested in Yahoo! Pipes might be interested in my book "Working with Yahoo! Pipes, No Programming Required".
It's not yet on Amazon (as I write this) but is available as paperback or PDF from http://www.lotontech.com/it_books.htm
Posted March 03, 2008
hypnoplace wrote...
Awesome tutorial, and thank you so much for sharing your valuable knowledge.
Posted February 21, 2008
clouda9 wrote...
Not sure if I did this right...huge learning curve. I guess time will tell :) Great big hug for sharing this information! **Awaiting approval on the Plexo :)
Posted January 29, 2008
Frankster wrote...
Great lens, fluff. Thanks! 5 stars and I'm a FAN! You should add it to the plexo in my How to Squidoo....
Posted January 28, 2008
flaminglacer wrote...
Thank you - I can get my aging head round it now! Squid Angel blessing for an excellent lens....
Posted December 06, 2007
flowergardener wrote...
Yea! I finally did it! Here's my first pipe:
http://pipes.yahoo.com/pipes/pipe.info?_id=CFLySpei3BG3RYloy6ky6g
Now, I'm going to try to make one for my dog collectibles and gifts lenses! Thank You!
Posted December 05, 2007
adez7 wrote...
Pipes is a very powerful tool and a great asset to have in ones arsenal of tricks. Combined with Dapper and others there is no limit as to what you can do. THis is a five star lens and a great resource. Thanks :)
Posted December 05, 2007
dougeby wrote...
Thanks for this great info and the developed pipes. Will come in handy!
Doug
Posted December 05, 2007
More RSS Stuff
-
Lensmaster Tools and Utilities
-
SQUIDUtils.com Utilities and Tools for the Expert Lensmaster SquidUtils is the place to find complementary utilities and tools for your Lenses on Squidoo. Here you can see all of your lenses and groups, with shortcuts to some of the most vital func...
-
How To Get Deep Links with RSS Feeds
-
There are a number of online syndication services that will display an RSS feed as a web page. These will typically contain deep-links to pages on to the originating website, which is a valuable source of backlinks. I originally wrote about thi...
-
Amazon RSS Feeds for Your Squidoo Lens
-
I have this special service for Squidoo Lensmasters whom are also Amazon Associates - RSS Feeds containing products from Amazon. You can put these feeds into your lens to show books, DVD, electronics and more from specific categories. "But Why?&q...
by thefluffanutta
(more) Explore related pages
Real Estate Feed Marketing- For real estate agents and marketers, RSS feeds are an easy way to keep updated...
- Domain Name Registration: Supercharge Squidoo with Google Apps (2 of 11)
- If you're like me, you love building Squidoo lenses. But after a while, it's tou...
- Google Talk (Chat): Supercharge Squidoo with Google Apps (4 of 11)
- Google Talk or Chat is the Google instant messaging system. It's a powerful prod...
- Start Page: Supercharge Squidoo with Google Apps (6 of 11)
- A Squidoo lens does well when it offers great content. But without regular updat...
- Supercharge Squidoo with 'Google Apps for your Domain' (1 of 11)
- If you're like me, you probably got hooked on Squidoo after building your first...
How to make it into top 2000 lensrank- Most squidoo users know that reaching the top 2000 in lensrank can be difficult!...
