Dapps from RSS

Dapper solves a very hard problem: getting structured content out of a website that only provides an unstructured interface (i.e., HTML). However, many sites today provide their content in more structured form using the RSS and Atom standards. With today’s release, you can now create a Dapp from any existing RSS or Atom feed on the web.

Now why would you want to do such a thing? Structure already structured information? Make RSS out of RSS?

Once an RSS feed is available in Dapp form, you can harness all the power of the Dapper platform, including email alerts, gadgets for personalized homepages (Netvibes, Google, Pageflakes), Flash widgets, and filtering. You can even link an RSS Dapp to any other Dapp to create a new webservice out of the two (e.g. take listings from Craig’s List and put them on a Google Map). Furthermore, our SDKs allow you to programmatically access these RSS Dapps just like any other.

Many of you have asked for this feature over the last six months and we really hope that having access to it makes it even easier to reuse, manipulate, and take advantage of content from all over the web.

In several days, we’ll release the second part of this feature which will allow you to specify dynamic variables for the RSS URL. This will allow you to very easily work with sites like Craig’s List, where URLs contain arguments specifying search terms and location, as if the site had a fully fledged API.

Finally, those of you who produce content and have a feed can now very easily provide your content in new formats. See the “distribute your content” section for more information.

This was a big change to the code base and we’d really appreciate your bug reports. Thanks!

Dapper Version 2!

Today is a happy day!  After a long time in the making, Dapper version 2 is ready for everyone to enjoy.

The new version entails a massive overhaul to Dapper’s user interface in an attempt to address the many insightful comments and suggestions that users like you have submitted over the last six months.  We hope that you’ll find the interface more intuitive, faster, and easier to use.

Another big new feature is the Dapper widget.  You can now create an embeddable Flash widget for any content on the web.  We think you’ll love this.

We’re very eager to hear your reactions, comments, feedback (and, of course, bug reports) so please be in touch.

Enjoy!

Dapp Monitoring

Since Dapper launched in August, many of you have approached us asking “what happens when a Dapped website changes? Will the Dapp still work?” The answer has always been that if the website changes a bit, or even quite a bit, the Dapp will continue to function. If the website changes a lot, the Dapp will break. Today we’re releasing a feature that alerts the creator of a Dapp when a Dapp stops working as expected and allows him or her to edit the Dapp and fix it. Please let us know how it works for you and if you encounter any difficulties.

For what it’s worth, only about 1.3% of all Dapps are failing today, and we anticipate that, as we improve Dapper’s technology, this number will decrease. Also in the future, we plan to implement automatic correction of Dapps where possible, but in the mean time we can rely on one another to keep the Dapp pool healthy.

As always, thank you for your feedback. It leads us to implement features like these and we couldn’t do it without your input.

Distribute Your Content

Good news for content providers: today we’re unveiling a set of features aimed at making it easy to distribute your content to your users and others on the web. These tools include a wizard which enables you to provide your content to your users in new feed formats, including RSS, Google Gadgets, and more as “chicklets” on your website. Just choose the content you want to distribute and the formats for distributing it, and the wizard will generate the HTML code that you can embed directly into your site or blog. The feeds are simply transformed Dapps, so they are always up to date.

Another exciting part of today’s release is the ability to ascribe licenses to your content allowing you to determine how users can or cannot use your content through the Dapper service. These licenses include several Creative Commons licenses, for those of you familiar with the organization. We’re very happy to be working with the good people at Creative Commons and are sure that together we can do great things. By licensing your content, you can gain significant exposure and traffic for your website as well as empower valuable applications all over the web to use your content legitimately.

We’re excited by these new offerings and we hope that they will bring value to all of you who are out there producing great content. As always, we aim to empower you by enabling you to sensibly distribute your content through the Dapper service. Your feedback is critical in this process and we’re very open to hearing from you. Please contact us with any comments, questions, concerns, or bug reports.

View the press release here.

Netvibes Modules

Netvibes users rejoice!  You can now take content from anywhere on the web, including sites that do not provide RSS fees, and add it to your Netvibes page without having to program.   For instance, take the latest messages from your Facebook account, your SkypeOut account balance, or search results from you favorite site and instantly include it in Netvibes.

To create a Netvibes module, simply create a Dapp for the website you like (search first, someone else may have already created one).  On the “how to use” page for the Dapp, select “Netvibes Module” and then configure it however you like.

In the future we intend to expand the transformer’s capabilities to provide even more interesting and useful Netvibes modules.  In the mean time, we hope you find this useful. Please let us know what you think.

Login Dapps

It has been a while since our last update, but we’ve been very busy. We’re excited to announce that today we’re releasing a significant update to Dapper that allows you to create Dapps for websites that require a username and password. This means that now you can create Dapps for your web-mail accounts, your web banking, your online college course calendar, and any any other data that sits behind a login form. Over the past couple months we’ve received many requests for this functionality, and so after a lot of work, we’re happy to announce that it’s now available. The possibilities are truly endless – RSS for your webmail, an email alert to notify you when you’re running out of money in your bank account, a Google module that shows you how much SkypeOut credit you have in your account, and more – as usual, your creativity is the limit.

To demonstrate the power and ease of this functionality, we’ve built Snag, an aggregator for several of the most popular social networks. It’s a proof of concept, but we think you’ll find it useful.

So, how does all this work and how can you get started making and using these new Dapps? The answer is that it’s no different than any other Dapp. When you’re creating a new Dapp for a site that requires you to log in, use the Dapper Virtual Browser. Go through the login process in the Virtual Browser as you would on the site itself, and then add pages to your basket as usual. When you proceed to define your fields and groups, you’ll be asked for your username and password. Supply the credentials you ordinarily give to the site, and Dapper will log in to the site as you, and perform its analysis. When you save the Dapp, your username and password are not saved in the Dapp (for your security), and as such you will have to supply them whenever you use the Dapp (whether it is the XML you are retrieving, or working with one of transformers like RSS or JSON). There are two ways you can do this: you can supply them as POST arguments or as GET arguments in the URL. The latter is very easy to do from the “how to use” page of any Dapp. Using this page to generate a URL will automatically encrypt your username and password so that the URL you are given does not contain your credentials in clear text. If you want to encrypt credentials programmatically (for use in a service you build, for instance), you can use our credentials encryption web-service to encrypt your users’ credentials once and reuse the encrypted credentials repeatedly. Additional details appear in the FAQs. We’ll soon be supporting HTTPS, and when we do, if you POST your credentials, they’ll be encrypted automatically, and the data returned to you will also be encrypted.

It’s important to note that this is an early release of the login functionality and that we plan to continue working on it to improve it. You’ll find that it is rough around the edges. Many sites still don’t work (please let us know when you find one) and various functionality is not supported (e.g. sites that require more than just one username and one password, using the Dapp linker on login Dapps, etc.). However, the functionality is rich and opens many doors, and so we’re releasing it in its current form with the hopes of improving it with your help. A lot of code was added and changed to support this, so if you find bugs, please tell us and we’ll get right on it. The PHP5 SDK has been updated to support login Dapps – the PHP4 and Java SDKs will be updated soon.

You can also read our updated FAQs if you have additional questions (or just drop us an email). As always, thanks for using Dapper and for any feedback you can provide.

iCalendar Support

Ever wanted to add your favorite upcoming concerts to your calendar, automatically? The weather? Your favorite blog’s posts? Some other time-based data out there on the web? Until today, unless the website provided an iCalendar interface, you were at a loss.

We’re pleased to announce that you can now use Dapper to take information from any webpage and put it into Google Calendar, Apple iCal, Mozilla Sunbird, Microsoft Outlook, and any other program that supports the iCalendar standard. Even better, as you’d expect, the data is dynamic and constantly updates from the website via Dapper.

To use the feature, just find the Dapp for the website you’re interested in (or create a Dapp if one doesn’t already exist), then choose the iCalendar option. There you define which fields contain the date and what you want to show up as the description of the event.

One great example is to take news items from apple.com from recent days and make an iCalendar for it. Try it out using the Apple News Dapp.  You can generate the URL for the iCalendar and insert it into your calendar program.

If you have any questions, run into any trouble, or want to recommend a related feature, don’t hesitate to contact us.

Sandbox Extension and Private Dapps

For those of you who have asked for the ability to keep Dapps in your sandbox for longer periods of time (for the sake of editing them or deleting them after the 24 hour period), we’re happy to announce that this is now feasible. When viewing a Dapp that is currently in your sandbox (e.g. after creation), you can click the option to keep the Dapp in the sandbox indefinitely. When you are happy with the Dapp, you can release it to the public by publishing it. The Dapp will remain in your sandbox, open to editing, until you publish it.

This feature also addresses those of you who have asked for the ability to keep your Dapps private. By keeping a Dapp in your sandbox, it remains visible only to you and to those to whom you give the URL.

Enjoy and please do not hesitate to send along any feedback.

Linking Dapps

We’re happy to announce a new feature: Dapp linking.  This feature lets you links two or more Dapps together.  The ouput from the first Dapp is used as input to the second Dapp (and so on).  For instance, if you have a Dapp which takes a zip code and returns movies playing in that area, and another Dapp which takes a movie title and returns reviews, you can link the two together.  The end result is a new Dapp which, in this example, takes a zip code and returns a list of movies playing in the area and reviews of each movie.

To link two Dapps together, find the first Dapp and select “Use this Dapp” – from the next page, click “Link to Another Dapp” and you’re on your way.  As always, your feedback and bug reports are greatly encouraged.

New Features

We’re pleased to release a couple of minor, but useful, features today:

1) Improvements to the “Create a Dapp” process: Now when you save your Dapp, you will immediately see what the output looks like and be able decide whether or not you’re satisfied.

2) YAML transformer (sample): YAML is a data exchanged format much like JSON. Programmers can use this format in their programs and can forget about dealing with XML. Packages for working with YAML are available for a variety of programming languages. Thank you to one of our users, David André, for requesting this feature.

3) Miscellaneous bug fixes and improvements. You should now see less cryptic error messages when you forget to supply required information for transformers. The “My Dapps” page has also been improved to show you Dapps that are in your sandbox first.

Thanks for everyone who has been providing feedback over the last couple days. It’s greatly appreciated. Please do keep it coming.

New Features

This week will see the release of several new and exciting features.  As always, your feedback would be greatly appreciated.

• Alerts – you can now choose to be alerted whenever a specific web page meets conditions you specify.  Just find or create a Dapp for the website you want to monitor, go to the Dapp's details, and click "Alert."  You will then be able to set your alert conditions to things like "alert me whenever at least three stories on the Digg homepage have over 1,000 diggs." 
• Full groups – when saving your Dapp, you can now set whether to exclude groups that do not have all the group fields present.  To do so, click "Advanced options" when saving your dapp and check the box that says "Full groups only."
• Contact us - we've made it even easier to contact us by providing a form on the website.  You can access it by click the "Contact" link in the footer of any page on the site.
• More Dapp metadata – coming soon, when saving a Dapp you will be able to supply descriptions and tags that can be used while searching.

We hope you enjoy these various features.  Keep the suggestions coming – you input is very valuable.