Have been playing with Yahoo!’s YQL this weekend, querying the Sydney Ferries website. Pretty amazing what it allows you to do, though the Sydney Ferries site wasn’t the best site to start playing with I guess. I did have a need to have the ferry timetable on my iPhone (especially the Neutral Bay service), so that’s why I put together Next Sydney Ferry this weekend.

The premise is pretty simple: when does the next ferry depart from Circular Quay? I had this wild idea to do cool stuff with it, but inspired by the simplicity of Next Manly Ferry, I thought I’d start out pretty simple too. And it certainly still is a work in progress with plenty of bugs.

NextSydneyFerry.com parses the timetables of the SydneyFerries.info site using YQL and JSONP. No luck with any API, so it’s pretty fragile reading in the HTML table data. Wish they made an effort marking up the data a bit more helpful (as in markup-as-an-api). One of the URLs even has a typo (“weekemd”). I additionally notice YQL not returning any results for the same query, while at other times it would, so they have some bugs too. It was an interesting YQL experiment, but since the data is not too dynamic, I will probably switch to using a more static datastore, which would also be a lot more responsive, and which would be lighter on your mobile dataplan. This is very much a first version application, with limited functionality (it shows you only the next and thereafter ferry, and only departing from Circular Quay), some noteworthy points:

• it uses the server’s time so tourists don’t have to have their mobiles set to local time.
• it updates the time in the background without a page refresh.
• It allows you to bookmark your daily service.

Sydney Ferries allows you to repurpose their content, as long as you add their copyright notice and don’t charge for it. But they could have made it a lot easier for people to do so.

Post from Jean-Jacques Halans MarkupAsAnApi blog.

Playing with YQL

We read these documents, we print them, bookmark them for later retrieval. We might copy/paste content into another document, constructing a new one.

If we wanted to automate this, we’d resort tot screen scraping. But this easily breaks, as there’s no standard or “contract” between the original site and the screen scraper.

Or we’d go for duplicating content into a new format like XML, with an agreed upon format. That way we could build product price aggregators using SOA Web Services with SOAP, WSDL,… Or use a REST architecture which takes us a bit closer back to our original HTTP request.

Most popular formats for sharing (XML) data is RSS and Atom, were again we duplicate the content we publish online.

But we’ve come a long way last couple of years, towards Web Standards, pushed by organisations like the Web Standards Project (WaSP) and Web Standards Group (WSG). They promote standards for separation of content, styling and behaviour, and the use of semantic HTML.

And then there’s the W3C, who promotes the Semantic Web as knowledge representation, using Resource Description Framework. RDF is a general method of modelling information making statements about resources in triples. Triples represent a subject-predicate-object expression, for example JJ – isBornIn – Belgium.

The W3C’s Web Ontology Language, or OWL, provides additional vocabulary and formal semantics, providing greater machine interpretability of Web content, but with added complexity.

But as of yet there isn’t much RDF data online, or ontologies are missing for many application domains. The W3C’s projects are rather academic, and aren’t close to any web developer’s mindset.

What is closer to the web developer though is semantic HTML, the correct use of heading levels and paragraphs to introduce structure, blockquotes and correct use of tables, for tabular data.

Now we add rich semantics, standardised Microformats. They are small pieces of metadata, within the markup, using CSS. They are discoverable, interpreted by machines.

Post from Jean-Jacques Halans MarkupAsAnApi blog.

Markup as an API

A common solution has been to proxy and scrape an application with a combination of XQuery and TagSoup (to fix the ugly, broken HTML, dontcha know), but it is possible to do this purely in the browser.

or with jQuery, as Jan Varwig describes:

Fortunately, just the day before, I discovered jQuery, a Javascript framework with strong support for finding DOM-Nodes via CSS, XPath and some custom selectors. The tricky part now was to get jQuery to access the DOM-Tree of the schedule page on kino.de.

Of course, screen scraping would be so much easier using Web Standards.

Post from Jean-Jacques Halans MarkupAsAnApi blog.

Scraping HTML with innerHTML or jQuery