Idle tinkering - products, barcodes, freebase & wikidi

A little bit more work was done on products.freebaseapps.com this weekend. Products.freebaseapps.com simply grabs all Consumer products and renders them out as RDFa - as good relations info.

I did a bit of work to make the site more human friendly and to encourage random users who come across it to start editing.

All of that done, I still can't for the life of me get the Google Rich Snippets tool to actually render anything happily. Couple that with a variety of RDFa validators and other tools being out of action/upset with the markup in a most confusing way; and you start to think this linked data stuff is tricky...

While testing  and pounding my head against a brick wall I did come across wikidi.

I explored a bit, but couldn't decide - the content they are providing is largely freebase sourced; but there has been other data integrated (ie: prices). It's under a creative commons licence, but there's no 'bulk download' or other functionality - so getting data back out usefully is difficult.
There's no RDF/RDFa or other structured markup either - so this site is a bit of a black hole. I'm not sure I like knowing my contributions to open data are going into a location like that.

LIXI forum 2010

Over the past two days, I have been lucky enough to attend the 2010 LIXI forum.

LIXI is a set of standards relating to how financial institutions can deal with loan applications, valuations, and a few other items.
Typically it's serialized into an XML version of a paper form; and gets sent around by SOAP to various workflow engines.

Previously, my involvement with LIXI has been in the valuations space, with ValEx being a big driver of LIXI usage; but no one really talking about the standards (or asking for improvements).

So, it was much to my surprise when I got to meet much of the industry over the last two days.

Probably one of the most interesting ideas put forward was by Christopher Joye. He proposed a national clearing house for data relating to credit:

Moving from the general to the specific, I would propose that the Commonwealth establish a central electronic ‘clearinghouse’ of all residential, personal and business credit originated in Australia. For simplicity’s sake, let’s call it the National Electronic Credit Register (NECR).

The main purpose of this information would be to provide much more accurate, real time, granular data to understand how individuals are actually fairing, rather than trying to guess at an aggregate level.
This is primarily to allow entities like the RBA to act quickly and accurately to help manage our economy.

I personally think it is a quite valid idea, as much of my frustration with how statistics and modeling takes place stems from the use of assumptions, or aggregate data.

There are obviously obstacles, not least of which is the potential for scope creep and how that relates to individual's privacy; but as a concept, it quite impressed me.

Other presenters touched on everything from the cost of software failure to a comparison of Australia's railway gauge mess to the current fragmentation of the LIXI standards.

The big move at the moment seems to be to standardize on a mechanism to transmit business rules.
Three vendors all have the same problem of trying to communicate changing credit policy for specific products from a lender to many brokers; and there are a number of solutions out there on the cusp of realization.

We saw practical demonstration of real time transmission of changed requirements (application A hits the save button, changing the age limit on a product, application B becomes aware of it within seconds).

Whilst people did not get into the technical details behind that particular demo, a technical training session on schematron was run.
I had initially dismissed the idea of schematron, but hearing Rick Jelliffe (one of the creators of XSD) talk about many of the concepts that drove the design; I'm more sold on it.

If you haven't used schematron, in short it differs from XSD by being much more relaxed. You write specific xpath rules to target segments of a document as being valid.
XSD on the other hand ends up being much more restrictive. For instance, it forces you to make choices about sequence of nodes and tends to fail validation if you have extra content.

The extra content item is a killer for the LIXI ecosystem, as everyone seems to have practice of grabbing a copy of the XSD at a given point in time; and if need be; doing minor modifications for their patch of it.
It's not on the web, because no one wants to expose a core banking system to the internet.
As an example, I introduced a simple additional Time element to ValEx's schema and the LIXI standards. It has taken months of testing with multiple organisations just to take a tiny step forward, and in the end a partner application still ended up ceasing to communicate with us.
The most painful part of it all? The only bits which "broke" in every single system were the XSD validation components. Every other bit of code simply ignored the additional XML.

With such a minor change requiring so much effort, you can understand how we've ended up in a fragmented state, and why XSD has been harmful.

I would be very keen to see the entire basis of validation (both at a syntax/expected data level and a business rules level) shift to schematron; or something else - much of the RDF world does not have the XSD problem, because no one uses XSD to validate!


Overall, I made quite a few contacts in all sorts of places, touching on topics such as the use of unique identifiers (ala Linked Data, even if we're not an RDF based standard); to improving physical documentation handling with our existing partners.

I look forward to the coming year, and the opportunities it may create.

Redland in PHP / Ubuntu

The PHP docs are a little lacking; so...

Install in Ubuntu: apt-get install php5-librdf

roll: great, the redland php bindings are also in darwin ports. just $port install redland-bindings +php5. time to integrate it in my project :)

Checking its installed:

clockwerx@clockwerx-desktop:~$ php -i | grep rdf
redland librdf version => 1.0.9

How do you use it? Here's a port of example1.c

<?php
$world=librdf_new_world();
librdf_world_open($world);

$uri=librdf_new_uri($world, "http://librdf.org/bindings/bindings.rdf");
if(!$uri) {
    die("Failed to create URI\n");
}

$storage=librdf_new_storage($world, "memory", "test", NULL);
if(!$storage) {
    die("Failed to create new storage\n");
}

$model=librdf_new_model($world, $storage, NULL);
if(!$model) {
    die("Failed to create model\n");
}

$parser_name = "";
$parser = librdf_new_parser($world, $parser_name, NULL, NULL);

if(!$parser) {
    die("Failed to create new parser\n");
}

function librdf_uri_as_string($uri) {
    return (string)$uri;
}

print "Parsing URI " . librdf_uri_as_string($uri) . "\n";
if(librdf_parser_parse_into_model($parser, $uri, $uri, $model)) {
    die("Failed to parse RDF into model\n");
    return(1);
}
librdf_free_parser($parser);


$statement2 = librdf_new_statement_from_nodes($world, librdf_new_node_from_uri_string($world, "http://www.dajobe.org/"),
                                             librdf_new_node_from_uri_string($world, "http://purl.org/dc/elements/1.1/title"),
                                             librdf_new_node_from_literal($world, "My Home Page", NULL, 0)
                                             );
librdf_model_add_statement($model, $statement2);

/* Free what we just used to add to the model - now it should be stored */
librdf_free_statement($statement2);


  /* Print out the model*/

  print "Resulting model is:\n";
  print librdf_model_to_string($model, $uri, "");

  /* Construct the query predicate (arc) and object (target) 
   * and partial statement bits
   */
  $subject=librdf_new_node_from_uri_string($world, "http://www.dajobe.org/");
  $predicate=librdf_new_node_from_uri_string($world, "http://purl.org/dc/elements/1.1/title");
  if(!$subject || !$predicate) {
    die("Failed to create nodes for searching\n");
  }
  $partial_statement=librdf_new_statement($world);
  librdf_statement_set_subject($partial_statement, $subject);
  librdf_statement_set_predicate($partial_statement, $predicate);


  /* QUERY TEST 1 - use find_statements to match */

  print "Trying to find_statements\n";
  $stream=librdf_model_find_statements($model, $partial_statement);
  if(!$stream) {
    die("librdf_model_find_statements returned NULL stream\n");
  } else {
    $count=0;
    while(!librdf_stream_end($stream)) {
      $statement=librdf_stream_get_object($stream);
      if(!$statement) {
        die("librdf_stream_next returned NULL\n");
        break;
      }
      
      echo("  Matched statement: ");
      print  librdf_statement_to_string($statement);
      print "\n";
      
      librdf_stream_next($stream);
      $count++;
    }
    librdf_free_stream($stream);  
    print "got " . $count . " matching statements\n";
  }


  /* QUERY TEST 2 - use get_targets to do match */
  print "Trying to get targets\n";
  $iterator=librdf_model_get_targets($model, $subject, $predicate);
if(!$iterator)  {
    die("librdf_model_get_targets failed to return iterator for searching\n");
}

$count=0;
while(!librdf_iterator_end($iterator)) {

    $target=librdf_iterator_get_object($iterator);
    if(!$target) {
      die("librdf_iterator_get_object returned NULL\n");
    }

    print "  Matched target: ";
print librdf_node_to_string($target);
    print "
";

    $count++;
    librdf_iterator_next($iterator);
}
librdf_free_iterator($iterator);
printf("got %d target nodes\n", $count);

librdf_free_statement($partial_statement);
/* the above does this since they are still attached */
/* librdf_free_node(predicate); */
/* librdf_free_node(object); */

librdf_free_model($model);

librdf_free_storage($storage);

librdf_free_uri($uri);

librdf_free_world($world);

Good Relations, meet Freebase

Freebase Products, with RDFa of Good Relations in them.

What does it mean?

Well, if you were to gather up all of the RDF about Dyson DC25 Upright Ball Vacuums, you could find a product listing (try the 'see also' links) from Ebay, Amazon, etc etc.

It also means that this will all show up in Yahoo's index, possibly Google's too.

FOAF from Freebase

I made a quick FOAF from Freebase application.

Here's mine.

You just pass in an ID and out comes RDF/XML. It's limited to things typed as people.

This will help mend the gap between freebase RDF and common vocabularies, and was pretty easy to do - what other RDF vocabularies exist out there which would be nice to map onto freebase?

Generating Nutritional Data RDF from USDA, Part 2

I has a bit of a whinge yesterday about copyright, nutrition data, and so forth.

Today, my inbox has a nice copy of the NUTTAB data I want, I've located SR21, I've had someone else point me at canadian nutritional data too.

To get the NUTTAB data, you have to email Food Standards Australia, but thats not a huge deal.


So; progress:
I've made a script to import the USDA SR21 data into a database (ie, mysql), and render it out as RDF.

Installing it

Pretty easy stuff! Its in PHP, and makes use of PEAR.

# Get the code:
$ svn co svn checkout http://freebase-owl.googlecode.com/svn/trunk/nutrition/

# 0. Install dependencies
$ sudo apt-get install php-pear mysql wget unzip
$ sudo pear install -fa MDB2 XML_Beautifier

# 1. Get the SR21 data, extract it
$ wget http://www.nal.usda.gov/fnic/foodcomp/Data/SR21/dnload/sr21.zip
$ unzip sr21.zip

# 2. Make configuration
$ cp config.php.dist config.php
$ vim config.php

# 3. Create a database of your choosing, with the same settings as configuration
mysql -u root -p

CREATE DATABASE usda;

# 4. Run the install script. This will take a while as it imports all data. If it fails, just DROP the database and start again
$ php install.php

# 5. Give it a shot from the command line. "1002" is the USDA food id.
$ php rdfizer.php 1002

$ php rdfizer.php 1002 > 1002.rdf

#  6. Generate the whole set:
php generate-all.php

The basic plan: import everything, render out individual items, publish them statically on the web. Maybe later, get someone to stuff them all into a SPARQL endpoint.

Rinse, repeat with Canadian, Australian data. Grow a common ontology for Food, Nutrients, etc.

You can view some of the output RDF, I've not generated the whole set yet as my poor computer is far too old and creaky to do so.

Additionally, there are lots of linked data connections I want to make.

I want to link the sources with pubmed, the units with... something (side note: there's not much in the way of unit and measurement ontologies I could find!), the USDA style names with wikipedia/dbpedia/freebase; the compound names (PROCNT - protein content) with... something.

How silly is this: there's no semantic web url for milligrams. The best I could do was a few related concepts, because someone at wikipedia decided to merge all of the sub-articles for measurements into the single unit (ie, mg to g).

Nutrition, Food Standards and Copyright

I have a simple wish: build a really simple calorie counter backed with some common products; so that I don't have to manually count calories.

I should be able to type "Rosella Condensed Tomato Soup", and it finds the right product for me and tells me that it has:

Energy 78.87189kcal
Protein 1.5g
Carbohydrate 16.8g
Sugars 12.3g
Fat 1.0g
Saturates 1.0g
Sodium 1,125.0g
Serving size 245.0g
Servings per package 4

... and I ate a whole can, meaning I ate 4 servings.

So, I set off looking around for things which already did this.

Nothing.

Update: Bigfoodlist is almost exactly what I want.

Apparently, you can get information from a bunch of sources, like Foodstandards.gov.au or the USDA; but after just a little bit of clicking my eyes bled from the web 1.0 of it all.

Finally, I find where I can get information in CSV form - by emailed request.

This work is copyright. You may download, display, print and reproduce this material in unaltered form only (retaining this notice) for your personal, non-commercial use or use within your organisation.

Apart from any use as permitted under the Copyright Act 1968, no part may be reproduced by any process without prior written permission from Food Standards Australia New Zealand (FSANZ).

Requests and inquiries concerning reproduction and rights should be addressed to the Information Officer, FSANZ, PO Box 7186, Canberra BC, ACT 2610 or email info@foodstandards.gov.au

I feel so very backwards that there isn't a simple, universal product database which stores this information.

This is the kind of thing I want from the semantic web - government data I can query so that I can control my calorie intake. Why is this idea so foreign to government? Why are all licenses by default set to "no".

We'll see how far I get with my can-I-have-information email, but past experience has led me to disappointment.

Update 2: Success! USDA SR21 is a bunch of data you can download and stick into a database. It would not be hard to RDFize this.

Simple Content Negotiation for RDF service application with PHP

Need to serve up HTML for most users, but have an RDF representation of your resource?

Simply pop in:

$accept = explode(',', $_SERVER['HTTP_ACCEPT']);
if (in_array('application/rdf+xml', $accept)) {
    header('Location: path/to/rdf/version.rdf');
    die();
}

Twine invites

Twine is a semantic web version of del.icio.us.

I've got a few invites floating around. Be aware, it's a real beta - that means rough around the edges.

At first, you might not see the value in it, at least not until you peek under the hood - every item you add is exportable as RDF.

The other side of that is like del.icio.us recommending what other people have tagged an item with, Twine will recommend related or similar items, based on picking out common people, themes, etc.

If you'd like to see, drop me a line.

xOperator, the Valuation Industry, XSLT, GRDDL and RDF

I just discovered the most amazing thing. It's a jabber/xmpp sparql interface, called xOperator.

It sits between you, the end user, and multiple other agents and datastores. It uses XMPP (Google talk!) to interact with you.

You say things like:

me: add ds DBPedia http://dbpedia.org/sparql

me: query SELECT ?name WHERE {
   ?musician skos:subject <http://dbpedia.org/resource/Category:Australian_musicians> .
   ?musician foaf:name ?name .
}

bot: (Alright, this is dummied up, but I *did* have it working with other stuff before)
"Ben Lee"@de
"Billy Thorpe"@de
"Brody Dalle"@de
"Charlie McMahon"@de
"Craig Nicholls"@de
"Daevid Allen"@de
"Delta Goodrem"@de
"Fritz Hart"@de
"James George Thirlwell"@de
"Joe Dolce"@de
"Johnny Diesel"@de
"Kasey Chambers"@de
"Kevin Johnson"@de
"Natalie Gauci"@de
"Percy Aldridge Grainger"@de
"Peter Garrett"@de
"Peter Allen"@de
"Rick Springfield"@de
"Richard Franklin"@de
"Sophie Monk"@de
"Steven Kilbey"@de
"Stevie Wright"@de
"Tina Arena"@de
"Toni Collette"@de

me: add template "Tell me about *" "PREFIX owl: <http: org="" 2002="" 07="">
PREFIX xsd: <http: org="" 2001="">
PREFIX rdfs: <http: org="" 2000="" 01="">
PREFIX rdf: <http: org="" 1999="" 02="">
PREFIX foaf: <http: com="" foaf="" 1="">
PREFIX dc: <http: org="" dc="" elements="" 1="">
PREFIX : <http: org="" resource="">
PREFIX dbpedia2: <http: org="" property="">
PREFIX dbpedia: <http: org="">
PREFIX skos: <http: org="" 2004="" 02="" skos="">


SELECT ?name WHERE {
   ?thing skos:subject <http: org="" resource=""> .
   ?thing foaf:name ?name .
}"

bot: template added

me: Tell me about German_musicians

me: Tell me about German_artists
bot: Store dbpedia answered:
     name
"A. R. Penck"@de
"Adolf Ziegler"@de
"Adolph Menzel"@de
"Albrecht Altdorfer"@de
"Alfred Harth"@de
"Alfred Rethel"@de
"Alfred Mahlau"@de
"André Butzer""@de
"Bruno Paul"@de
"Bärbel Bohley""@de
"Carl Grossberg"@de
"Carsten Höller""@de
"Charles Crodel"@de
"Christian Möller""@de
"Christoph Meckel"@de
"Christoph Ruckhäberle""@de
"Cosmas Damian Asam"@de
"Dieter Grossmann"@de
"Dieter Roth"@de
"Eberhard Bosslet"@de
"Eberhard Havekost"@de
"Egid Quirin Asam"@de
"Elsa von Freytag-Loringhoven"@de
"Ernst Deger"@de
"Hugo Höppener""@de
"Franz Joseph Spiegler"@de
"Franz Wilhelm Seiwert"@de
"Franz Xaver Feuchtmayer der Jüngere""@de
"Franz Xaver Feuchtmayer"@de
"Frieder Nake"@de
"Fritz Maurischat"@de
"Gerd Aretz"@de
"Gerd Arntz"@de
"Gert Heinrich Wollheim"@de
"Gert Neuhaus"@de
"Hans Haacke"@de
"Hannah Höch""@de
"Hans Grundig"@de
"Hans Richter"@de
"Heiko Daxl"@de
"Heinrich Aldegrever"@de
"Herma Auguste Wittstock"@de
"Herbert Holzing"@de
"Horst Janssen"@de
"Horst Antes"@de
"Ignaz Günther""@de
"Jankel Adler"@de
"Hans Arp"@de
"Jim Avignon"@de
"Johann Peter Melchior"@de
"Johann Baptist Straub"@de
"Johann Joachim Kändler""@de
"Johann Joseph Christian"@de
"Johann Melchior Dinglinger"@de
"Johann Baptist Zimmermann"@de
"Johann Michael d.J. Feuchtmayer"@de
"Johannes Baader"@de
"Jonas Burgert"@de
"Joseph Anton Feuchtmayer"@de
"Gotthard Johnny Friedlaender"@de
"Jonathan Meese"@de
"Joseph Beuys"@de
"Karin Ulrike Soika"@de
"Karl von Fischer"@de
"Karl Hubbuch"@de
"Klaus Herlitz"@de
"Konrad Peter Cornelius Klapheck"@de
"Kurt Wegner"@de
"Leo Kahn"@de
"Franz Karl Leopold von Klenze"@de
"Lore Lorentz"@de
"Ludwig Fahrenkrog"@de
"Martin Kippenberger"@de
"Manfred Mohr"@de
"Manfred W. Jürgens""@de
"Maria Innocentia Hummel"@de
"Mariele Neudecker"@de
"Werner Mathias Goeritz Brunner"@de
"Mary Bauermeister"@de
"Max Ernst"@de
"Michael Buthe"@de
"Matthäus Günther""@de
"Michael Sowa"@de
"Otto Eckmann"@de
"Otto Hupp"@de
"Ottmar Hörl""@de
"Otto Pankok"@de
"Otto Piene"@de
"Peter Vischer"@de
"Peter Dreher"@de
"Rebecca Horn"@de
"Ruben Talberg"@de
"Rosemarie Trockel"@de
"Sandra Rauch"@de
"Silvia Quandt"@de
"Thomas Eller"@de
"Thomas Schütte""@de
"Tilman Riemenschneider"@de
"Tomma Abts"@de
"Torsten Amft"@de
"Uwe Laysiepen"@de
"Veit Stoß""@de
"Virgil Solis"@de
"Wilfried Behre"@de
"Wilhelm Heine"@de
"Wilm Weppelmann"@de
"Wolfgang Joop"@de
"Wolfgang Müller""@de
"Wolfgang Tillmans"@de
"Evi Moechel"@de

(10:22:30 PM) clockwerx@livejournal.com/Home: Tell me about Australian_musicians
(10:22:32 PM) daniel.oconnor: Store dbpedia answered:
     name
"Ben Lee"@de
"Billy Thorpe"@de
"Brody Dalle"@de
"Charlie McMahon"@de
"Craig Nicholls"@de
"Daevid Allen"@de
"Delta Goodrem"@de
"Fritz Hart"@de
"James George Thirlwell"@de
"Joe Dolce"@de
"Johnny Diesel"@de
"Kasey Chambers"@de
"Kevin Johnson"@de
"Natalie Gauci"@de
"Percy Aldridge Grainger"@de
"Peter Garrett"@de
"Peter Allen"@de
"Rick Springfield"@de
"Richard Franklin"@de
"Sophie Monk"@de
"Steven Kilbey"@de
"Stevie Wright"@de
"Tina Arena"@de
"Toni Collette"@de

Big whoop you say.

Well, I've been working on both an XML_GRDDL package (php5), and the relevant GRDDL transformations to turn LIXI Valuation documents into the relevant RDF/XML representation.

I'm also re-deploying work's internal Jabber server in the not too distant future.

So:
1. We handle a pretty big chunk of all residential valuations within Australia.
2. A lot of that gets described adequately in LIXI
3. There's now or soon to be a GRDDL transformation or two for that
4. And a GRDDL parser for PHP
5. Add in xOperator
6. Add in robust jabber services
7. Mix in a few microformats as well

Viola:
My workplace is half a step away from becoming a knowledge enterprise as well as a valuation panel manager.

That's not bad for a few days off!

Imagine these future scenarios:

Working smarter with internal data and external clients:

Internal xOperator to answer questions like "show me similar valuations we've done to address xyz", and "show me everything valuer X has completed over $1million"

Paid commercial access to a SPARQL service as well, for larger clients who want to better understand the marketplace
Paid commercial access to an xOperator style service for smaller clients, who just want a couple of pre-built, on demand answers


Publishing old (> 12-24 months) LIXI data on the web
Merge our data up with other countries, like the US, for trend analysis
Provide anonymized public data, with http basic auth. to protect names, addresses, privacy
Merge diverse sets of data - ie, imagine EuroStat data for Australia, but with an atomic level of detail available about the raw input

Provide semi-automated valuations based on known data
Suburb X has a population of Y and an area of Z (db/wikipedia), factor that into calculations

Government, and Software, Department of Transport Day

I made a whinge into an email - the adelaide metro site sort of works, and lets me plan a route. It provides me with information, stuck into a PDF.

I can't remix it, it's not open, I can't copy and paste it in a meaningful way.

So, I wrote a letter:

• Can I have some data?
• I want to make a google transit type application

The answer came back:

• No, we don't allow derivative applications
  

Huh? You guys do know about jNomad, right? The Adelaide metro's timetable on your phone project?

I didn't want to take the no I got for an answer.

• I pay taxes
• Adelaide Metro is run by the Department of Transport, Energy, and Infrastructure
• Taxes made the data
• The Adelaide Metro site says just ask if you want to use our copyrighted data.

So, I wrote this:

Hi again,

First off, I just want to assure you I'm not trying to be pushy and rude, because I absolutely hate it when people do that to me. If anything below comes off like that, please be assured it wasn't my intent at all; I'm just somewhat enthusiastic about open data & web applications.

The South Australian Government - Department of Transport, Energy and Infrastructure - Public Transport Division (DTEI-PTD) does not currently distribute this type of data set for the purposes of development of derivative products, for example the nextbus application quoted below.

Are there any kind of circumstances in which you guys would?
I'm aware of what's been done with jnomad; but I don't have access to it (not owning a phone). I think it's a great idea to improve the use of public transport and keep the public informed.


I have a problem though. My problem is that the Adelaide metro site has a trip planner - and it's reasonably good. It does most of what I want, most of the time. For the rest, I have to pick up a phone and hassle your call centre staff.

The trip planner can work out that I want to go from A to B; and tell me the timetables to read to do it. Unfortunately, that's a lot of

1. Input data
2. Download multiple pdfs
3. Compare each
   
4. Find the right time, date, bus number
5. Repeat until I get it right

It's very hard for me to do something like: find a restaurant, find the relevant transport options, then make a booking. There's a considerable amount of effort involved in merging the two bits of data.

Using something like google maps/google transit ( http://www.google.com/transit); I can get a lot more information a lot quicker; and I can visualise it ("Oh, I can get off the bus right near X", and "The restaurant is located at X, I need directions from my house to there").

It's really frustrating to know that all of the data is there, tucked away, and have no easy way to use it together.



As a php developer; and knowing that the Adelaide metro site (and trip planner) is written in PHP ; it doesn't seem to be a huge (design/technical) problem to produce plain text data[1] from whatever database you guys are using behind the scenes.

I'd suggest that given a little over two weeks full time, a reasonably competent PHP developer could read information from that database and produce the data.

I can't imagine any scenarios in which sharing this data does harm to the Adelaide metro/transport department - most programmers don't run rival bus companies. If anything, I'd suggest this kind of thing achieves some of the spirit of the 2006 e-Government Strategy [2]. With careful licensing you could ensure non-commerical use only.

So; my questions are:

• What licencing agreements were reached with jnomad's solution provider Laborotech?
• Would the DTEI ever consider similar agreements with Google [3]?
• Would the DTEI ever consider similar agreements with private citizens (in the form a of developer key or written agreement)?
• Would the DTEI ever consider open licences for their data[4]?
• Would the costs of developing either of the above be considered prohibitive; or outside of the current IT resources available?
• Are there more appropriate channels to pester with these kinds of questions?
  

[1] http://code.google.com/transit/spec/transit_feed_specification.htm#transitFeedSubmit
[2] http://www.agimo.gov.au/publications/2006/march/introduction_to_responsive_government
[3] http://www.google.com/intl/en_ALL/help/faq_transit.html#adding_an_agency
[4] http://creativecommons.org/licenses/by-nc-sa/2.0/

which got me an invite:

Thanks Daniel for your comments. I would like to meet with you to discuss the content of this email further, and while we are at it show you the great things we are doing.

Wow! I thought. I've pierced the layers of red tape and communicated with this person, who has the ability to act on things.

So today, I went in for the meeting.

When I came out, I'd never felt so dismayed with government before in my life.


The subtext of the entire meeting went:

I didn't read your email fully. You are one of those annoying people, and I've brought you in here to shut you up. Since you are a young person, I also must speak to you like a 5 year old, which you obviously are, on account of your youth. Q.E.D.

The spoken meeting went more like:

• We are already doing that internally! Here's an internal web application, which I'll click through, on my side of the desk, and not actually show you. IMPREZZLED? I AM!
  
• Open data? What's that? Obviously, we can't let you use any of our data. I mean, who do you think you are, someone who's paid for this kind of thing? You are John Q. Public, and I can't imagine a single use for you to have this data. No use for you, despite the fact there's enough data to keep me in a manager's position at this department, and manage all sorts of people who spend all the time in the world working with it.
  
• What do you mean, "is Jnomad a commerical company who sought us out, and sold us a solution, or is it a company that responded to a tender"? We have a contract. It's spelt... k-o-n-t-r-u-c-t. You are john q public, and we don't make contracts with you, you need money before we'll talk.
  
• Don't worry, concerned member of the public. The government has it all under control.
  
• Since you are using big words in your questions, if I furrow my brow and talk about something else, I don't have to answer the original question.
• Oh, you're leaving after only 8 minutes of this? IF YOU EVER WANT A JOB YOU HAVE MY NUMBER! WOW I HAVE REALLY FIXED THE PROBLEMS OF THE WORLD TODAY!
  

There was only snippets of information that were actually valuable amounted to:

• We don't know what open government is
• Data is OURS, ALL OURS!
• ... and one other which I won't mention here, for a while.

I should have known, as this is the same government who used EDS to build a registration site - when launched, it had "click here" in plain text on the front page.

So I left, knowing that there was no way I could ever communicate what I was trying to this person.

In other news:

• Renewing my drivers' license didn't suck
• Except the website misled me about the cost (it said: $30 to renew, periods of 1-10 years available), not ($30 per year, so you can register for 10 years ($300))
• The new tram has an awesome tram stop
• It's neat that its free for some sections
  
• For all of the money spent on the damned thing, the tram seats are rock hard. The old tram was far more comfortable.

e-government, SOA and Microformats

The Australian Government appears to get e-government. I'm still waiting on an email reply, to a whole bunch of question I had though, so maybe the AGIMO isn't the most responsive government department...

Anyway, this leads me to ask:

• What government sites would benefit from basic microformats the most? (vcard, vevent, addresses)
• What raw data would benefit you the most?

For me, when it comes to work, it's getting hooked up to a title/deed search webservice - it's currently a complete pain and splintered between many different offices.

Personally, I like what the National Archive of Australia has done - rdf for all with AGLS!

Also useful for business is the ABRRegister SOAP services - exposing simple functionality to lookup ABNs.

So readers, what would make you as a private citizen quite pleased to have?

Why it's a Good Idea to use Sensible CSS

Imagine:
You are blind; and you want to read this blog. Do you have to wade through the huge left hand side menu of crap before you can find out any information. Wouldn't it be great if your screenreader could guess that a menu is a menu and content is content? (ignoring one unless you ask for it).

Meet SADIe.

1. Author creates site, marks up css with classes and names like menu and content and comments
2. Author or third party describes in generic terms what a menu is, what content is in an ontology. For site specific things, you can extend basic concepts like UnimportantInformation to create a meaningful description of your site. There are editors for this.
   
3. This is made available with a LINK tag to the RDF.
4. A semantic web enabled screenreader comes along. It knows what Content is; and can map the CSS styles for content to the Content object. It doesn't know what CloCkWeRXAboutPage is; but it knows CloCkWeRXAboutPage extends UnimportantInformation.
   
5. It reorganises the layout of the content to provide a better user experience for the end user. Unimportant stuff gets left behind, important stuff gets upped in priority

My only thought is the affect of spam / advertising if this took off.

PEAR is getting foaf...

Finally, PEARweb is getting foaf output - I'll have a reputable foaf file again.

Yay.

Lijit / Outfoxed

Outfoxed is something I encountered some time ago - it's just like StumbleUpon but provides a bit more under the hood.
It's just become lijit.com; and it's still a firefox extension. It's a bit hard to get started with it - but create an account and find the 'get the extension' link on the left hand side.

Drupal 4.7 and the foaf.module

I wanted the foaf module for drupal on 4.7; so instead of being a whinger; I just went out and did it.

Here's the patch:

Index: modules/foaf/foaf.module
===================================================================
RCS file: /cvs/drupal-contrib/contributions/modules/foaf/foaf.module,v
retrieving revision 1.13
diff -u -r1.13 foaf.module
--- modules/foaf/foaf.module 12 Jul 2006 17:13:51 -0000 1.13
+++ modules/foaf/foaf.module 23 Oct 2006 13:21:35 -0000
@@ -1,5 +1,5 @@
<?php
-// $Id: foaf.module,v 1.13 2006/07/12 17:13:51 walkah Exp $
+// $Id: foaf.module,v 1.6.2.4 2005/11/29 18:22:34 walkah Exp $

/**
 * Implements hook_help
@@ -11,8 +11,8 @@
      $output .= '<p>'. t('The FOAF module allows users to export FOAF documents based on their profile information. It also allows users to import profile information from external FOAF files, and even includes an option to let profiles auto sync when using distributed authentication.  The FOAF module can export your buddy list if the buddy list module is enabled.   You can download FOAF files in the view tab of a user profile if FOAF is enabled.') .'</p>';
      $output .= t('<p>You can</p>
<ul>
-<li>read the <a href="%external-http-en-wikipedia-org-wiki-FOAF" title = "explanation of FOAF from wikipedia the free encyclopedia">FOAF wikipedia explanation</a>.</li>
-<li>read the <a href="http://www.socialtext.net/foafnet/index.cgi?basic_specs" title = "a programmers level description of this social networking standard">FOAFNet specificiation</a>.</li>
+<li>read the <a href="%external-http-en-wikipedia-org-wiki-FOAF" title="explanation of FOAF from wikipedia the free encyclopedia">FOAF wikipedia explanation</a>.</li>
+<li>read the <a href="http://www.socialtext.net/foafnet/index.cgi?basic_specs" title="a programmers level description of this social networking standard">FOAFNet specificiation</a>.</li>
<li>view <a href="%profile"> user profiles</a> and select a user to see their FOAF file.</li>
<li>enable the buddy list module at <a href="%admin-modules" > administrator >> modules</a>.</li>
', array('%external-http-en-wikipedia-org-wiki-FOAF' => 'http://en.wikipedia.org/wiki/FOAF', '%external-http-www-socialtext-net-foafnet-index-cgi' => 'http://www.socialtext.net/foafnet/index.cgi?basic_specs', '%profile' => url('profile'), '%admin-modules' => url('admin/modules'))) .'</ul>';
@@ -55,9 +55,15 @@
      break;
    case 'view':
      drupal_set_html_head("\n".'<link rel="meta" type="application/rdf+xml" title="FOAF" href="'.url('foaf/'.$user->uid, NULL, NULL, TRUE).'" />');
-      $output = '';
      if (_foaf_allow_export($user)) {
-          return array(t('Personal Information') => l(t('Download <acronym title="Friend of a Friend">FOAF</acronym>'), 'foaf/'.$user->uid, NULL, NULL, NULL, FALSE, TRUE));
+       
+        $title = t('Personal Information');
+
+        $link = l('Download FOAF', 'foaf/'.$user->uid, NULL, NULL, NULL, FALSE, TRUE);
+        $output = array();
+        $output[$title][] = array('value' => $link);
+
+        return $output;
      }
      break;
    case 'form':
@@ -69,31 +75,41 @@
}

function foaf_settings() {
-  $output = '';
-
-  $output.= form_radios(t('Export <acronym title="Friend of a Friend">FOAF</acronym> for users by default'), 'foaf_export_default', variable_get('foaf_export_default', 0), array(0 => t('no'), 1=> t('yes')), t('This can be overridden by individual users'));
+  $form = array();
+  $form["foaf_export_default"] = array();
+  $form["foaf_export_default"]["#title"] = t('Export <acronym title="Friend of a Friend">FOAF</acronym> for users by default');
+  $form["foaf_export_default"]["#type"]  = 'radios';
+  $form["foaf_export_default"]["#default_value"]  = variable_get('foaf_export_default', 0);
+  $form["foaf_export_default"]["#options"]  = array(0 => t('no'), 1=> t('yes'));
+  $form["foaf_export_default"]["#suffix"] = t('This can be overridden by individual users');
+ 
+  $formItemTemplate = array('#type' => 'select');
+
+  $fields = array('foaf_name', 'foaf_title','foaf_firstName',
+                    'foaf_surname', 'foaf_Organization',
+                    'foaf_phone', 'foaf_aimChatID', 'foaf_icqChatID',
+                    'foaf_msnChatID', 'foaf_yahooChatID', 'foaf_jabberChatID',
+                    'foaf_workplaceHomepage');
+  $fields["vCard:Street"]   = 'foaf_street';
+  $fields["vCard:Locality"] = 'foaf_locality';
+  $fields["vCard:Region"]   = 'foaf_region';
+  $fields["vCard:PCode"]    = 'foaf_pcode';
+  $fields["vCard:Country"]  = 'foaf_country';
+  $fields["bio:keywords"]   = 'foaf_keywords';
+
+  $form["settings"] = array("#title" => t('Field Mappings'), "#type" => "fieldset");
+  foreach ($fields as $name => $field) {
+    if (!is_string($name)) {
+      $name = str_replace('_', ':', $field);
+    }
+    $form["settings"][$field] = array_merge($formItemTemplate,
+                                array('#title' => t('Field for ' . $name),
+                                      '#default_value' =>  variable_get($field, false),
+                                      '#options' => _foaf_profile_fields()
+                                ));
+  }
 
-  $mapping.= form_select(t('Field for foaf:name'), 'foaf_name', variable_get('foaf_name', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for foaf:title'), 'foaf_title', variable_get('foaf_title', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for foaf:firstName'), 'foaf_firstName', variable_get('foaf_firstName', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for foaf:surname'), 'foaf_surname', variable_get('foaf_surname', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for foaf:Organization'), 'foaf_Organization', variable_get('foaf_Organization', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for foaf:phone'), 'foaf_phone', variable_get('foaf_phone', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for foaf:aimChatID'), 'foaf_aimChatID', variable_get('foaf_aimChatID', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for foaf:icqChatID'), 'foaf_icqChatID', variable_get('foaf_icqChatID', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for foaf:msnChatID'), 'foaf_msnChatID', variable_get('foaf_msnChatID', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for foaf:yahooChatID'), 'foaf_yahooChatID', variable_get('foaf_yahooChatID', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for foaf:jabberID'), 'foaf_jabberID', variable_get('foaf_jabberID', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for foaf:workplaceHomepage'), 'foaf_workplaceHomepage', variable_get('foaf_workplaceHomepage', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for vCard:Street'), 'foaf_street', variable_get('foaf_street', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for vCard:Locality'), 'foaf_locality', variable_get('foaf_locality', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for vCard:Region'), 'foaf_region', variable_get('foaf_region', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for vCard:Pcode'), 'foaf_pcode', variable_get('foaf_pcode', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for vCard:Country'), 'foaf_country', variable_get('foaf_country', ''), _foaf_profile_fields());
-  $mapping.= form_select(t('Field for bio:keywords'), 'foaf_keywords', variable_get('foaf_keywords', ''), _foaf_profile_fields());
-
-  $output.= form_group(t('Field Mappings'), $mapping);
-  return $output;
+  return $form;
}

function foaf_xmlrpc() {
@@ -349,6 +365,7 @@
  while ($row = db_fetch_object($result)) {
    $output[$row->fid] = $row->title;
  }
+
  return $output;
}

GeoURL is back

I must have been asleep at my post, because this one slipped right by me. GeoURL is back and better than ever. Woo!

There's even a firefox extension. And mixed in RDF! Whoo!

Cool

For a while I've had an idea about GIS and photo annotation. From my lips to Trevor F Smith's creation. I'm at work with dialup at the moment, but I want to use it.

Questions:

• Can I collaborate with my peers to annotate my city?
• Can I get the data out as XML (with suitable inverse functional properties) which I can then turn into RDF?
• How Cool Is Trev (and does he mind me calling him Trev)?!

It's HERE!

Check out the publicity post.

See more progress on: See a 43 things Web Services API & RDF output

The Semantic Web & Porn: The solution PICS never was.

Do any of you fellow web folk remember a thing called Platform for Internet Content Selection (PICS)?

It was a content labelling system for the web. It was mainly focused on accurately labelling content to provide you with choice before you were assaulted with pictures of people doing nasty things with food items.

Yes, we've all seen goatse.cx.

There were several approaches undertaken with PICS, from HTTP headers...

HTTP/1.0 200 OK
Date: Thu, 30 Jun 1995 17:51:47 GMT
Last-modified: Thursday, 29-Jun-95 17:51:47 GMT
Protocol: {PICS-1.1 {headers PICS-Label}}
PICS-Label:
(PICS-1.1 "http://www.gcf.org/v2.5" labels
on "1994.11.05T08:15-0500"
exp "1995.12.31T23:59-0000"
for "http://www.greatdocs.com/foo.html"
by "George Sanderson, Jr."
ratings (suds 0.5 density 0 color/hue 1))
Content-type: text/html

... to META tags ...

<META http-equiv="PICS-Label" content='

(PICS-1.1 "http://www.gcf.org/v2.5"

labels on "1994.11.05T08:15-0500"

       until "1995.12.31T23:59-0000"

       for "http://w3.org/PICS/Overview.html"

ratings (suds 0.5 density 0 color/hue 1))

'>

But it never really was too effective. That's because it was a bitch to label every page with something hard to read like that, and if you didn't know how to make your webserver do things niftily, you were screwed.

When MCF came about, and turned into RDF, PICS was the cool application for it.

Separate W3C working groups are developing a new label format, called RDF; the Resource Description Framework, based on XML. RDF labels will be able to express everything that PICS labels can express, but will also permit string and structured values, and some other nifty features. The latest information on this available at http://www.w3.org/RDF.

... but then, no one really cared for a good few years.

So why this post?

Because PICS is back and more RDFish than ever before!

It works like CSS does - one file to describe them all.

<link rel="meta" href="labels.rdf" type="application/rdf+xml" title="Content labels" />

... and in labels.rdf, we find:

<?xml version="1.0"?>
<rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
xmlns:label="http://www.w3.org/2004/12/q/contentlabel#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:icra="http://www.icra.org/rdfs/vocabularyv03#">

<rdf:Description rdf:about="">
<dc:creator rdf:resource="http://www.icra.org" />
<label:authorityFor>http://www.icra.org/rdfs/vocabularyv03</label:authorityFor>
</rdf:Description>

<label:Ruleset rdf:ID="Ruleset">
<label:hostRestriction>icra.org</label:hostRestriction>
<label:hostRestriction>icra.org.uk</label:hostRestriction>
<label:hostRestriction>icra.tv</label:hostRestriction>
<label:hostRestriction>rsac.org</label:hostRestriction>
<label:hasDefaultLabel rdf:resource="#label_1"/>
</label:Ruleset>

<label:ContentLabel rdf:ID="label_1">
<rdfs:label>No nudity, no sex, no violence, no potentially offensive language, no user-generated content, no other potentially harmful activities</rdfs:label>
<icra:cz>1</icra:cz>
<icra:lz>1</icra:lz>
<icra:nz>1</icra:nz>
<icra:oz>1</icra:oz>
<icra:sz>1</icra:sz>
<icra:vz>1</icra:vz>
</label:ContentLabel>
</rdf:RDF>

So how do you get it working for you?

Well, you can either read W3C tech specs or you can use this handy dandy reference (mine has more mention of nudity, buttocks, and exposed genitals).

<label:ContentLabel rdf:ID="label_1">
 <rdfs:label>Human readable description of rating goes here. WARNING: This contains geek pr0n.</rdfs:label>
 <!-- Category nx: Nudity -->
   <icra:nz>1</icra:nz> <!-- No Nudity -->

 <!-- Category cx: User Generated Content -->
   <icra:cz>1</icra:cz> <!-- No User-Generated Content -->

 <!-- Category lx: Language -->
   <icra:lz>1</icra:lz> <!-- No Language Content -->
 <!-- Category ox: Potentially Harmful Activites -->
   <icra:oz>1</icra:oz> <!-- No Potentially Harmful Activites Content -->
 <!-- Category sx: Sexual Material -->
   <icra:sz>1</icra:sz> <!-- No Sexual Material Content -->
 <!-- Category vx: Violence -->
   <icra:vz>1</icra:vz> <!-- No Violent Content -->
</label:ContentLabel>