Missing Maps, Ebola and Learn OSM

I've spent a large amount of time over the past weeks mapping for Ebola. You can see some of my work, along with many others in Freetown, Kayes and more.

Often it has been an article of faith that the requesting organisations find mapping contributions useful; with feedback few and far between. For example, we saw some usage during Typhoon Haiyan, but it's hard to know if you are being truely effective.

This past fortnight, that doubt has been firmly put to bed. MSF has launched Missing Maps, proactively calling on the global community not to donate money, but to donate time.

Additionally, a thread on the HOTOSM mailing list has nicely overlapped in demonstrating value - and of course, personal anecdotes go a long way to making things relatable.

Given this renewed push from the people consuming open street map data in crisis situations, I began to wonder how I could turn my efforts from "dedicated mapper" to something more magnified.

I was lucky to have Nick Tailguy commit to a refreshed OSM Tasking manager guide and talk to the mailing lists - I've since contributed a number of small edits that will hopefully go on to encourage new mappers to confidently contribute in times of need.


To really kick it up a notch, I'm considering outreach to universities in ebola impacted locations - but I could really use the help of a dedicated few to push me further here; if the idea truely has value.

Shaprgram: a Govhack entry for crowdsourcing photos of spatial data

#Govhack is over, and I'm still recovering from it. I didn't know what to expect, but ran into Kieran and Chris from the AdelaideRB meetup.

We decided to team up, and kicked around a few ideas.

Friday

We got to talking through the ideas and datasets. Some we didn't love, some we didn't just have time for. Things got a bit weird with all of the teams murmuring in hushed tones, protecting their ideas. We decided the point of govhack was openness, so we shared our non starters. I was surprised to see the huge rush of people within milliseconds, obviously stalking the #govhack twitter feeds.

It came down to do a visualisation/data exploration tool or make a reusable tool that improves data. I managed to sway the rest of the team towards reusable tool, pointing out that schema and data production was a complete pain; and anything we built might not survive beyond the underlying data being regenerated.

We spent the rest of the time setting up on #Slack, Trello, Github and more. I sketched out some of the worst looking paper prototypes, then at home early into the AM bashed out a bunch of placeholders.

Saturday

Exhausted from the early morning work, I crawled in at around 11am to find terrible scaffold turning into a much nicer rails app.

We made a few choices that would come back to haunt us, including using https://github.com/mutablestate/ink3-rails - works great in dev, but absolutely hates being compiled in production mode, as we later found out.

We wrote some CSV and Shapefile importers, working with rgeo and postgis to add collections of points. Chris pointed out postgres' JSON field support solved one of the flexible schema problems we had (I was dreaming up nasty hacks and ways to kick ActiveRecord square in the pants), and by mid afternoon we had a pretty good prototype working.

A stampede took place, and all of the other teams vanished before we knew what was going on. 

I took the chance to catch up with Alex, and we got talking to some of the data.sa.gov.au folks. 

I found this pretty valuable - we touched on some of the reluctance to open up data that exists, the need for productive narratives & collaborations to highlight the value of open data, and the need for data set requests.

Chris was bitten by the urge to work into the early hours, and pushed us a lot further along.

When I got home, I thought we were looking pretty good - at least until I demo'd it to a non developer. Nothing like a live demo to find bugs.

Sunday

The final day - it came down to polish and deploy, which turned out to be a lot harder than it should have been.

Due to the previously mentioned asset compilation problems, there was frenzied cursing and kicking of the asset pipeline; as well as a fair bit of effort being put into infrastructure. I'd started out with a bit of chef to cook a machine, but found the change/deploy/test loop was tedious.

A few cosmetic and usability fixes were done, our team video recorded, and then disaster: most of the data sets we chose were in arbitrary reference systems (GDA94, etc), and we'd accidentally broken our transformations into WSG84.

Literally in the last 10 minutes, we ended up live editing to strongly encourage WSG84 friendly data sets to be added and to prove our code was happy end to end.

Results

Shaprgram is live, if a little buggy. It lets you take any dataset in WSG84 hosted on a data.*.gov.au site and crowdsource photos.

Here's our entry - don't forget to vote!

Who's this for

Shaprgram is aimed at government data producers who may not have the full resources of a geospatial team at their beck and call, who need to crowdsource information about their assets; and anyone that can benefit from visual imagery to supplement raw data.

Use cases

Adelaide City Council produces a number of data sets regarding picnic sites, which is published on their website and as open data. Unfortunately no one was able to snap a few photos at the time, and google streetview doesn't reach these places.

Using this tool, anyone with a shapefile or CSV can put up a list of assets that need photographs, and download a simple .zip file with an enhanced dataset.

Asset tracking

Just spent a few hard earned rate payer dollars on nice new benches? Use this tool to capture a visual representation of the asset at the time of installation; and during routine tasks - letting you see how your assets survive in the wild, when they need repair or more.

Much more

• Parking spaces & restricted parking - all parking inspectors carry cameras, making it trivial to document your city
• Signs - need to capture the signed speed limits across multiple authorities? Use this tool to complete your dataset
• Tree health - Let members of your community tell you when a tree is ill, growing too much or more - find out *before* it becomes a costly issue to rectify.

OpenStreetMap and Keepright: Fixing all of the redaction errors in Melbourne

The OSM license change that lead to redaction (here's a decent animation of the automated reverts) happened well over two years ago now; and a lot of remapping has taken place.

Unfortunately, humans aren't as good as some of the QA tools we have available, and that means a lot of valuable mapping is left untagged.

Keepright has been helping me find all of the untagged ways in Adelaide and Melbourne; both cities are now effectively 'clean'. Anything where the last edit was the redaction bot usually only needs a few tags or to be deleted if it's a duplicate.

I'd encourage any other mappers who feel their area is 'complete' to view it through a tool like Keepright - even if you pick as few as 5-10 errors to fix, you can make a big impact.

Haiyan, part 2

Interestingly, the majority of Tacloban is now mapped, as well as many other areas.

Someone's run stats: 245 volunteers who created nearly 440,000 map changes within the aforementioned extent. It's now up to 600,000+

More exciting than that, however, is the sheer number of emergency responders using OpenStreetMap data.
For example; since we mapped just about every building in Tacloban, satellites have been redirected to capture updated images.
The two are being explicitly combined. You can see a really detailed map of what survived, and what did not:

http://emergency.copernicus.eu/mapping/ems-product-component/EMSR058_02TACLOBANCITY_GRADING_DETAIL03/1

You can also see a mashup of geotagged photos from reports on the ground, backed up by OSM tiles:
https://haiyan.crowdmap.com/

While we can't change the fact thousands have died, there's a quite reasonable hope that accurate, up to date geospatial information can change the situation; enabling people to prioritise with data driving decisions in both the immediate and longer term recovery efforts.

Local Government Areas and IDs

The ABS have made their data creative commons; but it's still a pain to get to the data you want.

I'm trying to reconcile LGAs against our work application, freebase, and the census data - but it's a pain.

To make things a bit easier for others treading this path, here's a spreadsheet of LGAs and ABS ids for them, 2006

Geocoding, Google Earth and Work

So I'm pretty excited at the moment, as we're finally rolling out our geocoding extras for our work application, ValEx.

This means you can see any of the 3.6 million or more Australian addresses we know about in Google Earth; search by proximity; find comparible sales to your subject property; or see your appointments for the day.

Frankly, it's all kind of amazing.

However all of that was just swept from my mind by Google Earth is embeddable in your browser.

Government, and Software, Department of Transport Day

I made a whinge into an email - the adelaide metro site sort of works, and lets me plan a route. It provides me with information, stuck into a PDF.

I can't remix it, it's not open, I can't copy and paste it in a meaningful way.

So, I wrote a letter:

• Can I have some data?
• I want to make a google transit type application

The answer came back:

• No, we don't allow derivative applications
  

Huh? You guys do know about jNomad, right? The Adelaide metro's timetable on your phone project?

I didn't want to take the no I got for an answer.

• I pay taxes
• Adelaide Metro is run by the Department of Transport, Energy, and Infrastructure
• Taxes made the data
• The Adelaide Metro site says just ask if you want to use our copyrighted data.

So, I wrote this:

Hi again,

First off, I just want to assure you I'm not trying to be pushy and rude, because I absolutely hate it when people do that to me. If anything below comes off like that, please be assured it wasn't my intent at all; I'm just somewhat enthusiastic about open data & web applications.

The South Australian Government - Department of Transport, Energy and Infrastructure - Public Transport Division (DTEI-PTD) does not currently distribute this type of data set for the purposes of development of derivative products, for example the nextbus application quoted below.

Are there any kind of circumstances in which you guys would?
I'm aware of what's been done with jnomad; but I don't have access to it (not owning a phone). I think it's a great idea to improve the use of public transport and keep the public informed.


I have a problem though. My problem is that the Adelaide metro site has a trip planner - and it's reasonably good. It does most of what I want, most of the time. For the rest, I have to pick up a phone and hassle your call centre staff.

The trip planner can work out that I want to go from A to B; and tell me the timetables to read to do it. Unfortunately, that's a lot of

1. Input data
2. Download multiple pdfs
3. Compare each
   
4. Find the right time, date, bus number
5. Repeat until I get it right

It's very hard for me to do something like: find a restaurant, find the relevant transport options, then make a booking. There's a considerable amount of effort involved in merging the two bits of data.

Using something like google maps/google transit ( http://www.google.com/transit); I can get a lot more information a lot quicker; and I can visualise it ("Oh, I can get off the bus right near X", and "The restaurant is located at X, I need directions from my house to there").

It's really frustrating to know that all of the data is there, tucked away, and have no easy way to use it together.



As a php developer; and knowing that the Adelaide metro site (and trip planner) is written in PHP ; it doesn't seem to be a huge (design/technical) problem to produce plain text data[1] from whatever database you guys are using behind the scenes.

I'd suggest that given a little over two weeks full time, a reasonably competent PHP developer could read information from that database and produce the data.

I can't imagine any scenarios in which sharing this data does harm to the Adelaide metro/transport department - most programmers don't run rival bus companies. If anything, I'd suggest this kind of thing achieves some of the spirit of the 2006 e-Government Strategy [2]. With careful licensing you could ensure non-commerical use only.

So; my questions are:

• What licencing agreements were reached with jnomad's solution provider Laborotech?
• Would the DTEI ever consider similar agreements with Google [3]?
• Would the DTEI ever consider similar agreements with private citizens (in the form a of developer key or written agreement)?
• Would the DTEI ever consider open licences for their data[4]?
• Would the costs of developing either of the above be considered prohibitive; or outside of the current IT resources available?
• Are there more appropriate channels to pester with these kinds of questions?
  

[1] http://code.google.com/transit/spec/transit_feed_specification.htm#transitFeedSubmit
[2] http://www.agimo.gov.au/publications/2006/march/introduction_to_responsive_government
[3] http://www.google.com/intl/en_ALL/help/faq_transit.html#adding_an_agency
[4] http://creativecommons.org/licenses/by-nc-sa/2.0/

which got me an invite:

Thanks Daniel for your comments. I would like to meet with you to discuss the content of this email further, and while we are at it show you the great things we are doing.

Wow! I thought. I've pierced the layers of red tape and communicated with this person, who has the ability to act on things.

So today, I went in for the meeting.

When I came out, I'd never felt so dismayed with government before in my life.


The subtext of the entire meeting went:

I didn't read your email fully. You are one of those annoying people, and I've brought you in here to shut you up. Since you are a young person, I also must speak to you like a 5 year old, which you obviously are, on account of your youth. Q.E.D.

The spoken meeting went more like:

• We are already doing that internally! Here's an internal web application, which I'll click through, on my side of the desk, and not actually show you. IMPREZZLED? I AM!
  
• Open data? What's that? Obviously, we can't let you use any of our data. I mean, who do you think you are, someone who's paid for this kind of thing? You are John Q. Public, and I can't imagine a single use for you to have this data. No use for you, despite the fact there's enough data to keep me in a manager's position at this department, and manage all sorts of people who spend all the time in the world working with it.
  
• What do you mean, "is Jnomad a commerical company who sought us out, and sold us a solution, or is it a company that responded to a tender"? We have a contract. It's spelt... k-o-n-t-r-u-c-t. You are john q public, and we don't make contracts with you, you need money before we'll talk.
  
• Don't worry, concerned member of the public. The government has it all under control.
  
• Since you are using big words in your questions, if I furrow my brow and talk about something else, I don't have to answer the original question.
• Oh, you're leaving after only 8 minutes of this? IF YOU EVER WANT A JOB YOU HAVE MY NUMBER! WOW I HAVE REALLY FIXED THE PROBLEMS OF THE WORLD TODAY!
  

There was only snippets of information that were actually valuable amounted to:

• We don't know what open government is
• Data is OURS, ALL OURS!
• ... and one other which I won't mention here, for a while.

I should have known, as this is the same government who used EDS to build a registration site - when launched, it had "click here" in plain text on the front page.

So I left, knowing that there was no way I could ever communicate what I was trying to this person.

In other news:

• Renewing my drivers' license didn't suck
• Except the website misled me about the cost (it said: $30 to renew, periods of 1-10 years available), not ($30 per year, so you can register for 10 years ($300))
• The new tram has an awesome tram stop
• It's neat that its free for some sections
  
• For all of the money spent on the damned thing, the tram seats are rock hard. The old tram was far more comfortable.

Google Maps to have microformats

Microformats in Google Maps.

Pretty neat. If you don't know what this means, I'll tell you.

You would:

• Search for pizza in collinswood
• Find your favourite pizza place
• Click a button
• Add it magically to your address book

Want to see how? Go and install Operator, and then do the above steps.

PhotoSynth, Seadragon, BeOS and Jef Raskin's THE

My friend James pointed to TED demo of seadragon/photosynth.

Think of it as a two part thing

• Seadragon is a rendering engine for images, which is really fast and only needs as much memory as your screen resolution needs


• PhotoSynth is automatic linking of images to make 3D models

Go and watch the demo if you have no idea what I just said.

There's a live demo too.

The interaction with SeaDragon reminded me of Jef Raskin's zooming user interface (8mb swf)

Pretty sweet.

I always wanted to build something like it, after playing with a photo+wireframing tool (It was like a poor man's google sketchup) when I was about 13. You'd get an image, draw a cube, and texture it, get another image, texture, hey presto, easy digital model. Unfortunately, adobe bought it and snuffed it, and I have no freakin' idea on how to build something like they have.

The performance of SeaDragon reminded me of a BeOS demo video - a lot of things going one at once.

Note to self

KML rendered with Google Maps is a great way to add value to your application.

GeoURL is back

I must have been asleep at my post, because this one slipped right by me. GeoURL is back and better than ever. Woo!

There's even a firefox extension. And mixed in RDF! Whoo!

Cool

For a while I've had an idea about GIS and photo annotation. From my lips to Trevor F Smith's creation. I'm at work with dialup at the moment, but I want to use it.

Questions:

• Can I collaborate with my peers to annotate my city?
• Can I get the data out as XML (with suitable inverse functional properties) which I can then turn into RDF?
• How Cool Is Trev (and does he mind me calling him Trev)?!

Patently Insane, Continued

No less than 30 minutes after I had a bit of a go at patents, I found something that really annoys me.

The gist of it? Microsoft claims to have 'invented' a system of inserting imprecise numbers into URLs to represent longitude and latitude by concatinating them.

For the non geekish: Every point on the world has a longitude and latitude. These things are pretty precise... For instance, I live near longitude: 138.5831, latitude: -34.9502. Microsoft says that if I put those numbers in a URI in a certain way, and that URI is about whatever is there, they can sue me.

Summary of Patent: IfYouWriteLikeThisItsPatented... except with long/lat numbers. Joined together words are out of bounds.

To challenge them, it would cost me an arm and a leg. Not this very week was I renewing my interest in a geospatial annotation and mapping thing inspired by OpenGuides...

And now... NOTHING.