Shaprgram: a Govhack entry for crowdsourcing photos of spatial data

#Govhack is over, and I'm still recovering from it. I didn't know what to expect, but ran into Kieran and Chris from the AdelaideRB meetup.

We decided to team up, and kicked around a few ideas.

Friday

We got to talking through the ideas and datasets. Some we didn't love, some we didn't just have time for. Things got a bit weird with all of the teams murmuring in hushed tones, protecting their ideas. We decided the point of govhack was openness, so we shared our non starters. I was surprised to see the huge rush of people within milliseconds, obviously stalking the #govhack twitter feeds.

It came down to do a visualisation/data exploration tool or make a reusable tool that improves data. I managed to sway the rest of the team towards reusable tool, pointing out that schema and data production was a complete pain; and anything we built might not survive beyond the underlying data being regenerated.

We spent the rest of the time setting up on #Slack, Trello, Github and more. I sketched out some of the worst looking paper prototypes, then at home early into the AM bashed out a bunch of placeholders.

Saturday

Exhausted from the early morning work, I crawled in at around 11am to find terrible scaffold turning into a much nicer rails app.

We made a few choices that would come back to haunt us, including using https://github.com/mutablestate/ink3-rails - works great in dev, but absolutely hates being compiled in production mode, as we later found out.

We wrote some CSV and Shapefile importers, working with rgeo and postgis to add collections of points. Chris pointed out postgres' JSON field support solved one of the flexible schema problems we had (I was dreaming up nasty hacks and ways to kick ActiveRecord square in the pants), and by mid afternoon we had a pretty good prototype working.

A stampede took place, and all of the other teams vanished before we knew what was going on. 

I took the chance to catch up with Alex, and we got talking to some of the data.sa.gov.au folks. 

I found this pretty valuable - we touched on some of the reluctance to open up data that exists, the need for productive narratives & collaborations to highlight the value of open data, and the need for data set requests.

Chris was bitten by the urge to work into the early hours, and pushed us a lot further along.

When I got home, I thought we were looking pretty good - at least until I demo'd it to a non developer. Nothing like a live demo to find bugs.

Sunday

The final day - it came down to polish and deploy, which turned out to be a lot harder than it should have been.

Due to the previously mentioned asset compilation problems, there was frenzied cursing and kicking of the asset pipeline; as well as a fair bit of effort being put into infrastructure. I'd started out with a bit of chef to cook a machine, but found the change/deploy/test loop was tedious.

A few cosmetic and usability fixes were done, our team video recorded, and then disaster: most of the data sets we chose were in arbitrary reference systems (GDA94, etc), and we'd accidentally broken our transformations into WSG84.

Literally in the last 10 minutes, we ended up live editing to strongly encourage WSG84 friendly data sets to be added and to prove our code was happy end to end.

Results

Shaprgram is live, if a little buggy. It lets you take any dataset in WSG84 hosted on a data.*.gov.au site and crowdsource photos.

Here's our entry - don't forget to vote!

Who's this for

Shaprgram is aimed at government data producers who may not have the full resources of a geospatial team at their beck and call, who need to crowdsource information about their assets; and anyone that can benefit from visual imagery to supplement raw data.

Use cases

Adelaide City Council produces a number of data sets regarding picnic sites, which is published on their website and as open data. Unfortunately no one was able to snap a few photos at the time, and google streetview doesn't reach these places.

Using this tool, anyone with a shapefile or CSV can put up a list of assets that need photographs, and download a simple .zip file with an enhanced dataset.

Asset tracking

Just spent a few hard earned rate payer dollars on nice new benches? Use this tool to capture a visual representation of the asset at the time of installation; and during routine tasks - letting you see how your assets survive in the wild, when they need repair or more.

Much more

• Parking spaces & restricted parking - all parking inspectors carry cameras, making it trivial to document your city
• Signs - need to capture the signed speed limits across multiple authorities? Use this tool to complete your dataset
• Tree health - Let members of your community tell you when a tree is ill, growing too much or more - find out *before* it becomes a costly issue to rectify.

Rails, how developers grow, and what's wrong with it

Rails is a great tool for automating the heck out of pumping out a quick and dirty site in short order.

It's a terrible tool for learning. The community isn't like the IRC communities I've been involved in. It's catering for a lot of developers early in their journey.

This is what I see on /r/rails or hacker news or being actively practised far too often:

• I just discovered service objects
• Test Driven Development/Unit Tests are dead, because it's too hard to stub out ActiveRecord.
• Uh, guys, I started using after_save hooks and now my persistence layer is working with Sidekiq and the Mail server to kill all humans.
• DHH said something outrageous! 60% of us are rushing to take his word as gospel right now; cause like, uh, nuts to the status quo!
• Oh neat I discovered modules, and that classes are objects, so... like, I'm not going try to do evil... but... here's a gem I wrote that adds a search to all of the defined classes, because it's not like that could cause conflicts; right?

and it makes me cranky. It's actively harmful for people who don't have seasoned peers near them, ready to slide over a polite link on SOLID; or talk to you about the Principle of least astonishment.

Surely we can do better.

Where are the articles on

• If you are testing your active record instances, that's not a unit test. Go extract your domain models and test those please.
• Here's 10 terrible ways to write a component/gem/etc; because it's the people using your code that have to put up with your mistakes when your github repo becomes abandonware.
• How to open up a github project to collaborators when you don't have the time to maintain your code anymore.
• How to write a decent readme that sums up your gem/repo/etc.
• What is Single Responsibility Principle? How does it apply within the context of ActiveRecord, Controllers and triggering after-create/after-save events?
• Why are we OK to use FactoryGirl for tests (obviously useful), but much less willing to use the Factory Pattern to assemble our object graph; when convention breaks down in the face of complexity?
• KISS: Why it's often worth writing code you might feel is imperfect if it helps your team collectively understand what the heck you are doing.
• Why version control systems have a blame feature, and the startling reality the next guy looking after your system can track you down and beat you to death with your own severed legs.
• Stop reading the ember.js documentation now, and just go look at the source code, it'll save us  all a load of time.
• Magic is generally bad unless you are all wizards; but like Lord of the Rings it's freakin' easy for one or more of you to turn evil.

Interestingly, many these are framework and language agnostic - in my experience though, many framework centric types tend to only know their framework and community; so it distorts their understandings. 

It's not their fault - otherwise smart people have set out to solve a number of problems, then tried to generalise them having already learnt a significant amount of things the hard way - they just keep forgetting to share their learnings.

The best resources I had when I was learning was a mix of IRC (there is nothing like asking a stupid question and realising you just did so infront of the guy who invented the web, or the language you are using right now), and wikis like Portland Pattern Repository.

Dog of a week... and it's only monday?

Feeling a bit overwhelmed already

1. I don't know why I have to argue about DI, but I find I do. This neatly gets away from fat controllers, fat models and finally the the stuff that matters.
2. I pulled together a quick ABN input on the weekend. I figured I've built the thing enough times to do it right, and other Australian web devs might have the same underlying issue.
3. Many, many pull requests this week, contributing to Fat Free CRM, Chef recipes and even my much unloved fork of tipsy.
4. All of Adelaide, most of Melbourne, Tasmania, Brisbane and Sydney have had a lot of attention from keepright.ipax.at - Adelaide has the cleanest map data of the lot, but Melbourne isn't far behind.
   My main focus: unclosed ways, redaction errors, sport tags without a physical associated tag - mostly things that won't show up in the rendering.
5. Took the opportunity to work from home today, so spent my lunch break constructing gravel path.

Consuming SOAP services with Ruby / Rails

Savon is a ruby soap library, and I mostly like it. I've been using the latest Savon to do integrations with SSRS at work.

Getting started
As practical advice, I strongly recommend you go and find a decent pre-built SOAP IDE. I've used XMLSpy and Oxygen before, which have a slightly wider focus than just pure SOAP; but in recent times I've come to like SoapUI quite a lot.

Compared to other tools, it will:

• Take a WSDL
• Generate a sample request for each action
• Generate code for you in you happen to be writing in Java - and stub services
• Allow you to easily set different endpoints (ie: prod, staging, dev) without stuffing around with WSDLs.
• Allow you to develop a number of premade requests which will return expected responses.

In short, it's a heck of a tool with a low price point: free.

SSRS

If you aren't developing in C# or similar, MSDN isn't very helpful. Not having had to... enjoy... windows servers for some time, I found a lot of time was spent:

• Working out if I was authenticating correctly (browser to web UI, browser to WSDL, SOAP UI to WSDL, SOAP UI to SOAP Action, Actual Code to Soap Action)
• Blundering around looking for configuration files in a file manager, instead of using google to find a likely file path and using vim until success was achieved.
• Finding all kinds of Authentication that should work, simply don't. In the end, if the SOAP MSDN articles had simply pointed me to Configure Basic Authentication on the Report Server, I think I have achieved more.
• Discovering the docs were incorrect re what WSDLs were actually available vs what the server published, and what worked vs the configuration of 'Native Mode'.

Once you get over the successive hurdles of Auth, identifying the correct service, and guessing that ListChildren is the method you want to get a list of reports, it's fairly straight forward.

Gotchas with Savon
A few of the soap calls wanted specific soap headers. Savon only supports global soap headers, and once you've built the object it's hard to mutate it.

Tips:
1) To get more control, ditch the Savon::client() factory method - it's not really doing too much you can't do yourself.
https://github.com/savonrb/savon/blob/master/lib/savon.rb

2) If you need to monkey with the global options, you can often extend your client to make them accessible.
https://github.com/savonrb/savon/blob/master/lib/savon/client.rb

class ExtendedSoapClient < Savon::Client
   attr_writable :globals
end

3) It's worth stealing from Javaland in most cases - for each configured Savon Client, do up a quick FooServiceProxy. Inject your client into the constructor, and for all of the management of WSDLs, go stick that in your environment config.
If needed, go build yourself a SSRSServiceFactory to handle all of this wiring up.

The primary advantage - you get a reasonably clean API that can return either hashes, or if you go a little further you can generate models for the SOAP responses and have those appropriately populated.

It's important to remember these proxy classes exist only to translate a SOAP request from your code, to the transport layer, and to return a simple, meaningful result to your code.

That means avoiding a lot of logic in favour of making an API explicit.

Activerelation, stop being so clever.

I nearly flipped by lid today, when doing what should have been a simple activerelation query to get the  'last record'.

Foo.order('column')
SELECT * FROM foo ORDER BY column ASC;


Foo.order('column ASC').last
SELECT TOP 1 * FROM foo ORDER BY column DESC;



Foo.order('column DESC').last
SELECT TOP 1 * FROM foo ORDER BY column ASC;


Foo.order('column DESC').first
SELECT TOP 1 * FROM foo ORDER BY column DESC;


Foo.order('column ASC').last
SELECT TOP 1 * FROM foo ORDER BY column ASC;

Foo.order('column ASC').limit(1)
SELECT TOP 1 * FROM foo ORDER BY column ASC;

Foo.order('column DESC').limit(1)
SELECT TOP 1 * FROM foo ORDER BY column DESC;

Something tells me I'll never like SQL server's TOP 1 syntax, as I spent ages just staring at the end of the queries and wondering "why the heck isn't it choosing the last record, I told it to sort DESC!"

I miss my LIMITs.

Flattening your models in JSON

You have a graph of objects, with some of those external lookup codes.

class ViciousAnimal < ActiveRecord::Base
  has_one :classification
end

# A simply code, description pair that captures the Smithson Nasty Bite Classification (WEAK_BITE, OW, WHERE_IS_MY_LEG)
class Classification < ActiveRecord::Base
end

When you are using something like knockout, quite often it's more useful for you to have application.classification.description present, but unless you do two serialisations to JSON, and build the relationship between the two JSON objects that's a bit annoying. In this scenario, I'll typically create an implentation like:

class ViciousAnimal < ActiveRecord::Base
  has_one :classification

  def classficiation_description
     classification.description
  end

  def to_json(options = {})
    super(options.merge({:methods => :classification_description}))
  end
end

This allows you to begin to flatten your object graph into something easier to work with on the UI.

What are the gotchas?

When you customise your to_json method, don't always expect rails to invoke it. For example, if you are passing back a number of objects, you'd typically do:

render :json => {
  :something => true,
  :vicious_animal => @vicious_animal
}

However, you'll be surprised to find none of your own implementation of to_json appears to be called. only whatever the Hash's to_json method supports.

To work around this, you often have to dumb your ActiveRecord instance down to a plain old hash:

render :json => {
  :something => true,
  :vicious_animal => ActiveSupport::JSON.decode(@vicious_animal.to_json)
}

Hardly the prettiest scenario.

How do you make that scale?

The above approach is only really good for a handful of descriptors - if you suddenly find half of your model is devoted to flattening out the graph, I would recommend extracting said code into a ViciousAnimalJSONSerializer, which knows how to orchestrate the mapping between the object graph and flattened view.

Rails provides ActiveModel, which gives you a good starting point for cleaner mapping code on plain objects. The benefit of getting yourself familiar with these sorts of mapping libraries is fairly high - for example, it would be fairly trivial to implement an n-triples mapper, to/from text serialization, or just about anything else you can imagine.

Ruby 1.8.*, UTF-8 and Oracle / Oracle Enhanced / OCI-8

... is a mess.

For future reference:

1. Check select value from nls_database_parameters where parameter = 'NLS_CHARACTERSET'; is AMERICAN_AMERICA.AL32UTF8 or similar
2. In your boot.rb, prior to any gems being loaded, add

   require 'rubygems'
   
   # Set up gems listed in the Gemfile.
   ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__)
   ENV['NLS_LANG'] = 'AMERICAN_AMERICA.AL32UTF8'
   
   require 'bundler/setup' if File.exists?(ENV['BUNDLE_GEMFILE'])
   

3. Curse repeatedly at Oracle's database wide character sets, rather than MySQL's table based character sets - if you are migrating, you have to migrate every schema being used.
4. Export your data, change the character set by ALTER DATABASE CHARACTER SET, re-import.

Things I miss in ruby

Type hints.
In php I could have type hints to recommend the interface I wanted to be injected into my method or ask for certain dependencies.

Sure, I can do it myself but that seems painful compared to what php gave me out of the box.
You don't have to be a jerk about it, but a little structure can go a long way.

Dependency injection.
For some reason many of the rails community read a post against DI frameworks, and decided it was ok not to do DI. Because if you duck type it a frame work is a philosophy.
The results speak for themselves. Clearly a level seven ruby developer wrote the ruby CAS rails plugin, which had no tests, is a super singleton, and does whatever the hell it wants with your controller. Define two alternative logout uris? Not on its watch!
I admit the design is from 2006 or earlier when not so many people were enlightened, but I am still stuck with it.

On a library designed by people who develop for computer literate universities.

Cut the cord people, stop doing magic and just make a simple utility class or three.

Defensive programming.
There seems to be a jerk who said not to program defensively.
Thanks, that guy. The result is execution of an app continues well pat the danger input point because of a general principle.
Think about it for a minute, with the observer pattern.
Some input happens, events trigger. Now there are fifteen event listeners and they all freak out half way through executing because off some failed expectation.
A simple exception to check the input often points directly at the problem, but in ruby land you just let execution continue in general.
Most dissuading about this, those that discuss it on the Web tend to agree that you should check input and bug out.
The point is not to fail silently. But rails land doesn't care. It is going to make poor code poorly while screaming look at my ruby FU.
If rails developers, by and large left the power offered by ruby well enough alone and coded for other humans, life would be far, far better.
KISS > DRY

Help Wanted: PHP / Rails developers in Adelaide

If you have skills in PHP 5, or have built something decent in ruby on rails, then we're looking for you.

In short...

• We have a Really Big Web Application which supports about 25% of the residential home loan valuations done in Australia
• We've got a lot of smaller projects on the drawing board where rails experience is a definate plus
• We're in Adelaide.

... and we need you. Interested? Email a resume and a few links to things you're proud of, and we'll see what we can do.

Bad rails

Can someone help this patch get a move on?

Escaping `table names` for mysql - so reserved sql keywords don't cause errors.

One Small Step For CloCkWeRX...

...One Giant Leap Backwards for productivity.

After fooling about I've decided Ruby isn't for me. Yet. It's fine for rapid application development, but the documentation doesn't coddle me enough. Also, what am I thinking to pump out a web application in a language I don't grok!

I've also discovered a game. No, not a game, the game. Morrowind is the third in the Eldar Scrolls something or rather. I'm hooked. My roommate shifted the TV & XBox right next to where I usually code.
I'm stuck.
Quite literally.
I nearly lost my job because of the irregularities introduced into my sleeping patterns! :S

Of late, my coding efforts have been on

• PointOfSale - XUL/JS/PHP 5/RDF


• Fresh FM's new site - Customising Drupal


• Fresh FM's intranet - Documentation galore


• Refactorying Fresh Framework - migrating the code for the work website to new machines introduced a lot of bugs I ignored before.


• Cursing Gallery 2 - Running direct from the CVS is no fun :(

Hopefully in the next few weeks I'll have more time to devote to PointOfSale, and indirectly PEAR::RDF as I track down bugs by the dozen. Stay tuned for some actual code related blogging!

What's in your fridge?

What's in your fridge?
Originally uploaded by CloCkWeRX.

Rubylicious. Sample interface... now to turn it into a working, clickable application with Rails.

I spent a while on this... I love the idea: it makes me hungry. Stay tuned, I've got to earn a bit more from the adwords before hosting costs can be covered :)

Rolling With Ruby On Rails

Rolling with Ruby On Rails was the first thing I ever did in Ruby.

Wow.

Wow.

Just... really, really... wow.

Ruby doesn't impress me too much. Rails... where I can tell what's not Ruby... is simply amazing. PHP has a lot to learn from this.

The gist of it? I was able to whip up something that made forms straight from the database. Hell, it made the entire application from the database. I define my fields in the database and it makes the forms.

ZOOSH! (Zoom, and Whoosh... but it's Ruby on Rails, so they only need to do it once).

Everything is facetted... it's PHP's Smarty built into the application framework. Database abstraction... built in. Forms manger... built in. Want a new facet? Override the old, and whammo!

I'm going to have a go at getfridged with Ruby.