Monday, May 13, 2013

Consuming GeoRSS and querying OpenStreetMap

In Adelaide, there was an unexpected fire. It was a bushfire, and I'd just been mapping the area earlier that day - ironically, tracing some of the forests which were being burnt out at that very point in time.

I decided to go and see what was available for fire alerts around the nation.

It turns out the majority of fire authorities publish GeoRSS. I took the feeds and aggregated them, plus put in a transformation of the data to JSON.

Using Leaflet, I added a rendering of particular fires.

Using the OpenStreetMap Overpass API, I queried for "all areas in a 500m bounding box", and again using Leaflet, rendered those polygons onto a map.

Here's the Project:
https://github.com/CloCkWeRX/burning-down

Here's a few screenshots:

Coverage, for most of Australia. WA doesn't provide openly licensed data, and South Australia's does not contain lat/long info.
The few you see in those states are from Sentinel, a satellite based system for detecting 'hot spots' that may indicate a fire.

Here, a number of resident buildings are marked as 'at risk' from a fire. In reality, these are probably safe, due to fires in these areas tending to be contained to residential areas.
On the other hand, you can certainly consider adding an Open Street Bug report if someone's house may have burned down!

This is more interesting - there's a fire or other emergency right next to a national park. National parks are made from trees, which in my experience tend to catch on a whole lot of fire!

Obviously this sort of incident is well in hand if the fire authorities are aware of it, but it is interesting to see that fuel sources are being picked up.

My next efforts here are going to be around encouraging edits near the area - if there isn't much detail on water sources (dams, swimming pools, rivers, lakes); it certainly makes sense to identify those if there's already been a fire once.

I also made a bit of effort to aggregate the feed output into sqlite - I might go run up a basic server and run this as a service at some later point, as aggregating this data seems useful.

The thing I like most about this - it's trivial to take any GeoRSS feed and query OpenStreetMap for nearby things - I've done fairly wide ranging queries with a narrow bounding box so far; but those are easy enough things to tweak.

Neat ad-hoc networking trick

I didn't realise it for a long time, but instead of relying on DNS or directly entering IP addresses, most machines in my home network actually self organise. Many of you will already know this, but for those that dont...

Most devices support a thing called Multicast DNS - apple devices also do this.

Here's the article:
http://en.wikipedia.org/wiki/.local

This is why things like Transmission, SickBeard, XBMC and SSH keys are useful: you can simply point to xbmc.local:8080 and get to a basic remote control.

This lets you simply refer to whatever the machine is and start talking to it. On a local wifi network, it's great.

Wish:
Ubuntu made this much clearer in some fashion - the first time you join a network, it should flash a welcome message. If it hears about a particular host, it should notify "xbmc.local is available" or similar, indicating a few machines it has detected.

Enhanced by Zemanta

Wednesday, May 08, 2013

Making code review on BitBucket suck less

I'm a big fan of code review, I feel that almost everything should be given a once over. I'm used to it in various work places, I'm especially used to it when working in an open source project.

BitBucket is a follower, not a leader, and has managed to get a lot of what makes Github work well added - inline comments are great.

BitBucket has lead in one area though - much like a 'merge pull request', on each change there's an Approve button.


If you are happy with a commit, it's a click away - you don't even have to say 'LGTM', if not, have a discussion.


This adds a handy tick in the commits list, letting you see what is accepted or what needs to be talked about.

There are two problems with this though:

  • Every time you hit the button, notifications are sent
  • The review is per commit, not per feature or branch. If you use a simple 'all in' model, it's hard to group a stream of work - later fixes are applied that invalidate your review, all in the same push.

The best answer I have for issue 1 is to create email rules. If it has to works "commit approved" from bitbucket.org, mark as read and archive it.
This dramatically changes the signal to noise ratio, but lets you still leverage the visual indications of review having taken place.

For the second, feature branches and pull requests are the next best bet - BitBucket allows you to do a pull request from a feature branch on your repository to master.

This lets review take place on sets of code and features; and works extremely well for integrating stable changes only.


Sunday, May 05, 2013

Called it!


http://clockwerx.blogspot.com.au/2012/10/mapping-rosewater-arg-and-much-more.html and http://clockwerx.blogspot.com.au/2012/05/things-i-want-to-build.html meets
Augmented Reality Games using open data.

Thursday, April 25, 2013

A quick guide to code review

In my opinion, it's good practice to have a strong culture of peer review - it ensures your team knows what is going on, catches simple issues, and makes you a better group of developers.

Bitbucket and Github both provide a quite intuitive interface for reviewing code. You can simply look at a list of recent commits, and find a particular line to add a comment.

If you are new to git, I'd recommend you find an existing project and try this out.
  1. Find the commits tab - https://github.com/jaz303/tipsy/commits/master for example
  2. You can see a number of commits, with references like 3545aa6ae2 - these are hashes that uniquely identify the change.
    It shows the commit message, author, as well as date/time.
  3. Viewing one brings up the review user interface.

Code review is a fairly simple process. Different teams do different things, but I generally do:
  • Does the code change meet the description?
  • Is the code self describing / self explanatory?
  • Are there tests?
If you have questions, or don't understand something - ask. It may not be that you don't understand, it may be an oversight by the original developer.

Give it a go on an open source project today!

Wednesday, April 24, 2013

A first time using Git, with Bitbucket - add existing code.

If you are new to Git, it can be a bit much to get your head around. The content below assumes you are making use of http://bitbucket.org/ - a good alternative for small commercial teams, which is roughly as good as Github in a number of areas.

Git (Windows)
First, let's get a git client. If you are on windows, I recommend one called msysgit. It gives you access to a start menu item, known as Git Bash - this will look and feel a lot like a Linux terminal.

  1. http://code.google.com/p/msysgit/downloads/list 
  2. Grab Git-1.8.1.2-preview20130201.exe, the "Official installer for windows"
  3. Install, next, next, next, leave the line endings settings to whatever it defaulted to, next some more, SUCCESS!
Bitbucket

Much of the bitbucket UI for signup, create repository, etc is fairly self explanatory. Let's go ahead and create a repository.

When you've done that successfully, you'll be confronted with a screen:

  • I am starting from scratch
  • I have code to import

Selecting "I have code to import" will give you a number of git commands as examples - to initialize a repository, to add your content, and to add a remote.

There are two main protocols that git can make use of - https and ssh. The https URLs look like

https://you@bitbucket.org/you/repository_name.git

... and for the moment, are the simplest to understand. If you already know about SSH and SSH keys, use the other protocol.

When you add a remote, you give it a name - in this case, origin. This is just a shortcut for the bitbucket server - you can choose any name you like; however there is a widespread convention to use origin with the main repository.

For windows users,

  • Start
  • Git Bash - this is much like the windows command line (cmd), with a few different commands.
  • Find your existing code - cd /c/path/on/your/hard/disk - note the direction of the slashes!
In Git Bash, instead of seeing a prompt like c:\>, the prompt ends in a $. 

To get to your c:\ drive
$ cd /c/

To get to another path
$ cd /c/path/to/location

If you aren't sure, you can hit the TAB key repeatedly to autocomplete your filenames or show you a list of suggestions.

Once you have located the right directory with your existing code, we want to initialize a repository

$ git init


Next, add a remote called origin pointing at bitbucket.
$ git remote add origin https://you@bitbucket.org/you/repository_name.git

Tell Git to monitor the files - this is like an svn add.
$ git add .

To see the affect of git add, do
$ git status
# On branch master
#
# Initial commit
#
# Changes to be committed:
#   (use "git rm --cached ..." to unstage)
#
#       new file:   A.xml
#       new file:   B.java
#       new file:   C.wsdl
#       new file:   D.wsdl
#
you@yourcomputer ~/path/to/local/code (master)


You can see the code is now being tracked. It wants you to commit.

So, perform your first commit:
$ git commit -m "Importing my code" -a

[master (root-commit) 0bba05d] Importing my code
 4 files changed, 127 insertions(+)


All done, right? Not quite. You've made a commit, but this is only local to your repository. Think of it as the save command in a word document - it doesn't publish anything, but it lets you undo/redo changes.
From here you can do as many changes and as many commits as you want.

$ git commit -m "fix typo"
$ git commit -m "remove swearwords"

Also, you probably saw some warnings like:

*** Please tell me who you are.

Run
  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

It's worth doing this, as each commit is attributed to a user - you'll only have to do this once. Don't worry too much: your commit will still have succeeded.


Anyway, you can check the status again with
$ git status
# On branch master
nothing to commit, working directory clean

It should tell you if there are any untracked files (git add them), or uncommmited files (git commit them) - but ideally, it should look like the above.


To publish your changes, you need to push them. For simplicity, we'll also set a default destination (via -u) - the origin remote repository; master branch.

$ git push -u origin master
Password for 'https://you@bitbucket.org':
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 3.81 KiB, done.
Total 6 (delta 1), reused 0 (delta 0)
remote: bb/acl: you is allowed. accepted payload.
To https://you@bitbucket.org/you/repository_name.git
 * [new branch]      master -> master
Branch master set up to track remote branch master from origin.
you@yourcomputer ~/path/to/code (master)


... and check bitbucket.org - your code should be published!


Great - but what just happened? If you look at the diagram, we've done everything on the left hand side - have a local, have a remote, and pushed changes. It's now possible for someone else to clone your repo, or fork it, and pull changes.

You can see that there can be many repositories - sometimes teams will maintain a 'core' repo, and people on the outside take a 'fork'.




Saturday, April 06, 2013

New OpenStreetMap Editor available.

http://ideditor.com/ is pretty neat - it's a bit slower than potlatch, but try tracing the roof of a house.


Here's the full blog post. It certainly looks like the kind of editor you want for any kind of property data capture - imagine doing a kerbside valuation and pulling this up on an ipad to explore / add data.





Wednesday, March 27, 2013

Bugs and annoyances with the Adelaide Metro Card Site

1. You cannot mark a card as lost or consolidate balances. I am now on my third card.

2. Add one item to the cart, it redirects you to the list view. Add a duplicate due to a spotty connection - say... found on a bus in motion - it silently redirects you to the referring page but still adds the item.

3. The site will charge you for multiple things which are identical.

4. Deleting an item from the cart results in a 'temporarily unavailable' action, and does not remove content from your card.

5. It is so painful to complete the act of purchasing, that I am twenty minutes in to the process. A bus driver laughed when I suggested I could recharge my expired card and reswipe it.

It seems like a fairly obvious set of things to cover during testing. I cant understand how a simple cart + cc gateway can take so long (Nov 2012 to now) and be so buggy.

Thursday, March 21, 2013

Consuming SOAP services with Ruby / Rails

Savon is a ruby soap library, and I mostly like it. I've been using the latest Savon to do integrations with SSRS at work.

Getting started
As practical advice, I strongly recommend you go and find a decent pre-built SOAP IDE. I've used XMLSpy and Oxygen before, which have a slightly wider focus than just pure SOAP; but in recent times I've come to like SoapUI quite a lot.

Compared to other tools, it will:

  • Take a WSDL
  • Generate a sample request for each action
  • Generate code for you in you happen to be writing in Java - and stub services
  • Allow you to easily set different endpoints (ie: prod, staging, dev) without stuffing around with WSDLs.
  • Allow you to develop a number of premade requests which will return expected responses.
In short, it's a heck of a tool with a low price point: free.

SSRS
If you aren't developing in C# or similar, MSDN isn't very helpful. Not having had to... enjoy... windows servers for some time, I found a lot of time was spent:
  • Working out if I was authenticating correctly (browser to web UI, browser to WSDL, SOAP UI to WSDL, SOAP UI to SOAP Action, Actual Code to Soap Action)
  • Blundering around looking for configuration files in a file manager, instead of using google to find a likely file path and using vim until success was achieved.
  • Finding all kinds of Authentication that should work, simply don't. In the end, if the SOAP MSDN articles had simply pointed me to Configure Basic Authentication on the Report Server, I think I have achieved more.
  • Discovering the docs were incorrect re what WSDLs were actually available vs what the server published, and what worked vs the configuration of 'Native Mode'.

Once you get over the successive hurdles of Auth, identifying the correct service, and guessing that ListChildren is the method you want to get a list of reports, it's fairly straight forward.

Gotchas with Savon
A few of the soap calls wanted specific soap headers. Savon only supports global soap headers, and once you've built the object it's hard to mutate it.

Tips:
1) To get more control, ditch the Savon::client() factory method - it's not really doing too much you can't do yourself.
https://github.com/savonrb/savon/blob/master/lib/savon.rb

2) If you need to monkey with the global options, you can often extend your client to make them accessible.
https://github.com/savonrb/savon/blob/master/lib/savon/client.rb

class ExtendedSoapClient < Savon::Client
   attr_writable :globals
end

3) It's worth stealing from Javaland in most cases - for each configured Savon Client, do up a quick FooServiceProxy. Inject your client into the constructor, and for all of the management of WSDLs, go stick that in your environment config.
If needed, go build yourself a SSRSServiceFactory to handle all of this wiring up.

The primary advantage - you get a reasonably clean API that can return either hashes, or if you go a little further you can generate models for the SOAP responses and have those appropriately populated.

It's important to remember these proxy classes exist only to translate a SOAP request from your code, to the transport layer, and to return a simple, meaningful result to your code.

That means avoiding a lot of logic in favour of making an API explicit.


My recommended pathway into Ruby / Rails


I was asked what I'd encourage a complete outsider to RoR to look into.

Here's my list - what would you add?

- Linux intro
  - User accounts
  - File permissions
  - su / sudo
  - Installing programs (apt)
  - What's a distro? (Ubuntu, Debian, etc)
  - Gnome desktop
    - Alternatives
  - Bourne Again SHell (BASH)
    - cd, ls, touch, vim, top, kill, pgrep, pkill, mkdir, chmod, chown
  - Basic bash scripts
- Git intro
  - Cloning
  - Commits
  - Log
  - Push
  - Pull
  - Fetch, Merge
  - Branches
  - Tags
  - Revert
  - Reset
- Ruby intro
  - Hello world
  - Functions, Control flow, Loops
  - Classes
  - Modules
  - Blocks! Lambas! Currying!
    - Chunky Bacon and Foxes!
  - DSLs
  - Libraries (gems)
  - Require / include path
  - Metaprogramming
  - OOP Intro
  - OOP: SOLID principles
  - OOP: TDD
- HTML intro
  - HTML5
    - what is it?
    - differences to HTML4/xhtml?
    - Data attributes
    - New inputs
  - Javascript basics
    - JQuery
    - AJAX
    - Backbone, Ember, etc
    - D3
  - CSS basics
    - CSS3
    - Responsive design
  - SVG basics
- JSON! YAML! Data structures & serializations.
- Rails intro
  - Generators
  - Rake
  - Organisation
  - Scaffolds
  - Convention, not configuration
  - Rspec
- Sinatra intro
- SASS / Less
  - What is it?
  - Bootstrap/Compass
- Continuous Integration
  - Jenkins
  - Travis CI
  - Post commit hooks
  - Rake tasks
- DevOps!
  - Chef
  - Puppet