This blog post has been a long time coming. Can't remember how long I've been promising to write it...
Now that libraryhack has started up and the wheels at mixandmash.org.nz are starting to spin for 2011, I guess it's about time I finished this off.
Enjoy. (or jump directly to the Cliff's Notes)
Our heroes set out on a journey
During November 2010, I worked with Joshua Smyth on a mashup entry for The Great NZ Remix and Mashup Competition (2010), a wicked competition run by DigitalNZ and supported by many partners and sponsors.
With $30,000 in total prizes and $10,000 for the winning mashup, we decided to turn a few ideas we'd been discussing into reality.
About a month before the competition was announced, Trade Me had just launched their API, and we'd spent a few beer-filled evenings just brainstorming ideas: tracking stolen guitars through serial numbers posted in auction descriptions/comments, analysing bidding trends for similar items over time and generating "buy/sell" indicators to maximise margin, and other ideas that were so implausible I can't even remember them.
One idea we settled on early is how cool it would be if we could walk around our suburb with a GPS-enabled iPhone or Android device and find out where open homes / rentals within walking distance were.
I'd also recently attended some inspiring geospatial data discussions at the GSoC Mentor Summit which was causing me to think how we could bring mapping to Trade Me and open government data. Listings with addresses (ie. Real Estate) were prime candidates.
M to the A to the Shing Up
So real estate it was. The content sources suggestions page at mixandmash.org.nz introduced us to the site that truly helped to turn our idea from "trademe map frontend" into "house-seeker mashup": koordinates.com
I can't praise koordinates enough. They rock. They rock more than David Bowie's sock.
They host a large number of public domain (as well as some commerical/pay) map layers that you can download and reuse. Most of these map layers come directly from NZ government or local government bodies like Environment Waikato.
They also have an API which allows direct vector or raster queries on map layers. This is what we used to mash school zone data in with the rental listings we were mapping from Trade Me , and for one piece of the electoral data we included.
The school zone (and school decile rating data) was an early suggestion from Stuart Lewis (who helped heaps with ideas and feedback but would always be too humble to say so) that we knew would be of great interest to families looking for houses to rent/buy. That was our first "open data" angle.
We had a plan... now we just needed to spend every night and weekend designing, implementing and perfecting it. Which we did. After we both each tinkered with Trade Me's API, Google geocoding/maps in our own way (I'm an open source / Java freak, Josh is a .NET kinda guy), we designed a PostGIS database, and then split up the rest of the tasks so that I was responsible for web services and the website, Josh was responsible for the Android app and helping me figure out arc tangents when I tried to get each listing's Streetview camera centered on the house.
Many cigarettes and bottles of Macs Gold later, we had a working product. People other than just my Mum were saying it was cool, so I was pretty happy.
The school zone additions were working really nicely, and we'd enhanced them by looking up decile ratings for each school listed. We'd also combined a list of current MPs with an electoral boundaries map to offer some data about representation in the area. OK, not as useful to a family looking for a good area to move into, but we hoped it helped emphasise that there's a lot of location-specific data not already offered on Trade Me that we could access.
We come close to being eaten by a grue
The original plan was to cover rentals, residentials, and open homes. Our prototype started with rentals only, since it was a smaller data set (and therefore easier to grab and work with) than residential sales, and the addresses and prices were generally more precise than sales listings.
At this stage, the Trade Me API did not give pre-geocoded coordinates with its search results (they do now), so we were manually geocoding every single new listing (there tend to be 17,000 - 20,000 rental listings at any given time) that came our way, using the Google Geocoding API. The rate-limit on these meant that we couldn't go from 0 - 20,000 listings in a single day... we could keep up-to-date with the rate of new Trade Me listings, but populating our initial database was difficult.
In other words, if I made a mistake storing or resolving addresses to latitude and longitude (not an exact science -- have a look at the kind of stuff that makes it into the address field of a Trade Me listing), it would be a long slog to get back up to date. I made a mistake. Twice. At this stage, it was clear that if we wanted "rooftop"-level confidence for most of the houses we were mapping, we'd need to keep focusing just on rentals and ignore sales, or we'd never catch up. Open homes were a small subset of sales, so we still got that going from the Android app.
We are eaten by a grue
That obstacle over, we suffered our next blow. Although it didn't impact our competition entry per se, it meant we wouldn't be the first people to launch a GPS-enabled houses-for-sale/rent smartphone app in New Zealand. Realestate.co.nz , with funding from Westpac , hired an excellent team of developers (who's name I have embarrassingly forgotten. Barnacle Barnes told me, and I forgot) and launched a slick-looking iPhone app to do precisely what we'd been working on 2 weeks before we were due to submit our entry to Mix & Mash.
Looking back, maybe it wasn't as bad as it felt then... it was iPhone only, we were doing Android. It used private data from realestate.co.nz, we were using open data from Trade Me. We knew we weren't ever going to achieve the slick look & feel of the realestate.co.nz app, but we were a bit gutted that we weren't the first to the finish line.
Triumphant music plays as our heroes emerge bruised and bloody, but victorious
We got our submission in, entering the "Best geo application for mobile" and "Best search experience" categories, with over 20 hours to spare. This was after spending all night trying to record a demo video in which I didn't swear or start mumbling to myself, and also after we convinced Jawyei to make a logo for us with promises of fame and fortune. We were finished!
Over the next 7 days, all the entries were judged. Now that we weren't allowed to work on our projects, all the people who'd entered the competition started to come out of the woodwork. There were some seriously cool mashups out there, and some cool mapping stuff that I hadn't thought to do (mostly non-Google stuff).
The diversity of entries was also better than I'd expected.
The day finally arrived when results would be announced. Live. Over Twitter. This was the first time most of the entries had been made public, so as well as being nervous, I was also having heaps of fun checking out all the other mashups.
Best geo application went to...... Robert Coup's* Yachter Mobile . Hard to be too upset about that, it looked bloody awesome.
Best search experience went to...... Kim Shepherd and Joshua Smyth for Home Sweet Home. Oh well, better luck next ye- WAIT A MINUTE THAT'S ME IUSFPHFIUWEYFIUWHFIUWHFF
(office mates will confirm that I maintained a stoic and almost ho-hum composure after this, but it wasn't too long until I made it clear that I was super-fucking-stoked)
You can see the full list of winners here. We didn't get the $5000 or $10000 "big prizes", but quite frankly, we didn't deserve to. Daniel Pietzsch's NZ Walks Info is a perfect example of intuitive interface design, nice-looking map textures and markers/clusters, and the interactive elevation profile / trail position was the icing on the cake. Check it out.
Our official prize was $2000, which was divvied up and helped reimburse Josh for the test Android device he'd had to buy, and has helped me cover hosting costs to keep the site running and updated.
That wasn't all, though: discovery of all these excellent NZ open government data sources and awareness of what can be done with them has to be the grand prize that every entrant took away with them. As a library system/repository developer, I'm fully behind the open access research movement , and open data is just another chapter in the same book. If the sponsors' aims included promoting reuse of open government data (and I'm pretty sure they did), they definitely succeeded.
Home Sweet Home will keep running**, and I have more plans for it. I also have some new ideas for the 2011 competition that I'll keep up my sleeve for now...
Cliffs Notes:
* I've only just realised this is the same 'rcoup' that was driving a lot of the development effort for eq.org.nz , a very important project that I wished I'd gotten more involved in.
** As Murphy would have it, it's currently a few days out of date due to some API problems. Nearly fixed, though.
Now that libraryhack has started up and the wheels at mixandmash.org.nz are starting to spin for 2011, I guess it's about time I finished this off.
Enjoy. (or jump directly to the Cliff's Notes)
Our heroes set out on a journey
During November 2010, I worked with Joshua Smyth on a mashup entry for The Great NZ Remix and Mashup Competition (2010), a wicked competition run by DigitalNZ and supported by many partners and sponsors.
With $30,000 in total prizes and $10,000 for the winning mashup, we decided to turn a few ideas we'd been discussing into reality.
About a month before the competition was announced, Trade Me had just launched their API, and we'd spent a few beer-filled evenings just brainstorming ideas: tracking stolen guitars through serial numbers posted in auction descriptions/comments, analysing bidding trends for similar items over time and generating "buy/sell" indicators to maximise margin, and other ideas that were so implausible I can't even remember them.
One idea we settled on early is how cool it would be if we could walk around our suburb with a GPS-enabled iPhone or Android device and find out where open homes / rentals within walking distance were.
I'd also recently attended some inspiring geospatial data discussions at the GSoC Mentor Summit which was causing me to think how we could bring mapping to Trade Me and open government data. Listings with addresses (ie. Real Estate) were prime candidates.
M to the A to the Shing Up
So real estate it was. The content sources suggestions page at mixandmash.org.nz introduced us to the site that truly helped to turn our idea from "trademe map frontend" into "house-seeker mashup": koordinates.com
I can't praise koordinates enough. They rock. They rock more than David Bowie's sock.
They host a large number of public domain (as well as some commerical/pay) map layers that you can download and reuse. Most of these map layers come directly from NZ government or local government bodies like Environment Waikato.
They also have an API which allows direct vector or raster queries on map layers. This is what we used to mash school zone data in with the rental listings we were mapping from Trade Me , and for one piece of the electoral data we included.
The school zone (and school decile rating data) was an early suggestion from Stuart Lewis (who helped heaps with ideas and feedback but would always be too humble to say so) that we knew would be of great interest to families looking for houses to rent/buy. That was our first "open data" angle.
We had a plan... now we just needed to spend every night and weekend designing, implementing and perfecting it. Which we did. After we both each tinkered with Trade Me's API, Google geocoding/maps in our own way (I'm an open source / Java freak, Josh is a .NET kinda guy), we designed a PostGIS database, and then split up the rest of the tasks so that I was responsible for web services and the website, Josh was responsible for the Android app and helping me figure out arc tangents when I tried to get each listing's Streetview camera centered on the house.
Many cigarettes and bottles of Macs Gold later, we had a working product. People other than just my Mum were saying it was cool, so I was pretty happy.
The school zone additions were working really nicely, and we'd enhanced them by looking up decile ratings for each school listed. We'd also combined a list of current MPs with an electoral boundaries map to offer some data about representation in the area. OK, not as useful to a family looking for a good area to move into, but we hoped it helped emphasise that there's a lot of location-specific data not already offered on Trade Me that we could access.
 |
| Example of a Home Sweet Home listing view |
We come close to being eaten by a grue
The original plan was to cover rentals, residentials, and open homes. Our prototype started with rentals only, since it was a smaller data set (and therefore easier to grab and work with) than residential sales, and the addresses and prices were generally more precise than sales listings.
At this stage, the Trade Me API did not give pre-geocoded coordinates with its search results (they do now), so we were manually geocoding every single new listing (there tend to be 17,000 - 20,000 rental listings at any given time) that came our way, using the Google Geocoding API. The rate-limit on these meant that we couldn't go from 0 - 20,000 listings in a single day... we could keep up-to-date with the rate of new Trade Me listings, but populating our initial database was difficult.
In other words, if I made a mistake storing or resolving addresses to latitude and longitude (not an exact science -- have a look at the kind of stuff that makes it into the address field of a Trade Me listing), it would be a long slog to get back up to date. I made a mistake. Twice. At this stage, it was clear that if we wanted "rooftop"-level confidence for most of the houses we were mapping, we'd need to keep focusing just on rentals and ignore sales, or we'd never catch up. Open homes were a small subset of sales, so we still got that going from the Android app.
We are eaten by a grue
That obstacle over, we suffered our next blow. Although it didn't impact our competition entry per se, it meant we wouldn't be the first people to launch a GPS-enabled houses-for-sale/rent smartphone app in New Zealand. Realestate.co.nz , with funding from Westpac , hired an excellent team of developers (who's name I have embarrassingly forgotten. Barnacle Barnes told me, and I forgot) and launched a slick-looking iPhone app to do precisely what we'd been working on 2 weeks before we were due to submit our entry to Mix & Mash.
Looking back, maybe it wasn't as bad as it felt then... it was iPhone only, we were doing Android. It used private data from realestate.co.nz, we were using open data from Trade Me. We knew we weren't ever going to achieve the slick look & feel of the realestate.co.nz app, but we were a bit gutted that we weren't the first to the finish line.
Triumphant music plays as our heroes emerge bruised and bloody, but victorious
We got our submission in, entering the "Best geo application for mobile" and "Best search experience" categories, with over 20 hours to spare. This was after spending all night trying to record a demo video in which I didn't swear or start mumbling to myself, and also after we convinced Jawyei to make a logo for us with promises of fame and fortune. We were finished!
Over the next 7 days, all the entries were judged. Now that we weren't allowed to work on our projects, all the people who'd entered the competition started to come out of the woodwork. There were some seriously cool mashups out there, and some cool mapping stuff that I hadn't thought to do (mostly non-Google stuff).
The diversity of entries was also better than I'd expected.
The day finally arrived when results would be announced. Live. Over Twitter. This was the first time most of the entries had been made public, so as well as being nervous, I was also having heaps of fun checking out all the other mashups.
Best geo application went to...... Robert Coup's* Yachter Mobile . Hard to be too upset about that, it looked bloody awesome.
Best search experience went to...... Kim Shepherd and Joshua Smyth for Home Sweet Home. Oh well, better luck next ye- WAIT A MINUTE THAT'S ME IUSFPHFIUWEYFIUWHFIUWHFF
(office mates will confirm that I maintained a stoic and almost ho-hum composure after this, but it wasn't too long until I made it clear that I was super-fucking-stoked)
You can see the full list of winners here. We didn't get the $5000 or $10000 "big prizes", but quite frankly, we didn't deserve to. Daniel Pietzsch's NZ Walks Info is a perfect example of intuitive interface design, nice-looking map textures and markers/clusters, and the interactive elevation profile / trail position was the icing on the cake. Check it out.
Our official prize was $2000, which was divvied up and helped reimburse Josh for the test Android device he'd had to buy, and has helped me cover hosting costs to keep the site running and updated.
That wasn't all, though: discovery of all these excellent NZ open government data sources and awareness of what can be done with them has to be the grand prize that every entrant took away with them. As a library system/repository developer, I'm fully behind the open access research movement , and open data is just another chapter in the same book. If the sponsors' aims included promoting reuse of open government data (and I'm pretty sure they did), they definitely succeeded.
Home Sweet Home will keep running**, and I have more plans for it. I also have some new ideas for the 2011 competition that I'll keep up my sleeve for now...
Cliffs Notes:
- We entered this site and won a prize in The Great NZ Remix and Mashup Competition
- Many hilarious hurdles were met along the way (including maths and stuff)
- It was hard work and we only just finished it all in time, but was definitely worth it
- DigitalNZ, the sponsors, and the judges all did an awesome job at running the competition and promoting it (and promoting the causes it stood for)
* I've only just realised this is the same 'rcoup' that was driving a lot of the development effort for eq.org.nz , a very important project that I wished I'd gotten more involved in.
** As Murphy would have it, it's currently a few days out of date due to some API problems. Nearly fixed, though.
