Anyway, we are now hosted on CatN‘s vCluster platform.
CatN is by the same people who provide colocation hosting for Geograph, so the move isn’t actually far (both Geographically, and Virtually .

Separating Channel Islands site into a more standalone project, should hopefully also make the site easier to manage – and give more flexibility to grow.

(Next project is moving Geograph Portals to vCluster too, but I’m going to have do some more work on the code first)

If you are interested then let me know, plan to make an order for blank hard disks early next week.

I plan to offer these at £60 a piece – postage to the UK included. Depending on postage worldwide, might be slightly more, quote on request. Which is pretty close to cost.

The drives will be 640G, and include about 490Gb of jpgs. (about 2222222 files) The database and code is included, but that can also be downloaded from online. Probably these.

… With an added twist, will send it completely free (I’ll finance it myself) if you promise to make an application to view the images off the disk (ie works entirely offline). This would have be Open Source so it could be included on future disks, and downloadable for holders of disks already). More details of this on request.

It ‘freeloaded’ on the back of the Britain and Ireland project and shared the same servers. Alas that is no longer sustainable, and causing issues to the other projects hosted there.

So looking for someone to step up and help with the administrative and hosting. It needs new hosting.

I believe it could fit in the free trial offer from Amazon http://aws.amazon.com/free/ – at least initially (for a year).

Would you be willing to signup? And look after raising funds to pay for it. That could be though ads, donations or sponsorship etc.

It doesnt have to be though Amazon, if can get reasonable hosting elsewhere.

I’ll take care of getting the site running in the new location, and provide lo-fi support to keep it running (the techy stuff). Just can’t justify the administration.

(Contact me via Geograph)

Now we have always been more that happy to produce and give out bulk dumps on demand to anyone who asks, we provide a live API, and even bittorrent based downloads of the images themselves (although admittedly they are a bit behind schedule!).

So in an effort to make it even quicker to get going on new ideas, we have began to put together a self service bulk data download site. So from the following page you can access MySQL dumps of the whole image database, as well as supplementary data such as extracted terms and words – ideal for making search engines and mashups, of course includes the full geographical locational data of the images.

data.geograph.org.uk/dumps

The only caveat is the data doesn’t include enough info to build a link to the image file on our servers. We are doing this because we don’t want to make it easy for people to spider large quantities of images, however contact us and we will see what we can do… we are normally happy to allow hotlinking of thumbnail images as long as we know about it.

This is only the beginning so let us know about your suggestions!

OMG, quite aside from what it shows, it looks beautiful. (To my untrained eye!)

Click the image for a glorious 1px per km version.

© Copyright 2009 Geograph Project and licensed for reuse under this Creative Commons Licence. (if you want to do anything with it!)

Key: Age less than in years: (0 means no photos yet)

There is also a whole series of rendered maps from the project.

Geograph Google Earth Clusters

It’s the same download link, if you already have a copy, just right click and select Refresh on the “Geograph Google Earth Clusters” feature.

This has a number of advancements:

• Filterable – click the main title in the Places panel to open configuration options in the popup balloon. Includes options to filter by keyword or Geograph contributor. (Requires Google Earth version 5)
• Cubes/Cloud – Geocubes offers two clustering options, can how choose between then on the same popup as above
• Thumbnails – Now shows a image thumbnail in the placemark balloons.
• Shows a few coarse clusters when not zoomed on the British Isles
• Source code available! Download the GPL licenced PHP source code here

But its not all rosey – this version seems to suffer from an annoying bug, when first loads sometimes the layer ‘freezes’ and doesnt automatically update as you move around. If that happens right click on the ‘Clusters’ feature in ‘Places’ and select Refresh, doing that once seems to fix it!

Thanks to Geocubes to the amazing service that powers this – highly recommended if you have larger numbers of features to display!

This is the though the hard work of Hansjörg Lipp, fighting though the various land mines we have left in the open source code that powers geograph. As developers we (well mainly me – I am a lazy programmer!) have taken a number of shortcuts it just getting it to work the quickest way for us in the British Isles, so while the intention its should be easy to start another country it was never as easy as it should be.

You can see the site here:

http://geo.hlipp.de/

which will hopefully be translated fully to Germany. There is also a English language version here:

http://geo-en.hlipp.de/

which still connects to the same database

You can read about the experience of working with the code here

Its still early days for the project (it only covers a few provinces of southern Germany so far), but its really exciting to see a new site based on the Geograph concept (and code!), we wish Hansjörg Lipp and his team every success. (I wont embarrass myself trying to put any German in here!)

oh carp, oh carp, oh carp (or words to that effect)

… why do I say that? Well the thing is I know that Geograph runs pretty close to the limit* hosting wise, and this milestone, could, should and probably will, make a big splash, suddenly have mental images of Geograph exploding in a fireball, … not good!

(note, the rest of this post uses a few technical terms, and is rather long, turn away now if that scares you…)

Now back track a bit, a few days before, knowing this was coming up, starting making some preparations just in case the million hits it big, essentially a way could cut off non essential parts of the site to save a bit of processing.

But what I know really hits us is static file serving – we host, well, close to a million images (edit: over a million images!), each having full size version, and two thumbnails, and then hundreds of thousands of map tiles, all told probably close to 7.5 million unique URLs. So then it hit me, why not off load some of that, to well … Google!

Disclaimer: this is not a public recommendation to do the same – I have many gray hairs to prove it.

From playing with Google Gadgets know they offer a proxy to serve your content – so your server doesn’t die if your gadget makes it big, so I know about this proxy, and its actually pretty easy to use, so I thought I could – if the need arose divert requests via their proxy. G are generally pretty forgiving about such things, I and I just hoped they didnt notice the little blip)

So in the early hours of Wednesday morning I set to work hot-wiring a function into the code so we can divert:

1) the big images (huge bandwidth – and heavy for as they have to be served via the whole pipeline -we don’t edge cache them )
2) the thumbnails – not huge in themselves, but lots and lots of them
3) javascript files
4) map tiles (cached pretty well but still need to come from the NAS)

(noteably didnt do the CSS or ‘chrome’ images, mainly as didnt have a easy way to divert requests gloablly, but also images in CSS files are relative to the CSS file, so wouldnt play nice via the proxy)

Update: one point forgot to note, only diverted traffic for non-registered users, mainly so it doesn’t defeat the cache they already had.

And then a hard decision, I know moderation is a pretty intensive process – it happens in bursts, usually the first time images are viewed – so the cache is getting primed – and moderators often go off and look at maps, the full image page, and even make changes to the image metadata (helping out with typos etc).

…. so in went a cutout switch for moderation. In reality it wouldn’t matter that much if the moderation period was extended, for the short time traffic was high. (in hindsight should of warned moderators about this, but didn’t want to put a dampener on the celebrations – had to keep a happy public face )

I’ve told others about the milestone, so know that the word can be spread, and leaving the hamsters some authority to start cutting back if the need arises, so finally about 4am roll into bed, full of anticipation for when the word really gets out.

… now in the cold light of day I find the site is still there (yey!), and Paul good to his word has started telling people about it, so I nervously watching monitoring graphs, and hovering over various cutout buttons – what every happens we must stay up today…

So drawing the story to a close, with careful and continuous tweaking of the limits we where able to survive – and in fact pass, with flying colours; there was no hint of an actual outage – even if we did loose some parts (including moderation – but as mentioned it was a decision) – and we able to keep going though 160% visitor levels over a normal day. (and subjectively the site felt faster than it has for a long time)

In the end I think we ended up serving up about 87% of the hits ourselves (about 79% of the javascript files – they are reused between pages) – but critically we only served about 62% of the bandwidth. It may not sound like much of a saving but it allowed us to keep going though nearly double the number of raw pages served, 548,323 in total that day! So a big thank you to Google – even if they will never know.

…. but of course there where lots of lessons learnt, for example it brought home just how close we are to capacity. I think we have enough hardware capacity to cope, but seeing as the project is in our spare time, we don’t spend as much time tweaking and using that effectively as perhaps we could. Do however have a few plans to move forward#.

A fun bit of trivia, actually most new traffic didn’t come direct from boingboing (or digg, or reddit), but a japanese news site! (which actually showcased a range of images)

All told probably one of the most nerve-wracking experiences with Geograph, but I think we got though it relatively unscathed.

* We regularly run within about 10% of a hard limit, where machines start crashing and other fun stuff… (we even cross its sometimes, but with multiple machines its not often noticed from the outside)

# For a while have been playing with ningx which looks likely to be able to replace Apache for static hosting (which is dismal, I’ve put many tricks in place to mitigate – more in a separate post sometime) – and generally do things a lot better. Or maybe simply squid will help. (so much to do, so little time… )

 © Copyright Dr Richard Murray and licensed for reuse under this Creative Commons Licence.

Update, In the Press: Offical (login required) (RSS), Paul’s Blog, Boing Boing, Reddit, and Digg. 

This is post is made with rather less zeal than perhaps it even deserves, but it is 1.44 in the morning here, the tipping point rather caught me unawares so wasn’t prepared! We’ll have something more offical out soon…

There are a number of interesting artefacts in this, if you are quick you can follow people as the hike long distance paths, but you can also see the people making concerted efforts to knock of hectads.

Of course the raw frames are available under Creative Commons, and a bonus prize to who can make an good mashup of these. The animations include both weekly frames, and monthly versions (labelled quick)

(the fire reference)