nearby.org.uk blog

… a website for more traffic that is. This is something a little off-topic perhaps for this blog, but it might be of interest to a few so will document a few tricks have learnt, in tweaking Geograph to cope with more traffic as it’s daily visitors and hits continue to climb. If you are not familiar with Geograph, or not a System-Admin (or budding - like me!), then probably can stop reading now!

First a little background, Geographs code started very humble, and coded to work off a single server, later with OS sponsorship we upgraded to multiple servers to cope with increasing traffic. This was done with a single larger server for Database and photo storage, and then multiple commodity webservers (with a front end load-balancer) More. This worked well for a number of months, but simply the DB/NAS server couldn’t cope with the increasing DB load, and bandwidth for serving all hundred of thousands of photos.

• Split the database, a small quick win is/was split off php sessions and gazetteer queries to a second Database. Sessions of course have lots of writes, so where tending to saturate the main db, this perhaps reduced its load by a 3rd!
• Cache Images on the separate servers. The servers aren’t big enough to house a copy of the full archive, but thumbnails are certainly more manageable. Seeing as thumbnails actually account for about 60-70% of the raw hits to the site, this is a potential win, as previously each server would have to seperatelly fetch individual images off the NAS. We use Apache as the webserver, so could easily create another simple VirtualHost to serve thumbnails, a empty DocumentRoot save images, with a simple 404 handler to fetch and store images not ready copied. This greatly reduces load on the NAS as its not having 3x servers fetching random thumbnails. (this also paves the way to move away from full-blown Apache simply for static content)
• Cache stuff in Memory - with Memcache. Related to the above point, quite a bit of load is actually random disk IO to determine image sizes as this requires reading the jpeg data. Caching this all in memory is good. Memcache can easily distribute it cache across multiple machines, so even losing a server means only part of the cache needs rebuilding. We also use ADODB as a database abstraction layer, with the latest version it has support to use Memcache for its caching, great! Last up is to do lots of application level caching on key places. Of course sessions and also the templating system (smarty) could benefit from memcache, but one step at a time!
• Optimise the HTTP headers. There are lots of tweaks and stuff here that can be done to lower the bandwidth and improve external cacheability of objects. This post is getting quite long so I think that might be a separate post…
• Optimize the slow queries. And last of not least, learn to love going though the log of slow database queries, and really stepping on the slow ones. This of course is an ongoing project. I found a script to summarize these, but it seems that mysql 5 at least comes with its own equally good one!

This list isn’t exhaustive, and of course is an on going project, always more can be done…

Over the weekend I attended OpenStreetMap’s first conference “State of the Map“, all in all a very enjoyable time, great to to listen to all the talks, and also chat with various mappers, meet up with various people I’ve only met before in cyberspace.

Hopefully it will inspire me to actually contribute, esp as frequent two ‘holes’ in the current data…

An interesting little snippet from Ed Parsons talk, is this slide, which shows KML/GeoRSS publishing as indexed by Google, somehow I think I reconsise the British Isles hotspot; geograph, which publishes many KML feeds, (about 600k (the Superlayer, and also a file per photo), of which about 300k are reported to be indexed in Google’s main index, so show up well in ‘User Generated Content‘ in Google Maps!)

Everyone (nearly) - me third from left

Andrew over at highearthorbit.com recently put together a proposal for a Geo Extension to Opensearch.

Naturally I couldn’t not add it to the Geograph OpenSearch definition, will have to keep an eye on the logs to see if there is any take up…

This also reminds me that nearby.org.uk is in need of some loving, there are some datasets locked away within that could be made searchable (of course geographically!) in this way.
One thing that struck me reading the proposal, is that a method to specify geographical extents that the engine works with. Thereby a sane search initiator could know that searching Geograph for locations in Austrialia is likely to end in tears….

Something like a simple:

<opensearchgeo:extents>
<georss:box>49,- 9.5,62,2.3</georss:box>
</opensearchgeo:extents>

Would probably do it

Update: Sorry, just relised the tags didnt show up, seems TinyMce inside wordpress, decodes entities on its own, so double(!) encoding it is then.

Wow, Google have again gone and introduced an almost draw-dropping new feature to Google Maps - Mapplets. Put simply they allow you to create mini-map-mashups, but as they run on maps.google.com, visitors can view your content along side the normal content of Google Maps - but even more interestingly visitors can load more than one Mapplet at a time, combining the data from each mashup onto ONE map!!!

And it turns out they are really easy to code for been based on Google Gadgets (which I haven’t used before) and Google Maps API (which I have :)), but I also congratulate Google on including in the package ‘Developer’ mapplets that make developing relatively painless.

Anyway onto the demos, just visit http://maps.google.com/preview (it’s a Developer orientated demo for the moment), click the ‘Mapplets’ tab, goto ‘Add Content’ and see what takes your fancy. To try out my (very humble!) demo, use the ‘Add by URL’ and paste in the following URL: http://www.geograph.org.uk/stuff/gmapplet0.xml

(more about maplets in the post linked in the first paragraph!)

All hail the new GGeoXml class, just introduced into the Google Maps API! Now you can easily (usually 1 or 2 lines!) add content that you have in KML/Z or GeoRSS to a Google Map in your API page.

This is the same functionality that has been on the main Maps site (well the GeoRSS support is new!) for a while: nice for this to migrate to the API.

TIP: to try it don’t forget to use &v=2.x in your api loader to get the latest version that includes the GGeoXml.

To celebrate here is a Geograph Demo - following some testing will make it an official feature.

Following the recent introduction of the Superlayer to explore Geograph Photographs in Google Earth, have updated so that transitions between ‘levels of detail’ are much nicer, as well as sporting new custom icons!

If you have previously downloaded it then you should get the update automatically - otherwise right click the superlayer in My Places and select ‘refresh’ to fetch the latest version. It seems sometimes GE doesn’t reload the styles - so if all you see is a mass of yellow push-pins, restart GE and it should then return to nice circles and blue camera icons.

Or just download it here:

Geograph Superlayer (Google Earth Version 4+ REQUIRED)

If you have Google Earth version 4 and broadband then you might be interested in clicking this link:

· Geograph SuperLayer ·

By opening the above link you should be able to view the whole Geograph image collection directly in Google Earth. This has been possible in a number of ways for a while, but this does so in a much more compelling way (i.e. a single small download!).

This exploits the new Region functionality of GE4 to only fetch and load detail as you zoom in, in this way the view starts depicting a course overview of the current coverage, zooming in reveals more detail until pictures themselves become visible. (the ‘Super’ is adopted from Google’s term of SuperOverlays - overlays that load via nested regions)

In particular this is all pre-compiled and compressed, so should be fairly quick to download and is friendly as possible on bandwidth. Overall the layer consists of 102,965 files at 450MB compressed so there is a lot of data available, however you will download a lot less than this, probably on the region of a couple hundred files totalling about 2MB in a typical session.

As a small refinement, when a icon represents a single photo (a camera icon) if a view direction is stored the icon will have the top point in the appropriate direction. Also once the thumbnail is visible you can double click the thumb and the View will rotate to orientate the photo correctly!

Also this is only the first version, hot off the presses, so it not perfect yet, in particular would like to use Geograph specific icons (any Graphic Designers reading???), for example to give more feedback to the user on squares with many images, and view direction - also GE sometimes gets confused at the amount of detail available - plan to experiment to see if something can do about that.

… Hopefully users of GE3 should degrade nicely to use the View-based refresh method of viewing the dataset.

Oh and yes this is the little kml project I mentioned, see also other ways of loading geograph data into Google Earth or Google Maps via KML files.

For a little project working on for Geograph (btw Second Birthday today!), that will end up producing quite complex (and repetitive) KML, I decided to take the plunge and create my own abstraction class.

There are already a few of these about, but I wanted something with just enough abstraction so don’t need xml in the php code, but without too much complexity, or a whole new syntax to learn. So the final output is basically a generic XML creation system, but tailored to KML generation, with a number of convenience methods for common bits of KML, (like outputting the right headers). This don’t do any schema enforcement, and assumes a good working knowledge of kml.

Anyway as the code is already GPL, thought would mention it here in case it useful to someone else.

Get the base class here (from the Geograph SVN repository)

There is also another file here, which extends basic the functionality, this is more specific to Geograph, but could still be useful.

And a few demo’s/tests used during development:

Demo1 - (view running demo - by default displays the KML, option to open in GE)
Demo2 - (view running demo)

And for a real world use of the code (been in active use for a week now)

Eagle eyed viewers might notice it claims kml2.0 but then goes on to allow use of 2.1 features. Yes I know this is ‘bad’ but is with reason. I have found to my expense that using 2.1 as the version actually has a few undesirable effects, eg making the Document visible in My Places tree, can’t mix and match styleUrl and style (to have common style but with a unique icon), and a few other subtle things. Whereas GE seems to allow the new features in a 2.0 kml file, I guess this is the GE equivalent to quirks and strict mode. (disclaimer: it’s a while since I tested this, so might be old news, I really should run some real world tests to confirm this is still the case)

Sometime last year I created a visualization of Geograph coverage as a series of 3D columns in Google Earth, well as that is now out of date, time to create updated versions:

: Geograph Points Per Hectad (*)
and Images Submitted Per Hectad (each around 200kb)

These are spectacular in themselves, but if you then create a series of these each timestamped with the dates of the images included, you can get an interactive animation…

So here is a version that downloads a monthly bargraph, since the project began, allowing full animation using the timeslider. Due to the size will only give the Animated Images Version which is the best anyway.

(due to the way of loading the data with nested network links will take time and patience with the timeslider to load them, all it seems to only really be possible by manually dragging the time-slider slowly across the timespan. )

Just need someone to create a fly though preview and upload to YouTube

(* Hectad is a Geograph term for a 10km by 10km square)

View at Geograph: SH7840 : Stone on Carnedd Iago
Another in the mini-series of Panoramic Images loaded into Google Earth, using DigitalUrban’s Panoramic Viewer tutorial.

This one is higher resolution than some of the others and ways in at 525kb,
but the extra resolution is useful to see the distant mountain ranges.

Anyway download it here: Google Earth Version