nearby.org.uk blog

Recently I have been playing with YSlow quite a bit, mainly for Geograph. A useful feature is it gives the load time for every page (without doing a full scan) in the status bar and I happened to visit this blog and the results where quite surprising.

Geopress which I have installed for adding GeoRSS to rss feeds and creating kml files, (and geotagging the post itself, which is custom), actually includes it’s javascript on each an every page! So 6kb for the core, 34kb for mapstraction, and then because I had blindly entered a API key for Google and Yahoo, get both of their apis thrown in (about 150kb I think), and no maps for all that effort!
I use a Google map for geocoding the post in the first place, so don’t want to blindly disable the whole lot, so as a quick bodge, I just commented out the following line in geopress.php

add_action(’wp_head’, array(’GeoPress’, ‘wp_head’));

leaving the ‘admin_head’, so that it would get included in the admin , which is just me.

If I knew more I would digg in and make it a preference, or better yet just make it auto-detected. But posting this here in the hope it useful to someone.

… a website for more traffic that is. This is something a little off-topic perhaps for this blog, but it might be of interest to a few so will document a few tricks have learnt, in tweaking Geograph to cope with more traffic as it’s daily visitors and hits continue to climb. If you are not familiar with Geograph, or not a System-Admin (or budding - like me!), then probably can stop reading now!

First a little background, Geographs code started very humble, and coded to work off a single server, later with OS sponsorship we upgraded to multiple servers to cope with increasing traffic. This was done with a single larger server for Database and photo storage, and then multiple commodity webservers (with a front end load-balancer) More. This worked well for a number of months, but simply the DB/NAS server couldn’t cope with the increasing DB load, and bandwidth for serving all hundred of thousands of photos.

• Split the database, a small quick win is/was split off php sessions and gazetteer queries to a second Database. Sessions of course have lots of writes, so where tending to saturate the main db, this perhaps reduced its load by a 3rd!
• Cache Images on the separate servers. The servers aren’t big enough to house a copy of the full archive, but thumbnails are certainly more manageable. Seeing as thumbnails actually account for about 60-70% of the raw hits to the site, this is a potential win, as previously each server would have to seperatelly fetch individual images off the NAS. We use Apache as the webserver, so could easily create another simple VirtualHost to serve thumbnails, a empty DocumentRoot save images, with a simple 404 handler to fetch and store images not ready copied. This greatly reduces load on the NAS as its not having 3x servers fetching random thumbnails. (this also paves the way to move away from full-blown Apache simply for static content)
• Cache stuff in Memory - with Memcache. Related to the above point, quite a bit of load is actually random disk IO to determine image sizes as this requires reading the jpeg data. Caching this all in memory is good. Memcache can easily distribute it cache across multiple machines, so even losing a server means only part of the cache needs rebuilding. We also use ADODB as a database abstraction layer, with the latest version it has support to use Memcache for its caching, great! Last up is to do lots of application level caching on key places. Of course sessions and also the templating system (smarty) could benefit from memcache, but one step at a time!
• Optimise the HTTP headers. There are lots of tweaks and stuff here that can be done to lower the bandwidth and improve external cacheability of objects. This post is getting quite long so I think that might be a separate post…
• Optimize the slow queries. And last of not least, learn to love going though the log of slow database queries, and really stepping on the slow ones. This of course is an ongoing project. I found a script to summarize these, but it seems that mysql 5 at least comes with its own equally good one!

This list isn’t exhaustive, and of course is an on going project, always more can be done…

Ever since releasing the MGRS layer for Google Earth, I’ve been asked a number of times (well about 10) about tweaking the numbers on the gridlines, to be more ‘MGRS like’. To be honest havent had a clue what this means, but a recent email made me twig whats going on.

Basically the numbers included the hundreds of KM, but in fact that number is already represented in the Grid Letters so doesnt make sence to include again (it was there because the layer was an almost direct port of the UTM code, which of course needs it).

So the server hosted version has been updated, and also the file inside the zip for the offline version (which are in fact identical code!), and if really interested heres the diff.

Enjoy!

(Very) Simplistic tool for trying out a OpenSearch Description file :

http://www.nearby.org.uk/opensearch/

… includes a Gmap to make defining the lat/long and/or bbox easy. (if the file has geo extensions)

(Just submit the same file again, if you update it. btw don’t submit a definition file you don’t want to become public, but remember it is called ‘open’

I realised after mentioning we updated the description file on www.geograph.org.uk, that it wasn’t fully tested, (hate testing, but not adverse to spending 2 hours scripting a tester), and in fact this tool highlighted a mistake!

Hope it of interest, feedback welcome,

Thanks to Frank for letting us know about the new SDK for the 3DConnexion devices, including the SpaceNavigator, I’ve been waiting for this for a while as wanted to try it for websites like Google Maps. Even nicer is the SDK includes a Javascript demo - so makes developing for Google Maps Easy!

Enough already, I want to see the Demo!

WARNING: Internet Explorer - and you might have to enable ActiveX controls to run. Do so at your own risk! I won’t intentionly do anything bad, but can’t promise that 3DConnexion won’t (but doubtful they would either) - Recommend you add Nearby to your trusted sites and then allow controls to run in the trusted zone.

Or just download the file to your local drive. (right click the above link and select Save As) and run it from a local file - didn’t have to jump thought the hoops to get it to run there!

Todo: The zooming isn’t as slick as it could be (possibly) and would be nicer to have more control of the speed. Also the silly (depending on your point of view!) warnings from IE7, which I wonder if there is a better way of loading the objects to make it more compatible.

For a little project working on for Geograph (btw Second Birthday today!), that will end up producing quite complex (and repetitive) KML, I decided to take the plunge and create my own abstraction class.

There are already a few of these about, but I wanted something with just enough abstraction so don’t need xml in the php code, but without too much complexity, or a whole new syntax to learn. So the final output is basically a generic XML creation system, but tailored to KML generation, with a number of convenience methods for common bits of KML, (like outputting the right headers). This don’t do any schema enforcement, and assumes a good working knowledge of kml.

Anyway as the code is already GPL, thought would mention it here in case it useful to someone else.

Get the base class here (from the Geograph SVN repository)

There is also another file here, which extends basic the functionality, this is more specific to Geograph, but could still be useful.

And a few demo’s/tests used during development:

Demo1 - (view running demo - by default displays the KML, option to open in GE)
Demo2 - (view running demo)

And for a real world use of the code (been in active use for a week now)

Eagle eyed viewers might notice it claims kml2.0 but then goes on to allow use of 2.1 features. Yes I know this is ‘bad’ but is with reason. I have found to my expense that using 2.1 as the version actually has a few undesirable effects, eg making the Document visible in My Places tree, can’t mix and match styleUrl and style (to have common style but with a unique icon), and a few other subtle things. Whereas GE seems to allow the new features in a 2.0 kml file, I guess this is the GE equivalent to quirks and strict mode. (disclaimer: it’s a while since I tested this, so might be old news, I really should run some real world tests to confirm this is still the case)

Following the addition of GeoRSS, earlier today while out photographing some PhotoSpots (TBA), I had an even better (possibly) idea, now that my WordPress stores Lat/Long, why not output KML too! The icing on the cake is you can also wrap it in a NetworkLink so that peoples Google Earth will auto update. (I am surprised couldn’t find this done already!)

And if that isn’t enough then I also release the code under GPL - note its the first time written anything for Wordpress, so might not be a tidy as could be!

Anyway see it in action on my series of PhotoSpots:

• As a Auto Updating Network Link
• or direct Google Earth KML file

• or even on Google Maps! (will also auto update!)

- as all these list the PhotoSpot category, each post includes a link to open the actual spot, so just click the world icon in the balloon to open

Download the Source or view the source: WordPress KML Generator and NetworkLink Wrapper

Installation: This is not a proper plugin, so just drop the .php files into the root of your WordPress folder and you should be done (but you can configure the icon to display if you wish by editing the wp-kml.php file)

Use: Normally you would only provide a link to the Wrapper - which in turn links to the feed itself, but the feed is useful like in the above example to plot directly on Google Maps. You can also specify a particular category to display like the above example or otherwise will include all geocoded posts.
Important: Requires the GeoPress extension be installed (to get geocode info)
Only tested on WordPress 2.0.1 - Works For Me - Your Mileage May Vary - no guarantee, implied or otherwise