nearby.org.uk blog

Following part 1 yesterday, have now plotted some maps per individual service.

Flash Earth Layer

Conclusions

1. Google Earth does request a link for the initial view on startup - that point in the Atlantic.
2. I guess North American installs have two slightly different default placemarks; has it changed, or maybe Canada gets a different one?
3. But that doesnt explain the lines from there - people must have their links set to periodically refresh?
4. Germany likes FlashEarth (or rather people like looking at Germany with FlashEarth link enabled)
5. Many people leave the links open even when not in use - e.g. the British Isles links show worldwide usage, notably over Russia.
6. MGRS notably has high usage in two areas.
7. The Far east likes to know what direction they facing.

Update: In a total forehead slapping momont realised these unprojected whole world images, are exactly what is used in GroundOverlays, so the the pages now include links to view the higher resolution images directly in Google Earth, duh!

Example: View In Google Earth (don’t forget to try adjusting the tranparency slider!) [Update, to fix broken link, sorry!]

Recently geohash.org was released; the premise: short easy links that encode a location. The website offers instant conversion, but as the algorithm is Public Domain, Lordelph was quick to implement a php class, and I could quickly add the conversion to my existing conversion API. Convert a postcode to geohash anyone?

Updating the documentation noticed the API was last updated in 2005 - gulp, how time flies! Anyway as a celebration have upped the throttling on the API, now three times the previous limit. Running on much improved hardware than when it was launched so maybe the limit can be tweaked even more, will run some analysis of its impact - it already seems the API is a minor part of the overall load on the server - particularly against the Google Earth layers.

For a project wanted to create a small script to make generation of network link wrappers to KML content easy, but then I thought could have it create ’short’ urls, ala tinyurl.com, to make reuse easy. But then thought if going to that trouble might as well make it a full blown mini application, so I give you

goKML.net :: Short Links to KML content

I think it turned out pretty well (esp for just an evenings hacking) - it even has an element of ‘design’ - which is saying something!

So if you want to create a network link wrapper with the minimum of fuss, and might enjoy having it as a short url, then give it a go.

Of course there are still a few things that could be done, but will leave these for another day…

• Auto discover the ‘title’
• Intelligence for ‘My Map’s - eg support multiple pages, and maybe some other KML types.
• Actually make the admin page useful - display stats (they are stored already) , and edit the link
• A public gallery of popular/new etc links - maybe even some sort of search - esp if we can do rudimentary indexing of the content (when fetch the title)
• Support for named links eg could choose gokml.net/fredsparty.kml
• Complete the support for the various network link options - particulaly making it easier to select the appropriate settings.
• Maybe even support for forwarding the link by email (and/or sending the admin link to yourself)
• Any thing you can think of - let me know!

Recently I have been playing a lot with Sphinx full-text search engine, in particular with regard to indexing the Geograph archive. (a bit of background - Geograph has a fairly good homegrown site text search - but its not full text, so many queries will not return that many results - not to mention been based on MySQL ‘like’, so is pretty slow - so a full text search is the next level). And I have to say I am liking it a LOT, in fact I would say I am a fanboy

So to that end of created a whole bunch of demos based around the flexible indexing it provides, location based searching is even possible!

At the most basic is simple text based search, one point of note, there is no pagination, simply add more keywords (including negative) or grid references to refine the selection.

Next is a ‘auto-complete’ style image finder, this is designed to find ‘that image’ quickly, in a similar way to the above but shows the results in a autocomplete box immediately!

A refinement of the first is search with location, this allows you limit the search to near a particular Grid References - this is particully cool in that there is Sphinx powered auto-complete for place names for finding GRs. (a real auto-complete not a like the search in the previous one pretending to be one)

This is all building towards the Illustrator demo. Which from a block of text attempts to find relevent images. The idea is that a (geolocated) news article, walking route, place description and such could be automatically have relevent(ish) images shown. (an example demo here)

(a few more ‘toys’ can be found in GeographTools!)…. Try them out and let me know how you get on…

I have learnt a lot about search indexing from this, including how to perform location searches in the index (I know latest versions of sphinx include a lat/long based geosearch - but I think this r-tree method in text has better scalability), and how to create an autocomplete function with sphinx. If anybody is interested in these, they will eventually make it into the geograph codebase, or let me know and I might make a separate post.

Interestingly (huh?), it was actually creating a ‘autocomplete’ textbox for finding trigpoints (which included the forerunner to the sphinx location search in but implemented in mysql), is actually what inspired me to actually go the trouble if figuring out how to install Sphinx on linux, which I have been interested in for a long time! - that is also now sphinx powered for text searches

As a side note have now reached the ‘linux sysadmin’ level that I can compile it on Geographes servers, yay! But I do worry for the sanity of others due to this (a little knowledge is a dangerous thing!)

Following the update in July for hiding the 100km numbers from the gridline labels, it seems the same fix broke the sub KM lines, thanks to a comment on the original post for letting me know.

Anyway, the online hosted version has been updated (and checked at all scales as best I can!), and the same file pushed into the downloadable zip for the offline version, so can download the latest code from the file. And just for completeness the diff.

Following the amazing interest in the networklink for embedding FlashEarth inside Google Earth 4.2, have performed a few updates…

1. The zoom is a LOT closer matched, this is thanks to the inspiring formula: $z = (-1.4956 * log($range/1000) + 16.529; (yes really!); brought to you by the wonder of plotting a graph in Excel, creating a trendline, turning on the formula display option. Probably could get an even better match with more patience, but probably about as good as it going to get really.
2. Removed the title and the directions links, makes the balloon looks a bit nicer, couldn’t get it to go transparent for me tho… suggestions welcome.
3. When the network link is now a folder so can choose the size of your popup The previous version is now the ‘medium’ one.

Though the magic of the web, you get 1) and 2) automatically, if want 3) just download it again:

Open in Google Earth (Required: GE 4.2 Windows)

Thanks again to everyone involved.

… a website for more traffic that is. This is something a little off-topic perhaps for this blog, but it might be of interest to a few so will document a few tricks have learnt, in tweaking Geograph to cope with more traffic as it’s daily visitors and hits continue to climb. If you are not familiar with Geograph, or not a System-Admin (or budding - like me!), then probably can stop reading now!

First a little background, Geographs code started very humble, and coded to work off a single server, later with OS sponsorship we upgraded to multiple servers to cope with increasing traffic. This was done with a single larger server for Database and photo storage, and then multiple commodity webservers (with a front end load-balancer) More. This worked well for a number of months, but simply the DB/NAS server couldn’t cope with the increasing DB load, and bandwidth for serving all hundred of thousands of photos.

• Split the database, a small quick win is/was split off php sessions and gazetteer queries to a second Database. Sessions of course have lots of writes, so where tending to saturate the main db, this perhaps reduced its load by a 3rd!
• Cache Images on the separate servers. The servers aren’t big enough to house a copy of the full archive, but thumbnails are certainly more manageable. Seeing as thumbnails actually account for about 60-70% of the raw hits to the site, this is a potential win, as previously each server would have to seperatelly fetch individual images off the NAS. We use Apache as the webserver, so could easily create another simple VirtualHost to serve thumbnails, a empty DocumentRoot save images, with a simple 404 handler to fetch and store images not ready copied. This greatly reduces load on the NAS as its not having 3x servers fetching random thumbnails. (this also paves the way to move away from full-blown Apache simply for static content)
• Cache stuff in Memory - with Memcache. Related to the above point, quite a bit of load is actually random disk IO to determine image sizes as this requires reading the jpeg data. Caching this all in memory is good. Memcache can easily distribute it cache across multiple machines, so even losing a server means only part of the cache needs rebuilding. We also use ADODB as a database abstraction layer, with the latest version it has support to use Memcache for its caching, great! Last up is to do lots of application level caching on key places. Of course sessions and also the templating system (smarty) could benefit from memcache, but one step at a time!
• Optimise the HTTP headers. There are lots of tweaks and stuff here that can be done to lower the bandwidth and improve external cacheability of objects. This post is getting quite long so I think that might be a separate post…
• Optimize the slow queries. And last of not least, learn to love going though the log of slow database queries, and really stepping on the slow ones. This of course is an ongoing project. I found a script to summarize these, but it seems that mysql 5 at least comes with its own equally good one!

This list isn’t exhaustive, and of course is an on going project, always more can be done…

Spured on by comments on Stefan’s Ogle Earth blog, have created this little page as a wrapper to the excellent service provided by geonames.org to automatically convert a RSS feed to KML. It also does RSS->GeoRSS and GeoRSS->KML.

http://www.nearby.org.uk/rss-to-kml.php

Hopefully should make it a bit easier to load a (Geo)RSS feed into Google Earth in the style of a RSS feed - it auto updates.

… really need more to talk about, been kinda quiet of late…

If you have Google Earth version 4 and broadband then you might be interested in clicking this link:

· Geograph SuperLayer ·

By opening the above link you should be able to view the whole Geograph image collection directly in Google Earth. This has been possible in a number of ways for a while, but this does so in a much more compelling way (i.e. a single small download!).

This exploits the new Region functionality of GE4 to only fetch and load detail as you zoom in, in this way the view starts depicting a course overview of the current coverage, zooming in reveals more detail until pictures themselves become visible. (the ‘Super’ is adopted from Google’s term of SuperOverlays - overlays that load via nested regions)

In particular this is all pre-compiled and compressed, so should be fairly quick to download and is friendly as possible on bandwidth. Overall the layer consists of 102,965 files at 450MB compressed so there is a lot of data available, however you will download a lot less than this, probably on the region of a couple hundred files totalling about 2MB in a typical session.

As a small refinement, when a icon represents a single photo (a camera icon) if a view direction is stored the icon will have the top point in the appropriate direction. Also once the thumbnail is visible you can double click the thumb and the View will rotate to orientate the photo correctly!

Also this is only the first version, hot off the presses, so it not perfect yet, in particular would like to use Geograph specific icons (any Graphic Designers reading???), for example to give more feedback to the user on squares with many images, and view direction - also GE sometimes gets confused at the amount of detail available - plan to experiment to see if something can do about that.

… Hopefully users of GE3 should degrade nicely to use the View-based refresh method of viewing the dataset.

Oh and yes this is the little kml project I mentioned, see also other ways of loading geograph data into Google Earth or Google Maps via KML files.

For a little project working on for Geograph (btw Second Birthday today!), that will end up producing quite complex (and repetitive) KML, I decided to take the plunge and create my own abstraction class.

There are already a few of these about, but I wanted something with just enough abstraction so don’t need xml in the php code, but without too much complexity, or a whole new syntax to learn. So the final output is basically a generic XML creation system, but tailored to KML generation, with a number of convenience methods for common bits of KML, (like outputting the right headers). This don’t do any schema enforcement, and assumes a good working knowledge of kml.

Anyway as the code is already GPL, thought would mention it here in case it useful to someone else.

Get the base class here (from the Geograph SVN repository)

There is also another file here, which extends basic the functionality, this is more specific to Geograph, but could still be useful.

And a few demo’s/tests used during development:

Demo1 - (view running demo - by default displays the KML, option to open in GE)
Demo2 - (view running demo)

And for a real world use of the code (been in active use for a week now)

Eagle eyed viewers might notice it claims kml2.0 but then goes on to allow use of 2.1 features. Yes I know this is ‘bad’ but is with reason. I have found to my expense that using 2.1 as the version actually has a few undesirable effects, eg making the Document visible in My Places tree, can’t mix and match styleUrl and style (to have common style but with a unique icon), and a few other subtle things. Whereas GE seems to allow the new features in a 2.0 kml file, I guess this is the GE equivalent to quirks and strict mode. (disclaimer: it’s a while since I tested this, so might be old news, I really should run some real world tests to confirm this is still the case)