nearby.org.uk blog

Geograph offers bulk Torrent downloads, of images. All good (we hope!), but a major reason for doing this is to utalise the power of p2p and share the bandwidth bill. We have bordering on 150Gb of image data to share, but everybody downloading that data from our servers is, well a LOT, not to mention slow, and using bandwidth that could otherwise be serving regular visitors. (the alternative is to serve it from home – but using the upstream connection for that size is also really slow…) 

But to do this is needs others to help with the seeding – in the long term. For a while now 90% of downloading is coming from a seed I have on a server, ie people are peering while downloading, but as soon as the download is completed… its ‘thanks for the fish’ and they are gone.

(we could seed from our homes too, but with the server offering a big enough pipe theres no need – yes aware of the irony in that statement – but we already paying for the server) 

So what can be done to help encourage others to stick around and help?  One consideration is the torrents include the raw files – rather than a tar or similar, making it easy to keep the torrent software pointed at the folder that actullly use the images – without duplication. 

What if the files where made smaller?  (but would mean many more torrents – and the assumption is people are pretty much wanting most images, so picking and choosing isnt really going to happen) 

One critism is the metadata in the files, is hard to keep uptodate – the metatdata is updated on the site from time to time. But for the most part that can be considered a different issue, we can offer bulk downloads of data – on a bespoke basis. (and have an API) 

I’ve looked around the interweb, and this does crop up occasionally. But our case is somewhat unique; its a specialist market (so really at most 1 or 2 downloaders at once), the files on the large size, and we’er in this for the long term (decades ideally!).