Open grids and distributed asset systems

One of the things I really love about OpenSim is that it gives us a platform for all kinds of experimentation. One of the experiments going on right now is the establishment of “open” grids using the Second Life protocol. An “open” grid, such as OSGrid, is one where anyone (subject to the grid owner’s agreement), can host their own region server and reserve a place for it in the grid map. Once the server is online, it appears in that grid’s world and can be flown/teleported into and out of just like any other region. This is in contrast to “closed” grids, such as the OpenLife and the Linden grids, where all region servers are hosted by a central entity (though it looks like the Linden grid is soon to open up in a limited fashion by allowing IBM to host region servers of its own).

By terming these “open” and “closed” grids I don’t mean that “closed” grids are evil in any way – I think there are very good arguments for running a grid as a “closed” grid. One of these is quality control – if you’re running an “open” grid, large chunks of your world may suddenly disappear if region servers you have no control of unexpectedly go offline. More insidiously, if the software running the region isn’t working as it should then the experience of users may be degraded in various ways. For example, all inventory requests in OpenSim are currently routed through the region server. This gives the region server provider an opportunity to maliciously or accidentally return an empty or garbled response to an inventory request, possibly rendering your inventory invalid until you clear your cache and relog).

Nonetheless, there are a number of exciting things about an “open” grid that I think make it an idea very much worth pursuing. Most of these stem from the fact that it leaves you in control of your own server. Don’t like the performance on your region? Then get more powerful hosting on the open market. Want to allow web users to spectate on events in your sim? Then add in an OpenSim module which streams flash video from a fake avatar acting as a camera (this is just a random idea – I’m pretty sure it doesn’t exist yet for OpenSim!).

So what, then, are the current challenges for an “open” grid? One of the chief problems is that up until recently, it looks like the second life protocol was designed with a “closed” grid in mind. This is manifest in the Linden viewer in various ways. For instance, to go back to the earlier inventory example, once bad inventory information has been passed to the viewer by a rogue region, I believe it can’t be replaced with new information until a relog (and possibly a cache clear). Actually, I’d love to know if this isn’t true but it’s rather difficult to determine since we can’t look at the viewer source (for reasons which have been exhaustively covered elsewhere) and Linden Lab hasn’t published all that much information on the protocol.

Also, to be fair, some of the newer protocol changes may make it easier to host an “open” grid. For instance, with the inventory example above, it certainly would have been true with the old method of sending inventory information directly in the client’s UDP session (which is connected to the region). However, with the new CAPS method of retrieving inventory (see here for an explanation of CAPS), it might be possible to provide a capability which routes inventory requests directly to a grid server rather than via the region server.

Although agent inventories are a challenge for an “open” grid, today I really want to talk about the asset issue. The architecture of OpenSim currently has a very centralized model where all asset data is hosted by an asset service run by the grid operator. For a small scale grid, this could be a single server (for example, the reference asset server provided by the OpenSim distribution), hooked up to a backend database.

As the grid grows and more regions come online, the load on the asset server increases. The chief increase is in texture data – additional regions have neat terrain and objects built on them with lots of interesting textures, and the extra regions attract more people who want lots of individual pieces of clothing.

Here are two possible responses to this problem. The first is to scale up your central asset service – throw a few more servers in and tune up your database. This is relatively nice and simple, but it may mean you have to start charging for uploads to reflect your storage and maintenance costs. This might be a perfectly acceptable business model (an “open” grid isn’t necessarily a non-commercial grid).

The second is to distribute the asset data. Instead of hosting everything yourself, allow asset uploaders to provide an alternative location from which to retrieve textures, scripts, sounds, etc. This could be as simple as placing them on a webserver. This reduces the load on a central asset server and may allow an “open” grid to scale to a much bigger size without needing a commercial or formal donations system.

Naturally, there are problems with such a scheme. Suppose you buy an item from a vendor hosting their own texture data. What happens if their asset service goes offline? If you want to allow other people to come and build on your region, how do you make sure their asset uploads are routed to your asset service (assuming that’s what you want) rather than their own? I don’t know currently whether such problems are surmountable.

Is there any way that a distributed asset system could be implemented in OpenSim today? Using the current Second Life protocol (which is the only one in town right now) this might be possible. All asset requests (whether through the client’s UDP session or via CAPS), are routed through a central point, which means we would have to store a pointer in the asset database in order to tell the client where it can find the data for a particular asset. Under CAPS, I believe this http location could be provided directly to the client, resulting in an interaction as illustrated below.

We still need to contact the central asset service for each request, but at least all the asset data doesn’t have to be stored at, and flow through, this single point. Ideally, one might want to embed the asset’s location directly in the asset metadata so that the request can be made to the external data source directly.

However, I don’t believe this is currently possible within the Second Life protocol.

So far this is just an idea, but it might be one worth exploring in more detail as “open” grids continbute to grow in size. I’ve taken a high level view of the distributed asset approach – almost certainly there will be issues once you get down to the nitty gritty of trying to make such a thing work. Constructive feedback is welcome – particularly in light of that fact that I’m not overly familiar with the plans for the evolution of the Second Life protocol (mainly because of a chronic shortage of time!).