jump to navigation

OpenSimulator’s login process and common login problems May 26, 2011

Posted by justincc in opensim, opensim-arch, opensim-tech-basics, secondlife, viewers, virtual-environments, virtual-worlds.
13 comments

Introduction

Today I’m going to write about how a viewer logs in to an OpenSimulator grid.  This is a considerably more complicated process than a simple website login.  If you ever need to find out why login isn’t working it really helps to know what’s going on under the hood.

Process

For simplicity’s sake, we’ll look at the standalone case, where all the regions and grid services are running under a single OpenSim.exe process.

Step 1: As the first step for logging in, the viewer sends an XMLRPC message to the Login Service URI containing the name, password, viewer version and other details.  The -loginuri parameter on the command line that tells the viewer where the login service is (here it’s 192.168.1.2 and the port number is 9000).  In viewers such as Imprudence the viewer’s grid manager can fetch this information from the grid info XML that the grid advertises at a well known URL.

Step 2: The login service uses these details to check that the password is correct.  If it is then it looks up the simulator that the user should be placed in (e.g. last or home).  In this case there’s only one region called “My Region”.  The login service will generate a random ‘circuit code’ and ask the simulator to record this code and set up an ‘agent’ in My Region.  The agent represents a user in an OpenSim region.

Step 3: Once the simulator has generated the agent, it returns a randomly generated ‘circuit code’ to the login service.  The login service will package this up together with the IP and port address of the region and return it to the viewer in a reply XMLRPC message.  The login service gets the region’s IP address and port from the ExternalHostName and InternalPort entries in the bin/config-include/Regions.ini file (config files there can also have other names as long as they end with .ini).  In this case the entry is

[My Region]
RegionUUID = dd5b77f8-bf88-45ac-aace-35bd76426c81
Location = 1000,1000
...
InternalPort = 9000
ExternalHostName = 192.168.1.2

The host name here is 192.168.1.2 (same as the login service since we’re on a standalone) and the internal port is 9000.  But in this case it specifies the port that the client should use for UDP messages between itself and the simulator (we’ll ignore HTTP capabilities in this post).  The ExternalHostName can also be SYSTEMIP, in which case the default IP of the machine hosting the simulator is used (which would also be 192.168.1.2).

Step 4: When the viewer receives the XMLRPC reply, it extracts the circuit code, simulator ip and port.  To make first contact with the simulator, it sends it a UseCircuitCode UDP message containing the circuit code.  The simulator compares this against the circuit code that the Login Service gave it for that client.  If they match then the simulator sends back and Ack packet and the two start talking to each other (i.e. the simulator gets sent terrain, object and texture information and can move around the region).  If they don’t match then the simulator logs a warning and ignores the UseCircuitCode message.

Problems

Whew, quite a process, eh?  As you can imagine, there’s a lot that can go wrong.  Let’s go through the possible problems.

Viewer has wrong loginuri

This is an easy one.  If the viewer is trying to login with the wrong uri (e.g. 192.168.1.3 in the example above) or wrong port (e.g. 9001) then you’ll get something like an “Unable to connect to grid” – nothing will ever reach the Login Service.

Viewer has wrong login credentials

Another easy one.  The Login Service will reject the credentials and tell the viewer, which will display a “Could not authenticate your avatar” message.

A firewall prevents the Login Service from replying to the viewer

In this case the viewer can send some initial TCP packets to the Login Service but can’t get anything back.  As above, the viewer will present an “Unable to connect to grid” message but this time after a longer pause until it times out on the Login Service connection.

Viewer receives misconfigured external host name from Regions.ini

Now it gets more complicated.  Suppose that instead of putting 192.168.1.2 in the My Region config I accidentally put 192.168.1.3 instead, which is a machine that doesn’t exist on my network.

[My Region]
RegionUUID = dd5b77f8-bf88-45ac-aace-35bd76426c81
Location = 1000,1000
...
InternalPort = 9000
ExternalHostName = 192.168.1.3

In this case, the first part of the login process works okay and the progress bar moves along in the viewer.  But when the Login Service returns the simulator information to the viewer, it returns the ExternalHostName of 192.168.1.3 instead of 192.168.1.2.  The viewer will make a number of attempts to contact this non-existent simulator for the second part of the login, and so appear to hang for a while on a message such as “Waiting for region handshake…” before failing with a “We’re having trouble connecting…”

In this case, since 192.168.1.3 has no machine a simple ping will reveal the mistake.  If there is a machine at that address or it’s the port number that is wrong then things are more complicated.  It’s difficult to diagnose problems here since UDP messages are connectionless, unlike TCP.  If you have a utility like netcat available on the viewer machine, you can try sending nonsense to the address and port given in Regions.ini.  For instance, above we could do

echo “foo” | nc -u 192.168.1.2 9000

and the simulator would print out a “Malformed data” message.

Viewer can’t reach region external host name

Now let’s suppose that the ExternalHostName and InternalPort are correct, but the viewer can’t reach that address for some reason (e.g. UDP messages to that port are blocked by a firewall).  You’ll see exactly the same symptoms as if the host name is misconfigured.  The diagnostics are also the same, with the addition that you need to thoroughly check your firewall and other network settings.

You can also see this if you’ve specified a public IP for ExternalHostName but you’re attempting a connection from within your LAN and your router does not support NAT loopback.  The easiest solution is to get a router that does support NAT loopback though you might also want to try the workarounds listed on that wiki page.

A firewall prevents the simulator from replying to the viewer

Unlike the firewall blocking the login service reply above, this time the first part of the login process will complete correctly and the simulator will even receive the UseCircuitCode message.  However, the Ack that it replies with (and any other UDP messages) is blocked by a firewall.  In the simulator log you will see messages such as

[LLUDPSERVER]: Ignoring a repeated UseCircuitCode from
2c3b8307-e257-4d1e-b12f-76f2b8f50ee9 at 192.168.1.3:1208
for circuit 546230463

as the viewer resends the UseCircuitCode packet another 3 times (while it displays the “Waiting for region handshake…” message.  Eventually, the viewer gives up and displays the “We’re having trouble connecting…” message.  In this case, you need to carefully check that your firewall allows outbound UDP messages from the simulator to the viewer’s IP address.

Conclusion

As you can see, the login process is complicated.  Much of this complexity exists so that in grid mode simulators can be hosted on different machines to the login service.

In grid mode, all of the above information still applies, with the addition that the login service and simulators communicate over a network rather than within a single process.  This is another point of failure.  If there’s a problem here then you should see an error in the login service log and the viewer will return with a “Unable to connect to grid” message.

Advertisements

Pretty OpenSim 0.7 diagrams and updated grid setup instructions July 30, 2010

Posted by justincc in hypergrid, opensim, opensim-dev, opensim-grid, opensim-tech-basics, opinion, secondlife, virtual-environments, virtual-worlds.
9 comments

Hi folks.  Just thought I’d let you know that I’ve once again undertaken the sisyphean task of updating the OpenSimulator configuration instructions 🙂  This time, I’ve updated the grid setup instructions to reflect the current situation in OpenSimulator 0.7.  In fact, I wrote a whole new set of steps based on my experiences in setting up a very small multi-machine grid from scratch, so hopefully they should be reasonably accurate!  Of course, any improvements are very, very welcome – feel free to just edit the wiki page!

I also included some diagrams to illustrate the basic differences between standalone and grid configurations.  Here’s the standalone diagram.

As you can see, in standalone mode both grid services (login, assets, inventory, etc.) and the region simulator itself are contained within the same process (OpenSim.exe).  Viewers simply connect to this one instance.

On the other hand, even a very simple grid setup is considerably more complicated.

In OpenSimulator 0.7 a process called Robust.exe runs all the grid services.  Instead of using their own local services, each simulator (OpenSim.exe) connects to this common set of services, usually across a network.  And once a network gets involved there’s considerably more to go wrong – simulators may encounter both network congestion and a delayed response from the central services if they’re overloaded with requests.

In grid mode, simulators also need to communicate with each other.  For instance, since all regions are on a common grid, an avatar may teleport to a region running on another machine, or cross an in-world border between regions hosted on different computers.  Both teleports and border crossings involve an intricate data handoff between the origin simulator and the destination simulator.  The less reliable the network between the simulators, the worse the teleport or crossing experience will be.  Naturallly, this is often worst in the case where these simulators are communicating over the Internet.

Another interesting thing to note about this diagram is that all communication between the viewer and the virtual environment (apart from initial login, which isn’t shown here) occurs through the region simulator that the avatar is currently in.  This applies not only to things such as walking about and creating objects (both of which naturally belong there anyway), but also to operations such as inventory retrieval and texture fetch.

This is one reason why the current grid model is a centralized one, even though OpenSim instances themselves are distributed across different machines.  Anyone wanting to add a new region simulator to a grid is forced to store their assets and inventory in the single set of services provided by that grid operator – they can’t choose some independent third party or provide their own data store.  Equally, anyone adding a simulator has to be trusted by the grid operator since the guy (or gal) running the simulator potentially gains access to the assets and inventory of that grid.

That’s what makes efforts like the hypergrid, http texture fetching and VWRAP so interesting – I think that they’re all trying to lever open the door to a genuinely distributed, more web-like network of virtual environments.  A network where anybody can set up their own simulator without requiring the trust of every other operator, yet receive visitors with the same ease that one can click on a web hyperlink today.  A network where visitors retain their own appearance and are safe in the knowledge that the items in their inventory are not exposed to the simulator that they happen to be visiting (unless they rez them there).  It’s a massive challenge but one that could really transform the landscape.

Anyway, enough architectural rambling :-).  The OpenOffice draw source files for the diagrams above are linked from the configuration wiki page if anybody wants to use them further.

Introducing OpenSim Inventory Archives (IARs) September 14, 2009

Posted by justincc in iars, opensim, opensim-dev, opensim-grid, opensim-tech-basics, secondlife, virtual-environments, virtual-worlds.
30 comments

(Updated Thursday 24th September 2009) – IAR load and save works again on grid mode again as of revision e1abc3 (also on this day)).  However, the user service also needs to be of this revision or later.

(Updated Friday 18th September 2009 – unfortunately it turns out that the added password requirement stops load and save iar working in grid mode.  A fix will come soon).

(Updated Tuesday 15th September 2009 since load iar and save iar commands also need the user’s password as of Git revision c5455d4 on the same day).

Some time ago I mentioned that I was working on a feature for OpenSim known as Inventory ARchives (IARs).  Well, after a longer period than I would have liked, this facility is now available on an experimental basis in the latest bleeding edge OpenSim code.

What is an IAR?

The short answer is that an IAR is the inventory equivalent of an OpenSim ARchive (OAR).

The long answer is that inventory archives are a way of saving selected inventory folders and items to an external file.  This file can then be reloaded later on or given to somebody else to load onto a completely different installation of OpenSim.  The IAR contains all the data necessary to fully restore the items, including textures, scripts and other contained assets.

The IAR facility isn’t intended to supplant tools such as Second Inventory, which extract and restore objects via the client interface.  Rather, it exists as a way for region owners (whether standalone or on a grid) who have access to the region console to backup and transfer inventory.

IAR is also a something of a rough and ready format.  Like OAR, it relies on undocumented object serializations.  It will also itself be subject to considerable evolution in the future.  Nonetheless, I intend that IAR formats will be backward compatible – IARs saved now should remain loadable into future versions of OpenSim.

How do I use an IAR?

IARs are loaded and saved from the region console, just like OARs.  Before doing any loading or saving, you have to be logged in to a region served by the region simulator.

Let’s run through an example.  Suppose that you have an inventory structure like so

and you want to save the contents of folderA to an IAR.  On the region console command line you would issue the command

save iar Justin Clark-Casey folderA userpassword

with your own avatar name and password instead of mine, of course 🙂  This would save folderA and its descendant folders to a file called user-inventory.iar.  If you want to specify a different filename then you can add it to the end of the command, for example

save iar Justin Clark-Casey folderA userpassword my-great-items.iar

If you need to specify a folder or item containing spaces, then you’ll need to surround the path with quotes (e.g. “Folder A” or “Folder X/Folder Y”).

Having saved the IAR, you decide that you’d like to make your items available to the world so you put the file on a website.  John Does comes along and thinks that your items sound really interesting, so he downloads the IAR to the bin/ directory of his own OpenSim installation, starts OpenSim up and logs in.

Now, let’s suppose that John has an inventory structure like this

and he wants to load the IAR into folderY in folderX.  To do this he issues the command

load iar John Doe folderX/folderY userpassword my-great-items.iar

on the region console.  After a short delay the items appear in his inventory ready for rezzing!

Example IAR

Here’s an example IAR (containing the inventory folders and items described above) that you can download and load into your own bleeding edge copy of Git-fresh OpenSim (since revision 5a64ca on 14th September 2009).  If you’d rather not suffer the pain of the bleeding edge, then the load and save IAR commands will be active in the next OpenSim release (which should be 0.6.7).  Load and save IAR commands are present in OpenSim 0.6.6 but they were not complete in this version and do not work properly.

Final Notes

The load and save IAR feature is currently experimental.  Bug reports and suggestions for improvement are welcome!  The best place for bug reports is OpenSim’s Mantis system and the best place for suggestions right now is as comments on this blog post.  Thanks!

There is also some more information on the load and save iar commands on the opensimulator.org IAR page.

Next time…

On my next post (hopefully coming soon), I’m going to talk about some of the current limitations of the IAR implementation and what can be done about these in the future.

One thing that I do want to mention right now is that due to technical limitations item creator names are not preserved unless a profile for that name exists on the target OpenSim system.  I’ll write some more about this problem (and suggestions for overcoming it) on my next IAR post.

Preventing user script execution in OpenSim applications December 5, 2008

Posted by justincc in opensim, opensim-applications, opensim-tech-basics.
2 comments

opensim-logo-short-padlockOpenSim aims to be a general virtual environment platform, useable as a basis for software applications as well as public virtual worlds.

However, for an application to use OpenSim with the Second Life protocol and client there are some restrictions that we may want to apply that would either not be desirable or would be impossible to enforce in a public virtual world.

For instance, suppose that you wanted to use OpenSim as part of a virtual business conferencing application.  You may want to allow users to be able to copy and execute pre-existing scripts (for example, so that they can display and flip between slides on a projector screen) but not let them be able to create scripts that send any user within range their business card.  This might be done to perserve the user experience or for various other reasons.

How might the ‘no auto-sending business cards restriction’ be enforced?  In this particular case, one could stop LSL methods such as llGiveInventory() from executing (this can’t currently be done in stock OpenSim but it would be easy enough to patch the code such that this method doesn’t do anything).  However, one might want to go a step further and stop any user script from executing.  The rest of this blog post explores how we can currently do that.

Of course, there are various trade-offs from going down this route.  On the plus side, preventing user scripts entirely would make scripting engine performance more predictable.  Support costs could be reduced since there won’t be any scope for user scripts to interact with other scripts and avatars in unpredictable ways.  This reduction in unpredictability would also make the user experience more uniform.

The flipside of a reduction in unpredictability is a lower level of direct customization for ordinary users.  They would no longer be able to directly write their own scripts to provide functionality that the original designers never thought of, either for public consumption or for personal use (e.g. huds).  It would still be possible to get these scripts in via an administrator but this is often a painful barrier to cross.

Suppose that you did want to go ahead with stopping users from running their own scripts.  In this case we can’t simply stop scripts belonging to ordinary users from executing at all since in the business conference application we still want them to be able to run pre-canned scripts.  One approach is to only allow scripts to execute that have certain checksums (which acts as fingerprints).  This could be quite good in a finalized application where no new scripts are being created or introduced.

However, the approach that we’re going to explore here is one where ordinary users are prevented from editing existing scripts or creating new ones in the first place.  In OpenSim (as of r7623, post 0.6), this can be done by tweaking some settings in the OpenSim.ini file.  First of all, you have to have permissions enforcement turned on with the following entry (it is actually turned off by default).

serverside_object_permissions = true

In order to allow administrators/gods to still be able to edit and create scripts, one also needs to make a setting to allow god mode.

allow_grid_gods = true

Finally, there are two settings to affect who can create and who can edit scripts. To restrict script creation, the settings need to be changed so that only administrators/gods have these abilities

allowed_script_creators = gods
allowed_script_editors = gods

save-permission-deniedNow when OpenSim is restarted, only users that are gods can create and edit scripts.  For other users, selecting the new script option in the Linden Labs client interface will have no effect and saving edited scripts will not be possible (this is a situation where one could possibly remove these options from a client altogether).

There’s currently no good core OpenSim way of conferring and removing god status from users, though external projects may provide one.  At the moment to make a user an administrator you have to directly change a setting in the user database, with an SQL command such as

update users set godLevel=400 where username="Justin";

400 is the magic number where a user is considered an administrator.

As this solution for stopping user script execution relies on preventing them from creating or editing scripts, it is not really useful in situations where the region is connected to a public grid where the are region or grid servers controlled by a third party, as scripts can then be inserted at the grid servers directly or simply be imported over the border from other region servers (this is a situation where a fingerprinting approach may be more useful).

At the moment I would regard this approach to preventing user script execution as experimental.  Although the obvious routes to creating and editing scripts have been sealed off, there may still be some obscure method that remains viable (though I hope this is unlikely).  Any help in plugging gaps that exist would be much appreciated.

C# scripting in OpenSim November 21, 2008

Posted by justincc in opensim, opensim-scripting, opensim-tech-basics.
43 comments

c-script-small2A month ago, I wrote a short article about how to add an OSSL (OpenSimulator Scripting Language) function to OpenSim. This assumed that you were writing a script using the Linden Scripting Language (LSL).

However, in principle LSL is not the only language that you can write scripts in for OpenSim.  LSL scripts in OpenSim are actually translated into C# before being compiled and executed.  Therefore, it’s possible to bypass LSL and instead write in C# directly (or in Visual Basic, Javascript or any language which has a .net compiler that can be added to OpenSim).  Indeed, some OSSL functions (such as osParseJSON()) will not currently work in LSL at all – the data it returns cannot be easily stuffed into any LSL data type.  Unfortunately, there is a

major caveat with allowing OpenSim to directly execute C# and other non-LSL scripts.  We’ll cover this later on.

But first of all, how does one write and execute a C# script in OpenSim?  The first step is to make sure that OpenSim is allowed to compile C#.  In your OpenSim.ini file, you will need the entry

AllowedCompilers = lsl, cs

in either the [ScriptEngine.DotNetEngine] or [XEngine] sections depending on which script engine your using (or both, if you want to be sure).

Now that this is done, we can enter a C# script.  Here is a very short example.

//c#
// justincc's short test script

string message = "Hello avatar!";
string xml = "<tag>ribbit</tag>";

public void default_event_state_entry()
{
    llSay(0, message);
}

public void default_event_touch_start(
    LSL_Types.LSLInteger total_number)
{
    System.IO.StringReader sr
        = new System.IO.StringReader(xml);
    System.Xml.XmlTextReader reader
        = new System.Xml.XmlTextReader(sr);

    llSay(0, reader.ReadElementString("tag"));
}

You can copy and paste this into a normal Second Life script within OpenSim.

As you can see, the structure is quite different from that of an LSL script.  The first line

//c#

is always required to tell OpenSim that the script should be compiled as a C# script.  WIthout this, the LSL compiler would be used instead (unless you’ve changed the default compiler in OpenSim.ini).

In the subsequent lines you can see that we don’t have the ‘default’ state section that one would find in an LSL script.  Instead, the functions that receive events need to have names in the format <state>_event_<event function>.  For instance,


state_entry() in the default state an LSL script
becomes public void default_event_state_entry().

touch_start(integer total_number)
becomes default_event_touch_start(LSLType.LSLInteger total_number).

We need to use LSLType.LSLInteger instead of C#’s int since this is the method signature that the script engine expects).

There is a more complex example of a C# script that uses osParseJSON() here.

So what are the advantages of writing scripts directly in C#?  The major one is that one gets can use the .NET APIs directly.  We can’t use C#’s using directives in the script so they need to be accessed using their full names (e.g. System.IO.StringReader rather than StringReader).  Among other things, this allows one to use much richer data types in C# that aren’t available in LSL, such as System.Collections.Generic.Dictionary.  One could also do such things as parsing XML using .NET (or mono’s) bundled XML parsers, as shown in the example above.

However, this advantage also forms C# scripting’s current major weakness.  At the moment, as well being able to use such helpful classes as System.Collections.Generic.Dictionary, it’s also possible to call methods such as System.File.IO.Delete(“path/to/really/important/system/file”).  Mantis 2610 has some more details of why this is.  Hopefully, this can be fixed at some stage, but at the moment I would have to recommend that you don’t enable C# scripting on a server unless you really, really, really trust everybody who has access to it.

Alternatively, one may also be able to prevent non-administrators from creating or editing scripts at all.  I hope to write about this at a later date, though these OpenSim facilities are still very experimental.

OpenSim Scripting Languages: LSL and OSSL October 24, 2008

Posted by justincc in opensim, opensim-dev, opensim-tech-basics, secondlife.
17 comments

Introduction

Despite our long term desire to see OpenSim become a general virtual environment platform, implementing all the functions in the Linden Scripting Language (LSL) for the Second Life environment has become one of our de facto aims.  This is very understandable for two reasons

  1. There is a large stock of Second Life scripts already written in LSL.  The more accurately our implementation of LSL, the easier it is to reuse that existing code.
  2. The syntax and ll functions of LSL effectively constitute a language specification.  This allows lots of different open source contributors to work in ‘bazaar‘ fashion.  Many different people can contribute patches that implement ll functions piecemeal, with no need to co-ordinate their efforts with other open source contributors.

However, LSL is imperfect.  Leaving aside any questions about the inconsistency of Linden Lab’s ll function implementations or the deficiencies of the language itself, there is also a lot of missing functionality.  For instance, in LSL one cannot teleport an agent (instead various imperfect workarounds have to be used).  Neither can one write data to notecards, or find out information about avatar group roles.

OpenSimulator Scripting Language (OSSL)

How could the problem of missing functionality be addressed?  If we leave aside the prospect of using completely different languages with their own function implementations (C#, Python, etc.), one way to do it is to extend LSL by adding extra functions that OpenSim understands.  This is what the OpenSimulator Scripting Language (OSSL) is.  It is effectively all of LSL plus extra functions that start with an os prefix.  os functions can be used in a script in exactly the same way as ll functions are used.  Examples of os functions are

  • osTeleportAgent() allows you to teleport an agent to another position and region.
  • osSetDynamicTextureURL() allows you to display an external or dynamically generated image inworld  (as used in this inworld webpage display script or my own ‘graffiti board’ script that displays text on a single texture in-world.
  • osParseJSON() allows you to parse JSON within a script.  This isn’t such a good example as the HashTable return type probably makes it unusable currently in LSL scripts (the example here is a script in compiled C#).  However, I think it could be extended to return a data structure that could be used in LSL.

How to implement an OSSL function

Implementing an OSSL function is a little fiddly – you need to make changes in 3 places.  First, let’s do the actual implementation itself.  Just as there exists an LSL_Api class in the OpenSim.Region.ScriptEngine.Shared.Api package that implements all the LSL functions, so there is an OSSL_Api class that implements OSSL functions.  To implement a new function, one would add a method with the same name in OSSL_Api.cs.  For example

public string osMyFunction()
{
    // We'll talk about this in the next section
    CheckThreatLevel(ThreatLevel.None, "osMyFunction");  

    // Let stats know that a script line has been executed
    m_host.AddScriptLPS(1);  

    return "Hello World!";
}

The method signature for this implementation needs to be added to the interface IOOSL_Api in OpenSim.Region.ScriptEngine.Shared (Api/Interface directory)

string osMyFunction();

Finally, a method to call the function via this interface must be added to the ScriptBaseClass (which underlies all LSL and OSSL scripts) in the OSSL_Stub.cs file in package OpenSim.Region.ScriptEngine.Shared.Api.Runtime.

public string osMyFunction()
{
    return m_OSSL_Functions.osMyFunction();
}

Then it’s a case of recompiling OpenSim and restarting the region server.  Before you restart, make sure that

AllowOSFunctions = true

is set in your OpenSim.ini.  If this isn’t set then no os function will run no matter what threat level is set (as discussed below).

Once you’ve restarted OpenSim, you can invoke osMyFunction() in an in-world script

default
{
    state_entry()
    {
        llSay(0, osMyFunction());
    }
}

In the future I hope it will become possible to add new functions via plugins rather than by altering the OSSL_Api.cs file.

Don’t threaten me

You may have noticed the line

CheckThreatLevel(ThreatLevel.None, "osMyFunction");

above.  Just as some ll functions are dangerous in a virtual world environment and need to be limited (e.g. llRezObject()), so os functions may also need to be controlled.  Therefore, each os function has a ‘threat level’ that is checked before execution.  Threat levels range from None (no threat at all), through to Moderate (intentional abuse may cause denial of service) and up to Severe (even casual use may cause region instability and/or data loss).  Our example osMyFunction() is rated None because it simply returns “Hello world”.  Something like osTeleportAgent() is currently rated High because no security checks are currently carried out before teleport.

Region adminstrators can change the level of os functions they prefer to allow by changing the OSFunctionThreatLevel attribute in OpenSim.ini.  They can also turn individual os functions on and off.

In a public virtual world context you might never want to allow functions with high threat levels.  However, in a virtual world where the people allowed in are trusted, or in application virtual environments allowing such functions may make a lot more sense.

Contributions welcome

OSSL is by no means complete.  So patches to implement functions that you wish existed in LSL are welcome – we’re always happy to consider them 🙂

OpenSim Tech Basics: OARs – Opensim region ARchives October 10, 2008

Posted by justincc in opensim, opensim-tech-basics.
20 comments

OAR Basics

OpenSim archives (OARs) are designed as a mechanism for transporting whole regions from one simulator instance to another.  This involves transporting not just XML information about the size and shapes of prims, but also the textures applied to their surfaces and the items (scripts, notecards, sounds, etc.) contained within them.  As we are transporting a whole region, OpenSim archives also contain terrain information.

Here is a very simple example archive, a screenshot from which can be seen below

So how would you load this archive?  First, you would need to start up an OpenSim instance with a blank region or one that you don’t mind losing (since loading an OAR will replace the entire region).  If your region simulator hosts more than one region you will have to make the blank region the ‘current region’ via the command

change region my-region

You can then load the archive with the command

load oar scene_oar.tar.gz

If your avatar is sitting in that region, you’ll first see any existing objects all disappear.  Then the terrain will be loaded from the archive and the archived prims will magically pop into existence (this is a bit like scenes from the Matrix except much, much slower :-).

Saving out an archive uses a similar command

save oar my-archive.tar.gz

More details can be found on the OpenSimulator wiki here.

OAR Internals

I had three chief aims when designing the file format for OpenSim archives.  More details on this can again be found on the wiki.

  1. Make it easy for people to read and write different OpenSim files within an archive.OpenSim archives are old style UNIX tar.gz files (in the future they may be zips – this just requires us to sort out some license issues.  Nonetheless, the tar.gz files can be opened on Windows with software such as 7-zip).  This means that anybody can unpack them.  Moreover, the files within the archive are laid out to try and make it easy to look at different types of data.  For instance, if you unpack the example oar above you’ll see
    archive.xml
    assets/
    objects/
    terrains/
    Inside the assets directory are all the assets for that archive.  Each asset filename starts with its UUID and ends with an extension appropriate to that filetype.  For instance, the single script in this archive has the filename

    e03b08f7-9bf5-43b7-839e-ba90d42d04a4_script.lsl

    This means that it can be read and even edited – though editing isn’t recommended if you’re going to reload the archive to a system which already has that asset (since assets are meant to never change).

  2. Make it easy to compose two region archives into a single region archive.Because assets and objects are separate files living in separate directories, in principle one could merge two regions into a single region simply by copying the contents of one directory into another.  The exception here is the terrains directory.This will be more useful for archives that are collections of objects rather than whole regions.
  3. Make it easy to compose archives from scratch.In principle one doesn’t need to create an OAR by saving the contents of an existing region.  It should be possible to create one from scratch.

OAR Futures

There’s still some way to go with region archives.  For instance, you can’t currently save multiple regions in a single archive, or parcel information.  Patches and bug reports are very welcome.  Also, I’d be very interested in any links to people making their OARs publicly available.

In the long term the road map is to implement inventory archives as well as region archives – there is now considerable code that can be reused, and people such as Kayne have already contributed preliminary code.  This facility isn’t really meant to compete with products such as Second Inventory – archives can only currently be used if one has adminstrative access to an OpenSIm region (either directly via the region console or through the XMLRPC Admin functionality).  However, I feel that it’s important to have a way of transporting inventory that is free and open, and doesn’t rely on the fiddly business of extracting and recreating data via the client protocol.

OpenSim Tech Basics : OpenSim programming guide beginnings July 5, 2008

Posted by justincc in opensim, opensim-dev, opensim-tech-basics.
1 comment so far

Recently in OpenSim we’ve started to hear more requests for information about getting started in coding OpenSim itself (as opposed to scripting within OpenSim).  This isn’t surprising, OpenSim is increasing in popularity and our current developer documentation is pretty sparse.  So this fortnight I chose to spend what blog writing time I have constructing the beginnings of an OpenSim programming guide for new developers.  Well, it’s actually called a hacking guide (in the best programming tradition), but I thought using that title for this blog post had the potential for causing needless excitement.  The aim of the guide is to make it easier for somebody who is already a reasonable programmer and who knows C# to get up to speed – I don’t think that OpenSim programming is really for people who don’t have some existing skills in these areas.

Actually, to be honest, I don’t think the guide in this very early state is very good at all right now.  Probably the only vaguely worthwhile part at the moment is the codebase overview.  However, even this is only really the sketchiest of outlines – OpenSim already has quite a few packages  and I find that writing such guides always takes much more time than I expect.  Nonetheless, it is in a wiki and it does provide some kind of skeleton for further sections, so hopefully it will be improved over time both with people filling out the rubbish bits and making additional documentation requests to our developer’s mailing list.

And I do think this kind of thing is important if we want to fulfill our aim of OpenSIm a general virtual worlds server that can be adapted to fulfill many people’s needs.  It’s all very well for us obsessives to work everything out by reading the code and experimenting, but if that’s the only route in to OpenSim programming then we may end up excluding a whole tranche of people who have fantastic and creative ideas but who aren’t the programming hardcases that the core developers are.

However, I believe there is a question as to whether it’s a little too early to start writing such a guide – OpenSim is still very much a rapidly moving target in terms of development and a good proportion of anything written could become out of date almost immediately after it’s done.  We’ll see :-).

So if you are somebody with C# programming experience who is interested in OpenSim, please feel free to take a quick look at the prototype guide as it exists right now, and hop on the OpenSim developer’s mailing list if there’s a particular thing you’d like to see.  As is common with open source projects, fulfillment (or even reply) is far from guaranteed, but if there is some development topic you’d like to see more information on I think it’s worth asking.  You could also ask directly here, though that does mean I might be the only core developer to see it 🙂

OpenSim Tech Basics : My God, it’s full of modules! May 8, 2008

Posted by justincc in opensim, opensim-dev, opensim-tech-basics, secondlife.
1 comment so far

Hello folks. Today I thought that I’d write about the modular nature of OpenSim. Modularity is of great importance to OpenSim since its goal is not explicitly to provide a complete Second Life service implementation. While the current Second Life protocol does provide a handy trellis up which OpenSim can grow (a little bit like POSIX did for Linux – though in this case the existing “standard” is both evolving and not very well documented), the guiding philosophy is to make OpenSim a platform for 3D environments in general, as stated on our front page. We hope that OpenSim will ultimately be useable as a base for many systems, covering the range from standalone 3D applications (such as data centre monitoring) to different types of virtual world grids. By changing or extending the OpenSim kernel using various modules and plugins, different metaverse systems and programs will hopefully be able to use the same base platform for different viewers, protocols and region features.

Thus, modularity is a crucial part of the OpenSim system. By a “module”, I mean a class or a set of classes (within a .NET dll) which can easily be integrated with OpenSim in order to either provide different types of core functionality, or provide extensions to the existing codebase. If one doesn’t count alternative grid service implementations, all the modules which I know to currently exist come bundled with OpenSim (such as the experimental SVN module which allows you to periodically backup your region to a subversion control system, and the dynamic texture module which uses .NET libraries to create a texture on the fly using draw commands). However, over time the real power of the system will come in allowing other people or organizations to write their own module code to make OpenSim do pretty much anything they want.

Since this is a tech article, I want to go on a brief tour of the modular system as it exists today. I’m going to cover three of the major types of module in OpenSim – database plugins, application plugins and region modules. We’ll take a look at each of these in turn.

OpenSim Database Plugins

These are the plugins which provide an interface between OpenSim and the database system that it’s using to store data. In an OpenSim distribution, these are found in the OpenSim.Data.* namespaces (such as OpenSim.Data.MySQL and OpenSim.Data.SQLite). Each of these namespaces is populated with classes which implement the storage interfaces for the various OpenSim services. So, for instance, OpenSim.Data.SQLite.SQLiteInventoryStore is a class which implements IInventoryData, translating calls to methods such as IInventoryData.GetInventoryFolders() to the necessary SQLite SQL commands required to retrieve the folder data).

This means that anybody can come along and implement the necessary interfaces to make OpenSim use another database. This was done with MSSQL, which also sits in our build tree. However, the problem that we currently have is that a database module writer ends up implementing lots of custom code in order to talk to the backend system – it’s much easier currently to copy, paste and adapt than it is to extract common module code. Getting the code right for the foibles of each individual database is difficult, and means that unless someone is actively maintaining a plugin, it easily falls into disrepair (as has in fact happened for some of the MSSQL plugins).

Luckily, there are various ways to tackle this. Core committer Neas Bade (sdague) is currently working on the production of a set of NHibernate plugins, following preliminary work by developers such as Impalah. NHibernate can handle all the necessary work in persisting our objects to the database. Moreover, it means that in the future, OpenSim should be able to use all the databases which NHibernate supports, which is much larger than the set we could ever hope to reliably maintain. So much of what I’ve just told you should one day become obsolete, unless you want to use a database which NHibernate doesn’t support 🙂

OpenSim Application Plugins

These are plugins which act over the whole OpenSim server rather than just for a single region. Application plugins implement the IApplicationPlugin interface which provides just two methods – Initialise(), which passes in a reference to OpenSimMain by which the application plugin can get the information it needs, and Close(). Application plugins are found and instantiated using the Mono.Addins system (which also works under Microsoft’s .NET implementation).

There are a couple of examples of application plugins in the OpenSim build tree. One of these is the Remote Admin plugin, which allows certain OpenSim functions (such as shutdown and region creation) to be controlled by XML RPC calls rather than from the console.

OpenSim Region Modules

These are modules which operate at the region level (hence the name!). As mentioned earlier, these cover both fairly core functionality (such as the Chat module which is needed to allow anybody to say anything!) and extension functionality (such as the SVN backup and dynamic texture modules). The modules are currently a little scattered about the OpenSim tree (though there is now the beginnings of a categorization in OpenSim.Region.Environment.Modules), but they all implement the IRegionModule interface – a simple interface whose most important function is to pass in scene data when the module is initialized.

Apart from this initialization, region modules generally get and send information in one of two ways – either by registering for scene events (which is often done by modules that extend existing functionality and don’t need to pass back any data), or by registering a module using an explicit interface which the core code can later call. An example of the latter is the Permissions module, which registers itself with an IScenePermissions interface. This interfaces contains methods such as CanRezObject() that the scene can invoke when it wants to know whether object rezzing is permitted for a given entity.

You can find others examples of region modules by looking through the OpenSim code tree. There is also some documentation on the OpenSim wiki, though this does need to be updated.

In conclusion

We’ve covered three of the major types of OpenSim modular functionality. In addition, though, there are also other types of modularity (such as the ability to substitute alternative scripting engines) that I haven’t gone through here.

At the moment, the module system in OpenSim is pretty immature. For instance, it’s questionable as to why we’re using different systems for application plugins and region modules. There’s also currently no proper way of turning modules on and off, something which needs to be addressed. But even so, if anyone does get interested in changing or extending a part of OpenSim that can’t be done in the current module system, please do let us know. I am, as I’m sure are other core developers, very interested in providing the interfaces and architecture needed to accomodate further modularity.

Oh, and if you do write any modules, please feel free to list them on our related software page!

OpenSim Tech Basics : My God, it's full of modules! May 8, 2008

Posted by justincc in opensim, opensim-dev, opensim-tech-basics, secondlife.
1 comment so far

Hello folks. Today I thought that I’d write about the modular nature of OpenSim. Modularity is of great importance to OpenSim since its goal is not explicitly to provide a complete Second Life service implementation. While the current Second Life protocol does provide a handy trellis up which OpenSim can grow (a little bit like POSIX did for Linux – though in this case the existing “standard” is both evolving and not very well documented), the guiding philosophy is to make OpenSim a platform for 3D environments in general, as stated on our front page. We hope that OpenSim will ultimately be useable as a base for many systems, covering the range from standalone 3D applications (such as data centre monitoring) to different types of virtual world grids. By changing or extending the OpenSim kernel using various modules and plugins, different metaverse systems and programs will hopefully be able to use the same base platform for different viewers, protocols and region features.

Thus, modularity is a crucial part of the OpenSim system. By a “module”, I mean a class or a set of classes (within a .NET dll) which can easily be integrated with OpenSim in order to either provide different types of core functionality, or provide extensions to the existing codebase. If one doesn’t count alternative grid service implementations, all the modules which I know to currently exist come bundled with OpenSim (such as the experimental SVN module which allows you to periodically backup your region to a subversion control system, and the dynamic texture module which uses .NET libraries to create a texture on the fly using draw commands). However, over time the real power of the system will come in allowing other people or organizations to write their own module code to make OpenSim do pretty much anything they want.

Since this is a tech article, I want to go on a brief tour of the modular system as it exists today. I’m going to cover three of the major types of module in OpenSim – database plugins, application plugins and region modules. We’ll take a look at each of these in turn.

OpenSim Database Plugins

These are the plugins which provide an interface between OpenSim and the database system that it’s using to store data. In an OpenSim distribution, these are found in the OpenSim.Data.* namespaces (such as OpenSim.Data.MySQL and OpenSim.Data.SQLite). Each of these namespaces is populated with classes which implement the storage interfaces for the various OpenSim services. So, for instance, OpenSim.Data.SQLite.SQLiteInventoryStore is a class which implements IInventoryData, translating calls to methods such as IInventoryData.GetInventoryFolders() to the necessary SQLite SQL commands required to retrieve the folder data).

This means that anybody can come along and implement the necessary interfaces to make OpenSim use another database. This was done with MSSQL, which also sits in our build tree. However, the problem that we currently have is that a database module writer ends up implementing lots of custom code in order to talk to the backend system – it’s much easier currently to copy, paste and adapt than it is to extract common module code. Getting the code right for the foibles of each individual database is difficult, and means that unless someone is actively maintaining a plugin, it easily falls into disrepair (as has in fact happened for some of the MSSQL plugins).

Luckily, there are various ways to tackle this. Core committer Neas Bade (sdague) is currently working on the production of a set of NHibernate plugins, following preliminary work by developers such as Impalah. NHibernate can handle all the necessary work in persisting our objects to the database. Moreover, it means that in the future, OpenSim should be able to use all the databases which NHibernate supports, which is much larger than the set we could ever hope to reliably maintain. So much of what I’ve just told you should one day become obsolete, unless you want to use a database which NHibernate doesn’t support 🙂

OpenSim Application Plugins

These are plugins which act over the whole OpenSim server rather than just for a single region. Application plugins implement the IApplicationPlugin interface which provides just two methods – Initialise(), which passes in a reference to OpenSimMain by which the application plugin can get the information it needs, and Close(). Application plugins are found and instantiated using the Mono.Addins system (which also works under Microsoft’s .NET implementation).

There are a couple of examples of application plugins in the OpenSim build tree. One of these is the Remote Admin plugin, which allows certain OpenSim functions (such as shutdown and region creation) to be controlled by XML RPC calls rather than from the console.

OpenSim Region Modules

These are modules which operate at the region level (hence the name!). As mentioned earlier, these cover both fairly core functionality (such as the Chat module which is needed to allow anybody to say anything!) and extension functionality (such as the SVN backup and dynamic texture modules). The modules are currently a little scattered about the OpenSim tree (though there is now the beginnings of a categorization in OpenSim.Region.Environment.Modules), but they all implement the IRegionModule interface – a simple interface whose most important function is to pass in scene data when the module is initialized.

Apart from this initialization, region modules generally get and send information in one of two ways – either by registering for scene events (which is often done by modules that extend existing functionality and don’t need to pass back any data), or by registering a module using an explicit interface which the core code can later call. An example of the latter is the Permissions module, which registers itself with an IScenePermissions interface. This interfaces contains methods such as CanRezObject() that the scene can invoke when it wants to know whether object rezzing is permitted for a given entity.

You can find others examples of region modules by looking through the OpenSim code tree. There is also some documentation on the OpenSim wiki, though this does need to be updated.

In conclusion

We’ve covered three of the major types of OpenSim modular functionality. In addition, though, there are also other types of modularity (such as the ability to substitute alternative scripting engines) that I haven’t gone through here.

At the moment, the module system in OpenSim is pretty immature. For instance, it’s questionable as to why we’re using different systems for application plugins and region modules. There’s also currently no proper way of turning modules on and off, something which needs to be addressed. But even so, if anyone does get interested in changing or extending a part of OpenSim that can’t be done in the current module system, please do let us know. I am, as I’m sure are other core developers, very interested in providing the interfaces and architecture needed to accomodate further modularity.

Oh, and if you do write any modules, please feel free to list them on our related software page!