jump to navigation

Introducing OpenSim Inventory Archives (IARs) September 14, 2009

Posted by justincc in iars, opensim, opensim-dev, opensim-grid, opensim-tech-basics, secondlife, virtual-environments, virtual-worlds.
trackback

(Updated Thursday 24th September 2009) – IAR load and save works again on grid mode again as of revision e1abc3 (also on this day)).  However, the user service also needs to be of this revision or later.

(Updated Friday 18th September 2009 – unfortunately it turns out that the added password requirement stops load and save iar working in grid mode.  A fix will come soon).

(Updated Tuesday 15th September 2009 since load iar and save iar commands also need the user’s password as of Git revision c5455d4 on the same day).

Some time ago I mentioned that I was working on a feature for OpenSim known as Inventory ARchives (IARs).  Well, after a longer period than I would have liked, this facility is now available on an experimental basis in the latest bleeding edge OpenSim code.

What is an IAR?

The short answer is that an IAR is the inventory equivalent of an OpenSim ARchive (OAR).

The long answer is that inventory archives are a way of saving selected inventory folders and items to an external file.  This file can then be reloaded later on or given to somebody else to load onto a completely different installation of OpenSim.  The IAR contains all the data necessary to fully restore the items, including textures, scripts and other contained assets.

The IAR facility isn’t intended to supplant tools such as Second Inventory, which extract and restore objects via the client interface.  Rather, it exists as a way for region owners (whether standalone or on a grid) who have access to the region console to backup and transfer inventory.

IAR is also a something of a rough and ready format.  Like OAR, it relies on undocumented object serializations.  It will also itself be subject to considerable evolution in the future.  Nonetheless, I intend that IAR formats will be backward compatible – IARs saved now should remain loadable into future versions of OpenSim.

How do I use an IAR?

IARs are loaded and saved from the region console, just like OARs.  Before doing any loading or saving, you have to be logged in to a region served by the region simulator.

Let’s run through an example.  Suppose that you have an inventory structure like so

and you want to save the contents of folderA to an IAR.  On the region console command line you would issue the command

save iar Justin Clark-Casey folderA userpassword

with your own avatar name and password instead of mine, of course 🙂  This would save folderA and its descendant folders to a file called user-inventory.iar.  If you want to specify a different filename then you can add it to the end of the command, for example

save iar Justin Clark-Casey folderA userpassword my-great-items.iar

If you need to specify a folder or item containing spaces, then you’ll need to surround the path with quotes (e.g. “Folder A” or “Folder X/Folder Y”).

Having saved the IAR, you decide that you’d like to make your items available to the world so you put the file on a website.  John Does comes along and thinks that your items sound really interesting, so he downloads the IAR to the bin/ directory of his own OpenSim installation, starts OpenSim up and logs in.

Now, let’s suppose that John has an inventory structure like this

and he wants to load the IAR into folderY in folderX.  To do this he issues the command

load iar John Doe folderX/folderY userpassword my-great-items.iar

on the region console.  After a short delay the items appear in his inventory ready for rezzing!

Example IAR

Here’s an example IAR (containing the inventory folders and items described above) that you can download and load into your own bleeding edge copy of Git-fresh OpenSim (since revision 5a64ca on 14th September 2009).  If you’d rather not suffer the pain of the bleeding edge, then the load and save IAR commands will be active in the next OpenSim release (which should be 0.6.7).  Load and save IAR commands are present in OpenSim 0.6.6 but they were not complete in this version and do not work properly.

Final Notes

The load and save IAR feature is currently experimental.  Bug reports and suggestions for improvement are welcome!  The best place for bug reports is OpenSim’s Mantis system and the best place for suggestions right now is as comments on this blog post.  Thanks!

There is also some more information on the load and save iar commands on the opensimulator.org IAR page.

Next time…

On my next post (hopefully coming soon), I’m going to talk about some of the current limitations of the IAR implementation and what can be done about these in the future.

One thing that I do want to mention right now is that due to technical limitations item creator names are not preserved unless a profile for that name exists on the target OpenSim system.  I’ll write some more about this problem (and suggestions for overcoming it) on my next IAR post.

Comments»

1. Zonja Capalini - September 14, 2009

That’s really fantastic news! 🙂 Even the limitation of having a restore of “save oar John Doe / file.iar” creating spurious base folders can be overcome with some few minutes of work. Now we’ll be able to have a backup of our inventory in a pendrive, alongside with the .oars for the regions 🙂

2. rightasrain - September 15, 2009

Justin–xlent. On the creator name point…we are making a registry for creator names to support builderbot (which I guess could also make IARs). I suppose you could use that in the IARs to maintain that field?

3. LaurentAkina - September 15, 2009

I have tried the iar command in a fresh instance of OpenSim (trunk) but nothing append after. Nothing in my bin folder.

My command is right ; save iar Laurent Akina AkinaInventory AkinaInventory.iar

I got the message in console : saving archive user-inventory.iar from inventory path AkinaInventory for laurent akina

What is wrong ?

4. justincc - September 15, 2009

@rightasrain – yes, this is the kind of thing that I’ve been thinking of, albeit so that user profiles can originate from anywhere on the web. I hope to write more about this soon and conversation about that would be very welcome.

@LaurentAkina – So you don’t see any log messages at all? You really do need to be at the very latest OpenSim revision for this (now c5455d4 today which adds the password requirement). If you have further problems, please could you open a bug report at http://opensimulator.org/mantis – thanks!

5. Quick Links « web3d-blog - September 16, 2009

[…] Introducing OpenSim Inventory Archives (Justin Clark-Casey) Title say it […]

6. Tony - September 16, 2009

@ justincc
you are doing it wrong, try
save iar Laurent Akina AkinaInventory

and then in your OpenSim folder it will be saved as

user-inventory_iar.tar.gz

i then just rename it tony something like tony.iar then can load it on another opensim real easy.

Thanks to justinc for this, it really is a great addition to opensim.

7. MetaNews Daily – 108 – 15th of September 2009 / Rezzed.TV - September 16, 2009

[…] Introducing OpenSim Inventory Archives (IARs) […]

8. justincc - September 16, 2009

@Tony – it looks like you are using the load and save iar commands in an older version of OpenSim (0.6.6 or previous). The command format is slightly different but they may work just fine.

9. Tony - September 17, 2009

oops sorry!
I do have version 0.6.6 yes but thats all i can see on the website 😦

10. This Week in OpenSim Dev - week ending Saturday 19th September 2009 | justincc.org - September 21, 2009

[…] made save and load Inventory Archive (IAR) commands available to users.  “save iar” on the region console saves user inventory, […]

11. Preserving content creator credit in OpenSim IAR transfers | justincc.org - September 25, 2009

[…] week I blogged about the new experimental Inventory Archive (IAR) format in OpenSim.  This week I want to explore how much content creator information it preserves and how […]

12. OpenSim versão 0.6.7 « Estrutura OPEN - October 15, 2009

[…] Foram introduzidos de forma experimental “arquivos de Inventario” (IAR). […]

13. Karen Palen - December 10, 2009

Just tried out your IAR and it works great with OpenSim Diva-r11651, Linux/Ubuntu 9.10!

A couple of minor problems:

1) If the syntax of the load iar or save iar is incorrect the command line never gets onto the Gnome Terminal scrollback stack. This is not a feature in Windows command line mode, but it is a very useful feature for someone like me who has to make several tries to get the entire line typo free!

2) The save iar identified 12 files with missing components, but did not appear to name them anywhere! I suspect that these were objects in which I accidentally deleted the texture files (OOPS!). The Clothing directory which I dumped already has 440 files in it, so it sure would be nice to identify any bad files so they can be fixed.

14. justincc - December 10, 2009

@Karen – many thanks for the feedback 🙂 Just to echo the reply I gave on the mailing list..

1) Yep, this is an issue that’s annoyed me too 🙂 I’m sure this could be tweaked so that exception causing commands are still saved in command history. Of course, if anyone else wants to make a patch… 🙂

2) Good idea. Unfortunately, it’s not a straightforward fix since only missing asset uuids are currently saved but it would be a helpful thing to have.

15. WhiteStar Magic - January 4, 2010

Some further thoughts on IAR to ponder.

What I see as outlying issues: (thinking out loud)

A) Is the attachment / dependency to the scene.
– A method of generating the IAR without having to have the avatar standing on the region… IE a Late Night Backup could be run from Console Command / Script such as command-script IARbackups.txt which could have A, B, C avatar Inventories backed up to IAR ??? without them being parked on the region.

B) A method by which an IAR can be password locked, so if Joe Avatar can do an IAR on a designated server they should be able to password it. Maybe an osFunction that would grab the In-World userid/UUID and request a password on touch and excecute the dump ? Optionally request a password for the actual IAR file itself.

C) a possible method to transfer said generate IAR from the server to another system by some means, either website or archive system. That could be ugly and likely not possible as they can get mighty big might fast. But the idea of transfering teh IAR to another designated destination/path like a Cloud Storage system or other service/server may be a good avenue for maintaining an archive/backup.

I think this is essential in large part, as Inventory and Assets are a key point issue in this environment and we cannot depend on the potential of a single point of failure. We need to have backups and we need our stuff.

In Closing, with the pending changes to Core, I feel the time is ripe to get a solid and stable way of using IAR for people. I am now generating IAR’s regularly to protect my materials.

Thanks for Everything your doing Justin ! 150% Goodness!
WhiteStar

16. justincc - January 4, 2010

@WhiteStar

it’s possible to do A) already but it isn’t supported in the code due to various security concerns.

B) could be done by scripting if suitable os functions were exposed

C) I need to implement the same IAR web installation as for OARs. Uploading would be pretty neat as well, I agree.

17. WhiteStar Magic - January 5, 2010

Thanks for the response Justin.

In Regards to Item A) If I have My Password and my Alts Password then standing on a region to backup inventory should be no issue. The simple fact of having the password should be sufficent cause without the password you get no access to Avatar-X Inventory.

Item B) would be a good Candidate for the proposed llTextBox function should it be available for capturing the requested info. The one osFunction I could think of, I would not want to have Pub Access for, you know the one… Maybe a subset of that ? particularly aimed at ONLY IAR generation ?? Just thinking out loud.

Item C) Given recent events in OSG Inventory … no more needs to be said I guess.

Thanks Justin, Hope your feeling better.
WS

18. WhiteStar Magic - January 6, 2010

Sorry, something else to Ponder in regards to IAR.

IAR would be an excellent tool for bringing in Items that a person has obtained.

Case Scenario: A Web Based Service similar to the old SLX type of systems. If a person could get an OAR and build a region, or get an IAR for an inventory item this would serve a HUGE Audience of people.

IMO, would there be a way to facilitate the Importing of an IAR from an external area such as a Local HDD or WebSite Storage, much like how OARS can be pulled from a website ? Of course this would require some type of extended function to accomplish but I think it would be a major PLUS overall.

Thanks in Advance for the consideration
WS

19. justincc - January 7, 2010

@WhiteStar – Yes, this kind of IAR import from web is not complicated to do and it is something that I hope to put in soon(ish).

20. Andre - April 18, 2010

i had big trouble with loading iar’s whats the best way when you have a “clean” version of opensim (with nothing in inventory)

thanks

21. justincc - April 18, 2010

@Andrew

load iar Andre Doe / PASSWORD my-great-items.iar

should work fine (will load your iar straight to your “My Inventory” folder).

22. Jorel - July 7, 2010

Hello there,

Working with 0.7 I noticed that in GRID mode we can use save IAR / OAR in server side (not robust btw), since that mode will allow anyone (we want or maybe not) to connect to our grid and do such backups, these functions seems to be the perfect way to steal content.

So, here are my questions and suggestions:

Is there a way to keep our content safe from other servers copying it?

Is possible to add a command to Robust that will disable save iar/oar for specific IPs or all, and be able to use save commands from robust?

Will that command save only content created by the user or fullperm?, (can be hacked editing opensim source?)

regards,
Jorel

23. WhiteStar - July 20, 2010

Hi Justin,

Due to recent issues with Inventory again, I am twigging you about my previous comments from January of this yr but also wondering about the following.

Would it be possible to add a “Notifier” into the IAR Module which, when IAR is completed being written, it sends an “alert completed @ TIME-DATE” or Possibly have it send a MSG on a Negative Channel (like region ready does) so an In-World Script can “hear it” and act on it for the user.

I’m looking at Coding up an In-World tool where a user can go and have an IAR Generated and then have the server post the IAR to a WebFileService where it can be retrieved from by the user.

I’m also wondering if there is a limitation as to concurrency with IAR Generation… Is it possible to generate an IAR for Avatar-B while Avatar-A is having one generated ? (presently is seems not to be the case), so I assume that this is One Person at a time, is this correct ??

Thanks in Advance for your patience.
WhiteStar

24. justincc - July 20, 2010

@Whitestar – There is an event called OnInventoryArchiveSaved on IInventoryArchiverModule that will notify a region module on save completion. It should be possible to use a region module to post these events on to scripts, though the implementation may be a little involved.

And you’re right – at the moment IAR generation is synchronous, unlike OAR generation which is synchronous. I do need to change this at some point.

As for other IAR issues, at some point I now plan to limit the number of simultaneous asset requests that an IAR/OAR can generate in order to alleviate loading problem.

25. justincc - July 20, 2010

@Jorel – Due to fundamental architectural constraints, it is not currently possible to properly secure inventory access from region servers – this is not an IAR specific issue. In the case of IARs, there are some safeguards but it is possible for the technically inclined to circumvent them.

If you want to keep your content absolutely safe on the server side then you have to trust every region simulator that connects to your grid services. Of course, this still leaves the way open for copying via the client connection (a la copybot).

IARs are effectively full-perm since anybody could open the save file and change the contents.

I would say that IP restrictions on who can connect to your grid services should be done at the firewall level rather than by extra service facilities.

26. Introduccion a OpenSim Inventory Archives (IARs) | justincc.org | SoftwareLibre para pobres - March 28, 2011

[…] Introducing OpenSim Inventory Archives (IARs) | justincc.org. […]

27. Jigs - April 23, 2011

I tried about every combination of load iar I could and I only get this error:

Could not find file “/Users/macpro/opensim/bin/user-inventory_iar.tar.gz

Any idea what the problem is?

28. justincc - April 25, 2011

@Jigs – It looks like you’re using a very old version of OpenSim since the default IAR name hasn’t been user-inventory_iar.tar.gz for a long time (it’s now user-inventory.iar).

Anyway, in your case it looks like you aren’t giving “load iar” a filename to load, so it’s looking for a file with the default IAR filename which isn’t there.

29. Sim-on-a-stick: Backup der Region und es Inventory « Dings Digital's Diary - December 23, 2012

[…] habe ich dazu noch nicht gefunden. Fürs Erste darum hier der Eintrag im OpenSim-Wiki und ein paar Hinweise von Justin Clark-Casey aus dem Jahr […]

30. Sim-on-a-stick: Backup der Region und des Inventory « Dings Digital's Diary - December 23, 2012

[…] habe ich dazu noch nicht gefunden. Fürs Erste darum hier der Eintrag im OpenSim-Wiki und ein paar Hinweise von Justin Clark-Casey aus dem Jahr […]


Leave a reply to justincc Cancel reply