 |
NG Design Document/Viewer Architecture/Asset Architecture
From Rex community wiki
Asset Architecture
To create visually striking content in a virtual 3D world, the authors need to work with a number of resources of different types and purposes. The amount of computer memory required is considerable for even small world sizes, which necessitates building a system that handles these assets in a robust and flexible way. OpenSim features a method for handling assets, on top which further development will base on. Based on the experience in working on the previous realXtend project, continuing asset system development should focus on the following features:
- Extensible Asset Types. OpenSim defines its own set of data types and reX defines lots more. We shouldn't assume that there would exist some fixed "maximal" set of data types that would suffice for everyone. Custom 3rd party modules might want to define their own. For example, an AI module might want to store its own pathfinding data or trained neural net data as an asset. Or perhaps a custom HUD system wants to save its window layouts in a format of its own, or a 3D VW marketplace might want to store an SQL dataset of item prices. We shouldn't resort to a "blob/other" or "xml/other" -way of storing custom data, but provide a "strongly typed" asset system that allows anyone to define their own types. Therefore, e.g. a naive mapping of integer IDs to a predefined set of asset types would be a big inconvenience.
- Asset References. This requirement is a bit technical in nature, but extremely important. It is listed here, because the current OpenSim server has a glaring lack in this respect. In one way or another, we need to have a support for assets to refer to other assets. This issue is described below in its own section.
- Easy to manage. Currently it is very burdensome to work with assets in an OpenSim world. This is at least because of:
- The old viewer doesn't support a proper interface or tools for doing so. This is elaborated in a later paragraph.
- The problems with the two points mentioned above. OpenSim asset design follows the "immutable assets" principle, which is difficult when you have references to assets.
- The server has a nonextensible world model. It doesn't support adding objects of custom types into the scene, for example.
Easy-to-manage also means that content needs to be easily transferrable between worlds, and between the server and the client. No information should be lost in progress.
- No inaccessible data. (Or "no wasted effort.") When a user has created a set of objects in a world, he should be able to export all of that to his own offline inventory as it was and import it to another world without any extra work. Currently there are lots of inaccessible data in the world that cannot be extracted, and you can't follow a practice of first creating objects in a sandbox world and then migrating them to the real one.
- Rights to assets. Assets should be associated with the information of ownership and modification rights.
- Seamless online asset editing. When an asset is modified, the result is instantly shown in the world. This allows quick iteration and also lets several people collaboratively modify the world. The amount of offline content creation that is required should be minimized, as this pushes the users away from the world.
- Aim for stateless editing. The system should minimize the need for the user to transition between edit states, like "enable/disable world/geometry/texture/etc.. edit mode".
Asset References
In the current OpenSim system, assets don't contain references to other assets. This is problematic, because the data the world consists of is referential by nature. This is illustrated by the following diagram, which shows an ideal breakdown of world data into separate asset types.
Diagram 1.
The diagram contains an example of how an imaginary house in a virtual world would be composed of assets of different types. The house contains some static geometry (walls, floor, roof and a fireplace), that is baked into one mesh. There's an openable, moving door, a physically movable table and two chairs, a particle simulation representing the fireplace. Finally, in the ceiling there's a light that automatically switches on and off depending on whether there are any people inside the house. This whole collection has been grouped into a single container called "House".
From a technical perspective, each block denotes an asset of its own. An asset has a name and below it there is a label that denotes its type. The blocks are also color-coded for clarity. An arrow between two assets marks that there is a referential dependency between the assets. This example shows that even a small "self-standing" object in the world requires quite a few pieces of data to represent it correctly, and what's more important, there are lots of references between different assets. The OpenSim asset system does not support this notion, and so modeling these dependencies in the previous realXtend project was a major difficulty.
A proper asset system needs a specification of how references are described. This is to say, even if an external 3rd party developer implements his own asset type (, the contents of which of course are not understood by the server or a nonsupporting client), the asset system should still be aware of what other assets that unknown asset refers to.
Instance Data and Asset Data
Another concept that is useful to introduce at this point is the one of world "instance" data versus world "asset" data. This is related to one of the issues where OpenSim falls short.
Instance data refers to the pieces of information stored in the current active world scene hierarchy of the server, i.e. the currently instantiated entities along with their components and all the parameters that are related to them. These are for example the position/orientation data for the component "Placeable", or say, the health and armor status for the component "EnemyGoblin".
Asset data refers to information that is not typically copied to the world several times. The data is integrally existing in the world, but is not associated with a physical position in the scene, and so is not a dynamically active part of it. This creates a feeling that asset data is static information, something that's nonmodifiable.
The above easily leads to a bad conclusion. Since asset data is static, instance data must be dynamic, and hence anything that is dynamic is not to be considered an asset. This is how the current OpenSim model is like. For authoring a collection of entities like shown in Diagram 1, this is a problem, since anything that is not an asset understood by the architecture, needs to have a special import/export functionality written in. To allow smooth moving of large amounts of data, the client software needs to support batch imports and exports, or a "resource creation playback" feature of sorts, where the client, one object at a time, creates it in the world, submits assets and connects the dependencies between the new assets. On the long run, this leads to two tiers of data - the "higher-class citizens" of data, information that is managed via the asset architecture, and a "lower-class citizens" of data, information that is only communicated through ad-hoc import/export interfaces.
This description leads to the following thesis: For each piece of information that can exist in the form of instance data, there must exist an asset type that can store the information. This helps achieve the "No wasted effort" requirement, because it gives the users enough flexibility that they can move any part of data they create between worlds, and any piece of data is offloadable to the client computer as well.
Easy Management
The two previous sections are partial reasons to why world management is difficult in an OpenSim world. With only a burdensome import/export style of managing data around, the user needs to perform lots of tedious tasks to set the world up how he wants. The user should have a powerful command over the objects he can manage, with features like the following:
- Selection and filtering aids. Involves tasks like:
- Ability to select not only objects, but components of objects.
- Save a list of selections. Have multiple lists of selections.
- Select all objects in my view.
- Select all textures/materials used by the objects I've currently selected.
- Find all scripts/lights/entities near me.
- Find all objects/materials using a given texture.
- View a graph of object dependencies, much like depicted in Diagram 1.
- Edit multiple. Should be able to work on several objects simultaneously without difficulty. Involves tasks like:
- Transform multiple.
- Set the mesh/material/texture/effect referenced by all objects in current selection.
- Modify multiple materials/entities/components/lights.
- Shallow clone selected entities/assets/components. For example, for a entity with a mesh, clones the entity but will share the mesh component.
- Deep clone. Duplicates the selection and the whole hierarchy of data referenced by it.
- No chores. I shouldn't have to repeat myself. For example this means that for skinned (animated) objects, I shouldn't need to tick any checkboxes after I've added the animsequence (no need to synchronize redundant data to match). Also, I shouldn't be bothered with having to do any tricks because of architecture restrictions to get things working. This means no pressing F5 to refresh an unsynced view, and no manually updating of references if I "modify" an immutable object (see the next chapter).
About the immutability of objects
The current OpenSim server and the Cable Beach asset architecture both follow the "immutable objects" principle. This means that after a new asset is created, it cannot be modified at all. If the user wants to alter an asset, he has to submit the modifications as a new asset, and change all the references to the old asset to point to the new asset. To hide the user the complexity of doing this manually, the user interface should be modelled so that it transparently supports modifications and performs this cloning in the background.
The benefits of immutability are the following:
- Easy to implement caching. An asset cache doesn't need to have any update mechanisms, because updates don't occur. The cache just adds new items and drops oldest or unused entries when it is full.
- No need for update notifications in the wire. There is no need to implement any AssetUpdate (or TextureUpdate, GeometryModify, MaterialChanged, etc..) messages since those won't occur (in a push model). Only CreateAsset -messages need to be submitted. This also means that if a pull model is employed, there is no need for polling for possible asset updates.
- Easier to decouple asset storage. As a consequence of the above, it is easier to move the asset system to another server, since the required protocol message set is smaller.
The drawbacks of immutability:
- Added complexity to support transparent modifications. We won't need to implement asset updates, but instead we need to write logic for behind-the-scenes asset duplication to allow transparent modifications. This gets difficult when the user does N modifications in sequence to one asset, since we would prefer to batch those N changes and apply them all at once. This makes the UI heavier, possibly more stateful (i.e., an edit mode, which is finished by a commit operation).
- Inconvenient with asset references. When there is a long chain of references between assets, modifying the leaf asset will result in duplication of the whole parent chain. This is because the asset references are an integral part of an asset as well, and to change the reference of a parent to a new child, a new asset has to be submitted. It was considered that perhaps this could be overcome with the convention that perhaps asset references could be mutable metadata of an otherwise immutable asset, but this breaks caching and questions the usability of immutability in the first place.
- Divides a line between static and dynamic. Assets like Texture, Geometry and Audio are 99% static in our use cases (realXtend is not Paint, 3DSMax or Audacity). Lots of other assets on the other hand (Script, Mesh, Entity, Light, Material, ParticleSystem, Avatar, ...) are dynamic by nature, i.e., they are modified in the world itself. Immutable assets -policy (along with a distributed asset server architecture) discourages saving Instance Data or collections of assets like Diagram 1 to an asset server, because modifying them becomes distant and heavy. As a result overcoming the "No wasted effort" is difficult.
Different asset architectures
A number of different proposals have been brought up for the use of an overall asset architecture. The different architectures are described here along with a comparison.
Centralized asset architecture
This is the current OpenSim model. All assets exist on the sim server and they are managed by logging in to the world with an avatar and performing asset imports and exports and by modifying the prims in the world.
Diagram 2.
This model is depicted in Diagram 2.
Pros
- Asset server "knows" all the assets. It can do physics simulations and queries easily since it's aware of all the data in the world.
- Synchronizing client views is easy. The sim server can send update notifications to all clients whenever an asset changes. This allows instantaneous collaborative editing, for instance.
- No need for immutable assets. Since updates are easy, there's no need really to limit to the Immutable Assets policy - it just doesn't give any big benefits. Note that OpenSim does this, probably just for making the implementation easier.
- Robust inventory handling. While OpenSim's concept of an inventory is a bit different (it's just a list of shortcuts), in a centralized system the sim server could easily contain arbitrary assets, like in Diagram 1. Since all assets are on the same sim server, there is no need to distinguish between shallow and deep asset storage. Shallow references are always safe and preferred, because the sim sees all data and can at all times verify that the references are valid.
Cons
- No asset access outside the sim. Assets cannot be accessed without logging to the sim server. The sim owns its assets and they cannot be shared between sims.
- Server-to-Client performance. If there are lots of clients requesting assets, the server upload bandwidth gets heavily stressed, which interferes the basic movement and action communications. Note that current OpenSim+SL has performance problems on the client end as well regarding the asset downloads, but that is due to engineering limitations (no throttling for UDP, image decoding consumes CPU) rather than by a "theoretical" hardware/wire communication limit.
Note that while this is how the current OpenSim architecture works, the asset system it implements is currently not flexible enough for allowing custom asset types, or asset references.
Distributed asset architecture
The Cable Beach project led by Intel proposes a following system.
Diagram 3.
To solve the downsides of a centralized asset architecture (asset sharing and performance), all asset management is offloaded to a two-stage server model. This is shown in Diagram 3. The Cable Beach metadata server stores all the metadata information about all the assets it knows and the actual assets are scattered to backend web servers utilizing for example standard HTTP-based data transfer. The model is depicted in Diagram 3. There are several clients who are connected each to a sim server of their own. A sim server references N Cable Beach asset metadata servers, which refer to multiple backend asset servers.
This model is further described in
Pros
- Offline access to assets. User doesn't have to log on to the sim to load assets to his own computer. The user can upload new assets to web without going through the sim, but to actually use the asset in the world, he needs to log in and build scene objects that refer to the assets.
- Less stressed server bandwidth. The client requests assets from external servers instead of the sim server, so the sim server needs less bandwidth to operate.
Cons
- Sim server becomes a client of the asset servers. To know about e.g. geometry data for proper physics simulation, the sim server needs to obtain the asset data itself from the asset servers. It is difficult to estimate what other asset types the server might be interested in, but it should be planned against that there could be multiple.
- Mutable assets are hard. Dynamic data is hard. Changing existing assets on an asset server requires that the asset server notifies the sim, which notifies the clients, or that the clients poll the asset servers, or that the asset server notifies clients, or a hybrid of these approaches. With a many-to-many-to-many mapping like in Diagram 3, this can become complicated. Therefore the easiest design decision is to lock down to immutable assets and try to solve the problems that lie there.
- No central UUID authority. With several asset servers, there could be random collisions, which is not a big catastrophy. The issue is that there will certainly be malevolently caused collisions for hacking purposes. This can probably be defended against by using white- & blacklisting, but the issue needs to be noted.
- Inventory?. Cable Beach seeks to store the personal inventory on the asset server as well. This work raises several questions. Would this mean that each user profile has its personal asset server so that the inventories of users persist when he logs in to different worlds? Then, it has to be noted that the set of asset servers that a sim uses is dynamic and partially determined by the currently logged-on users. Or, does the sim have one central asset server that it uses for all user inventories? Then, it has to be noted that there will be a 'master' asset server and 'secondary' asset servers of some sorts. It was hoped that the OpenSim inventory could become a real container for actual assets and scene objects, so that they could contain 'EntityBags' (copies of scene objects exported to a binary file for easy management), instead of being just a bookmark list like it now is. How this distributed system would allow this nicely, is not immediately clear.
A hybrid model
Both models described above fail in important ways. To allow mutable assets with easy modifications and asset references, but also to allow decoupling assets from the sim in order to reuse them in several sims, perhaps the best way would be to fuse both systems.
Diagram 4.
This design is presented in Diagram 4. In this design, the sim server holds assets on its own and there is no external layer of asset servers to store metadata about particular assets. The sim holds all the assets and the user inventories, but some of the assets may not be contained on the server, but just be shallow references to external sources.
Pros
- From the centralized architecture: 'Asset server "knows" all the assets', 'Synchronizing client views is easy', 'No need for immutable assets', 'Robust inventory handling'.
- From the distributed architecture: 'Offline access to assets'.
- Partially less stressed server bandwidth. Lots of big assets like textures and audio can be offloaded to external servers where applicable. It should be quite easy to offload enough (static) asset data so that the server is not spending too much bandwidth on serving asset data requests. Utopistically, perhaps the server can autonomously manage this offloading behind-the-scenes, but this can be difficult to achieve.
- Easy to implement. Curiously, this model seems to be easier to implement than proposal #2, but still allows the same benefits than #1 and #2. Instead of building a full-blown all-or-nothing offloading of assets to an external server, which is hard for dynamicity, we'll just contend to offloading the static immutable-style asset types, which fortunately are the ones that take up most of the data. Additionally, since the sim server is the metadata server, there is no need for a separate push/pull -model for managing synchronized client views.
Cons
- Not transparent to user. The user really has to know where he puts his assets, unless we implement an automatic offload system (yeah, right..). The good thing though is that he does have the flexibility to choose both, and can manage between the various pros and cons of each.
In overall, attaining the required level of decentralization to the system doesn't need much more than the MediaURL feature we already have built-in in the old realXtend viewer, although it should be extended for other types than textures and web pages as well. Using that feature one can offload all textures, audio and video from the sim server, which is already a major part of the whole bandwidth. Geometry data might also be offloadable (especially non-physics-simulatable), but we should avoid having to make the sim server a client of an asset server itself. Asset types like Materials, Entities, Lights, Particle Systems, Scripts, are such an integral part of the dynamic world that they should at all costs be kept close to the sim for easy managing.
This model can be seen as being the equivalent of the distributed asset architecture, but the sim server acts as the metadata server, and supports the use of several backend servers.
Asset Caching
To reduce amount of repeated, unnecessary network bandwidth usage, and to speed up loading of virtual worlds on subsequent visits, assets should be cached by the viewer as they are retrieved from the network.
The asset retrieval and caching system should be extensible, so that it can handle new kinds of assets as they are invented, and new ways to retrieve assets. Presently at least 3 ways of asset retrieval can be identified:
- Legacy OpenSim asset transfer through the UDP pipeline.
- Distributed asset handling through http - using the Cable Beach server for instance.
- Application-specific pre-packaged assets - may be installed from a CD/DVD for example, and be large in size.
Asset identification
The traditional OpenSim / realXtend way of identifying assets is assigning each asset an unique 128-bit identifier, UUID. However, when a distributed asset system is used, the URL of the asset server is needed in addition to the UUID. Additionally, we may want to support content-based asset identification, for example a SHA-1 hash based identifier calculated of the asset contents.
Future Work
Building towards a hybrid model for server-side asset handling is what is currently desired. This is however part of a bigger project that involves the OpenSim community as well. The possible rework for an asset system goes hand in hand with a rework that is desired for the scene model. If we want to really make use of a component-based scene design, we need to have the server adhere to the same scene model, which requires a lot of collaboration with the OpenSim community.
References
|
|
