Open Source Platform
for interconnected virtual worlds

NG Design Document/Viewer Architecture/User Interface

From Rex community wiki

1 Introduction
2 Input
3 Input use cases
- 3.1 Initializing an input plugin
- 3.2 Receiving input
4 User interface
5 User interface use cases
6 Compositing user interface and renderer output
7 2D user interface toolkit choices
- 7.1 Conclusions
8 UML diagrams
9 Citations

Introduction

The user interface (UI) is a critical part of the new viewer: it will determine the viewer look-and-feel and how the users will interact with it.

To support different kinds of user interfaces in addition to a traditional 2D one, such as 3D or just simple heads-up-displays for games, the UI also needs to be implemented as a plugin module.

It should however be noted that for another module to do anything useful with the UI plugin, it must take a rather heavy dependency to it, and therefore swapping UI plugins will be a lot of work, unless a general "least-common-denominator" approach is taken with the UI plugin API, which is not very attractive either.

To begin with development work and to get something with a full set of UI widgets running quickly, an established 2D UI toolkit such as Qt [1] will be used.

Receiving user input (keyboard, mouse, gamepad etc.) is a distinct function from maintaining the state of the user interface and displaying it graphically, and therefore can be implemented as a separate plugin. Though user interface libraries usually can also gather user input by themselves, a flexible user interface library should also allow input to be injected from outside sources. A good example of a functional crossplatform input library is OIS (Object-Oriented Input System) [2].

In the following text, the roles of the input plugin and the actual user interface will be handled separately.

Input

An input plugin will watch the input devices, in the typical case:

Keyboard
Mouse
Gamepad

and send events of the input received from them (mouse move, mouse button press, key press etc.) to be received by the other modules.

Because of the viewer framework handling events in a hierarchical manner (a module will always get a chance to handle events before its child modules) the handling of the input events may differ based on the state the viewer is in: for example moving the avatar vs. typing into a chat interface.

For nonstandard input devices, the input plugin may either send custom events, which the rest of the modules have to understand to be useful, or they can be mapped to "standard" input events.

Input use cases

Initializing an input plugin

A plugin for input device X is loaded.
The plugin checks for presence of the input device.
The device is found, so a corresponding InputDevice object is created, which will proceed to monitor the device's state and send input events.

Receiving input

The user moves the mouse wheel.
The InputDevice that corresponds to the mouse sends a mouse wheel event.

User interface

The user interface plugin implements the state, layout and graphical rendering of the actual user interface. If necessary (if the input plugin is separate), it receives events from the input plugin to know which parts of it are being manipulated. In turn, the user interface sends events, for example button clicks.

The user interface layout, look-and-feel and functionality should be easily modifiable via configuration files, for example in XML format. As little as possible should be hardcoded. Other modules can define their own windows and layouts, by using the user interface module.

The default 2D user interface plugin should support standard 2D UI widgets and elements, like windows, buttons, checkboxes, texture images, single- and multi-line text editors, tab windows and scrollbars.

User interface use cases

Initializing the default user interface layout

The user interface plugin is loaded and initializes itself.
It loads the UI layout and look-and-feel descriptions from XML file(s).
The initial set of widgets corresponding to the layout is created.

Instantiating a user interface layout for a world editing tool

The user activates a material editor tool, via a menu for example.
The material editor activates and instructs the user interface module to create a UI layout for itself from its own XML file.
The material editor UI is displayed.

Clicking a button widget

The user interface receives a mouse click event sent from the input plugin.
The user interface notices that the mouse click happened on top of a button widget.
The button widget sends an UI action event corresponding to the button click.

Compositing user interface and renderer output

Because both the rendering engine and the user interface exist as separate plugins, there has to be a way of coordinating how the graphical outputs of both will be combined. Several methods exist:

Rendering window embedded as a user interface widget: this is the way a rendering engine such as OGRE can be easily embedded into Qt or GTK+. It works, but does not for example allow transparency effects on widgets on top of the rendering window on all operating systems.

Direct composition in the frame buffer: in this method the rendering engine draws its output first into the frame buffer, then the depth buffer is cleared, so that the user interface can perform its own rendering on top. This requires that the user interface uses the same rendering API (for example Direct3D or OpenGL) as the rendering engine.

Texture compositing: the output of either the rendering engine or the user interface is rendered into a texture, and the other plugin will composite the final output using the texture. Depending on the way this is done, it may lead into performance loss due to the blitting overhead.

2D user interface toolkit choices

Qt [1] and GTK+ [3] were investigated as choices for the 2D user interface toolkit used by the default 2D user interface plugin.

Qt has an object-oriented interface out of the box. GTK+ is procedural instead, but through the wrapper library GTKmm [4] it too can be used in an object-oriented manner. Qt requires a Meta-Object Compiler to be run on widget classes, producing an additional generated C++ file: GTKmm has no such requirement.

GTK+ has a LGPL license. From version 4.5 onwards Qt will also be licensed through LGPL, making it a suitable choice for the new viewer. Previously only commercial and GPL open source licensing models were available for Qt.

GTK+ has a larger base footprint of shared libraries compared to Qt.

Both Qt and GTK+ implement their own concept of an application and an application main loop. This could be interpreted as problematic, if the user interface is to be just a plugin among other plugins, while the viewer core framework should retain control of the main loop and update logic. However, it can be made so that the viewer framework can, if necessary, relinquish control to the user interface plugin, and hook its main update function to the plugin.

Conclusions

The initial set of debug UI's was implemented using GTKmm, but compositing the user interface and the renderer output proved to be a problem. The 3D world view could have been embedded as one widget in the viewer main window, but then floating tool widgets above the world view would have been impractical to implement.

Therefore Qt was taken into use. Qt widgets can be rendered into system memory buffers, from where they can be blitted into GPU textures and therefore be composited in a flexible manner by the rendering module, using the texture compositing method described in the previous section. Additionally, Qt accepts injected input from outside sources, allowing a separate input plugin.