(was "Wonderland Testing 101: An Introductory Guide to Testing")

By Nicole Yankelovich (nicole@openwonderland.org)

This guide provides details about how to collect data essential for both troubleshooting and testing.

Troubleshooting#

When you run into a problem with Open Wonderland, the most effective way to get help is to provide the development team with data. The Data Collection section below explains a number of different ways to collect data. The best technique to use depends on your level of technical expertise and also on the type of problem. For example, if your Wonderland client hangs such that you can no longer use the menus, then it will not be possible to generate an error report using the in-world tool. Instead, you will need to capture information from the Java Console or from a Trace file. If you need help figuring out how to collect data for your given problem, the instructions below explain how to post a message to the Open Wonderland forum.

Posting Questions to the Open Wonderland Forum#

Open Wonderland is a welcoming open source community that invites all users to actively participate on the discussion forum:

http://groups.google.com/group/openwonderland

If you need any help figuring out how to troubleshoot a problem, what data to collect, or how to try to reproduce a problem, please do not hesitate to post a question. To be able to post questions using the web or by sending email (openwonderland@googlegroups.com), you will first need to create a Google account, if you don't already have one, and then join the group by clicking on the "Join this group" link in the right-hand column of the forum web page.

When you post a question, please include the following details:

• Operating system (Windows Vista, Windows 7, Mac OS, Ubuntu Linux, etc.)
• Make and model of your graphics card
• Type of network (home cable, home DSL, corporate network with firewall, etc.)
• Your level of technical expertise (end user, system administrator, Java developer)

In explaining the problem, try not to use generalities (eg, "Wonderland didn't work for me"). Instead, be as detailed as possible (eg, "I clicked the Login button and a window opened, but the window never showed the virtual world. It stayed black").

The best way to get help is to follow the Data Collection instructions below. The more data you can provide with your question, the better able we will be to help you quickly resolve the problem.

Things to Try First#

Here are a few steps to try when having trouble getting Wonderland to work.

Update Your Graphics Card Driver#

If you can log in to Wonderland, but the world crashes immediately or the graphics don't look correct, the first thing to try is updating your graphics card driver. How you do this is specific to each computer. Most computer manufacturers provide free graphics card driver updates on their support web site along with instructions on how to do the update.

Check Your Bandwidth#

Open Wonderland requires a reasonably good Internet connection to run. In particular, audio will sound choppy if you do not have adequate bandwidth. There are a number of free web sites that allow you to test the speed of your Internet connection. Here's one that is easy to use:

• SpeedTest.net

Simply click on the "Begin Test" button (and not on any other button - most are ads). If you're not at least in the middle of the blue range, you many not have a fast enough connection to have a good Wonderland experience.

Run the Client Test#

On the Open Wonderland community server launch page http://owf1.openwonderland.org:8080, you will find a green "Test Client" button towards the bottom of the page. This test application should give you a good idea if your computer will be able to run Wonderland. If there are problems, it will indicate what issues you need to address.

If the Test Client itself won't run, that indicates that you need to install or update your version of Java. Do this by clicking on the "Get Java..." link under the orange Launch button.

Clear Your Caches#

Wonderland makes extensive use of "caches" - data stored on your computer so that it does not have to be downloaded from the server each time. Sometimes data in one of these caches can be bad or corrupt, perhaps because of a network glitch that caused only a partial file to get stored. The types of problems you'll see if this happens are Wonderland starting up and only loading half a screen. Sometimes you might see "missing texture" messages overlaid on objects in the world.

There are two caches that you can try clearing to solve these type of problems. First is the Open Wonderland cache. To clear this cache:

• Click the Launch button to start Wonderland.
• When you get to the Open Wonderland login dialog, click on the "Advanced..." button.
• Then click on the "Cache" tab.
• Click "Clear Cache."
• Click on the "Ok" button.
• The operation may take some time, so be patient - the dialog will go away once the cache is cleared.
• Log in as usual.

Note that it will take longer to log in because Open Wonderland will have to re-download all the 3D assets.

The other cache that might need clearing is the Java Web Start cache. To clear this cache on Windows:

• Open your Control Panel.
• Open the Java Control Panel.
• In the General Tab, Click on the "View..." button to view Temporary Internet Files. The Java Cache Viewer window will open.
• Click on each "Open Wonderland..." entry and click the red "X" to remove the file.
• Close the Java Cache Viewer Window.
• Click "OK" to close the Java Control Panel.
• Log in to Wonderland as usual.

To clear the Java Web Start Cache on a Mac:

• Open your Applications folder.
• Open the Utilities subfolder.
• Double click on the Java Preferences app to open it.
• Click on the Network tab.
• Click on the "View Cache Files..." button (this may take a while)
• When the Java Cache Viewer opens, select all of the "Open Wonderland..." entries and click on the red "X" button to delete all these files.
• Close the Cache Viewer window and the Java Preferences window.
• Log in to Wonderland as usual.

As with clearing the Open Wonderland cache, it will take considerably longer to launch the application because Java Web Start will have to re-download all the Open Wonderland code and modules.

Data Collection#

Open Wonderland generates extensive error logs that are critical to identifying problems. If you need to troubleshoot a problem or if you are testing the system, the instructions below explain how to collect meaningful data, with the easiest techniques listed first.

Generate an Error Report In-World (All Users)#

To make it easy to collect troubleshooting or testing data, Open Wonderland provides an in-world error reporting tool. To use the tool:

• Select "Error Log" from the Help menu

• In the Log Viewer window that opens, click on "Error Report"

• In the Error Report window, click on "Submit..."

• A Comments window will open. Type in exactly what you were doing and click "Submit."

• Let your system administrator know that you have submitted an error report. These reports will appear in the "Error Reports" section of the Server Admin Console.

Generate an Error Report Using the Server Admin Console (System Admins)#

Sometimes if a client is hung, it is not possible to capture an error report in-world as described above. Also, sometimes it is simply easier for a system administrator to capture the error report rather than talk a user through the steps. You can easily capture an error report for any logged in user from the Wonderland Server Admin console:

• On the Wonderland launch page, click the "Server Admin" button
• Log in as the "admin" user using your admin password
• Click on "Connected Users" in the left navigation panel
• Each logged in user should be listed on this page.
• Click on "error report" in the Actions column associated with the person having an issue
• In the Comments dialog that opens, type in an explanation of the problem and then click "Submit"
• The Error Reports page will open with the most recent report at the top of the list
• Use the links in the Action column to look at or download the error report
• You can then email this report to your technical team or to the the Open Wonderland Forum (openwonderland@googlegroups.com)

Java Console (All Users)#

Sometimes it is more convenient to have Java open up a console window automatically every time you run Wonderland.

Showing Java Console on Windows

• Make sure Wonderland is not running
• From the Start menu, select Control Panel
• In the Control Panel, click on Java
• In the Java Control Panel, click on the Advanced tab
• Click on the + before Debugging and select "Enable logging" and "Enable tracing"
• Click on the + before Java console and click on Show console
• Click OK

Showing Java Console on Mac OS

• Make sure Wonderland is not running
• Navigate to Applications --> Utilities --> Java
• Open Java Preferences
• Click the Advanced tab
• Under the Debugging heading, check "Enable logging" and "Enable tracing"
• Under the Java console heading, click Show console
• Click Save and close the Java Preference dialog

Showing Java Console on Linux

• Make sure Wonderland is not running
• Navigate to Applications --> Settings --> Sun Java 6 Plugin Control Panel
• Click the Advanced tab
• Click the toggle before "Debugging" and select "Enable logging" and "Enable tracing"
• Click the toggle before "Java console" and select "Show console"
• Click OK

Note: These Linux instructions assume you have installed the Sun Java 6.0 Runtime (sun-java6-bin) or the Sun Java 6.0 JDK (sun-java6-jdk) packages on Ubuntu. If you didn't use a package manager, the Java Console configuration can be accessed from the Java Web Start application, which you can start from a terminal window:

javaws -viewer

If javaws is not in your default path, try:

/usr/jre/bin/javaws -viewer

or:

/usr/jdk/jre/bin/javaws -viewer

Using the Java Console

You'll only need to configure the Java Console to be visible once. After you do, every time you run a Java program, you will see a Java Console window open that prints logging messages. It should appear at about the same time as the Wonderland launching dialog:

Resize the Console window to make it relatively large so that you can see a good number of messages in the window. When you run into a problem, simply copy the entire log up to the point of the error and include it with your forum post or bug report. It's important to copy the log immediately after the problem occurs so that the error does not get lost in the middle of a very long file. After you copy it, you can continue working, but always copy the entire file for each individual problem found.

Often, errors in the log file can be identified by the word "Exception." Here's an example of a Console log that contains an exception:

Exceptions are sometimes harmless, but typically they are an indication that a problem has occurred.

If your Wonderland client hangs to the point that you are unable to copy the text in the Java Console, which sometimes happens, try taking a screen shot of the Console window instead. The relevant messages are likely to be displayed in the window if you have it sized large enough. If a screen shot is not possible, you can find the same messages in the Java "trace" file stored on your hard disk. Follow the instructions below for how to find the trace file on your system.

Trace Files (All Users)#

Java trace files contain the same information that is printed in the Java Console and in the Error Reports, so typically you will not need these. If, however, something goes wrong that causes you not to be able to open the Error Log or copy the text in the Java Console, you will need to submit the trace file instead. This can happen if Wonderland freezes up or if it crashes completely, taking the Console window down with the rest of the system.

Trace files are only created if you have followed the instructions above and clicked on "Enable tracing" in the Java Preferences dialog. If you have not done that, follow the instructions above for showing the Java Console before proceeding any further. If you did not have tracing enabled, you will, of course, have to reproduce the bug again in order to produce a trace file.

Trace files on Windows

On Windows, you can find the trace files by navigating to the Java Preferences "General" tab.

• From the Start menu, select Control Panel
• In the Control Panel, click on Java
• In the Java Control Panel, make sure the General tab is visible
• In the section "Temporary Internet Files," click on the "Settings..." button
• You will see a location section with a path to the cache directory
• Navigate to that "cache" directory, go up one level, and then navigate down to the "log" directory

In the log directory, trace files will be in the form:

javawsXXXXX.trace

where XXXXX is a long number. For example: javaws23173890315369271.trace. Be sure to sort the contents of the log directory by date to ensure that you are finding the most recent file.

Trace files on Mac OS

On Mac OS, the java trace files are in your home directory in the following sub-directory:

MacintoshHD/Users/<username>/Library/Caches/Java/log/javawsXXXXX.trace

where XXXXX is a long number. For example: javaws23173890315369271.trace.Be sure to sort the contents of the log directory by date to ensure that you are finding the most recent file.

Trace files on Linux

On Linux, the trace file is stored in your home directory:

$HOME/.java/deployment/log/javawsXXXXX.trace

where XXXXX is a long number. For example: javaws23173890315369271.trace. Be sure to sort the contents of the log directory by date to ensure that you are finding the most recent file.

jStack (System Admins, Developers, or Advanced Users)#

Sometimes it helps developers to find a problem if you can capture a "thread dump." Jstack is the Java thread stack trace dump tool. This is part of the Java Development Kit (JDK) which must be installed before you can follow the instructions below. Having the Java Runtime Enviroment (JRE) alone is not sufficient.

Assuming the JDK is installed, run Wonderland on your system, open a terminal, and type

jps

Your output will look something like this:

9955 Main
10092 Jps
9958 001bcb4c4209cff5b7272a4742e82ad596eaaaae
9961 softphone.jar

Copy the number before "Main." This is the Java process ID (pid) of your Wonderland client. Now type the jstack command using the pid obtained from the jps command:

jstack [pid]

You will get a lot of output that looks something like this:

nicolesmac:~ nicoley$ jstack 9955
2011-04-26 15:39:41
Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.1-b02-334 mixed mode):
"Attach Listener" daemon prio=9 tid=130eb0800 nid=0x15660f000 waiting on condition [00000000]
   java.lang.Thread.State: RUNNABLE
"pool-84-thread-1" prio=5 tid=101dd7800 nid=0x14b7fc000 waiting on condition [14b7fb000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <7e316fce8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
	at java.lang.Thread.run(Thread.java:680)

Copy and paste all the output into an email message, forum post, or bug report.

Audio Log (System Admins, Developers, or Advanced Users)#

If you are experiencing an audio problem, the problem can sometimes be determined by examining the audio log file, also known as the SIP Communicator log. The easiest way to get a log file for debugging audio problems is to run Wonderland and then select Tools --> Audio --> Log Audio Problem immediately after the audio problem occurs. This command should give you a dialog telling you that a log file has been created on your desktop. Look on your desktop for a file name like this one:

sipCommunicator.jp_61070.log

Server Logs (System Admins, Developers, or Advanced Users)#

From the web page with the Wonderland "Launch" button, you can access the Server Administration Console by clicking on the gray "Server Admin" button. From the default Manager Server page, you can access four log files (there might be more, depending on how your server is configured):

• Web Administration Server
• Darkstar Server
• Shared Application Server
• Voice Bridge

In all cases, you can access the log files by clicking on the "log" link in the Actions column.

If, for some reason, you cannot access the log files from the web site, these log files are also available from the Wonderland server log directory. The server log directory is in the user's home directory, a subdirectory called ".wonderland-server/<wonderland-version>". For example, for 0.5 development, the server log directory is in '~/.wonderland-server/0.5-dev'.

FINE logging (Systems, Developers, or Advanced Users)#

Often in the course of debugging, a developer will request a more detailed log to track down a problem. Most objects in Wonderland use the standard Java logging utilities described in the Java Logging Overview. By configuring the logging properties for the various Wonderland applications, it is possible to get more detailed logs for a particular object, subsystem, or even the whole application.

Usually, the developer will tell you what objects or packages to turn up logging for. Different Wonderland components use different configuration files to control logging, but all these configuration files use the same syntax (described in the link above). Below are the files relative to the root of the Wonderland workspace for various components:

Wonderland client

In core/src/classes/org/jdesktop/wonderland/client/jme/resources/logging.properties set (for example):

org.jdesktop.wonderland.client.assetmgr.AssetManager.level=FINE

After changing this value, re-run the Wonderland client to see the new logging messages in the Wonderland client output. Note that if you launch the client via Java Web Start, you will need to restart the Wonderland web server using "ant run-server" to redeploy the webstart logging configuration. You will also need to enable the Java Web Start console to see the output, as described above.

Wonderland web server

In web/server/web-logging.properties, set (for example):

org.jdesktop.wonderland.web.server.level=FINEST

After changing this value, re-run "ant run-server" to restart the web server with new logging values. Output will be in the Wonderland server log directory (~/.wonderland-server/0.5-dev/log) in a file named "web_server.log.0". The web server log is also available from the web administration UI.

Darkstar server

In core/src/darkstar_config/sgs-logging.properties, set (for example):

org.jdesktop.wonderland.server.cell.CellMO.level=FINE

After changing this value, re-run "ant run-server" to restart the Wonderland server with new logging values. Output will be in the Wonderland server log directory in a file named "darkstar_server.log". The Darkstar server log is also available from the web administration UI.

Testing#

The remainder of this document describes how you can help to improve the quality of Wonderland by testing the software and reporting bugs. Before you do any testing, it is crucial that you set up your system to collect data as described in the Data Collection section above. What data you need to provide depends on what you are testing and on your level of technical expertise. At a minimum, all testers should turn on their Java Consoles in order to show the messages that the Wonderland client prints out for debugging purposes. More advanced users should review the other types of log data described above.

The following sections begin by describing four types of testing: feature, usability, performance, and scalability testing. If you have not done a lot of testing, particularly in multi-user environments, the section on General Testing Tips provides some guidance on how to approach the testing process. Once you identify bugs or other problems, these should be recorded in the Issue Tracker. Before entering a bug report in the Issue Tracker, it is extremely helpful to have either a log file or a reproducible test case, and ideally both of these. In some cases, when you find a problem, it can help others in the community if you post the problem to the Wonderland Forum. And if you are curious about what happens with the bug reports after you file them, the final section explains the process the core team uses for prioritizing and fixing bugs.

If you don't want to read this whole section, here are the bare essentials:

• Set up your Wonderland environment by connecting to an existing server using Java Web Start, by downloading a binary build, or by building from source
• Be sure you are set up to collect data from the Java Console and from Java trace files
• Familiarize yourself with the Wonderland Server Administration web pages so you can easily find the server-side log files
• Test the features you care about the most and those you believe will be most frequently used
• Write up and share Regression Test Scripts that represent common tasks your users are likely to want to perform
• Be methodical in your testing, testing one feature at a time and adding complexity incrementally
• Test with multiple users as much as possible - we recommend testing in groups of two or three people with periodic tests involving up to 20 people
• Use the Issue Tracker to file bug reports, which should include:
  • Summary of the problem using search-friendly keywords
  • Description of the symptoms
  • A reproducible test case, if possible
  • One or more relevant log files
  • If the problem is related to artwork not loading or rendering properly, a sample art file
  • Information about your computing environment (see Adding Information About Your Computing Environment)
• Post a message to the Wonderland Forum if you find a bug that impacts multiple people or if you need help reproducing a serious problem.

Types of Testing#

There are several types of testing you can help with. We would strongly encourage everyone who is testing to do the first type:

Feature Testing with Regression Scripts

The one we expect most people to be able to help with is feature testing. That is, does the feature work as advertised? For example, let's say you are testing the drag-and-drop feature. You would collect a variety of .png, .jpg, .svg, and .kmz files and try dropping them into a Wonderland window. Then you would give those same files to someone else on a different platform and make sure they all appear properly in the world. An important thing to keep in mind when testing is that most features tend to work fine with a single user (because that's the way the developer does most of their own testing), so always try to test with at least one other person. Following our example, you might be able to see the image or model, but you need to make sure that everyone else not only sees it, but sees it in the same place as you see it and with the same colors and textures.

One challenge of feature testing in a new system that is not yet documented is that you may not know how the feature is supposed to work. Please do not hesitate to post a message to the forum asking for clarification. And once you do figure it out, considering contributing some documentation on the topic. Post a message to the forum and someone on the core team will work with you to find the appropriate place for the documentation.

Due to the inter-dependencies that exist in large systems such as Wonderland, a feature that works one day, may no longer work when some other part of the system is changed. A technique used to check that features continue to work as the system evolves is called regression testing. Regression testing involves defining tasks and then coming up with a regression script, an exact set of steps for accomplishing those tasks. Each time a new Wonderland binary build or release candidate comes out, you run through all your scripts, using the exact same set of tasks and steps to accomplish those tasks. In a successful test, all the features will still work as they are supposed to. A regression test should always start with a known system state. With Wonderland, you need to reset the server (remove any stored state), clear your user cache (in ~/.wonderland) and also clear your Java Web Start cache.

In thinking about regression scripts, try to come up with those tasks you believe are the most common activities your users will want to do. For example, if you are planning on teaching a class using Wonderland, write regression scripts that involve the set of steps you expect a student to go through when they first enter Wonderland, and the set of steps they need to complete a lesson.

It is also a good idea to define a range of tasks, from very simple to more complex. A simple task might involve changing the shirt and pants color of your avatar. A more complex task might be collaboratively drawing a diagram on a whiteboard with 3 other people or having one person add a 3D model to the world, while a second person positions the model.

Sharing regression scripts benefits everyone. Please use the following wiki page to add your scripts if you are willing to share them with the community: Regression Test Scripts.

Having a library of scripts is particularly useful for the core team. It allows us to see which types of tasks you believe will be most common. We can run through the scripts ourselves before issuing releases to make sure everything is still working as it should be. Of course, if your scripts depend on custom modules, then you'll need to include a pointer for downloading the module so that others can test with that code.

Usability Testing

Usability testing involves observing end users performing tasks with software. A good usability test will include end users who are representative of the end users you expect to use Wonderland. For example, if you are planning to teach a class using Wonderland, you might be able to recruit some students who took a non-Wonderland version of the class the previous year.

While some of you may be equipped to do formal, scientific usability testing, most people do not have the resources or time for anything so formal. But even informal usability testing can be extremely valuable. For those of you who have access to some willing end users, we recommend devising a few basic tasks. For example, a task involving two users might be: log on to Wonderland and collaborate with a colleague on drawing an organizational chart for your department using the in-world Whiteboard. Another task might be: find the gallery space and add several photographs and 3D models to the current exhibit.

Be sure to prepare your test participants appropriately. Give them any end-user documentation you plan to give to your users. If the participants have never used a virtual world before, you may also need to provide a half an hour of training and practice prior to the usability test.

When you observe users in a usability test, the goal is to intervene as little as you possibly can. It can be hard, but try to let the participant work out problems on their own, only helping them if they become completely stuck. You should focus on watching what the users do unaided and record any problems. Occasionally, it is acceptable to ask people why they did some action that seems unusual. It often helps to videotape the sessions, but take notes even if you tape the sessions, since it is time-consuming to go back and watch all that video.

At the end of the test, have a debriefing session with the participant to find out what they thought did and did not work well. While participants will undoubtedly stumble on software bugs, a more important goal of usability testing is to uncover ease-of-use problems. These include difficult to understand dialog boxes or menu item names, problems figuring out how to escape from a mode, problems remembering how to navigate, and so forth. If more than one person has the same problem, we consider these usability problems bugs, and encourage you to report them.

Performance Testing

In performance testing, you track how long it takes to do important operations. For example, how long does it take between the time you click the Login button and the time you are able to move your avatar? Or how long does it take between the time you click "Insert" in the Insert Object dialog and the time the object shows up in the world? One goal in this type of testing is to discover frequent operations that take an inordinately long time. This allows us to focus attention on tuning these common operations. Another goal is to find platform-specific performance problems. If you can create objects quickly from the Insert Object dialog on a machine with an Nvidia graphics card running Windows XP, but it takes twice as long with the same card on a comparable Windows Vista or Linux machine, then something may be wrong.

Scalability Testing

The last type of testing that we're concerned with is scalability testing. While most scalability testing is best done with automated scripts and simulated users, there is no substitute for a test with a large number of live users. In this type of testing, it is helpful to add users incrementally and then do some amount of regression testing using tasks you know work fine with a small number of users. For the test to be useful, you need to note how many users are in the world when performance starts to degrade or operations stop working properly.

Orchestrating scalability tests can be extremely tricky. Once you start getting large numbers of users, you run into the problem of people's computing environments not being set up properly to run Wonderland. Trying to troubleshoot these connection problems while running a test can be problematic. If at all possible, try to make sure that everyone participating in the test has run Wonderland successfully sometime prior to the test.

In addition, it is essential to have a separate form of backchannel communication set up with all the testers. An IM group chat or IRC channel works well for this. This will allow you to cue people when you would like them to come into the world. It will also allow you to communicate with them if and when things go wrong during the test. If the system slows down so much that the user interface is no longer usable, you need a way to tell people that the test is done for the day or that you would like to try again, starting with a few people entering the world at a time.

General Testing Tips#

Set up data collection

Before doing anything else, follow the instructions above to set up data collection. Even if you are able to reproduce a bug, the information is not likely to be useful without the corresponding log files which help the developers see where in the code the problem is occurring.

Work in small groups

The nature of development work is such that developers tend to test their code by themselves. Wonderland, however, is fundamentally a multi-user environment, so it is essential to test every feature with multiple users. Testing with a single user can be helpful, but it is more helpful to test with at least two people and preferably three.

Test one thing at a time

It's tempting to try testing a series of features all at once, particularly if there are multiple people in the virtual world, but this makes it difficult to reproduce problems. A good rule of thumb is to hold as many variables constant as you can. For example:

• Hardware
• Base Wonderland world
• Number of objects in the world
• Number of users in the world
• People's activities

If you are testing drag-and-drop, for example, only vary the files that you are going to drag and drop. Do all your tests on the same computer, using the same Wonderland world (eg, Garden Arches or Empty World) with the same set of additional objects in the world, always have 2 other users and make sure those users do nothing. Now try dragging and dropping different types of files into the world. Be sure to collect all your test files into a folder so you can try them all again under slightly different conditions. Once one test works, then make one single change. For example, try dragging and dropping the same files using a different operating system, or try dragging and dropping the same files while the other two users run around you in circles.

Add complexity incrementally

Once you have simple test cases that work, you can start to add complexity. Following the example above, a more complex task would be to have several people drag and drop items at the same time. Then try increasing the speed that they drop items into the world.

Adding more users to the equation is another way to add complexity. Again, do this incrementally. Don't go from two people to 10 people. If the test works with two people, try it with three or four people, then with 6, and then with 10.

If you are testing a variety of things and you have a set of tests that work when one person at a time carries out the test, try having a few people in the world, each doing a different one of the tests at the same time.

Keep track of every action you do

Even though you think you'll remember exactly what you did, when problems occur, it is very difficult to remember everything. It's a good idea to take notes and jot down each action that you take.

Another strategy is to use regression tests. Have the steps written down ahead of time so you and others can follow the script exactly. If you are using a script, try not to stray from it. Instead, write another script or a variation on the script and then follow that one.

Create a Reproducible Test Case

As soon as you find a problem, the most important thing to do is to immediately capture the Java Console log. Even if you cannot reproduce the problem, the log may contain important information for the developers.

Once you do that, try to reproduce the problem. Start everything over fresh and do exactly the set of steps you have written down in your notes. Keep all the variables constant that you held constant during the initial test.

If you can reproduce the problem, write down the exact set of steps. If you have time, experiment by eliminating steps. The goal is to create a reproducible test case with the fewest possible set of steps that will still trigger the problem. Shorter test cases help to narrow down the problem and they also save time for the person who is trying to fix the problem.

Unfortunately, the reality of testing is that it is often quite difficult to make problems happen again. Try three or four times to make the problem happen again. Make slight variations in what you try and have the other people in the world think carefully about what they might have been doing in the time leading up to the problem. Have each of them try to reproduce it.

If you can't reproduce the problem immediately, record everything you know about the problem. If you have time, try to reproduce it again another day. If you don't have the time or you can't reproduce it, add it to the Issue Tracker anyway. If you do add an unreproducible problem, please make a note in the Issue Tracker that you tried to reproduce the problem, but could not. Many times the log files will contain enough information for the developer to figure out what went wrong.

Report Problems in the Issue Tracker

Before you can use the Issue Tracker to do anything other than search the database, you will need to log in to the Google Code web site. If you do not already have a Google account, you'll need to create one.

You can browse through the table of open issues, use the Search pull-down to limit the amount of data, or type in a query in the search field.

A prerequisite to entering a bug in the Issue Tracker is to have a log file captured immediately after the problem occurred or a reproducible test case. Preferably you will have both.

Before you enter a new bug, it's a good idea to search and make sure that your bug has not already been reported. If your problem does not appear to be in the issue database, you should add a "new issue" by clicking on the "New issue" link in the upper left of the Issues web page.

Select the appropriate description from the Template pull-down.

In the Summary field, include a several word description of the problem. Good summaries include keywords that people are likely to search on when checking to see if the issue already exists. Here are some examples of good issue summaries:

• Avatar continuously walks in circles
• Portal cell does not trigger properly
• Client hangs on File --> Exit on OpenSolaris
• OutofMemory exception in x11 app sharing
• ComboBox menus do not display

In the Description field, use the default template to provide a description of the problem. First describe the symptom (e.g., When I dragged a .kmz model of a fire engine into the Wonderland window, the model did not appear). Then explain what you were doing when the problem occurred. Ideally, here is where you would include the reproducible test case (e.g., First open the "Gestures" from the Window menu and click on wave. Without moving your avatar, drag the attached fire engine model into the window.)

At the end of the Description, please include as much detail about your computing environment as possible. See "Note About Adding Information about your Computing Environment" below for details about the information that we would like in this section.

Then use the Attach a file option to attach any error logs, images, PDFs, or 3D models that are relevant to understanding or solving the problem.

Special Note About Artwork and Application Files

In order to debug graphics problems or problems with applications, it is often necessary to have a copy of the 3D model, the 2D graphics file, the PDF file, or any other type of data you are trying to add to the system. We recognize that you may not want to share this type of file with the public. If this is the case, you can send us the file using a free file transfer web service such as: YouSendIt.com.

In the email field please use the email alias:

info@openwonderland.org

And include a subject in the form:

Art File from <your Google Code user name>

If you chose to send us your file using this mechanism, please make a note in the issue Description field that you have done so and be sure to include the exact file name.

Note About Adding Information about your Computing Environment

If you did not use the in-world Error Reporting tool which automatically gathers all this information for you, we ask that in the Description field, you include as much of the following information as possible:

• The exact version of Wonderland you are testing - If you downloaded a binary version of Wonderland, the version number is either the version number of Wonderland for a stable release or the date a nightly build was released on. So if you are using the Wonderland 0.5 dev5 build, then "dev5" is the right version. Similarly, if you downloaded the 2009-07-21 build of Wonderland, then you can specify the date as the version number. If you are building Wonderland from source, use "svn info" to get the source version you are using. For example:

   # svn info
      URL: https://wonderland.dev.java.net/svn/wonderland/trunk
      Revision: 2911
      This indicates that you are using revision 2911 of the trunk. 
      If you are using a branch (which has a different URL), make 
      sure to mention that. You can also find the revision number in 
      the upper right corner of the Wonderland Server Administration web page. 

• Details of the platform you are using - On Windows, you can typically find this information in the Control Panel under something like "Systems and Maintenance --> Welcome Center." On a Mac, you can find it in the Apple menu under "About this Mac." Please record the following information:
  • Make and model of the computer (include server information if the problem involves the server as well as the client)
  • Make and model of the graphics card in the computer
  • Operating system name and version number

• Details about your network connection
  • Internet or intranet
  • Type of connectivity (cable, FIOS, ...)
  • Specify if you are behind a firewall or a NAT-ed router, or connecting using VPN
  • If you problem seems related to bandwidth, include your connection speed (use the Speedtest.net to help figure this out)

• Java version - The Java version number is printed on the second line of the Java Console in the form "Using JRE Version..." You can also find the information by typing "java -version" in a terminal window. Alternately, you can open your Java Preferences dialog and get the information from there. On Windows, open the Java control panel, navigate to the Java tab and then click the "View..." button in the Java Application Runtime Settings box. When this opens, look in the User tab and record the top-most value in the Products column. On a Mac, use the Finder to navigate to the Java Preferences item in Applications --> Utilities --> Java. Navigate to the General tab and record the top item listed under "Java Applications."

When you are done, do not forget to click the "Submit Issue" button to record your bug.

Posting Bugs to the Forum#

If you have found a significant bug, it is a good idea to post a message to the Wonderland Forum describing the problem and providing a pointer to the issue in the Issue Tracker.

This alerts others to the problem and gives them an opportunity to add additional comments to the issue if they have experienced the same problem.

Posting to the forum is particularly a good idea if you have found a severe problem but cannot reproduce it. This way others can be on the lookout for the problem and hopefully someone will be able to come up with a reproducible test case.

Fixing the Bugs#

Anyone who is able is invited to pitch in and help fix bugs. The best way to start is to attach a patch to the issue in the Issue Tracker. The core team reviews new issues as they are entered into the system and re-assesses the priority given to them.

As a problem is being worked, it may be helpful to have additional information. The developer may contact the person who submitted the report and ask for some additional data or help testing a possible solution. Developers also often add notes to the Issue Tracker on their progress, so you can check back on your entries to see if any notes have been added, or set your preferences such that changes to issues are automatically e-mailed to you.