OSCORE

This web-page documents a project for the UCSB MAT-233 Java Fall 2004 course; Graham Wakefield, Dec 2004.

Overview

Development Timeline

Data Structure: Score & OSC

Scheduler & Threading

Overview

OSCORE is a Java application intended to provide a visual interface for scheduling OSC messages, to be used for example as a graphical score-controller for OSC-enabled applications including SuperCollider, Max/MSP, Pd, Csound, Reaktor, Traktor, Intakt, Bidule, CPS, Open Sound World, RTMix, Squeak etc. In this project I shall be primarly considering Max/MSP and SuperCollider as the destination applications for the OSC messages, in order to generate structured sounds.

OSC & JavaOSC

OSC is a hierarchical network-based messsaging protocol developed at CNMAT, UC Berkeley. It provides an open-ended, dynamic, URL-style symbolic naming scheme, numeric and symbolic arguments, pattern matching language, high resolution time tags, message bundling and a query system. For more information: http://www.cnmat.berkeley.edu/OpenSoundControl/.

This project makes use of the JavaOSC code developed by Chandrasekhar Ramakrishnan, which is an OSC-friendly wrapper of the native Java UDP network protocol.

Why

My impetus for this project was a conviction that, while tools such as Max/MSP & SuperCollider are very useful for prototyping musical and DSP ideas, both lack in providing a simple linear timing structure that could be used by the composer to write complete works.

Although SuperCollider is often used in projects involving the algorithmic generation of score events, I believe it could be equally fruitfully used in a more traditionally deterministic way. Max/MSP on the other hand does provide a graphical timeline interface, however it is unstable and poorly designed, rendering it impractical.

While it is true that both Max/MSP and SuperCollider both provide mechanisms for text-based score control, I have always believed that a graphical interface offers a much richer means for working with sound/time structures.

Java seems to be an ideal platform to develop such an application in, as it is renowned for its extensive and well-designed 2D graphics and GUI APIs, in addition to providing cross-platform implementation and networking facilities.

Core requirements

These are the requirements I have set out for the minimal practical version of this application, based on some self-determined use-cases:

A data structure for the score, and a data structure for the OSC namespace.
Several common OSC data-types to be supported, including 32-bit floating-point values, and arbitrary strings.
Score events may be grouped to create a new score event, thus enabling manipulation of events en masse.
A GUI application including:
- a graphical score interface (including drawing, manipulating, deleting)
- text-based score interface (including drawing, manipulating, deleting)
- OSC namespace interface
- configuration interface
OSC message scheduling system for playback
Score-file loading & saving - possibly XML, which is inherently hierarchical & lexically transparent

It may not be possible to achieve all of these requirements by the end of the MAT 233 course, however I hope to continue developing this project into a publicly distributable version during the following quarter.

Development Timeline

11.8

Concept design:

Use cases
Conditions & limiting cases for simplest demonstration
Future scalability requirements

First shot at class hierarchy design

Proof of concept tests

Send/receive OSC in Java shell app
Simple graphics (lines etc.)

11.15

Develop

Framing GUI
OSC communicator

Design Class Hierarchy

11.22

Develop

OSC classes
Score classes
GUI classes
Drawing classes
FileIO classes

11.29

Testing & refinements

12.6

Deadline for submission of code & documentation & presentation materials

Data Structure: Score & OSC

Score data: Mark class

The data structure for the score is quite complex in order to support the open-endedness and scalability I desired. Firstly, I chose to make all score events instances of a class, which I named Mark. As there are many different kinds of potential score event, I make the class Mark be data-type agnostic, planning to subclass this class for each specific data-type (e.g. MarkFloat, MarkString, MarkMIDI etc.). The subclass types will affect how the score events are rendered on screen, and what kinds of transformations can be applied.

As a core requirement is that any score events may be grouped and contained within other score events, I opted for a hierarchical tree-based structure for the score data. This was achieved by ensuring that every Mark instance contains a Vector property named marks, containing references to all the Mark instances contained within it. Thus any Mark can be a group. Every Mark also contains a reference to its containing Mark via the property parent.

As a result of this data structure, there is a single Mark that has no parent, which is the topmost Mark in the score data structure, referenced via the static member of the Mark class named root. In addition the Mark class contains a static Vector property named allmarks, which contains a reference to every mark in the score, no matter at what depth. Having a flattened version of the data structure was useful in determining a OSC message playlist, and in generating an editable text-based version of the score.

Marks may also contain a reference to an OscNode instance (see below), to determine the target location in the OSC namespace to send their content to. The nodes of marks in a group are not necessarily the same - this is part of the usefulness of abstracting the score-hierarchy from the synthesis-hierarchy.

At present, only the subclasses MarkString (for text messages) and MarkDouble (for 64-bit floating point numbers) have been implemented.

ScoreAsTableModel

The ScoreAsTableModel class, which extends the Swing abstract class AbstractTableModel, is a flattened version of the score data in the form of an Object[][] array of arrays, suitable for tabular presentation in a JTable. I created this because I thought some users might find it useful to view the score data as an event list - and an event list might provide useful tools such as searching & replacing, sorting, filtering, etc.

OSC datastructures

To recreate the hierarchical OSC namespace in a Java data structure, I extended the provided Java class DefaultTreeModel to create my class OscModel, and extending the class DefaultMutableTreeNode to create my class OscNode.

This had the advantage of providing standard parent-child methods I might require, and also enabling me to display the OSC namespace in a directory-style Swing component I created called OscTree, which extends JTree.

At present, there is no scope for editing the OSC namespace of a score through the OscTree component, however the inherited JTree methods will provide ample basis for implementing this in the future.

UML class diagrams

Scheduler & Threading

Clearly any practical score tool needs to have an accurate timing system. OSC supports a timetag attribute to each message bundle; this timetag is a 64-bit big-endian integer, representing the number of milliseconds since January 01, 1970.

So long as all networked applications have access to a reasonably accurate UTC system clock, events can be scheduled to occur correctly. I will refer to this as system time. Of course, Java's access to the system clock has it's own element of jitter, up to approximately 50 ms. But it is the synthesis tool that must be highly accurate, not the scheduler!

The score itself has its own time scale, represented by the x-axis, where the leftmost co-ordinate is the beginning of the piece. I will call this absolute score time. I'm using the term absolute, because at a future date I wish to enable the score to be sped up or slowed down at will, calling for a relative score time.

The current implementation loops the score continuously. The scheduler runs intermitently, collecting data from the score to make a list of the next events to be performed, and broadcasting them to the network. Because of the looping, some of the events to be scheduler in the future may be from earlier in the score, so I need a third representation of time, which I will call schedule time.

Scheduler Queue and Latencies

I implemented scheduling and broadcasting as independent threads in order to let Java manage the system resources appropriately for the unpredictable chunks of data that may be found in a score. This is appropriate for looped playback, or editing during playback (in contrast, a completely deterministic playback could precalculate the entire queue schedule beforehand).

The scheduler operates using a FIFO queue system implemented using the Vector class. The scheduler reads ahead in the score data from the (relative score) timestamp of the last element in the queue, up to a user-defined amount of time (the look-ahead period). The effects of looping are taken into account. Any score events found are appended to the end of the queue.

At this point, the scheduler thread goes to sleep for a predetermined amount of time, the sheduler sleep period.

Meanwhile, the broadcaster thread reads through the queue from the beginning, broadcasting events over the network, until it finds an event that is too far into the future to warrant sending. This distance into the future is the horizon period.

At this point the thread goes to sleep for a predetermined amount of time, the broadcaster sleep period.

The broadcaster also runs as a high priority thread. If it encounters any events that are too late to be broadcast, it makes a log of the error, and may suggest increasing the horizon time or the lookahead time.

Clearly, the total system latency may be up to a sum of all these periods. Reducing the sleep periods of either thread will increase CPU usage. Reducing the lookahead or horizon periods may introduce dropouts.

Another factor that may introduce larger and less predictable latencies; the UDP network latency. At this stage I have not implemented any means to estimate network latency, but instead have given the user access to managing this. If any events arrive at the synthesis application too late (i.e with timetags in the past), it indicates that the network latency estimate is too low. The application also stores the estimate latency period in the Config object.

System (real) time:	1	2	3	4	5	6	7	8	9
GUI cursor position				I
Scheduler read position						I
Scheduler lookahead							I
Broadcaster position						I
Receiver receiving position					I
Receiver synthesizing position				I

Scheduling: Playhead class

This class reads ahead into the score to format OSC events and put them in the event que for the broadcaster. I implemented it as a subclass of TimerTask, which simplifies the thread sleep scheduling somewhat. TimerTask runs as a thread that takes its own execution time into account by referencing the system time, thus it can be relied on to not drift out of periodicity (though it still incurs jitter).

The Playhead maintains its own internal representation of real time, which is projected into the future by the total latency. Based on this time, and the effects of looping, it reads from the Mark.allmarks vector all the marks that need to be scheduled, converting each into an OSC packet using the OscBun class, along with the appropriate timestamp. This OSC packet is then placed in the global queue, named Oscore.queue.

The algorithm for calculating the appropriate timestamp, taking into account lauch time, playback rate, thread intervals, latency estimate and looping was very difficult to calculate; I hope my explanation above helps to clarify what is happening in the code.

Psuedocode for the Playhead thread run() loop:

// check if this is the first time it it run
		
	// assign the default values of loop times etc.
	// set the playheadBegun = the current date in milliseconds
	
// update the local time markers (long milliseconds)
// readpointer, readendpointer, etc
		
// prepare the time loop markers (double pixel coordinates):
		
// sort all the events in the score (Mark.allmarks) by time
		
// calculate the date in milliseconds that these events must be after
		
// loop while the read pointer is before the read-end pointer
		
	// get the next Mark from Mark.allmarks
			
	// if this mark is before readpointer
			
		// skip it
				
	// else if it is after readpointer
			
		// if it is before readEndPointer
				
			// work out what it's future timestamp will be
			// create an OSCPacket out of it
			// store the OSCPacket in the Oscore.queue
					
		// advance the readPointer to this Mark's position
				
	// move to the next Mark
			
	// if we reach the end of the list or the end of the loop 
			
		// wrap to the beginning
		// reset the local loop time pointers
		// schedule events a further loopDuration into the future
				
// end of while loop
		
// update the history marker; localNow = Now

Broadcasting: OscSender class

This class reads from the Oscore.queue vector, in a FIFO order, and processes sends each OSC packet it finds over the network using an OSCPort instance (taken from JavaOSC). OscScheduler also runs as a subclass of TimerTask, synchronised to the Playhead through the same Timer instance (ensuring thread safety).

Psuedocode for the OscSender thread run() loop:

// get current time
			
// while there are objects in the Oscore.queue
	
	// pull the first element out of the queue (FIFO)
	
	// check if the timestamp is too late & note in dropouts++
	
	// otherwise, send it using OSCPort.send()

// end while loop

GUI

Application layout

The main application class Oscore extends JFrame. In the constructor method, Oscore creates a JTabbedPane to contain various sub-panes of the application, followed by a JLabel acting as the application status bar:

The Score Editor pane is the principal graphical access to the score data model. It represents score events & groupings graphically, and in future will provide a range of view, create & transform tools. The OSC namespace is displayed to the left, colour-coded to match the colours of the grouped events in the canvas to the left.

The OSC Editor pane will provide access to the OSC namespace in more detail, showing the data types of each node, statistical information, and the means to edit the namespace, ping the server. At present this is not implemented.

The Score Text pane shows the score in a timetag-sorted plain text message list. Editing is not possible at present.

The Event View pane shows the flattened score in a JTable. Editing is not possible at present. It periodically fails to load the score data also, I haven't been able to trace where this bug is coming from yet.

The Configuration pane will provide access to changing score-wide and application wide-settings in the ScoreConfig class such as change the OSC server address & port details etc; it is not yet implemented.

The Help pane will provide a HTML based help guide for the application. At present it displays this document.

Networking

Until the ConfigPanel is implemented, the user can't change the target OSC port or host; it defaults currently to localhost on port 8001.

The scheduled broadcasting of OSC commands is described above. I wrote wrappers for the JavaOSC OSCMessage and OSCBundle classes as OscMsg and OscBun.

OscBun provides some shortcut constructors useful for making a default message that just sends the system clock time, to aid synchronisation. OscMsg can't send timetags, only OscBun can, so pretty much all OSC sending except for time-agnostic messages like system calls will use OscBun. I also wrote a couple of utility functions such as getFirstPacket(), which converts the OscBun to an intelligible string.

OscBun takes timetags as Date objects. In a future implementation, when I rewrite the JavaOSC classes for my own needs, I might change this to long. Though OSC supports bundles of bundles, and JavaOSC also, I couldn't think of a use-case for it (since all bunlded content must have the same timetag) so I'm just using a OscBun(OscMsg) structure.

FileIO

Not yet implemented. Serializing the Mark.root and its children, to write as an obejct to file, would facilitate easy loading & saving of score data, however I would prefer to store the score, OSC namespace and c onfig settings all together in an XML document if possible.

Future Development

Here are some ideas of features and areas I will try to explore in extending this application.

Application Design

A much more detailed Exception handling structure would be a very good idea, and perhaps a system log to help trace errors.

Deployment

Test & build on different platforms (Windows, Linux)
Consider building as a Max/MSP MXJ object

GUI & usage

Score event group transforms, e.g. stretch (parameter/time, linear/nonlinear, local/global), shear, shuffle, reverse, etc.
Quantisation tools (to meter/pattern, parameter/time, linear/nonlinear, local/global).
Performance area for real-time generation of events, and composing on-the-fly for live use.
Undo histories & gesture histories
Context-help

Data Structure

More score event types, based on the OSC suggested types: MarkFloat, MarkInt, MarkString, MarkBinary, MarkInt64, MarkFloat64, MarkAscii, MarkRGBA, MarkMIDI, MarkMIDINote, MarkMIDICtrl, MarkTrue, MarkFalse, MarkBool, MarkNil, MarkInfinitum...
Score event meta-types; for example, instrument redefinitions for SuperCollider, e.g. MarkSynthDef, or score-control events such as MarkBeginLoop, MarkEndLoop, MarkTimeChange etc.

Scheduling

In a later implementation I could use JNI calls to the system to achieve greater accuracy, but for the time being I'm happy to work with it as it stands. In fact, because the score events are scheduled for future times, the 50ms error in Java's clock time is unimportant; it just adds 50ms of latency buffer requirement.

FileIO

Exporting score files to XML (including an <oscnamespace> node, a <config> node and a <scoredata> node), to TXT (Csound-like note list)
Importing scores from other types, e.g. Csound note lists, MIDI files, etc. This would involve querying the user in how to interpret the various data found, perhaps with some predefined templates.

Networking

Support for multiple target servers
Inclusion of a simple IM chat window, for networked performance
Receiving OSC data from other applications or performance hardware (e.g. Lemur) and automatically inserting into the score

Source Code

Right-click and select Save as... on this link to download all source code as a ZIP file. This ZIP archive contains:

Compiled (OSX) OSCORE application
Example score files
All Java source files
JavaOsc source files
XCode project files
Max/MSP application (OSX) & source, for testing OSC