Image Encoding and Quality

The PureWeb SDK automatically handles the nitty gritty details of the imaging pipeline, and therefore you do not need to write complex code to handle the encoding, quality and rate of the images that you are remoting in your views. You do, however, have control over quality and encoding format, and you can modify these settings relatively easily.

Sending images in a more efficient format or lower quality means that the size of each view update will be smaller, which can be used to conserve bandwidth, reduce latency, and improve performance.

When defining the encoding configuration for a view, there are two settings that you can adjust in a client encoder format:

  • Encoding format: this is the mime type for the image (.jpeg, .png, and so on).
  • Quality: this is an integer (between 0 and 100) that defines the fidelity (overall clarity and number of visual artifacts) of the images to be generated by the service.

A key feature of the encoder configuration is the interactivity mode. When setting the client encoder configuration, you set two encoder formats: the interacting encoder format, and the non-interacting (or full-quality) encoder format. Then, in your service application, you have the option of toggling a boolean flag which indicates the current interactivity mode. The PureWeb APIs use the corresponding encoder format for the specified interactivity mode.

Each view in each client can have its own image encoding configuration. For example, if you have an HTML5 client and an iOS client both with two views, the iOS client can use a different encoding configuration for each view, while the HTML5 client can use two completely different encoder configurations for its two views.

Trying different settings in the Diagnostics Panel's Options tab will help you determine what works best in your application to achieve the optimal balance between image quality and system performance, without having to change the code or recompile between each try.

Once you know which configuration you need, you can set it permanently in the client code using the EncoderConfiguration available within the view.

Using the encoder configuration API is entirely optional. The default configuration is JPEGs at 70% quality for non-interactive mode, and JPEGs at 30% quality for interactive mode (if enabled). Interactive mode is not enabled by default.

It is possible to by-pass the imaging pipeline altogether, for example to use a different image format. This level of customization, however, is outside of the scope of this guide.

About Encoding Formats

The SDK currently supports the following encoding formats (mime-types): PNG, JPEG, Tiles and, to a certain extent in the Flex and HTML5 client SDKs, H.264. Each format has its pros and cons:

  • PNG images, unlike JPEG and tiles, use loss-less encoding, and therefore changing the quality value for this mime type has no effect: the image will look the same, whether quality is set to 40 or 100.
  • When using JPEG or PNG, the application updates the entire view. Tiles, on the other hand, are small header-less JPEG images which only update the parts of the view that have changed since the last update; this sends noticeably less data to the clients.
  • Tiles require more computational effort on the part of the computer running the service application. Although this encoding format is supported in all client platforms except HTML5, tiles perform best when the client application is Flex; on other platforms, JPEGs perform best.
  • H.264 is a popular video compression format which delivers high quality images at a fraction of the data size of other formats. Although H.264 can be a real boon to reduce bandwidth consumption, it is not suitable for every application. H.264 requires more time for both encoding and decoding than either JPEG or PNG, and this will reduce the frame rate of your application.

Using H.264

H.264 is a video encoding format that provides a reduction in the bandwidth required to stream video data compared to the other encoding formats available in the SDK. The SDK comes with a built-in H.264 encoding implementation. It can also be used with x264, which is a more performant third-party implementation that can be obtained separately.

Both H.264 encoding implementations are supported in the C++ and .NET service APIs on Windows. The built-in implementation is also supported in the C++ service APIs on Linux. H.264 decoding is supported in the HTML5 client APIs only at this time.

Although H.264 support is feature-complete, it has not been fully optimized, and as such, it may not be ideal for production purposes.

It should also be noted that the feature currently works well only in Chrome, Firefox and Edge. In other browsers, including Internet Explorer, Safari, Mobile Safari and Mobile Chrome, it does not perform well, or does not work at all.

Licensing

The built-in H.264 encoding implementation is included with the PureWeb SDK licensing.

x264 is licensed under GNU GPLv2 and must be appropriately licensed for commercial and demo use. Note that x264 licensing is handled independently of PureWeb SDK licensing, please visit http://x264licensing.com/ for information on how to license x264 for use with your application.

The x264 dll files are not distributed as part of the PureWeb SDK install. If they are not present, the built-in H.264 encoding implementation will be used.

The x264 source code is not required for use in the PureWeb SDK.

If you would like to see this code, it can be obtained from
git://git.videolan.org/x264.git, version c628e3bacabb3ec6aff74332c76a7879b80a7a3d.

Setting the Image Encoding

The Diagnostics Panel’s graphical interface makes it easy to change the encoding configuration and see the impact of these changes on the fly. However, changing the configuration in the Diagnostics Panel is not a permanent operation; for the changes to be permanent, it is necessary to change the encoder configuration programmatically.

Changing the encoder configuration is simply a matter of creating or modifying two EncoderFormat objects within the EncoderConfiguration. Each view exposes a reference to EncoderConfiguration. The code below illustrates how to accomplish this. (See next section below for a discussion about why you may want a different encoding format for interactive and fullQuality modes.)

HTML5

Create variables for full and interactive quality; the snippet below illustrates how this could be done in the Scribble sample application:

var scribbleView = new pureweb.client.View({id: 'ScribbleView', viewName: 'ScribbleView'});
var interactive = new pureweb.client.EncoderFormat('video/h264', 10);
var fullQuality = new pureweb.client.EncoderFormat('image/jpg', 100);
var encoderConfig = new  pureweb.client.EncoderConfiguration(interactive, fullQuality);
scribbleView.setEncoderConfiguration(encoderConfig);

iOS

If you need to specify your own encoding configuration, use the <PWViewDelegate> protocol:

PWDiagnosticViewDelegate : NSObject <PWViewDelegate>

You can then implement the PWEncoderConfiguration method:

- (PWEncoderConfiguration *)preferredEncoderConfigurationForView:(PWView *)view
{
    PWEncoderFormat *fullQuality = [PWEncoderFormat formatWithMimeType:@"image/jpeg" quality:90];
    PWEncoderFormat *interactiveQuality = [PWEncoderFormat formatWithMimeType:@"image/tiles" quality:45];
    PWEncoderConfiguration *config = [PWEncoderConfiguration configurationWithInteractiveQuality:interactiveQuality fullQuality:fullQuality];
    return config;
}			

The EncoderFormat further exposes a parameters dictionary as an extension point. These parameters are also made available on the service side, as EncoderParameters on the RenderTarget interface. This allows developers to make potential rendering decisions based on the encoding format and quality that will eventually be used.

Setting the Image Interactivity Mode

You can set a boolean flag in your service application to indicate whether a user is currently interacting with a view. The PureWeb SDK then takes into account the current interactivity state for a given view when updating the image, and sends images encoded in encoding format specified by the client for the given interactivity state. When a user is interacting with a view, it is likely that the view is being updated very rapidly, to reflect the changes made by the user. Therefore, a popular approach is to send lower-quality images when the user is interacting with a view, and to restore full quality when interaction stops.

To determine whether to use the settings for interactive or non-interactive modes, the application relies on the value for the SetViewInteracting parameter on the ViewManager. If this value is set to true, the application will use the interactive mode settings, if it is set to false, the application will use non-interactive (full quality) mode settings.

In the example below, SetViewInteracting is set to true when the left button is pressed , and returned to false when that button is released.

C++

void CScribbleView::OnLButtonDown(UINT nFlags, CPoint point)
{
    m_startpt.x = point.x;
	m_startpt.y = point.y;
	CScribbleApp::StateManager().ViewManager().SetViewInteracting("ScribbleView", true);
}
void CScribbleView::OnLButtonUp(UINT nFlags, CPoint point)
{
    m_startpt = -1;
	CView::OnLButtonUp(nFlags, point);
	CScribbleApp::StateManager().ViewManager().SetViewInteracting("ScribbleView", false);
}

.Net

protected override void OnMouseDown(MouseEventArgs e)
{
    if (e.Button == MouseButtons.Left || e.Button == MouseButtons.Right)
	{
	    // Tell the StateManager we are interacting with this view
		Program.StateManager.ViewManager.SetViewInteracting(ViewName, true);
		Capture = true;

		BeginStroke();
		m_currentStroke.Add(e.Location);
		DrawCurrentStroke();
		RemoteRender();
	}			

Java

public void mousePressed(MouseEvent e)
{
    isLeftButtonDown = e.getButton() == MouseEvent.BUTTON1;
	if (isLeftButtonDown)
	{
	    setInteracting(true);
		mousePos.x = e.getX();
		mousePos.y = e.getY();
		path.moveTo((float) mousePos.x, (float) mousePos.y);
		remoteRender();
	}
}