5.0 Documentation

 

Fine-tune the image encoding

By default, the service sends JPEG images at 70% quality for non-interactive mode, and at 30% quality for interactive mode (when enabled).

You can override the defaults. In addition to JPEG, the APIs support PNG and H.264. Each view in each client can have its own encoding configuration.

When working with H.264, there are various requirements and licensing considerations, discussed in this section.

It is also possible to use other encoding formats, but this requires that you bypass the imaging pipeline.

If you would like to experiment first with various configurations, the Options tab in the Diagnostics Panel is a great tool to see the effect of your encoding changes without the need to edit the code or recompile between each try.

How-to

Enable the interactivity mode

APIs:

ViewManager.SetViewInteracting() | C++ | .Net

The interactivity mode is a feature that allows you to use a different encoding configuration for the images when the user is interacting with a view. This is entirely optional.

This feature is generally used to send images of lower quality during interaction, since the images are being updated very rapidly (several per second) and the end user doesn't really see these intermediate renderings. This is a great way to improve performance without sacrificing the quality of the images that the user actually does see.

To implement this, you need to tell the service when user interaction begins and when it stops. This is achieved by setting the value for the SetViewInteracting flag to true (to turn interactive mode on) or false (to turn interactive mode off). For example, you could enable the interactive mode on a mousedown event, and turn it back off on a mouseup event.

C++

void MyViewClass::OnLButtonDown(UINT nFlags, CPoint point)
{
    m_startpt.x = point.x;
	m_startpt.y = point.y;
    // Tell the StateManager that the user is interacting with the view
    MyApp::StateManager().ViewManager().SetViewInteracting("Spectacular3D", true);
}
void MyViewClass::OnLButtonUp(UINT nFlags, CPoint point)
{
    m_startpt = -1;
	CView::OnLButtonUp(nFlags, point);
    // Tell the StateManager that the user finished interacting with the view
    MyApp::StateManager().ViewManager().SetViewInteracting("Spectacular3D", false);
}

.Net

protected override void OnMouseDown(MouseEventArgs e)
{
    if (e.Button == MouseButtons.Left || e.Button == MouseButtons.Right)
    {
        // Tell the StateManager that the user is interacting with the view
        Program.StateManager.ViewManager.SetViewInteracting("Spectacular3D", true);
        Capture = true;
        (...)
    }
}

protected override void OnMouseUp(MouseEventArgs e)
{
    if (e.Button == MouseButtons.Left || e.Button == MouseButtons.Right)
    {
        // Tell the StateManager that the user finished interacting with the view
        Program.StateManager.ViewManager.SetViewInteracting("Spectacular3D", false);
        Capture = false;
        (...)
    }
}

Once the interactive mode is set, you can specify a separate image encoding configuration for this mode, as explained in the next section.


Change the encoding configuration

APIs:

EncoderFormat | HTML5

EncoderConfiguration | HTML5

View.SetEncoderConfiguration() | HTML5

The service provides an EncoderConfiguration object that determines the default image encoding for all views. On the client, you can override the service-provided default for a view by setting a custom EncoderConfiguration object.

You could create your own EncoderConfiguration object on the service, although this is not common; this would set a new default configuration for all views in all clients.

Each EncoderConfiguration object contains two instances of EncoderFormat, one for the non-interactive (or full quality) mode, and one for the interactive mode. (Values for the interactive mode will be ignored if this mode has not been enabled on the service).

Each EncoderFormat, in turn, contains one or more arguments: the format (or mime type) is the only one which is mandatory. The APIs support JPEG (the default), as well and PNG and H.264. It is common to also specify a quality value (an integer between 0 and 100 that defines the overall image fidelity). And finally, you can also add custom parameters of your own, as described further down this page.

You can choose either an image format (JPEG/PNG) or a video format (H.264) as the mime type for interactive and non-interactive modes, but not a mix of both. In other words, you can set non-interactive mode to PNG and interactive mode to JPEG (or vice versa), but if you choose H.264, both EncoderFormat configurations must bet set to that mime type.

H.264 images can be hardware-encoded or software-encoded, and there are licensing considerations for both implementations. See Working with H.264 for more information.

Setting a quality value for PNG images has no effect, since PNGs use loss-less encoding.

To set a view-specific encoding configuration:

  1. Create an EncoderFormat object for the full quality mode, and a second EncoderFormat object for the interactive mode. (The one for the interactive mode will be ignored if this mode has not been enabled on the service.)

    HTML5

    var fullQuality = new pureweb.client.EncoderFormat('image/png', 100);
    var interactive = new pureweb.client.EncoderFormat('image/jpg', 20);
    
  2. Create an EncoderConfiguration object using the two EncoderFormat you created in the previous step.

    HTML5

    var encoderConfig = new  pureweb.client.EncoderConfiguration(interactive, fullQuality);
    
  3. Assign the new EncoderConfiguration object to a specific view. This is done using View.SetEncoderConfiguration().

    HTML5

    Spectacular3D.setEncoderConfiguration(encoderConfig);
    

Work with H.264 encoding

By using H.264 with the PureWeb SDK, you will be subject to other third party licenses such as the AVC/H.264 Patent Portfolio License with MPEG-LA. PureWeb does not provide you with coverage for these licenses. It is your sole responsibility to obtain such licenses and ensure that you are in compliance with them.

Note that we do not support H.264 for Internet Explorer 11. We recommend using other supported web browsers.

H.264 images can be hardware-encoded or software-encoded. By default, service APIs will send hardware-encoded images using NVENC SDK 8.1, which has the following minimum requirements (for applications that do not meet these requirements, the APIs will fall back to a software-encoded implementation, see Fallback options.)

  • NVIDIA Kepler/Maxwell/Pascal/Volta GPU(for more information on supported GPUs, see the NVIDIA developer web site)
  • Driver version 390.77 or higher (on Windows) or driver from 390.25 or higher (on Linux)
  • CUDA 8.0 Toolkit

The default H.264 encoding has been formally tested on AWS G2 Windows instances using driver 354.41, and AWS G3 instances using driver 370.12. For Linux, only G3 instances are supported, and official testing was done using the 390.46 driver.

Fallback options

If the computer running the service application does not meet the requirements above, the service will automatically fall back to a software-encoded implementation. This will be either x.264 (if the libraries are present -- currently only available on Windows), or open-source UMC otherwise.

Using x264 is strongly recommended whenever possible, as the PureWeb API's UMC implementation is intended for development use and not ideal for production purposes.

To acquire the x264 library for your PureWeb application, please contact support@pureweb.com.

A commercial license agreement with CoreCodec Inc. is required for use of this library. By requesting this library, you agree that your contact information will be provided to CoreCodec solely for such licensing purposes and that it is your responsibility to acquire the license.

Note that the commercial license for x264 is in addition to any other third party licenses that you will be subject to for using H.264, such as the MPEG-LA AVC/H.264 Patent Portfolio License.

More details on x264 licensing can be found here.

Installing the x264 library

Once you have the x264 library:

  1. Copy the files into your service application's directory.
  2. Ensure that they are included when rebuilding and/or deploying your application, for example by adding them during a pre- or post-build step. The service will load the files at runtime.
  3. Restart the PureWeb server.

Verifying the settings

To confirm which encoding configuration is used in your service application, look for the encoder initialization message in the log. For example:

H264Encoder x264Encoder: Initialized

Environment variables

The following environment variables may help you fine-tune how H.264 works in your application:

  • PW_IMAGEPROCESSING_FORCE_H264_ENCODER: Set this environment variable to override any detection in image encoding and force the service to use the specified encoder (x264, umc, or nvenc). This could be used for instance to force x264 even if H.264 is supported by the graphics card. If you specify an unsupported format, the standard fallback options will apply (nvenc will fall back to either x264 or umc depending on installed libraries, and x264 will fall back to umc).
  • PW_IMAGEPROCESSING_USE_HW_DEVICEID: Set this environment variable to force a particular device ID by NVENC (device must support CUDA); the variable must be set prior to executing RegisterView. Can be useful if running K-class cards. Expects a positive integer; will default to 0 if the specified number is not valid.

Add custom encoding parameters

When you specify an encoder format on the client, you can add your own custom encoding parameters. The service can then use this information to make rendering decisions.

For example, let's say that you would like the images in a given view to display a watermark. On the client, you could add a custom boolean parameter called "useWatermark". On the service, you would check the value for this parameter when renderView is called, and if it's true, you would execute some code to add a watermark to the images being rendered.

Write the parameters on the client

APIs:

EncoderFormat | HTML5

When you instantiate an EncoderFormat object, you can add custom parameters as the third, optional, argument. The parameters are expressed as an array of key-value strings. You can convert the parameters to different data types when you retrieve them from the service.

HTML5

var MyCustomEncoderFormat = new pureweb.client.EncoderFormat('image/jpg', 100, {'useWatermark':true});

Under the hood, this adds the key-value pairs to the parameter map on RenderTarget.

Read the parameters from the service

APIs:

RenderTarget.Parameters | C++ | .Net

To retrieve the custom encoding parameters on the service, simply call the Parameters method on the RenderTarget class.

C++

String useWatermark = target.Parameters()["useWatermark"];

.Net

String useWatermark = target.Parameters["useWatermark"];


Override the default image format

APIs:

ViewManager.setViewImageFormat() | C++ | .Net

Each framework provides default values for the characteristics of an image (pixel format, scan line order and alignment):

  C++ C#
PixelFormat Bgr24 Rgb24
ScanLineOrder TopDown TopDown
Alignment 4 4

If you need to override these defaults, you can do this using the SetViewImageFormat method.

ViewImageFormat viewImageFormat;
viewImageFormat.PixelFormat = PixelFormat::Rgb24;
viewImageFormat.ScanLineOrder = ScanLineOrder::TopDown;
viewImageFormat.Alignment = 4;
PureWebCommon::StateManager().ViewManager().SetViewImageFormat("MyAwesomeView", viewImageFormat);

Bypass the imaging pipeline

Some applications have very specific needs not met by the default imaging pipeline. For example if you want to:

  • use a different image format other than ones provided by the APIs,
  • use a custom encoder format,
  • pass arbitrary binary data (such as already compressed images, documents, and so on).

For these situations, it is possible to bypass the imaging pipeline. However, this requires that you provide a custom renderer implementation on your client, and is not a trivial undertaking.

Here are some pointers to get you started; please refer to the API library reference for your chosen platform for more information.

  • On the service, you will need to create a custom implementation of EncoderFormat. Also, when implementing RenderView, use either RenderTarget.ContentInfo or RenderTarget.Type (instead of the usual RenderTarget.Image).
  • On the client, you will need to create your own implementations of CustomRenderer, which is an extension of ViewRenderer.