Stream graphics from the service

In a nutshell, streaming graphics in PureWeb consists of taking graphics (2D or 3D) rendered by the service and piping them to PureWeb's imaging pipeline so that they can be displayed remotely in a view container on the client.

PureWeb takes care of the technical intricacies under the hood (image scaling, colorspace conversion, noise reduction, data compression, and so on). It also automatically captures user input within views, and keeps the view size in sync between the service and the client.

This section walks you through the basic workflow for streaming graphics.

Steps

Create a view class

APIs:

(I)RenderedView | C++ | .Net | Java

In order to work with PureWeb views, you must implement PureWeb's IRenderedView interface (called RenderedView in java) in a class in your service application. This will be your "view class". The quickest way to do this is to extend IRenderedView to an existing class (the class where you are rendering the graphic that you want to stream to the remote clients). Here's what this might look like, using C++ on Qt:

class MyExistingClass : public QWidget, public CSI::PureWeb::Server::IRenderedView

This approach works fine if you are just exploring PureWeb and want to reach quickly the point where you are streaming a single view to a remote client.

However, most real-world applications have many views, and extending IRenderedView separately in several different classes would not be efficient. For these, the better approach is to create a generic view class at the onset of the project. This generic view class will work as an adapter to which you can refer from other classes. To use this approach, you must be familiar with adapter patterns.

You do not need to start from scratch, however. The .Net and Java Scribble samples have generic view classes (RemotedControl.cs and RemotedPanel.java, respectively) that are set up this way. The examples at the bottom of this page illustrate how they are used. We encourage you to use or reference these sample classes in your own projects.


Register a view

APIs:

ViewManager.RegisterView() | C++ | .Net | Java

Registering a view consists of a single line of code that you add to your view class. It fulfills two purposes:

  • It gives the view a unique name for identification purposes. When referring to a view, both the service and the client must use that view's unique name. In the example below, we chose the name "Spectacular3D".
  • It registers the view with the StateManager, so that PureWeb can do its magic under the hood.

C++

MyViewClass::StateManager().ViewManager().RegisterView("Spectacular3D", this);

.Net

MyProgram.StateManager.ViewManager.RegisterView(“Spectacular3D”, this);

Java

stateManager.getViewManager().registerView(“Spectacular3D”, this);

If you are using a generic view class, register the view upon construction of this class. This ensures that the lifespan for the handler class equals the registration period of the view it handles.

Otherwise, register the view where desired; that may be the constructor of the class or another initializing function.


Pipe a graphic to the client view

APIs:

(I)RenderedView.RenderView() | C++ | .Net | Java

RenderTarget | C++ | .Net | Java

Image | C++ | .Net | Java

PureWeb does the actual work of streaming the service-side graphic to the client-side view container. The graphic that you stream can be any 2D or 3D image data expressed as a bitmap.

In order for this to work, you must first tell PureWeb where to get the image to send. You do this by implementing RenderView. The general idea is that RenderView provides a RenderTarget parameter; this parameter, in turn, provides an Image object, which you must populate with the actual image data.

This sends the graphic into PureWeb's imaging pipeline for processing (compression and optimization) so that it can automatically be decoded in the client.

 

Graphics will be streamed using the default encoding configuration and image format (JPEG images at 70% quality), but you can override these defaults if needed. See Fine-tune the image encoding.

You can also send metadata with the image.

When you implement RenderView, you are telling PureWeb where in the service's image rendering code it can find the image data that will be sent to the client. Consequently, the level of difficulty in implementing this method is proportional to the complexity of the image rendering code in your service.

This snippets below show how RenderView is implemented in the Scribble sample application. Because the images on the Scribble canvas are 2D drawings that are constantly kept up-to-date, the code is relatively simple.

C++

void MyViewClass::RenderView(CSI::PureWeb::Server::RenderTarget target)
{
    ByteArray imageBytes = target.RenderTargetImage().ImageBytes();
    ByteArray::Copy((void*)this->image.bits(), imageBytes, CSI::SizeType(0),
      target.RenderTargetImage().ImageBytes().Count());
}         

.Net

This snippet shows part of the RemotedControl.cs generic view class, which is shared by both the Scribble and Asteroids sample applications.

The code which actually creates the offscreen bitmap can be found in ScribbleControl.cs, not shown below. See Putting it all together, further down, for a more complete example.

public class RemotedControl : UserControl, IRenderedView
{
    private Bitmap m_offscreen;
	
    public virtual void RenderView(RenderTarget target)
	{
        var image = target.Image;
        
        if (m_offscreen != null)
        {
            image.DrawUnscaled(m_offscreen);
        }
    }
}

Java

This snippet shows part of the RemotedPanel.java generic view class, which is shared by both the Scribble and Asteroids sample applications.

The code which actually creates the graphic can be found in ScribblePanel.java, not shown below. See Putting it all together, further down, for a more complete example.

public class RemotedPanel extends JPanel implements RenderedView
{
    public void renderView(RenderTarget target)
    {
        paintComponent(target.getImage().getBitmap().getGraphics());
    }
}

Stream image updates

APIs:

ViewManager.RenderViewDeferred() | C++ | .Net | Java
ViewManager.RenderViewImmediate() | C++ | .Net | Java

Once you have defined RenderView, your application needs to know when to execute it and stream a new image. To do this, you add either the RenderViewImmediate or RenderViewDeferred method to one or more event handlers. As their name indicates, the main difference between the two methods is timing. In one case, the call to RenderView is immediate, in the other instance, it is deferred.

Using the deferred method is the most common, as it is less resource-intensive: the service does not need to spend time churning image bits that may not be needed by the client right away. Deferring the call to RenderView is a service-side decision which does not impact the displaying of views in the client application. Image updates are seamless to the end users.

You call this method whenever it makes sense to update the images, this could be a paint event handler, on mouse move, on click, etc.

C++

MyViewClass::StateManager().ViewManager().RenderViewDeferred("Spectacular3D");

.Net

MyProgram.StateManager.ViewManager.RenderViewDeferred("Spectacular3D");

Java

stateManager.getViewManager().renderViewDeferred("Spectacular3D");

Add view sizing information

APIs:

(I)RenderedView.GetActualSize() | C++ | .Net | Java
(I)RenderedView.SetClientSize() | C++ | .Net | Java

When the client application first launches, PureWeb automatically captures and stores the size of the client-side view's container. From there, in the background it will take care of keeping the service-side and client-side views in sync and responding appropriately to resizing events (for example if the user resizes the browser window or rotates the mobile device).

As a developer, you only need to choose whether the size of the view on the service should reflect the size of the view on the client, or the other way around.

If you want the view size to be based on the client (this is by far the most common approach), then you use SetClientSize. To match the client-side view to the size of the view on the service, you would use GetActualSize instead.

 

Both setViewSize and getActualSize must be present; simply stub out the one that you are not using.

SetClientSize

This method will be called whenever a client changes the view size. In the function, you provide basic logic to compare the size of the service view to that of the client view, and resize the service view if they are not equal. The Scribble sample application has been implemented this way.

C++

void MyViewClass::SetClientSize(CSI::PureWeb::Size clientSize)
{
   QSize newSize = QSize(clientSize.Width, clientSize.Height);
   if (size() != newSize)
    {
        this->parentWidget()->resize(newSize);
        resize(newSize);
        update();
    }
}	

.Net

The definition of SetClientSize simply makes use of C# control properties to get and set the dimensions of the panel.

public virtual void SetClientSize(System.Drawing.Size clientSize)
{
    if (this.ClientSize != clientSize)
    {
        this.ClientSize = clientSize;
        this.Invalidate();
    }
}	

Java

The definition of setClientSize simply makes use of the JPanel’s setSize()/getSize() functions to get and set the dimensions of the panel.

public void setClientSize(Dimension clientSize)
{
    if (!getSize().equals(clientSize))
    {
        setSize(clientSize);
        invalidate();
    }
}	

GetActualSize

This method calculates the size of the buffer provided by RenderView; if this size and the client size are different, PureWeb will scale the client appropriately, minimizing bandwidth where possible. The Asteroids sample application has been implemented this way,

C++

CSI::PureWeb::Size ScribbleArea::GetActualSize()
{
   return CSI::PureWeb::Size(width(),height());
}	

.Net

public virtual System.Drawing.Size GetActualSize()
{
    return this.ClientSize;
}	

Java

public Dimension getActualSize(){
    return getSize();	
}	

Handle user input

APIs:

(I)RenderedView.postMouseEvent() | C++ | .Net | Java
(I)RenderedView.postKeyEvent() | C++ | .Net | Java

If you want your views to be interactive and respond to user input events, you must implement PostKeyEvent and/or PostMouseEvent as part of your IRenderedView implementation.

The code for this is very dependent on which framework you use for your service (Windows Forms, Microsoft Foundation Class (MFC), Qt, etc.). It also tends to be relatively long, as it must account for all types of user input events supported by the application.

For these reasons, we cover this step in its own separate section. For now, you can just stub out these methods; this will get you more quickly to the point where you can see the graphic displayed on the client (although, of course, you will not be able to interact with the graphic yet). Or, you can jump to Respond to user input in views to see what's involved; that topic also covers how to handle touchscreen input.

C++

public void PostKeyEvent(PureWebKeyboardEventArgs keyEvent)
{
// Will define later.
}

public void PostMouseEvent(PureWebMouseEventArgs mouseEvent)
{
// Will define later.
}

.Net

public void PostKeyEvent(PureWebKeyboardEventArgs keyEvent)
{
// Will define later.
}

public void PostMouseEvent(PureWebMouseEventArgs mouseEvent)
{
// Will define later.
}

Java

public void postKeyEvent(PureWebKeyboardEventArgs keyEvent)
{
    // Will define later.
}
						
public void postMouseEvent(PureWebMouseEventArgs mouseEvent)
{
    // Will define later.
}

Example

Putting it all together

Below is the graphics streaming code from the Scribble sample application.

C++

// Constructor
ScribbleArea::ScribbleArea(QWidget *parent)
    : QWidget(parent),
    myPenWidth(2)
{
    setAttribute(Qt::WA_StaticContents);
    modified = false;
    scribbling = false;

    // Register the view
    PureWebCommon::StateManager().ViewManager().RegisterView("ScribbleView", this);
}

// Destructor
ScribbleArea::~ScribbleArea()
{
    PureWebCommon::StateManager().ViewManager().UnregisterView("ScribbleView");
}

// The service application's draw functionality (doesn't use PureWeb APIs)
void ScribbleArea::drawLineTo(const QPoint &endPoint)
{
    QPainter painter(&image);
    painter.setPen(QPen(myPenColor, myPenWidth, Qt::SolidLine, Qt::RoundCap, Qt::RoundJoin));
    painter.drawLine(lastPoint, endPoint);
    modified = true;

    int rad = (myPenWidth / 2) + 2;
    update(QRect(lastPoint, endPoint).normalized()
                                     .adjusted(-rad, -rad, +rad, +rad));
    lastPoint = endPoint;
}

// PureWeb::IRenderedView
// This is the part that streams the images and keeps the view size in sync.

void ScribbleArea::RenderView(CSI::PureWeb::Server::RenderTarget renderTarget)
{
    // Copy scribble image into PureWeb image.
    // Note: Can only direct copy byte array if image format is QImage::Format_RGB888.

    ByteArray imageBytes = renderTarget.RenderTargetImage().ImageBytes();
    ByteArray::Copy((void*)this->image.bits(), imageBytes, CSI::SizeType(0), imageBytes.Count());
}
					
void ScribbleArea::SetClientSize(CSI::PureWeb::Size clientSize)
{
    QSize newSize = QSize(clientSize.Width, clientSize.Height);

    if (size() != newSize)
    {
        this-....parentWidget()->resize(newSize);
        resize(newSize);
        update();
    }
}
					
CSI::PureWeb::Size ScribbleArea::GetActualSize()
{
    return CSI::PureWeb::Size(width(),height());
}	
					
void ScribbleArea::PostMouseEvent(const CSI::PureWeb::Ui::PureWebMouseEventArgs& mouseEvent)
{
    // Will define later.
}
					
void ScribbleArea::PostKeyEvent(const CSI::PureWeb::Ui::PureWebKeyEventArgs& keyEvent)
{
    // Will define later.
}

// Service application's paint event handler
// PureWeb's call to RenderView is added here
void ScribbleArea::paintEvent(QPaintEvent *event)
{
    QPainter painter(this);
    QRect dirtyRect = event->rect();
    painter.drawImage(dirtyRect, image, dirtyRect);

    PureWebCommon::StateManager().ViewManager().RenderViewDeferred("ScribbleView");
}

.Net

The ScribbleControl.cs class is responsible for generating the images to stream:

public partial class ScribbleControl : RemotedControl
{
    private Bitmap m_offscreen;
    private Stroke m_currentStroke = null;
						
    private void DrawCurrentStroke()
    {
        if (m_currentStroke != null)
        {
            using (Graphics graphics = this.CreateGraphics())
            {
                m_currentStroke.Draw(graphics, m_pen);
                if (m_offscreen != null)
                {
                    m_currentStroke.Draw(Graphics.FromImage(m_offscreen), m_pen);
                }
            }
        }
    }

    protected override void OnPaint(PaintEventArgs e)
    {
        base.OnPaint(e);
        PaintStrokes(e.Graphics, this.Size);
    }

    private void PaintStrokes(Graphics graphics, Size size)
    {
        foreach (Stroke stroke in m_strokes)
        {
            stroke.Draw(graphics, m_pen);
        }
    }

    // Mouse events such as this one call RemoteRender,
    // which in the RemotedControl.cs class is used to update images
    // by calling RenderViewDeferred
    protected override void OnMouseDown(MouseEventArgs e)
    {
        if (e.Button == MouseButtons.Left || e.Button == MouseButtons.Right)
        {
            BeginStroke();
            m_currentStroke.Add(e.Location);
            DrawCurrentStroke();
            RemoteRender();
        }

        base.OnMouseDown(e);
    }
}

The RemotedControl.cs class is responsible for streaming images from all view classes, including those generated in ScribbleControl.cs.

// Generic class for implementing remote rendering in all view classes
public class RemotedControl : UserControl, IRenderedView
{
    #region Fields
    IRemoteRenderer m_remoteRenderer;
    string m_viewName;
    bool m_canDeferRendering = true;
    bool m_hasPendingRemoteRender;

    #endregion
					
    #region Private Methods
    // The service application's draw functionality (doesn't use PureWeb APIs)
    // Raises the E:System.Windows.Forms.Control.Paint event.
    protected override void OnPaint(PaintEventArgs e)
    {
        if (!m_hasPendingRemoteRender)
        {
            m_hasPendingRemoteRender = true;
            Action action = () =>
            {
                if (CanDeferRendering)
                    RemoteRender();
                else
                    RemoteRenderImmediate();

                m_hasPendingRemoteRender = false;
            };
            this.BeginInvoke(action);
        }
        base.OnPaint(e);
    }
    #endregion
					
    private Bitmap m_offscreen;
    public void SetOffScreen(Bitmap b)
    {
        if (b != m_offscreen)
        {
            m_offscreen = b;
            RemoteRender();
        }
    }

    // PureWeb::IRenderedView
    // This is the part that streams the images and keeps the view size in sync.

    public virtual void RenderView(RenderTarget target)
    {
        var image = target.Image;
        try
        {
            var startTime = System.DateTime.Now;
            m_hasPendingRemoteRender = true;

            if (m_offscreen != null)
            {
                image.DrawUnscaled(m_offscreen);
            }

            var endTime = System.DateTime.Now;
            Trace.WriteLine("RenderView: " + (endTime - startTime).Milliseconds + " ms");
        }
        finally
        {
            m_hasPendingRemoteRender = false;
        }
    }
					
    public virtual void SetClientSize(System.Drawing.Size clientSize)
    {
        if (this.ClientSize != clientSize)
        {
            this.ClientSize = clientSize;
            this.Invalidate();
        }
    }
					
    public virtual System.Drawing.Size GetActualSize()
    {
        return this.ClientSize;
    }
					
    public void PostKeyEvent(PureWebKeyboardEventArgs keyEvent)
    {
        // Will define later
    }

    public void PostMouseEvent(PureWebMouseEventArgs mouseEvent)
    {
        // Will define later
    }
					
    [DefaultValue("View")]
    public string ViewName
    {
        get { return m_viewName; }
        set
        {
            m_viewName = value;
        }
    }
					
    // To stream an updated image
    public virtual void RemoteRender()
    {
        if (m_remoteRenderer == null)
        return;

        m_remoteRenderer.RenderViewDeferred(m_viewName);
    }
}

Java

The ScribblePanel.java class is responsible for generating the images to stream:

@SuppressWarnings("serial")
public class ScribblePanel extends RemotedPanel
{
    private final StateManager stateManager;
    private GeneralPath path;

    // Initialize the panel
    public ScribblePanel(String viewName, StateManager stateManager)
    {
        super(viewName);

        this.stateManager = stateManager;
        path = new GeneralPath();
    }
				
    // The service application's draw functionality (doesn't use PureWeb APIs) 
    @Override
    protected void paintComponent(Graphics g)
    {
        super.paintComponent(g);

        // Render accumulated scribbles
        Graphics2D g2D = (Graphics2D)g;
        g2D.setColor(new Color(scribbleColor.toArgb()));
        g2D.setStroke(mouseHandler.scribbleStroke);
        g2D.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
        g2D.draw(path);
    }
				
    // Register the view with the state manager
    @Override
    public void addNotify()
    {
        super.addNotify();

        remoteRenderer = stateManager.getViewManager();
        stateManager.getViewManager().registerView(viewName, this);
    }
}

The RemotedPanel.java class is responsible for streaming images from all view classes, including those generated in ScribblePanel.java.

// Generic class for implementing remote rendering in all view classes
// Extends the basic Swing JPanel to handle remote rendering.				
@SuppressWarnings("serial")
public class RemotedPanel extends JPanel implements RenderedView
{
    protected RemoteRenderer remoteRenderer;
    protected final String viewName;
    protected boolean canDeferRendering;
    protected boolean hasPendingRemoteRender;

    // Initialize a new instance of the RemotedPanel class.
    public RemotedPanel(String viewName)
    {
        this.viewName = viewName;
        canDeferRendering = true;
    }
					
    // Paint the remoted panel
    @Override
    protected void paintComponent(Graphics g)
    {
        if (!hasPendingRemoteRender)
        {
            hasPendingRemoteRender = true;

            // execute remote render on the UI thread

            UiDispatcherUtil.beginInvoke(new Runnable() {
                @Override
                public void run()
                {
                    if (getCanDeferRendering())
                        remoteRender();
                    else
                        remoteRenderImmediate();

                    hasPendingRemoteRender = false;
                }
            });
        }		
        super.paintComponent(g);
    }				

    // PureWeb's RenderedView
    // This is the part that streams the images and keeps the view size in sync.

    @Override
    public void renderView(RenderTarget target)
    {
        try
        {
            hasPendingRemoteRender = true;
            paintComponent(target.getImage().getBitmap().getGraphics());
        }
        finally
        {
            hasPendingRemoteRender = false;
        }
    }
					
    @Override
    public void setClientSize(Dimension clientSize)
    {
        if (!getSize().equals(clientSize))
        {
            setSize(clientSize);
            invalidate();
        }
    }
					
    @Override
    public Dimension getActualSize()
    {
        return getSize();
    }
					
    @Override
    public void postKeyEvent(PureWebKeyboardEventArgs keyEvent)
    {
        // Will define later
    }
					
    @Override
    public void postMouseEvent(PureWebKeyboardEventArgs mouseEvent)
    {
        // Will define later
    }
}