Picking using a lasso

Introduction

This tutorial completes the Picking example. This time we'll do a wide region picking, using one of the two methods described hereafter:

The toolbar icon allows you to choose between rectangular area picking and lasso picking.

Visualizing the picking region

Let's start by describing how the picked region is being displayed. In the example, by holding down the shift key, the user can start dragging the mouse to define either a rectangle or a freeform lasso shape. In the tutorial, we create two shapes to visualize the results: one contour shape (drawn using a white color based material) and one interior polygon shape (drawn using a semi-transparent material). The setup code for the border shape is below:

g_pick_border = RED::Factory::CreateInstance( CID_REDLineShape );
if( !g_pick_border )
  RC_TEST( RED_ALLOC_FAILURE );

RED::IShape* iborder_shape = g_pick_border->As< RED::IShape >();
RED::ILineShape* iborder_line = g_pick_border->As< RED::ILineShape >();

float vertex[ 12 ] = { 0.0f, 0.0f, 0.0f,
                       0.0f, 0.0f, 0.0f,
                       0.0f, 0.0f, 0.0f,
                       0.0f, 0.0f, 0.0f };

int lindex[ 8 ] = { 0, 1, 1, 2, 2, 3, 3, 0 };

RC_TEST( iborder_line->SetArray( RED::MCL_VERTEX, vertex, 4, 3, RED::MFT_FLOAT, iresmgr->GetState() ) );
RC_TEST( iborder_line->AddSegments( lindex, 4, iresmgr->GetState() ) );
RC_TEST( iborder_shape->SetMaterial( g_white_matr, iresmgr->GetState() ) );

RC_TEST( ipicking_camera->AddShape( g_pick_border, iresmgr->GetState() ) );

Both shapes are added to a picking camera that is drawn after all other cameras in the RED::VST_FRONT list of cameras in the VRL. Then, the picking mechanism uses a two step pass once the shift key has been pressed down:

The code is here (working for both types of areas at the same time):

switch( g_pick_step )
{
  // Initialize the picking area definition:
  case 0:
  {
    if( g_shift_key_down == true && evt._mouse_left == true )
    {
      // Clear all previously picked entities in the overlay camera:
      RC_TEST( ioverlay_camera->GetRootShape( root ) );
      RED::IShape* isroot = root->As< RED::IShape >();

      RC_TEST( isroot->GetChildrenCount( count ) );
      for( i = count - 1; i >= 0; i-- )
      {
        RC_TEST( isroot->GetChild( child, i ) );
        RC_TEST( RED::Factory::DeleteInstance( child, iresmgr->GetState() ) );
      }

      g_xrect = mx;
      g_yrect = my;
      g_xpos = mx;
      g_ypos = my;

      g_lasso_vertex.clear();
      RC_TEST( g_lasso_vertex.push_back( (float)g_xpos ) );
      RC_TEST( g_lasso_vertex.push_back( (float)g_ypos ) );
      RC_TEST( g_lasso_vertex.push_back( 0.0f ) );

      g_last_xpos = g_xpos;
      g_last_ypos = g_ypos;

      g_pick_step++;
    }

    break;
  }

  // Extend the picking area:
  case 1:
  {
    if( g_shift_key_down == true && evt._mouse_left == true )
    {
      g_xpos = mx;
      g_ypos = my;

      if( g_last_xpos != g_xpos || g_last_ypos != g_ypos )
      {
        g_last_xpos = g_xpos;
        g_last_ypos = g_ypos;

        RC_TEST( g_lasso_vertex.push_back( (float)g_xpos ) );
        RC_TEST( g_lasso_vertex.push_back( (float)g_ypos ) );
        RC_TEST( g_lasso_vertex.push_back( 0.0f ) );
      }
    }

From this point, we have all the necessary informations to proceed with the picking operation by itself.

Rectangular area picking

Picking using a rectangular area is performed by a REDsdk built-in function:

if( g_lasso == false )
{
  RC_TEST( iwindow->FramePickingRectangle( picklist, vrl,
                                           ( g_xpos < g_xrect ) ? g_xpos : g_xrect,
                                           ( g_xpos > g_xrect ) ? g_xpos : g_xrect,
                                           ( g_ypos < g_yrect ) ? g_ypos : g_yrect,
                                           ( g_ypos > g_yrect ) ? g_ypos : g_yrect,
                                           NULL ) );
}
else
{
  RC_TEST( LassoPicking( picklist, camera, picking_camera ) );
}

This is RED::IWindow::FramePickingRectangle. This method is not using any hardware based rendering workflow, unlike the lasso picking method that'll be detailed hereafter. The rectangle picking method does only return object level RED::ShapePath informations, unless specified otherwise in the method (and in this case, thousands of results can be returned, so be careful on performances!).

Lasso picking

Due to the complexity of the lasso shape, we have to use a hardware based rendering workflow in order to proceed with the lasso picking operation. Analytical solutions will be too long to process to remain interactive. The principle of the method is to use an auxiliary VRL to perform a masked rendering of the scene to identify shapes in the mask:

Depth levelling using the lasso shape as a mask

Geometries in the mask region appear rendered using colors

Finally, from the rendered colors that are found in the visualized image, we can get the list of shape instances to highlight.

Practical implementation of the lasso picking workflow

First, we need to setup the rendering workflow detailed above. This requires parsing the scene graph of our model to collect several informations:

// Collect scene graph informations:
// ---------------------------------

RED::ShapePath path;
RED::Map< RED::Object*, int > lasso_matr;
unsigned int colorid = 0;

RC_TEST( ParseSceneGraphForLasso( root, path, lasso_matr, g_lasso_path_to_id, g_lasso_id_to_path, colorid ) );


// Create a shape coloring shader using instance counters:
// -------------------------------------------------------
//  o This shader outputs one color for every shape path that'll use an instance counter color.

RED::ShaderProgramID vshid, pshid;
RED::ShaderString vsh, psh;

RED::RenderCode rcode;
rcode.BindChannel( RED_VSH_VERTEX, RED::MCL_VERTEX );

vsh.VertexShaderStart();
vsh.VertexTransform( "result.position", "state.matrix.program[2]", "vertex.attrib[0]" );
vsh.ShaderEnd();

RC_TEST( iresmgr->LoadShaderFromString( vshid, vsh ) );

psh.PixelShaderStart();

// Output the object ID using the instance color, but discard elements with color.w = 0xff.
psh.Param( "color", 0 );
psh.Temp( "filter" );
psh.Add( "SGE filter.x, color.w, { 0.999 }.x;\n" );
psh.Add( "KIL -filter.x;\n" );
psh.Add( "MOV result.color, color;\n" );

psh.ShaderEnd();

RC_TEST( iresmgr->LoadShaderFromString( pshid, psh ) );

RED::RenderShader rsh_color;
RC_TEST( rsh_color.SetRenderCode( rcode, RED_L0 ) );
RC_TEST( rsh_color.SetVertexProgramId( vshid, RED_L0, resmgr ) );
RC_TEST( rsh_color.SetPixelProgramId( pshid, RED_L0, resmgr ) );

RED::RenderShaderParameter instance_color( "instance_color", 0, RED::RenderShaderParameter::PSH );
instance_color.SetReference( RED::RenderShaderParameter::REF_INSTANCE_COUNTER );
RC_TEST( rsh_color.AddParameter( instance_color, RED_L0 ) );


// Create the shape coloring material configurations:
// --------------------------------------------------
//  o Each material in the scene graph receives a coloring shader in a specific layerset. Then we connect
//    an instance counter callback to perform the association between a shape instance (identified by a
//    path) and a color, during the rendering, for the material configuration using that specific layerset.

RED::Object* matr;
RED::Vector< RED::RenderShaderParameter* > list_param;

RED::LayerSet lasso_layer;
RC_TEST( lasso_layer.AddLayer( RED::Object::GetIDFromString( "MyLasso" ) ) );

for( lasso_matr.begin(); !lasso_matr.end(); lasso_matr.next() )
{
  matr = lasso_matr.current_key();
  if( matr )
  {
    RED::IMaterial* imatr = matr->As< RED::IMaterial >();

    RC_TEST( imatr->RegisterShader( rsh_color, iresmgr->GetState() ) );
    RC_TEST( imatr->AddShaderToPass( rsh_color.GetID(), RED::MTL_PRELIT, RED::LIST_LAST, lasso_layer, iresmgr->GetState() ) );
  }
}

Then, we setup the masking rendering pipeline:

// Create the lasso mask material shaders:
// ---------------------------------------
//  o We reuse the mask shapes (border and contour) to 'dig' into the depth buffer during the lasso mask
//    rendering pass. We setup the masking material needed there into a 'lasso_layer' layerset configuration.

// Creating and registering the state shader:
RED::StateShader mask_ssh;
RC_TEST( mask_ssh.SetDepthTest( RED::StateShader::ON ) );
RC_TEST( mask_ssh.SetDepthFunction( RED::StateShader::ALWAYS ) );
RC_TEST( mask_ssh.SetDepthMask( RED::StateShader::ON ) );
RC_TEST( mask_ssh.SetFaceCulling( RED::StateShader::NONE ) );

// Creating and registering the render shader:
rcode.Reset();
rcode.BindChannel( RED_VSH_VERTEX, RED::MCL_VERTEX );

vsh.VertexShaderStart();
vsh.VertexTransform( "result.position", "state.matrix.program[2]", "vertex.attrib[0]" );
vsh.ShaderEnd();

RC_TEST( iresmgr->LoadShaderFromString( vshid, vsh ) );

psh.PixelShaderStart();
psh.Add( "MOV result.color, { 0 }.x;\n" );
psh.Add( "MOV result.depth.z, { 1 }.x;\n" );
psh.ShaderEnd();

RC_TEST( iresmgr->LoadShaderFromString( pshid, psh ) );

RED::RenderShader mask_rsh;
RC_TEST( mask_rsh.SetRenderCode( rcode, RED_L0 ) );
RC_TEST( mask_rsh.SetVertexProgramId( vshid, RED_L0, resmgr ) );
RC_TEST( mask_rsh.SetPixelProgramId( pshid, RED_L0, resmgr ) );

// Assigning masking shaders to our mask shapes materials:
RED::IShape* iborder_shape = g_pick_border->As< RED::IShape >();
RC_TEST( iborder_shape->GetMaterial( matr ) );
RED::IMaterial* iborder_matr = matr->As< RED::IMaterial >();

RC_TEST( iborder_matr->RegisterShader( mask_ssh, iresmgr->GetState() ) );
RC_TEST( iborder_matr->AddShaderToPass( mask_ssh.GetID(), RED::MTL_PRELIT, RED::LIST_LAST, lasso_layer, iresmgr->GetState() ) );
RC_TEST( iborder_matr->RegisterShader( mask_rsh, iresmgr->GetState() ) );
RC_TEST( iborder_matr->AddShaderToPass( mask_rsh.GetID(), RED::MTL_PRELIT, RED::LIST_LAST, lasso_layer, iresmgr->GetState() ) );

RED::IShape* isurface_shape = g_pick_surface->As< RED::IShape >();
RC_TEST( isurface_shape->GetMaterial( matr ) );
RED::IMaterial* isurface_matr = matr->As< RED::IMaterial >();

RC_TEST( isurface_matr->RegisterShader( mask_ssh, iresmgr->GetState() ) );
RC_TEST( isurface_matr->AddShaderToPass( mask_ssh.GetID(), RED::MTL_PRELIT, RED::LIST_LAST, lasso_layer, iresmgr->GetState() ) );
RC_TEST( isurface_matr->RegisterShader( mask_rsh, iresmgr->GetState() ) );
RC_TEST( isurface_matr->AddShaderToPass( mask_rsh.GetID(), RED::MTL_PRELIT, RED::LIST_LAST, lasso_layer, iresmgr->GetState() ) );


// Create an auxiliary VRL and render image:
// -----------------------------------------
//  o Turned off until we have to render for a lasso selection.
//  o This VRL contains the scene camera and a another camera that renders the lasso mask
//    prior to rendering the scene with colors as IDs instead of the regular scene materials.
//  o The lasso mask camera 'digs' the depth buffer so that only geometries within the lasso
//    shape are passing the depth test and are rendered.

RED::Object* lasso_vrl;

RC_TEST( iwindow->CreateVRL( lasso_vrl, width, height, RED::FMT_RGBA, iresmgr->GetState() ) ); 
RED::IViewpointRenderList* ilasso_vrl = lasso_vrl->As< RED::IViewpointRenderList >();

RC_TEST( ilasso_vrl->SetClearColor( RED::Color( 0.0f ), iresmgr->GetState() ) );
RC_TEST( ilasso_vrl->SetClearDepth( 0.0f, iresmgr->GetState() ) );

RC_TEST( ilasso_vrl->Activate( false, iresmgr->GetState() ) );

RC_TEST( ilasso_vrl->InsertViewpoint( camera, RED::VST_SCENE, RED::LIST_LAST,
                                      0, 0, width, height, 0.0f, 0.0f,
                                      RED::VSP_ANCHOR_FIXED, RED::VSP_SIZE_STRETCHED_AUTO_RATIO, 
                                      iresmgr->GetState() ) );

RC_TEST( ilasso_vrl->InsertViewpoint( picking_camera, RED::VST_SCENE, RED::LIST_LAST,
                                      0, 0, width, height, 0.0f, 0.0f,
                                      RED::VSP_ANCHOR_FIXED, RED::VSP_SIZE_STRETCHED, 
                                      iresmgr->GetState() ) );

Once everything is properly set, we can do the rendering of the picking. This involves several steps:

  1. Turn off the master VRL, turn on the lasso VRL.
  2. Set the instance counter callback on the scene camera.
  3. Assign the lasso layerset to render objects with color IDs.
  4. Draw: Force the instance counter to update all IDs. First camera = picking camera that 'digs' a hole into the depth buffer. Second camera = scene camera rendered using lasso_layer materials (color = ID of shape path).
  5. Readback the render image of the VRL.
  6. Extract selected objects IDs.
  7. Loop back to step 4 for the wished number of depths.
  8. Restore original setup.

We achieve a "deep" picking by rendering several passes 4 and 7. Each time, we remove IDs of the already picked objects from the list of valid IDs. This is the shape coloring shader that discards invalid ID values to make sure these don't appear more than once.

// 1. Turn off the master VRL, turn on the lasso VRL.
RC_TEST( ivrl->Activate( false, iresmgr->GetState() ) );
RC_TEST( ilasso_vrl->Activate( true, iresmgr->GetState() ) );


// 2. Set the instance counter callback on the scene camera.
RC_TEST( icamera->SetInstanceCounterCallback( InstanceCounterCB, NULL, iresmgr->GetState() ) );


// 3. Assign the lasso layerset to render objects with color IDs.
RED::LayerSet lasso_layer;
RC_TEST( lasso_layer.AddLayer( RED::Object::GetIDFromString( "MyLasso" ) ) );

RC_TEST( icamera->ApplyLayerSet( lasso_layer, iresmgr->GetState() ) );
RC_TEST( ipicking_camera->ApplyLayerSet( lasso_layer, iresmgr->GetState() ) );


// 4. Draw.
int d, dmax = 5;

g_already_picked.clear();

for( d = 0; d < dmax; d++ )
{
  // 4.1 Force the update of the entire scene graph to initialize instance counters.
  RED::Matrix* matx;
  RED::ITransformShape* iroot = root->As< RED::ITransformShape >();
  RC_TEST( iroot->GetMatrix( matx, iresmgr->GetState() ) );

  // 4.2 Draw.
  RC_TEST( iresmgr->EndState() );

  RC_TEST( iwindow->FrameDrawing() );

  iresmgr->BeginState();


  // 5. Readback the render image.
  RED::IImage2D* ilasso_image_2D = ilasso_vrl->GetRenderImage()->As< RED::IImage2D >();
  RC_TEST( ilasso_image_2D->GetPixels() );

  // Uncomment the line below if you want to see each picked layer:
  //RED::ImageTools::Save( ilasso_vrl->GetRenderImage(), true, RED::String( "c:\\render_%1.png" ).Arg( d ), false, true, 1.0f );

  // 6. Extract selected objects IDs.
  int i, j, n, id, id_last;
  unsigned char* lpix;
  RED::Map< RED::ShapePath, int > to_highlight;

  id_last = -1;

  lpix = ilasso_image_2D->GetLocalPixels();

  for( j = 0; j < height; j++ )
  {
    for( i = 0; i < width; i++ )
    {
      n = 4 * ( j * width + i );

      if( lpix[ n ]     != 0 ||
          lpix[ n + 1 ] != 0 ||
          lpix[ n + 2 ] != 0 ||
          lpix[ n + 3 ] != 0 )
      {
        id = lpix[ n ] + ( lpix[ n + 1 ] << 8 ) + ( lpix[ n + 2 ] << 16 ) + ( lpix[ n + 3 ] << 24 );

        if( id != id_last )
        {
          id_last = id;
          if( g_lasso_id_to_path.find( id ) != NULL )
          {
            const RED::ShapePath& path = g_lasso_id_to_path.current();
            RC_TEST( to_highlight.insert( path, 0 ) );
          }
          else
            RC_TEST( RED_FAIL );
        }
      }
    }
  }

  for( to_highlight.begin(); !to_highlight.end(); to_highlight.next() )
  {
    RC_TEST( picklist.push_back( to_highlight.current_key() ) );
    RC_TEST( g_already_picked.insert( to_highlight.current_key(), d ) );
  }


  // 7. Loop back:
}


// 8. Restore everything.
RC_TEST( ivrl->Activate( true, iresmgr->GetState() ) );
RC_TEST( ilasso_vrl->Activate( false, iresmgr->GetState() ) );
RC_TEST( icamera->SetInstanceCounterCallback( NULL, NULL, iresmgr->GetState() ) );
RC_TEST( icamera->ApplyLayerSet( RED::LayerSet::ALL_LAYERS, iresmgr->GetState() ) );
RC_TEST( ipicking_camera->ApplyLayerSet( RED::LayerSet::ALL_LAYERS, iresmgr->GetState() ) );