Skip to content
GenManip

Large-Scale Data Generation

In this section, we introduce how to perform large-scale data generation using GenManip. This section should be read together with the Advanced Data Generation section, and essentially serves as a concrete application scenario of the Advanced Data Generation section.

Prerequisites

Section titled “Prerequisites”

To perform large-scale data generation, you need to download some assets. You can download the assets using the following command:

Terminal window
python standalone_tools/download_assets.py --dataset objaverse-subset

You can refer to the Object Annotation section to annotate your objects and save them as a pickle file. For the assets mentioned above, we have already performed the annotations in advance and provided them in the Codebase, which you can use directly. For your own annotations, they should likewise be saved in the assets/objects directory.

About Large-Scale Generation

Section titled “About Large-Scale Generation”

For large-scale generation, we aim to introduce as much randomization as possible and randomly generate Pick-and-Place data over as many objects as possible.

Next, we will explain the design of the configuration files step by step.

First, for randomization, we use:

domain_randomization:
  cameras:
    config_path: configs/cameras/fixed_camera_robotiq_s2r_3L_align_twoObs.yml
    type: fixed
  object_data_path: objaverse_annotation_refined_pnp
  random_environment:
    has_wall: true
    hdr: true
    robot_base_position:
      random_range: 0.05
    robot_eepose: false
    table_texture: true
    table_type: true
    wall_texture: true
  rewrite_instruction: true

That is, we set the camera file to a calibrated camera configuration, and use the object annotation information from objaverse_annotation_refined_pnp.pickle. For domain randomization, we enable walls, HDR (dome light), randomization of the robot base position, as well as randomization of table textures and table types. At the same time, we enable instruction rewriting, meaning that during data generation, the instruction will be automatically rewritten based on the annotation information of the grasped and placed objects.

For object configuration, we use:

object_config:
  background_0: &id003
    clip_range:
      max: 0.2
      min: 0.05
    filter_rule: []
    max_cached_num: 30
    is_not_rigid: true
    path: object_usds/objaverse_usd
    type: load_object_from_path
  background_1: *id003
  background_2: *id003
  background_3: *id003
  background_4: *id003
  background_5: *id003
  background_6: *id003
  background_7: *id003
  background_8: *id003
  obj1_0:
    clip_range:
      max: 0.15
      min: 0.05
    filter_rule:
    - can_grasp
    max_cached_num: 30
    option:
    - adjust_thickness
    path: object_usds/objaverse_usd
    type: load_object_from_path
  obj2:
    clip_range:
      max: 0.35
      min: 0.2
    filter_rule:
    - is_container
    max_cached_num: 30
    option:
    - force_top
    path: object_usds/objaverse_usd
    type: load_object_from_path

As shown above, 11 objects are introduced in total: 9 background objects and 2 target objects. For all objects, we use the load_object_from_path type and load them from the object_usds/objaverse_usd directory. For background objects, we set is_not_rigid, so that after loading, the objects will not change their positions due to applied forces. For the three different categories of objects, we set different clip_range values; the final object size is randomly sampled from the intersection of the annotated size and the clip_range. For each object category, 30 objects are preloaded, and no more are loaded to avoid excessive memory usage.

For obj1 and obj2, they must be annotated as graspable (can_grasp) and as containers (is_container), respectively. In addition, obj1 is set with adjust_thickness, meaning that after loading, the object thickness will be adjusted to be smaller than the gripper width. force_top is a deprecated option and is no longer used.

After setting up the objects, we also need to configure the layout:

layout_config:
  ignored_objects: []
  type: random_custom_tableset
  in_order: true
  custom_tableset:
    obj1_0: &id001
      type: global_range
      random_range_angle: 360
      random_range_h: 0.7
      random_range_w: 0.4
      random_range_x: -0.2
      random_range_y: -0.35
    obj2: *id001
    background_0: &id002
      type: global_range
      random_range_angle: 360
      random_range_h: 20
      random_range_w: 20
      random_range_x: -10
      random_range_y: -10
    background_1: *id002
    background_2: *id002
    background_3: *id002
    background_4: *id002
    background_5: *id002
    background_6: *id002
    background_7: *id002
    background_8: *id002

We use the random_custom_tableset type throughout, and place each object sequentially according to the order defined in custom_tableset (in_order: true). For each object, we use the global_range type. For obj1 and obj2, we expect them to be placed within the robot’s workspace, as configured above. For background objects, they only need to be placed on the table, so we set an overly large range; the program will automatically adjust the range to the intersection between the specified range and the table surface.

generation_config:
  reset_tcp: true
  action_path:
    mode: auto
    robot: 0
  articulation: []
  goal:
  - - obj1_uid:
      - obj1_0
      obj2_uid:
      - obj2
      position:
      - top
      fixed_position: true
  mode: random_all
  planner: curobo
instruction: ''

Next, we configure the generation-related settings. The task is to place obj1 on top of obj2, and we use curobo as the planner. Here, we leave instruction empty, because the instruction will be automatically generated during the rewriting stage.

This is followed by setting a series of additional configuration options:

mode: manual
num_episode: 1000000
restart_per_success: 100
preprocess_config:
- config:
    config: default
    type: convexDecomposition
  type: collider
robots:
- config:
    gripper_type: robotiq
  default_joint_positions:
  - 0.0
  - -0.785
  - 0.0
  - -2.356
  - 0.0
  - 1.57079
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  type: franka
table_uid: aa49db8a801d402dac6cf1579536502c
task_name: system1_scaling_14kobj_object_container_fgrandpos_bgrandpos_fgclip_NOrobotbaserandpos_bgcache10_bgclip_3L2obsAlign_refineActionSpace
usd_name: scene_usds/sim2real_scenes_align_for_scaling/base

Here, we use RoboTiq as the gripper. After every 100 successfully generated samples, the program will restart the object preloading process to ensure diversity even within a single process, rather than generating data only from the same preloaded 90 objects. The remaining configurations are straightforward and require no special explanation.

The complete configuration file can be found in the Codebase, and you can use it directly.

Terminal window
# Make sure the AnyGrasp service is running
python demogen.py -cfg configs/tasks/scaling_up/system1_scaling_14kobj_object_container_fgrandpos_bgrandpos_fgclip_NOrobotbaserandpos_bgcache10_bgclip_3L2obsAlign_refineActionSpace.yml
python render.py -cfg configs/tasks/scaling_up/system1_scaling_14kobj_object_container_fgrandpos_bgrandpos_fgclip_NOrobotbaserandpos_bgcache10_bgclip_3L2obsAlign_refineActionSpace.yml

If you need to use your own assets, then annotate them and replace the folder path in the object configuration with your asset path.