Basic Usage Tutorial

In the Quickstart section, we introduced how to use client.py to test the simulated environment. This section explains how to evaluate your own models on GenManipBench.

Overview

GenManip is designed so that users can evaluate their models with minimal concern about the internal details of Isaac Sim or GenManip itself. Once your environment is correctly configured, you only need to communicate with GenManip through the defined protocol — that is, make sure your input and output follow the required communication format.

On top of this, GenManip supports a variety of benchmarks that evaluate models across multiple dimensions. You can find detailed descriptions of each benchmark in the Benchmarks section. For any benchmark, you can follow this general workflow:

# Download assets and start server
python ray_eval_server.py -cfg GenManipSuite/GenManip-Package-Basic

# Run client
python standalone_tools/client.py --worker_ids 0 # Let the server create an Isaac Sim test instance

Detailed benchmark-specific information can be found in the corresponding Benchmarks pages.

Integrating Your Model

You can refer to the file standalone_tools/client.py, which outlines the communication logic. This script retrieves data from the Isaac Sim benchmark environment and returns a static action (e.g., example joint positions and end-effector poses for different robot arms). For more information about the interface and communication protocol, please refer to the Communication Protocol section.

It is important to note that the client.py script is completely standalone — it has no external dependencies. This makes it very easy to integrate into your own project and add your model’s inference logic. For example, you could modify the main loop as follows:

...
if __name__ == "__main__":
    args = parse_args()
    host = args.host
    port = args.port
    worker_ids = args.worker_ids
    base_url = f"http://{host}:{port}"

    # Create workers on server here, make sure they are created before stepping
    client = EvalClient(base_url, worker_ids)
    print(f"Created workers {worker_ids} on server {base_url}.")

    # wrap the eval loop in a try-finally to ensure cleanup
    try:

        obs = client.reset()

        while True:

            action = {
                i: fake_action(args.arm_type, args.gripper_type, args.control_type)
                i: model.inference(process_data(obs[i])) # Or use batch inference
                for i in worker_ids
            }

            start = time.time()
            obs, done = client.step(action)
            print(f"workers {worker_ids} Step time: {time.time() - start:.4f} seconds")

            if done:
                # finished all evaluations
                break
            if obs is None:
                break
            if obs[worker_ids[0]]["obs"]["reset"]:  # type: ignore
                # model.reset()
                model.reset()
                pass
    finally:
        client.kill_workers()
        print("Client cleaned.")