Humanoid
Description
This environment is based on the environment introduced by Tassa, Erez and Todorov in “Synthesis and stabilization of complex behaviors through online trajectory optimization”. The 3D bipedal robot is designed to simulate a human. It has a torso (abdomen) with a pair of legs and arms. The legs each consist of two links, and so the arms (representing the knees and elbows respectively). The goal of the environment is to walk forward as fast as possible without falling over.
Action Space
The action space is a Box(1, 1, (17,), float32)
. An action represents the torques applied at the hinge joints.
 Num  Action  Control Min  Control Max  Name (in corresponding XML file)  Joint  Unit 
—–——————————————————————————————————
 0  Torque applied on the hinge in the ycoordinate of the abdomen  0.4  0.4  hip_1 (front_left_leg)  hinge  torque (N m) 
 1  Torque applied on the hinge in the zcoordinate of the abdomen  0.4  0.4  angle_1 (front_left_leg)  hinge  torque (N m) 
 2  Torque applied on the hinge in the xcoordinate of the abdomen  0.4  0.4  hip_2 (front_right_leg)  hinge  torque (N m) 
 3  Torque applied on the rotor between torso/abdomen and the right hip (xcoordinate)  0.4  0.4  right_hip_x (right_thigh)  hinge  torque (N m) 
 4  Torque applied on the rotor between torso/abdomen and the right hip (zcoordinate)  0.4  0.4  right_hip_z (right_thigh)  hinge  torque (N m) 
 5  Torque applied on the rotor between torso/abdomen and the right hip (ycoordinate)  0.4  0.4  right_hip_y (right_thigh)  hinge  torque (N m) 
 6  Torque applied on the rotor between the right hip/thigh and the right shin  0.4  0.4  right_knee  hinge  torque (N m) 
 7  Torque applied on the rotor between torso/abdomen and the left hip (xcoordinate)  0.4  0.4  left_hip_x (left_thigh)  hinge  torque (N m) 
 8  Torque applied on the rotor between torso/abdomen and the left hip (zcoordinate)  0.4  0.4  left_hip_z (left_thigh)  hinge  torque (N m) 
 9  Torque applied on the rotor between torso/abdomen and the left hip (ycoordinate)  0.4  0.4  left_hip_y (left_thigh)  hinge  torque (N m) 
 10  Torque applied on the rotor between the left hip/thigh and the left shin  0.4  0.4  left_knee  hinge  torque (N m) 
 11  Torque applied on the rotor between the torso and right upper arm (coordinate 1)  0.4  0.4  right_shoulder1  hinge  torque (N m) 
 12  Torque applied on the rotor between the torso and right upper arm (coordinate 2)  0.4  0.4  right_shoulder2  hinge  torque (N m) 
 13  Torque applied on the rotor between the right upper arm and right lower arm  0.4  0.4  right_elbow  hinge  torque (N m) 
 14  Torque applied on the rotor between the torso and left upper arm (coordinate 1)  0.4  0.4  left_shoulder1  hinge  torque (N m) 
 15  Torque applied on the rotor between the torso and left upper arm (coordinate 2)  0.4  0.4  left_shoulder2  hinge  torque (N m) 
 16  Torque applied on the rotor between the left upper arm and left lower arm  0.4  0.4  left_elbow  hinge  torque (N m) 
Observation Space
Observations consist of positional values of different body parts of the Humanoid,
followed by the velocities of those individual parts (their derivatives) with all the
positions ordered before all the velocities.
By default, observations do not include the x and ycoordinates of the torso. These may
be included by passing exclude_current_positions_from_observation=False
during construction.
In that case, the observation space will have 378 dimensions where the first two dimensions
represent the x and ycoordinates of the torso.
Regardless of whether exclude_current_positions_from_observation
was set to true or false, the x and ycoordinates
will be returned in info
with keys "x_position"
and "y_position"
, respectively.
However, by default, the observation is a ndarray
with shape (376,)
where the elements correspond to the following:
 Num  Observation  Min  Max  Name (in corresponding XML file)  Joint  Unit 
 —  —————————————————————————————————————  —  —  ——————————–  —–  ————————– 
 0  zcoordinate of the torso (centre)  Inf  Inf  root  free  position (m) 
 1  xorientation of the torso (centre)  Inf  Inf  root  free  angle (rad) 
 2  yorientation of the torso (centre)  Inf  Inf  root  free  angle (rad) 
 3  zorientation of the torso (centre)  Inf  Inf  root  free  angle (rad) 
 4  worientation of the torso (centre)  Inf  Inf  root  free  angle (rad) 
 5  zangle of the abdomen (in lower_waist)  Inf  Inf  abdomen_z  hinge  angle (rad) 
 6  yangle of the abdomen (in lower_waist)  Inf  Inf  abdomen_y  hinge  angle (rad) 
 7  xangle of the abdomen (in pelvis)  Inf  Inf  abdomen_x  hinge  angle (rad) 
 8  xcoordinate of angle between pelvis and right hip (in right_thigh)  Inf  Inf  right_hip_x  hinge  angle (rad) 
 9  zcoordinate of angle between pelvis and right hip (in right_thigh)  Inf  Inf  right_hip_z  hinge  angle (rad) 
 19  ycoordinate of angle between pelvis and right hip (in right_thigh)  Inf  Inf  right_hip_y  hinge  angle (rad) 
 11  angle between right hip and the right shin (in right_knee)  Inf  Inf  right_knee  hinge  angle (rad) 
 12  xcoordinate of angle between pelvis and left hip (in left_thigh)  Inf  Inf  left_hip_x  hinge  angle (rad) 
 13  zcoordinate of angle between pelvis and left hip (in left_thigh)  Inf  Inf  left_hip_z  hinge  angle (rad) 
 14  ycoordinate of angle between pelvis and left hip (in left_thigh)  Inf  Inf  left_hip_y  hinge  angle (rad) 
 15  angle between left hip and the left shin (in left_knee)  Inf  Inf  left_knee  hinge  angle (rad) 
 16  coordinate1 (multiaxis) angle between torso and right arm (in right_upper_arm)  Inf  Inf  right_shoulder1  hinge  angle (rad) 
 17  coordinate2 (multiaxis) angle between torso and right arm (in right_upper_arm)  Inf  Inf  right_shoulder2  hinge  angle (rad) 
 18  angle between right upper arm and right_lower_arm  Inf  Inf  right_elbow  hinge  angle (rad) 
 19  coordinate1 (multiaxis) angle between torso and left arm (in left_upper_arm)  Inf  Inf  left_shoulder1  hinge  angle (rad) 
 20  coordinate2 (multiaxis) angle between torso and left arm (in left_upper_arm)  Inf  Inf  left_shoulder2  hinge  angle (rad) 
 21  angle between left upper arm and left_lower_arm  Inf  Inf  left_elbow  hinge  angle (rad) 
 22  xcoordinate velocity of the torso (centre)  Inf  Inf  root  free  velocity (m/s) 
 23  ycoordinate velocity of the torso (centre)  Inf  Inf  root  free  velocity (m/s) 
 24  zcoordinate velocity of the torso (centre)  Inf  Inf  root  free  velocity (m/s) 
 25  xcoordinate angular velocity of the torso (centre)  Inf  Inf  root  free  anglular velocity (rad/s) 
 26  ycoordinate angular velocity of the torso (centre)  Inf  Inf  root  free  anglular velocity (rad/s) 
 27  zcoordinate angular velocity of the torso (centre)  Inf  Inf  root  free  anglular velocity (rad/s) 
 28  zcoordinate of angular velocity of the abdomen (in lower_waist)  Inf  Inf  abdomen_z  hinge  anglular velocity (rad/s) 
 29  ycoordinate of angular velocity of the abdomen (in lower_waist)  Inf  Inf  abdomen_y  hinge  anglular velocity (rad/s) 
 30  xcoordinate of angular velocity of the abdomen (in pelvis)  Inf  Inf  abdomen_x  hinge  aanglular velocity (rad/s) 
 31  xcoordinate of the angular velocity of the angle between pelvis and right hip (in right_thigh)  Inf  Inf  right_hip_x  hinge  anglular velocity (rad/s) 
 32  zcoordinate of the angular velocity of the angle between pelvis and right hip (in right_thigh)  Inf  Inf  right_hip_z  hinge  anglular velocity (rad/s) 
 33  ycoordinate of the angular velocity of the angle between pelvis and right hip (in right_thigh)  Inf  Inf  right_hip_y  hinge  anglular velocity (rad/s) 
 34  angular velocity of the angle between right hip and the right shin (in right_knee)  Inf  Inf  right_knee  hinge  anglular velocity (rad/s) 
 35  xcoordinate of the angular velocity of the angle between pelvis and left hip (in left_thigh)  Inf  Inf  left_hip_x  hinge  anglular velocity (rad/s) 
 36  zcoordinate of the angular velocity of the angle between pelvis and left hip (in left_thigh)  Inf  Inf  left_hip_z  hinge  anglular velocity (rad/s) 
 37  ycoordinate of the angular velocity of the angle between pelvis and left hip (in left_thigh)  Inf  Inf  left_hip_y  hinge  anglular velocity (rad/s) 
 38  angular velocity of the angle between left hip and the left shin (in left_knee)  Inf  Inf  left_knee  hinge  anglular velocity (rad/s) 
 39  coordinate1 (multiaxis) of the angular velocity of the angle between torso and right arm (in right_upper_arm)  Inf  Inf  right_shoulder1  hinge  anglular velocity (rad/s) 
 40  coordinate2 (multiaxis) of the angular velocity of the angle between torso and right arm (in right_upper_arm)  Inf  Inf  right_shoulder2  hinge  anglular velocity (rad/s) 
 41  angular velocity of the angle between right upper arm and right_lower_arm  Inf  Inf  right_elbow  hinge  anglular velocity (rad/s) 
 42  coordinate1 (multiaxis) of the angular velocity of the angle between torso and left arm (in left_upper_arm)  Inf  Inf  left_shoulder1  hinge  anglular velocity (rad/s) 
 43  coordinate2 (multiaxis) of the angular velocity of the angle between torso and left arm (in left_upper_arm)  Inf  Inf  left_shoulder2  hinge  anglular velocity (rad/s) 
 44  angular velocitty of the angle between left upper arm and left_lower_arm  Inf  Inf  left_elbow  hinge  anglular velocity (rad/s) 
Additionally, after all the positional and velocity based values in the table,
the observation contains (in order):
 cinert: Mass and inertia of a single rigid body relative to the center of mass (this is an intermediate result of transition). It has shape 14*10 (nbody * 10) and hence adds to another 140 elements in the state space.
 cvel: Center of mass based velocity. It has shape 14 * 6 (nbody * 6) and hence adds another 84 elements in the state space
 qfrc_actuator: Constraint force generated as the actuator force. This has shape
(23,)
(nv * 1) and hence adds another 23 elements to the state space.  cfrc_ext: This is the center of mass based external force on the body. It has shape 14 * 6 (nbody * 6) and hence adds to another 84 elements in the state space. where nbody stands for the number of bodies in the robot and nv stands for the number of degrees of freedom (= dim(qvel)) The (x,y,z) coordinates are translational DOFs while the orientations are rotational DOFs expressed as quaternions. One can read more about free joints on the Mujoco Documentation. Note: Humanoidv4 environment no longer has the following contact forces issue. If using previous Humanoid versions from v4, there have been reported issues that using a MujocoPy version > 2.0 results in the contact forces always being 0. As such we recommend to use a MujocoPy version < 2.0 when using the Humanoid environment if you would like to report results with contact forces (if contact forces are not used in your experiments, you can use version > 2.0). ### Rewards The reward consists of three parts:
 healthy_reward: Every timestep that the humanoid is alive (see section Episode Termination for definition), it gets a reward of fixed value
healthy_reward
 forward_reward: A reward of walking forward which is measured as
forward_reward_weight
* (average center of mass before action  average center of mass after action)/dt. dt is the time between actions and is dependent on the frame_skip parameter (default is 5), where the frametime is 0.003  making the default dt = 5 * 0.003 = 0.015. This reward would be positive if the humanoid walks forward (in positive xdirection). The calculation for the center of mass is defined in the.py
file for the Humanoid.  ctrl_cost: A negative reward for penalising the humanoid if it has too
large of a control force. If there are nu actuators/controls, then the control has
shape
nu x 1
. It is measured asctrl_cost_weight
* sum(control^{2}).  contact_cost: A negative reward for penalising the humanoid if the external
contact force is too large. It is calculated by clipping
contact_cost_weight
* sum(external contact force^{2}) to the interval specified bycontact_cost_range
. The total reward returned is reward = healthy_reward + forward_reward  ctrl_cost  contact_cost andinfo
will also contain the individual reward terms ### Starting State All observations start in state (0.0, 0.0, 1.4, 1.0, 0.0 … 0.0) with a uniform noise in the range of [reset_noise_scale
,reset_noise_scale
] added to the positional and velocity values (values in the table) for stochasticity. Note that the initial z coordinate is intentionally selected to be high, thereby indicating a standing up humanoid. The initial orientation is designed to make it face forward as well. ### Episode End The humanoid is said to be unhealthy if the zposition of the torso is no longer contained in the closed interval specified by the argumenthealthy_z_range
. Ifterminate_when_unhealthy=True
is passed during construction (which is the default), the episode ends when any of the following happens:  Truncation: The episode duration reaches a 1000 timesteps
 Termination: The humanoid is unhealthy
If
terminate_when_unhealthy=False
is passed, the episode is ended only when 1000 timesteps are exceeded. ### Arguments No additional arguments are currently supported in v2 and lower.env = gym.make('Humanoidv4')
v3 and v4 take gym.make kwargs such as xml_file, ctrl_cost_weight, reset_noise_scale etc.env = gym.make('Humanoidv4', ctrl_cost_weight=0.1, ....)
 Parameter  Type  Default  Description   ——————————————–  ———  —————  ————————————————————————————————————————————————————————  xml_file
 str "humanoid.xml"
 Path to a MuJoCo model  forward_reward_weight
 float 1.25
 Weight for forward_reward term (see section on reward)  ctrl_cost_weight
 float 0.1
 Weight for ctrl_cost term (see section on reward)  contact_cost_weight
 float 5e7
 Weight for contact_cost term (see section on reward)  healthy_reward
 float 5.0
 Constant reward given if the humanoid is “healthy” after timestep  terminate_when_unhealthy
 bool True
 If true, issue a done signal if the zcoordinate of the torso is no longer in thehealthy_z_range
 healthy_z_range
 tuple (1.0, 2.0)
 The humanoid is considered healthy if the zcoordinate of the torso is in this range  reset_noise_scale
 float 1e2
 Scale of random perturbations of initial position and velocity (see section on Starting State)  exclude_current_positions_from_observation
 bool True
 Whether or not to omit the x and ycoordinates from observations. Excluding the position can serve as an inductive bias to induce positionagnostic behavior in policies  ### Version History  v4: all mujoco environments now use the mujoco bindings in mujoco>=2.1.3
 v3: support for gym.make kwargs such as xml_file, ctrl_cost_weight, reset_noise_scale etc. rgb rendering comes from tracking camera (so agent does not run away from screen)
 v2: All continuous control environments now use mujoco_py >= 1.50
 v1: max_time_steps raised to 1000 for robot based tasks. Added reward_threshold to environments.
 v0: Initial versions release (1.0.0)

Declaration
Swift
public let model: MjModel

Declaration
Swift
public var data: MjData

init(forwardRewardWeight:
ctrlCostWeight: healthyReward: terminateWhenUnhealthy: healthyZRange: resetNoiseScale: ) Declaration
Swift
public init( forwardRewardWeight: Double = 1.25, ctrlCostWeight: Double = 0.1, healthyReward: Double = 5.0, terminateWhenUnhealthy: Bool = true, healthyZRange: ClosedRange<Double> = 1.0...2.0, resetNoiseScale: Double = 0.01 ) throws

Declaration
Swift
public typealias ActType = Tensor<Float64>

Declaration
Swift
public typealias ObsType = Tensor<Float64>

Declaration
Swift
public typealias RewardType = Float

Declaration
Swift
public typealias TerminatedType = Bool

Declaration
Swift
public func step(action: ActType) > (ObsType, RewardType, TerminatedType, [String : Any])

Declaration
Swift
public func reset(seed: Int?) > (ObsType, [String : Any])

Declaration
Swift
public static var rewardThreshold: Float { get }

Declaration
Swift
public static var actionSpace: [ClosedRange<Float>] { get }

Declaration
Swift
public static var stateSize: Int { get }