-
Notifications
You must be signed in to change notification settings - Fork 51
Description
Please provide the below details to ensure we understand your needs
Hello,
Recently, we hit some issue of modifying the QP to RTR. But from the log, we couldn't find any useful info to dig further. So, do consider to log the parameters when QP opertaions?
error log:
main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:420: non-zero status: 22 ibv_modify_qp failed.
The variables in the attr should be logged:
` attr.qp_state = IBV_QPS_RTR;
attr.path_mtu = port_attr->active_mtu;
attr.dest_qp_num = ep_handle->qpn;
attr.rq_psn = 0;
if (port_attr->lid == 0) {
ib_get_gid_index(&ftable, device->context, portid, port_attr->gid_tbl_len,
&device->gid_info[portid - 1].local_gid_index, ibrc_state->log_level,
ibrc_state->options);
ftable.query_gid(device->context, portid, device->gid_info[portid - 1].local_gid_index,
&device->gid_info[portid - 1].local_gid);
attr.ah_attr.is_global = 1;
attr.ah_attr.grh.dgid.global.subnet_prefix = ep_handle->spn;
attr.ah_attr.grh.dgid.global.interface_id = ep_handle->iid;
attr.ah_attr.grh.flow_label = 0;
attr.ah_attr.grh.sgid_index = device->gid_info[portid - 1].local_gid_index;
attr.ah_attr.grh.hop_limit = 255;
attr.ah_attr.grh.traffic_class = ibrc_state->options->IB_TRAFFIC_CLASS;
} else {
attr.ah_attr.dlid = ep_handle->lid;
attr.ah_attr.is_global = 0;
}
attr.max_dest_rd_atomic = NVSHMEMT_IBRC_MAX_RD_ATOMIC;
attr.min_rnr_timer = 12;
attr.ah_attr.sl = ibrc_state->options->IB_SL;
attr.ah_attr.src_path_bits = 0;
attr.ah_attr.port_num = portid;
flags = IBV_QP_STATE | IBV_QP_AV | IBV_QP_PATH_MTU | IBV_QP_DEST_QPN | IBV_QP_RQ_PSN |
IBV_QP_MIN_RNR_TIMER | IBV_QP_MAX_DEST_RD_ATOMIC;
status = ftable.modify_qp(ep->qp, &attr, flags);
NVSHMEMI_NZ_ERROR_JMP(status, NVSHMEMX_ERROR_INTERNAL, out, "ibv_modify_qp failed \n");
`