Perftest: Add support for Send with Immediate verb

[Yue-ByteDance]

Currently, perftest supports RDMA Write / Write-with-Immediate / Send /
Read and Atomic operations, but lacks Send-with-Immediate support. This
commit adds Send-with-Immediate functionality, allowing users to test it
via ib_send_bw with --send_with_imm flag.

Additionally, to verify the completion logic for operations with
immediate data is correct, completion events are now validated to ensure
the IBV_WC_WITH_IMM flag is set.

[mrgolin]

Generally looks good to me but I think we should add a check in run_iter_bw* and run_iter_lat that completion really has imm data. Something like:

if (user_param->use_send_with_imm && !(wc[i].wc_flags & IBV_WC_WITH_IMM)) {
        // error
        ...

Also can you please describe your changes in the commit message?

[Yue-ByteDance]

Generally looks good to me but I think we should add a check in run_iter_bw* and run_iter_lat that completion really has imm data. Something like:

if (user_param->use_send_with_imm && !(wc[i].wc_flags & IBV_WC_WITH_IMM)) {
        // error
        ...

Also can you please describe your changes in the commit message?

No problem, I will push a new version later

[Yue-ByteDance]

@mrgolin I've updated the code and the commit message with your suggestion, could you please take a look at the new version? Thanks!