NAME | SYNOPSIS | DESCRIPTION | NOTES | RDMA VERBS | CLIENT OPERATION | SERVER OPERATION | RETURN CODES | SEE ALSO | COLOPHON

RDMA_CM(7)              Librdmacm Programmer's Manual             RDMA_CM(7)

NAME         top

       rdma_cm - RDMA communication manager.

SYNOPSIS         top

       #include <rdma/rdma_cma.h>

DESCRIPTION         top

       Used to establish communication over RDMA transports.

NOTES         top

       The RDMA CM is a communication manager used to setup reliable,
       connected and unreliable datagram data transfers.  It provides an
       RDMA transport neutral interface for establishing connections.  The
       API concepts are based on sockets, but adapted for queue pair (QP)
       based semantics: communication must be over a specific RDMA device,
       and data transfers are message based.

       The RDMA CM can control both the QP and communication management
       (connection setup / teardown) portions of an RDMA API, or only the
       communication management piece.  It works in conjunction with the
       verbs API defined by the libibverbs library.  The libibverbs library
       provides the underlying interfaces needed to send and receive data.

       The RDMA CM can operate asynchronously or synchronously.  The mode of
       operation is controlled by the user through the use of the rdma_cm
       event channel parameter in specific calls.  If an event channel is
       provided, an rdma_cm identifier will report its event data (results
       of connecting, for example), on that channel.  If a channel is not
       provided, then all rdma_cm operations for the selected rdma_cm
       identifier will block until they complete.

RDMA VERBS         top

       The rdma_cm supports the full range of verbs available through the
       libibverbs library and interfaces.  However, it also provides wrapper
       functions for some of the more commonly used verbs funcationality.
       The full set of abstracted verb calls are:

       rdma_reg_msgs  - register an array of buffers for sending and
       receiving

       rdma_reg_read  - registers a buffer for RDMA read operations

       rdma_reg_write - registers a buffer for RDMA write operations

       rdma_dereg_mr  - deregisters a memory region

       rdma_post_recv  - post a buffer to receive a message

       rdma_post_send  - post a buffer to send a message

       rdma_post_read  - post an RDMA to read data into a buffer

       rdma_post_write - post an RDMA to send data from a buffer

       rdma_post_recvv  - post a vector of buffers to receive a message

       rdma_post_sendv  - post a vector of buffers to send a message

       rdma_post_readv  - post a vector of buffers to receive an RDMA read

       rdma_post_writev - post a vector of buffers to send an RDMA write

       rdma_post_ud_send - post a buffer to send a message on a UD QP

       rdma_get_send_comp - get completion status for a send or RDMA
       operation

       rdma_get_recv_comp - get information about a completed receive

CLIENT OPERATION         top

       This section provides a general overview of the basic operation for
       the active, or client, side of communication.  This flow assume
       asynchronous operation with low level call details shown.  For
       synchronous operation, calls to rdma_create_event_channel,
       rdma_get_cm_event, rdma_ack_cm_event, and rdma_destroy_event_channel
       would be eliminated.  Abstracted calls, such as rdma_create_ep
       encapsulate several of these calls under a single API.  Users may
       also refer to the example applications for code samples.  A general
       connection flow would be:

       rdma_getaddrinfo
              retrieve address information of the destination

       rdma_create_event_channel
              create channel to receive events

       rdma_create_id
              allocate an rdma_cm_id, this is conceptually similar to a
              socket

       rdma_resolve_addr
              obtain a local RDMA device to reach the remote address

       rdma_get_cm_event
              wait for RDMA_CM_EVENT_ADDR_RESOLVED event

       rdma_ack_cm_event
              ack event

       rdma_create_qp
              allocate a QP for the communication

       rdma_resolve_route
              determine the route to the remote address

       rdma_get_cm_event
              wait for RDMA_CM_EVENT_ROUTE_RESOLVED event

       rdma_ack_cm_event
              ack event

       rdma_connect
              connect to the remote server

       rdma_get_cm_event
              wait for RDMA_CM_EVENT_ESTABLISHED event

       rdma_ack_cm_event
              ack event

       Perform data transfers over connection

       rdma_disconnect
              tear-down connection

       rdma_get_cm_event
              wait for RDMA_CM_EVENT_DISCONNECTED event

       rdma_ack_cm_event
              ack event

       rdma_destroy_qp
              destroy the QP

       rdma_destroy_id
              release the rdma_cm_id

       rdma_destroy_event_channel
              release the event channel

       An almost identical process is used to setup unreliable datagram (UD)
       communication between nodes.  No actual connection is formed between
       QPs however, so disconnection is not needed.

       Although this example shows the client initiating the disconnect,
       either side of a connection may initiate the disconnect.

SERVER OPERATION         top

       This section provides a general overview of the basic operation for
       the passive, or server, side of communication.  A general connection
       flow would be:

       rdma_create_event_channel
              create channel to receive events

       rdma_create_id
              allocate an rdma_cm_id, this is conceptually similar to a
              socket

       rdma_bind_addr
              set the local port number to listen on

       rdma_listen
              begin listening for connection requests

       rdma_get_cm_event
              wait for RDMA_CM_EVENT_CONNECT_REQUEST event with a new
              rdma_cm_id

       rdma_create_qp
              allocate a QP for the communication on the new rdma_cm_id

       rdma_accept
              accept the connection request

       rdma_ack_cm_event
              ack event

       rdma_get_cm_event
              wait for RDMA_CM_EVENT_ESTABLISHED event

       rdma_ack_cm_event
              ack event

       Perform data transfers over connection

       rdma_get_cm_event
              wait for RDMA_CM_EVENT_DISCONNECTED event

       rdma_ack_cm_event
              ack event

       rdma_disconnect
              tear-down connection

       rdma_destroy_qp
              destroy the QP

       rdma_destroy_id
              release the connected rdma_cm_id

       rdma_destroy_id
              release the listening rdma_cm_id

       rdma_destroy_event_channel
              release the event channel

RETURN CODES         top

       =  0   success

       = -1   error - see errno for more details

       Most librdmacm functions return 0 to indicate success, and a -1
       return value to indicate failure.  If a function operates
       asynchronously, a return value of 0 means that the operation was
       successfully started.  The operation could still complete in error;
       users should check the status of the related event.  If the return
       value is -1, then errno will contain additional information regarding
       the reason for the failure.

       Prior versions of the library would return -errno and not set errno
       for some cases related to ENOMEM, ENODEV, ENODATA, EINVAL, and
       EADDRNOTAVAIL codes. Applications that want to check these codes and
       have compatibility with prior library versions must manually set
       errno to the negative of the return code if it is < -1.

SEE ALSO         top

       rdma_accept(3), rdma_ack_cm_event(3), rdma_bind_addr(3),
       rdma_connect(3), rdma_create_ep(3), rdma_create_event_channel(3),
       rdma_create_id(3), rdma_create_qp(3), rdma_dereg_mr(3),
       rdma_destroy_ep(3), rdma_destroy_event_channel(3),
       rdma_destroy_id(3), rdma_destroy_qp(3), rdma_disconnect(3),
       rdma_event_str(3), rdma_free_devices(3), rdma_getaddrinfo(3),
       rdma_get_cm_event(3), rdma_get_devices(3), rdma_get_dst_port(3),
       rdma_get_local_addr(3), rdma_get_peer_addr(3), rdma_get_recv_comp(3),
       rdma_get_request(3), rdma_get_send_comp(3), rdma_get_src_port(3),
       rdma_join_multicast(3), rdma_leave_multicast(3), rdma_listen(3),
       rdma_migrate_id(3), rdma_notify(3), rdma_post_read(3)
       rdma_post_readv(3), rdma_post_recv(3), rdma_post_recvv(3),
       rdma_post_send(3), rdma_post_sendv(3), rdma_post_ud_send(3),
       rdma_post_write(3), rdma_post_writev(3), rdma_reg_msgs(3),
       rdma_reg_read(3), rdma_reg_write(3), rdma_reject(3),
       rdma_resolve_addr(3), rdma_resolve_route(3), rdma_set_option(3)
       mckey(1), rdma_client(1), rdma_server(1), rping(1), ucmatose(1),
       udaddy(1)

COLOPHON         top

       This page is part of the rdma-core (RDMA Core Userspace Libraries and
       Daemons) project.  Information about the project can be found at 
       ⟨https://github.com/linux-rdma/rdma-core⟩.  If you have a bug report
       for this manual page, send it to linux-rdma@vger.kernel.org.  This
       page was obtained from the project's upstream Git repository
       ⟨https://github.com/linux-rdma/rdma-core.git⟩ on 2019-09-26.  (At
       that time, the date of the most recent commit that was found in the
       repository was 2019-09-25.)  If you discover any rendering problems
       in this HTML version of the page, or you believe there is a better or
       more up-to-date source for the page, or you have corrections or
       improvements to the information in this COLOPHON (which is not part
       of the original manual page), send a mail to man-pages@man7.org

librdmacm                        2010-07-19                       RDMA_CM(7)

Pages that refer to this page: cmtime(1)ibacm(1)mckey(1)rcopy(1)rdma_client(1)rdma_server(1)rdma_xclient(1)rdma_xserver(1)riostream(1)rping(1)rstream(1)ucmatose(1)udaddy(1)udpong(1)rdma_connect(3)rdma_create_ep(3)rdma_create_event_channel(3)rdma_create_id(3)rdma_dereg_mr(3)rdma_get_recv_comp(3)rdma_get_send_comp(3)rdma_listen(3)rdma_migrate_id(3)rdma_post_read(3)rdma_post_readv(3)rdma_post_recv(3)rdma_post_recvv(3)rdma_post_send(3)rdma_post_sendv(3)rdma_post_ud_send(3)rdma_post_write(3)rdma_post_writev(3)rdma_reg_msgs(3)rdma_reg_read(3)rdma_reg_write(3)