Purpose: This script detects and prints the latencies for all PING
and PONG operations, this can be used for detecting latencies
between kernel call stack used for ping and pong. Can be used for
detecting kernel latencies problems on these functions.

Arguments:
	MIN_LAT:
		Provide "-D MIN_LAT=1000" if we need to track packet with latency 1000 usec or more
			"-D MIN_LAT=0" if we need to track all packets
	DEBUG:
		Provide "-D DEBUG" as additional argument for additional debug logs
	None:
		If MIN_LAT is not provided, the defualt value of MIN_LAT will be 500000 usec
(Refer sample output for example).

Additionally provide uek5 as argument if running kernel is UEK5 - "./rds_ping.d -D uek5"
If running kernel is UEK8 run it as "./rds_ping.d -D uek8"

Output format: each line lists
	<Time>		- Actual timestamp
	<Connection>	- A joint string with <saddr,daddr,Qos>
	<Req>		- Ping or Pong
	<SrcPort>	- Source Port
	<sendq_wt>	- sendq_wt = rds_ib_xmit - rds_sendmsg [or rds_send_pong for pong]
	<comp_wt>	- comp_wt = rds_message_put - rds_ib_xmit
	<recvq_wt>	- recvq_wt = rds_ib_inc_copy_to_user [or rds_send_pong for pong] - rds_recv_incoming
	<total>		- total = [rds_ib_inc_copy_to_user - rds_sendmsg for ping] or [rds_message_put - rds_recv_incoming for pong]


Sample output with DEBUG and MIN_LAT flag:
# ./rds_ping.d -DDEBUG -DMIN_LAT=0
==============================================================================================
| rds-ping application                       |                         user space            |
==============================================================================================
          v                       ^
          |                       |
  +------------------+    +-------------------------+            +------------------+
  |    rds_sendmsg   |    | rds_ib_inc_copy_to_user |            |  rds_send_pong   | <------
  +------------------+    +-------------------------+            +------------------+       |
          |                       |                                     |                   |
          v                       ^                                     v                   |
  +------------------+        +--------------------+             +------------------+       |
  |   rds_ib_xmit    |        | rds_recv_incoming  |             |   rds_ib_xmit    |       |
  +------------------+        +--------------------+             +------------------+       |
          |                       ^                                     |                   |
          v                       |                                     v                   |
  +--------------------+          |                         +--------------------+          |
  |  rds_message_put   |          ------------------------  |   rds_message_put  |          |
  +--------------------+                                    +--------------------+          |
          |                                                                                 |
          |                                                                                 |
          |                                                                                 |
          |                                          +-------------------+                  |
          |----------------------------------------- | rds_recv_incoming | ------------------
                                                     +-------------------+
 |---------------------Ping-----------------------| |=================Pong==================|


====Time measurements on Ping side====

         Δtime = sendq_wt       Δtime = comp_wt                                 Δtime = recvq_wt
        /                \   /                \                             /                  \
       /                  \ /                  \                           /                    \
+-------------+      +-------------+      +-----------------+      +-------------------+     +-------------------------+
| rds_sendmsg |  =>  | rds_ib_xmit |  =>  | rds_message_put |  =>  | rds_recv_incoming | ... | rds_ib_inc_copy_to_user |
+-------------+      +-------------+      +-----------------+      +-------------------+     +-------------------------+
      \                                                                                           /
       \ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Δtime = total _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /


====Time measurements on Pong side====

               Δtime = recvq_wt         Δtime = sendq_wt       Δtime = comp_wt
             /                \      /                \    /                 \
            /                  \    /                  \  /                   \
+-------------------+      +---------------+      +-------------+      +-----------------+
| rds_recv_incoming |  =>  | rds_send_pong |  =>  | rds_ib_xmit |  =>  | rds_message_put |
+-------------------+      +---------------+      +-------------+      +-----------------+
           \                                                                   /
            \ _ _ _ _ _ _ _ _ _ _ _ _ Δtime = total _ _ _ _ _ _ _ _ _ _ _ _ _ /


DEBUG Enabled
Note: Displaying only packets with lat >= MIN_LAT (0 USEC)
[Time                ] Connection                             Req  SrcPort     sendq_wt      comp_wt     recvq_wt        total
[2026 Aug 11 07:49:31.20492214315594] DEBUG[rds_sendmsg] <192.168.50.186,192.168.50.187,0> ping sport=17332 dport=0
[2026 Aug 11 07:49:31.20492214358881] DEBUG[rds_ib_xmit] <192.168.50.186,192.168.50.187,0> ping sport=17332 dport=0
[2026 Aug 11 07:49:31.20492214498930] DEBUG[rds_message_put] <192.168.50.186,192.168.50.187,0> ping sport=17332 dport=0
[2026 Aug 11 07:49:31.20492214583586] DEBUG[rds_recv_incoming] <192.168.50.186,192.168.50.187,0> ping sport=17332 dport=0
[2026 Aug 11 07:49:31.20492214636856] DEBUG[rds_ib_inc_copy_to_user] <192.168.50.186,192.168.50.187,0> Ping sport=17332 dport=0 comm=rds-ping sendmsg_ts=20492214315594 xmit_ts=20492214358881 sendcomp_ts=20492214498930 recv_inc_ts=20492214583586 copy_to_user_ts=20492214636856
[2026 Aug 11 07:49:31] <192.168.50.186,192.168.50.187,0>    Ping   17332           43          140           53          321
[2026 Aug 11 07:49:32.20493214827393] DEBUG[rds_sendmsg] <192.168.50.186,192.168.50.187,0> ping sport=44787 dport=0
[2026 Aug 11 07:49:32.20493214884039] DEBUG[rds_ib_xmit] <192.168.50.186,192.168.50.187,0> ping sport=44787 dport=0
[2026 Aug 11 07:49:32.20493215006922] DEBUG[rds_message_put] <192.168.50.186,192.168.50.187,0> ping sport=44787 dport=0
[2026 Aug 11 07:49:32.20493215126962] DEBUG[rds_recv_incoming] <192.168.50.186,192.168.50.187,0> ping sport=44787 dport=0
[2026 Aug 11 07:49:32.20493215219113] DEBUG[rds_ib_inc_copy_to_user] <192.168.50.186,192.168.50.187,0> Ping sport=44787 dport=0 comm=rds-ping sendmsg_ts=20493214827393 xmit_ts=20493214884039 sendcomp_ts=20493215006922 recv_inc_ts=20493215126962 copy_to_user_ts=20493215219113
[2026 Aug 11 07:49:32] <192.168.50.186,192.168.50.187,0>    Ping   44787           56          122           92          391
[2026 Aug 11 07:49:33.20494215409752] DEBUG[rds_sendmsg] <192.168.50.186,192.168.50.187,0> ping sport=64267 dport=0
[2026 Aug 11 07:49:33.20494215466670] DEBUG[rds_ib_xmit] <192.168.50.186,192.168.50.187,0> ping sport=64267 dport=0
[2026 Aug 11 07:49:33.20494215597623] DEBUG[rds_message_put] <192.168.50.186,192.168.50.187,0> ping sport=64267 dport=0
[2026 Aug 11 07:49:33.20494215681488] DEBUG[rds_recv_incoming] <192.168.50.186,192.168.50.187,0> ping sport=64267 dport=0
[2026 Aug 11 07:49:33.20494215769315] DEBUG[rds_ib_inc_copy_to_user] <192.168.50.186,192.168.50.187,0> Ping sport=64267 dport=0 comm=rds-ping sendmsg_ts=20494215409752 xmit_ts=20494215466670 sendcomp_ts=20494215597623 recv_inc_ts=20494215681488 copy_to_user_ts=20494215769315
[2026 Aug 11 07:49:33] <192.168.50.186,192.168.50.187,0>    Ping   64267           56          130           87          359
[2026 Aug 11 07:49:34.20495215961293] DEBUG[rds_sendmsg] <192.168.50.186,192.168.50.187,0> ping sport=6583 dport=0
[2026 Aug 11 07:49:34.20495216058242] DEBUG[rds_ib_xmit] <192.168.50.186,192.168.50.187,0> ping sport=6583 dport=0
[2026 Aug 11 07:49:34.20495216200771] DEBUG[rds_message_put] <192.168.50.186,192.168.50.187,0> ping sport=6583 dport=0
[2026 Aug 11 07:49:34.20495216288933] DEBUG[rds_recv_incoming] <192.168.50.186,192.168.50.187,0> ping sport=6583 dport=0
[2026 Aug 11 07:49:34.20495216366431] DEBUG[rds_ib_inc_copy_to_user] <192.168.50.186,192.168.50.187,0> Ping sport=6583 dport=0 comm=rds-ping sendmsg_ts=20495215961293 xmit_ts=20495216058242 sendcomp_ts=20495216200771 recv_inc_ts=20495216288933 copy_to_user_ts=20495216366431
[2026 Aug 11 07:49:34] <192.168.50.186,192.168.50.187,0>    Ping    6583           96          142           77          405

Remote side
DEBUG Enabled
Note: Displaying only packets with lat >= MIN_LAT (0 USEC)
[Time                ] Connection                             Req  SrcPort     sendq_wt      comp_wt     recvq_wt        total
[2026 Aug 11 07:49:31.446861240891737] DEBUG[rds_recv_incoming] <192.168.50.186,192.168.50.187,0> pong sport=17332 dport=0
[2026 Aug 11 07:49:31.446861240924847] DEBUG[rds_send_pong] <192.168.50.186,192.168.50.187,0> pong sport=17332 dport=0
[2026 Aug 11 07:49:31.446861241026243] DEBUG[rds_ib_xmit] <192.168.50.186,192.168.50.187,0> pong sport=17332 dport=0
[2026 Aug 11 07:49:32.446862241415267] DEBUG[rds_recv_incoming] <192.168.50.186,192.168.50.187,0> pong sport=44787 dport=0
[2026 Aug 11 07:49:32.446862241448438] DEBUG[rds_send_pong] <192.168.50.186,192.168.50.187,0> pong sport=44787 dport=0
[2026 Aug 11 07:49:32.446862241527960] DEBUG[rds_ib_xmit] <192.168.50.186,192.168.50.187,0> pong sport=44787 dport=0
[2026 Aug 11 07:49:33.446863241982434] DEBUG[rds_recv_incoming] <192.168.50.186,192.168.50.187,0> pong sport=64267 dport=0
[2026 Aug 11 07:49:33.446863242015685] DEBUG[rds_send_pong] <192.168.50.186,192.168.50.187,0> pong sport=64267 dport=0
[2026 Aug 11 07:49:33.446863242097751] DEBUG[rds_ib_xmit] <192.168.50.186,192.168.50.187,0> pong sport=64267 dport=0
[2026 Aug 11 07:49:34.446864242572296] DEBUG[rds_recv_incoming] <192.168.50.186,192.168.50.187,0> pong sport=6583 dport=0
[2026 Aug 11 07:49:34.446864242621071] DEBUG[rds_send_pong] <192.168.50.186,192.168.50.187,0> pong sport=6583 dport=0
[2026 Aug 11 07:49:34.446864242697198] DEBUG[rds_ib_xmit] <192.168.50.186,192.168.50.187,0> pong sport=6583 dport=0


Sample output (without DEBUG flag and without mentioning MIN_LAT Value, by default the MIN_LAT value is taken as 500000 USEC)
#./rds_ping.d
Note: Displaying only packets with lat >= MIN_LAT (500000 USEC)
[Time                ] Connection                             Req  SrcPort     sendq_wt      comp_wt     recvq_wt        total
[2025 May  5 06:26:56] <192.168.100.39,192.168.100.124,0>    Ping   63137      4823013      1658106           49      6481807
[2025 May  5 06:26:56] <192.168.100.39,192.168.100.124,0>    Ping   32694      3822892      1658120          121      5481765
[2025 May  5 06:26:56] <192.168.100.39,192.168.100.124,0>    Ping    9503      2821739      1658118          147      4480638
[2025 May  5 06:26:56] <192.168.100.39,192.168.100.124,0>    Ping   15538      1820617      1658116          170      3479536
[2025 May  5 06:26:56] <192.168.100.39,192.168.100.124,0>    Ping   57903       819415      1658114          187      2478350
[2025 May  5 06:26:56] <192.168.100.39,192.168.100.124,0>    Ping   12973           30      1476262          203      1477128

Sample output with MIN_LAT flag set through ARGS, without DEBUG flag:
# ./rds_ping.d -DMIN_LAT=0
Note: Displaying only packets with lat >= MIN_LAT (0 USEC)
[Time                ] Connection                             Req  SrcPort     sendq_wt      comp_wt     recvq_wt        total
[2025 May  2 16:12:01] <192.168.100.117,192.168.100.180,0>    Ping   57353          224           55          207          769
[2025 May  2 16:12:02] <192.168.100.117,192.168.100.180,0>    Ping   33426          111          117          140          516
[2025 May  2 16:12:03] <192.168.100.117,192.168.100.180,0>    Ping   25577          232          132          158          575
[2025 May  2 16:12:04] <192.168.100.117,192.168.100.180,0>    Ping   55819           88          101           75          358

Remote side:
# ./rds_ping.d -DMIN_LAT=0
Note: Displaying only packets with lat >= MIN_LAT (0 USEC)
[Time                ] Connection                             Req  SrcPort     sendq_wt      comp_wt     recvq_wt        total
[2025 May  2 16:12:01] <192.168.100.117,192.168.100.180,0>    Pong   57353           78           35           52          165
[2025 May  2 16:12:02] <192.168.100.117,192.168.100.180,0>    Pong   33426           75           27           19          122
[2025 May  2 16:12:03] <192.168.100.117,192.168.100.180,0>    Pong   25577           31           15            5           53
[2025 May  2 16:12:04] <192.168.100.117,192.168.100.180,0>    Pong   55819           48           14            6           70


