It's found that server testhost01 is not pingable, and by locating the port on switch, below symptoms were found:
#1, port is is up and connected, and mac is learnt on the interface on both switch/L3 router
ucf-a1z6-as50-swi-1#sh int Gi1/27
GigabitEthernet1/27 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet Port, address is 5057.a8c4.99da (bia 5057.a8c4.99da)
Description: testhost01-net0
ucf-a1z6-as50-swi-1#sh mac address-table interface Gi1/27
vlan mac address type protocols port
-------+---------------+--------+---------------------+--------------------
124 0010.e00d.a2ba dynamic ip GigabitEthernet1/27
ucf-a1z6-rtr-1# sh ip arp vrf all | i 0010.e00d.a2ba
10.245.168.147 0.896955 0010.e00d.a2ba Vlan124
#2, we are continuously observing block transmit queue on the switch port
Oct 12 10:52:31.764: %C4K_HWPORTMAN-4-BLOCKEDTXQUEUE: Blocked transmit queue HwTxQId1 on Switch Phyport Gi1/27, count=583830
Oct 12 10:52:31.764: %C4K_HWPORTMAN-4-BLOCKEDTXQUEUE: Blocked transmit queue HwTxQId3 on Switch Phyport Gi1/27, count=583831
Oct 12 10:53:50.706: %C4K_HWPORTMAN-4-BLOCKEDTXQUEUE: Blocked transmit queue HwTxQId2 on Switch Phyport Gi1/27, count=532423
ucf-a1z6-as50-swi-1#sh clock
11:01:00.229 UTC Fri Oct 12 2018
#3, the drops on the switch port interface are increasing, and no traffic on the interface
ucf-a1z6-as50-swi-1#sh int gi1/27 | i drop|res|flap
Hardware is Gigabit Ethernet Port, address is 5057.a8c4.99da (bia 5057.a8c4.99da)
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 601929
ucf-a1z6-as50-swi-1#sh int gi1/27 | i drop|res|flap
Hardware is Gigabit Ethernet Port, address is 5057.a8c4.99da (bia 5057.a8c4.99da)
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 602075
ucf-a1z6-as50-swi-1#sh int gi1/27 | i load|rate
reliability 255/255, txload 1/255, rxload 1/255
Queueing strategy: fifo
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
From above, we can see continuous pause frames being sent to the switch causes the transmit buffer on switch port to fill up. When the transmit buffer on switch port is full, it drops further traffic destined to the server.
And checking from server side, the host was not hung, and we tried rebooting host but issue persisted. Also, LED status of Gi1/27 is not amber either.
Based on above, we planned to try with below:
1#, move the cable from Gi1/27 to Gi1/7 which is of the same vlan on the switch and in "not connected" status.
2#, if above doesn't help, then most probably the NIC of server is faulty. We can try replace the NIC card or connect secondary NIC to Gi1/27 earlier unplugged.
Finally, we tried option #1, and the issue got resolved.