Journey Of A Packet through TCP/IP Part 2 - From Wireless Interface to IP Stack ·

Table of Contents

In the previous blog post, I explored how a packet is formed and received at the physical layer using a Raspberry Pi Zero 2W with an SDIO Wi-Fi module. In this part, I will go into how the packet travels from the wireless interface to the IP stack within the embedded Linux operating system.

Overview of the Journey
#

From our last discussion we know that once the packet is received at the physical layer, it is handed over netif_rx() function in the Linux kernel. Before we proceed further, we need to know about the key datastructure involved in this journey - the sk_buff (socket buffer).

Understanding `sk_buff`
#

The sk_buff structure is the fundamental data structure in the Linux kernel networking stack. It holds everything associated with a packet: the headers, the payload, and the metadata.

Zero-Copy Operations: It’s designed to allow protocol layers to add or remove headers by just moving pointers, without copying the payload.
Metadata: It carries essential routing info, timestamps, and device details.

The structure of sk_buff helps to add or remove headers as the packet traverses through different layers of the network stack. It also maintains pointers to the data, allowing efficient access and manipulation of the packet contents. It is designed to be flexible and efficient, enabling the kernel to handle high-speed network traffic effectively. When a new packet is received, the kernel allocates an sk_buff structure to hold the packet data and metadata. The packet is the enqueued in the list of received packets for further processing. The sk_buff structure is designed to create a linked list of packets, allowing the kernel to efficiently manage multiple packets in transit.

The layout of the sk_buff structure is complex. You can find it in include/linux/skbuff.h file of the Linux kernel source code. Below is a simplified version of the sk_buff structure highlighting the key data and pointers:

struct sk_buff {
	union {
		struct {
			/* These two members must be first to match sk_buff_head. */
			struct sk_buff		*next;
			struct sk_buff		*prev;

			union {
				struct net_device	*dev;
				/* Some protocols might use this space to store information,
				 * while device pointer would be NULL.
				 * UDP receive path is one user.
				 */
				unsigned long		dev_scratch;
			};
		};
		struct rb_node		rbnode; /* used in netem, ip4 defrag, and tcp stack */
		struct list_head	list;
		struct llist_node	ll_node;
	};

	struct sock		*sk;

	union {
		ktime_t		tstamp;
		u64		skb_mstamp_ns; /* earliest departure time */
	};
	/*
	 * This is the control buffer. It is free to use for every
	 * layer. Please put your private variables there. If you
	 * want to keep them across layers you have to do a skb_clone()
	 * first. This is owned by whoever has the skb queued ATM.
	 */
	char			cb[48] __aligned(8);
    /* ... many other members ... */
};

An sk_buff rarely exists in isolation; it is usually part of a queue (e.g., a receive queue, a transmit queue, or a retransmission queue). The next and prev pointers allow the kernel to link multiple sk_buff structures together in a doubly linked list. The dev points to the net_device (e.g., eth0, wlan0).

On Rx: The device that received the packet
On Tx: The device that will send the packet

The sk_buff struct itself is just a metadata; it points to a separate block of memory that holds the actual packet content.

unsigned char *head, *data;
sk_buff_data_t tail, end;

*head points to the start of the allocated memory buffer for the packet, while *data points to the start of the actual packet data within that buffer. The tail and end as name suggest, point to the end of the data and the end of the allocated buffer, respectively.

Control buffers are stored in

unsigned char cb[48];

This area is used by various layers of the networking stack to store temporary data related to the packet. This memory is free for use by whichever layer currently “owns” the packet. IP Layer: Overwrites it with struct inet_skb_parm for IP options. TCP Layer: Casts this to struct tcp_skb_cb to store sequence numbers and flags.

Ownership and reference is handled by following fields:

struct sock *sk;
refcount_t users;
void (*destructor)(struct sk_buff *skb);

The sk pointer points to the socket associated with the packet, if any. It is NULL if the packet is not associated with a socket. The users field is a reference count that keeps track of how many parts of the kernel are currently using the sk_buff. If users > 1, the packet is “cloned”. When the reference count drops to zero, the destructor function is called to free the sk_buff and its associated resources.

Now when new packet is received, the kernel allocates an sk_buff structure to hold the packet data and metadata. The packet is then enqueued in the list of received packets for further processing.

From `netif_rx()` to IP Layer
#

Once the packet is moved from sdio driver to core networking stack via brcmf_sdio_readframes() function.

In this function the sk_buff is allocated using dev_alloc_skb() function which is wrapped inside brcmu_pkt_buf_get_skb function. This function allocates a new sk_buff structure and reserves space for the packet data.

Buffer alignment is using

skb_pull(pkt, head_read);
pkt_align(pkt, rd->len_left, bus->head_align);

skb_pull() advances the data pointer, reserving space at the head for the header pkt_align() aligns the buffer for DMA/hardware requirements.

These function seems to be specific to the brcmf driver and are used to ensure that the packet data is properly aligned in memory for efficient access by the hardware.

ret = brcmf_sdiod_recv_pkt(bus->sdiodev, pkt); call reads the packet data from the SDIO interface into the sk_buff’s data area.

After this the packet also contains the SPCI header which is removed using

__skb_trim(pkt, rd->len);
skb_pull(pkt, rd->dat_offset);

__skb_trim() trims the sk_buff to the actual length of the received data. And only the valid ethernet frame remains in the sk_buff data section.

After brcmu_pkt_buf_get_skb(rd->len_left + head_read + head_align):

dev_alloc_skb() allocates buffer
skb_put(len) sets tail to fill entire allocated space

┌───────────────────────────────────────────────────┐
│ head_read + head_align + rd->len_left (all "data")│
└───────────────────────────────────────────────────┘
▲                                                   ▲
head/data                                          tail

After skb_pull(pkt, head_read):

Moves data pointer forward, creates headroom for header

┌───────────────────────────────────────────┐
│ head_read   │  head_align + rd->len_left  │
└───────────────────────────────────────────┘
▲              ▲                            ▲
head          data                         tail

After pkt_align(pkt, rd->len_left, head_align):

Aligns data pointer for DMA requirements

┌───────────────────────────────────────┐
│ head_read + alignment │ rd->len_left  │
└───────────────────────────────────────┘
▲                       ▲               ▲
head                   data            tail

After brcmf_sdiod_recv_pkt()

SDIO DMA fills the buffer:

┌────────────────────────────────────────────────────┐
│  headroom  │ SDIO Frame Data (no header yet) + pad │
└────────────────────────────────────────────────────┘
▲            ▲                                       ▲
head         data                                   tail

After skb_push(pkt, head_read) + memcpy(header):

Moves data pointer BACK to include previously cached header

┌───────────────────────────────────────────────────────┐
│ headroom │ SDPCM Hdr │ Remaining Frame Data + padding │
└───────────────────────────────────────────────────────┘
▲          ▲                                            ▲
head      data                                         tail

After __skb_trim(pkt, rd->len):

Sets tail to actual frame length (removes padding)

┌───────────────────────────────────────────────────────┐
│ headroom │ SDPCM Header │Network Payload (Ethernet/IP)│
└───────────────────────────────────────────────────────┘
▲          ▲                                            ▲
head      data                                          tail

After skb_pull(pkt, rd->dat_offset) - FINAL STATE:

Strips SDPCM protocol header, payload ready for network stack

┌───────────────────────────────────────────────────────┐
│ expanded headroom │   Network Payload (Ethernet/IP)   │
└───────────────────────────────────────────────────────┘
▲                   ▲                                   ▲
head                data                                tail

and finally the packet is handed over (or queued) to the core networking stack using netif_rx() function after jumping some hoops.

The main function that enqueues the received packet for further processing is netif_rx_internal() using enqueue_to_backlog() function call.

Backlog Processing in Softirq
#

Once the packet is enqueued in the backlog, it is processed in the context of a software interrupt (softirq). The softirq responsible for processing received packets is the NET_RX_SOFTIRQ. The net_rx_action() function is called to handle the packets in the backlog.

net_rx_action() is attached as handler to NET_RX_SOFTIRQ using open_softirq() during kernel initialization at boot time. When the softirq is triggered, net_rx_action() processes the packets net/core/dev.c.

	// net/core/dev.c line 12233
	open_softirq(NET_RX_SOFTIRQ, net_rx_action);

Now net_rx_action(void) function creates two lists : list and repoll the first list is for packets to be processed immediately, while the second list is for packets that need to be re-polled later. You can see it at line 7012 to line 7020 of net/core/dev.c

	LIST_HEAD(list);
	LIST_HEAD(repoll);
	...
	list_splice_init(&sd->poll_list, &list);
	local_irq_enable();

The two list solves the problem of looping forever if a list still has packet and need to be pushed back to backlog again. With two lists the function can process all packets in the first list without worrying about new packets being added to it. Any packets that need to be re-polled are added to the second list which is processed later.

In net_rx_action() the __napi_poll(n, &do_repoll) function via wrapper napi_poll() is called for each NAPI context.

Like the net_rx_action() , process_backlog() function is also attached to int (*poll)(struct napi_struct *, int); funtion pointer of napi_struct during NAPI initialization on line 12205 of net/core/dev.c file.

	sd->backlog.poll = process_backlog;

The process_backlog() function is the NAPI poll function for the per-CPU "backlog" queue. It drains packets one by one and sends them up the stack via __netif_receive_skb function which inturn calls __netif_receive_skb_core.

Central Packet “Switchboard” : `__netif_receive_skb_core`
#

__netif_receive_skb_core() is the central switchboard of the Linux kernel’s network receive path. It takes a packet (skb) from the driver (or NAPI) and figures out who should get it next: packet sniffers, bridging logic, or network protocols (IPv4, IPv6, ARP).

We are interested in TCP/IP so we will focus on how the packet is handed over to the IP layer. I will skip the vlan, bridge, XDP and packet sniffer part for now. For us the line 5700 to 5707 is the most important part of the function.

	type = skb->protocol;
	if (likely(!deliver_exact)) {
		deliver_ptype_list_skb(skb, &pt_prev, orig_dev, type,
				       &ptype_base[ntohs(type) &
                   PTYPE_HASH_MASK]);
	}

A hash table containing handlers for protocols like IPv4 (ip_rcv), IPv6 (ipv6_rcv), and ARP.

The hash table is defined as ptype_base in net/core/dev.c file. And the mask and size are defined as below:

/*
 *	The list of packet types we will receive
 * (as opposed to discard)
 *	and the routines to invoke.
 *
 *	Why 16. Because with 16 the only overlap we get on a
 *	hash of the
 *	low nibble of the protocol value is RARP/SNAP/X.25.
 *
 *		0800	IP
 *		0001	802.3
 *		0002	AX.25
 *		0004	802.2
 *		8035	RARP
 *		0005	SNAP
 *		0805	X.25
 *		0806	ARP
 *		8137	IPX
 *		0009	Localtalk
 *		86DD	IPv6
 */
#define PTYPE_HASH_SIZE	(16)
#define PTYPE_HASH_MASK	(PTYPE_HASH_SIZE - 1)

When a packet arrives the function:

gets the protocol type from skb->protocol For example 0x0800 for IPv4.
calculates the index: hash = ntohs(skb->protocol) & 0xF.
iterates the list at ptype_base[hash].
compares ptype->type with skb->protocol.
If they match, it delivers the packet to ptype->func (e.g., ip_rcv).

The handler for IPv4 packets is ip_rcv(), which is registered in the hash table during kernel initialization in inet_init() function found in net/ipv4/af_inet.c file.

static struct packet_type ip_packet_type __read_mostly = {
    .type = cpu_to_be16(ETH_P_IP),
    .func = ip_rcv,
    .list_func = ip_list_rcv,
};

Welcome Packet to IP World: `ip_rcv()`
#

Once the packet reaches the ip_rcv() function, It has made its way to the IP layer of the networking stack. The ip_rcv() function is responsible for processing incoming IPv4 packets. I will keep the journey here and continue in the next part of the blog post. What happens to the packet inside IP layer. And the packet’s journey to TCP layer and finally to the application layer.

Overview of the Journey #

Understanding sk_buff #

From netif_rx() to IP Layer #

Backlog Processing in Softirq #

Central Packet “Switchboard” : __netif_receive_skb_core #

Welcome Packet to IP World: ip_rcv() #