System call: How it works, why expensive, how to reduce them
We will be focusing on the x86 instruction set. A running program on CPU can be stopped under three conditions:
It has finished execution
There is a timer interrupt
It wants to yield control to the OS because of a system call
When the program wants to make a system call, it issues a trap instruction, where the program moves from user mode to kernel mode (i.e., ring 3 to ring 0).
GS is a variable which in user mode, that points to thread-local storage, but in kernel mode it points to the per-CPU data, which is a special portion available per CPU to provide faster access to certain data. When a trap instruction is issued, swapgc happens. This per cpu block, also known as the pcpu_hot, has few variables such as user stack pointer and top of the stack of the kernel.
Data that is saved on the kernel stack after a trap instruction
Trap frame: So firstly, the program saves a trap frame, which provides information about which current instruction the program is running in user mode. Other system call variables, such as rax, which stores the system call int, then rsi, rdi, rdx, r10, r9, r8, these variables are saved on the kernel stack.
RIP: Then the current instruction pointer is saved, rip.
Then the callee information is saved, and we can begin switching to the kernel stack. Currently, user code is executing; now we will execute on the kernel stack. For this, the current value of the user rsp is stored in the gs user stack pointer. Then the rsp is switched to the top of the kernel stack, and then at that end the value of the GS stack user stack pointer is pushed on the top of the kernel stack.
Things that are expensive in context of system call.
we have to save registers from one process and load the variables of several registers of one other process.
the modern TLB’s translation lookaside buffer stores the ASID, which is like the address space ID, to prevent need of flushing the TLB. It can happen that if it doesn’t store, then we have to flush because these TLB or another process might be mapping to the physical pages, which are entirely different.
Pipeline flushing
because modern CPU has 20 to 30 stages in the pipeline where they have instructions that are doing something in parallel. Some might be fetching, some might be doing executing. If the current instruction is going into syscall, all these instructions have to be flushed.
Another thing is that CPUs nowadays are super scalar, so they can perform four to five instructions in parallel, which is just four to five instructions going in and being processed per cycle. All these instructions need to be flushed, and a 20-cycle pipeline takes 20 cycles to be filled up again, which is a very expensive process.
then is branch prediction. Branch prediction buffer takes some time to build up so that for if and conditional statements, the CPU can predict which branch will be executed and can start executing them optimistically.
Another thing is the written stack buffer. Sometimes it can happen that the process’s CPU might know what to return, but the address of that instruction might not be known before. The CPU tries to correct that. This thing also needs to be flushed out and built up again for the process when it’s picked up again. https://blog.codingconfessions.com/p/what-makes-system-calls-expensive
Reducing no of system calls
System call batching, multiple reads and writes into a single system call, so that we can reduce the number of context switches. Another is using IO_URING.
Another thing is that we can totally skip hardware and just use zero-cost copying using mmap, which maps the file into the address space of the process.
Caching system call result for system call where the result doesn’t change much.
Moving executable code under the kernel mode so that less context switching is needed. We can just get the result directly from the kernel code execution with epbf.
Use vDSO: For calls like clock_gettime, prefer the vDSO path to avoid kernel entry.
Links:
https://blog.codingconfessions.com/p/what-makes-system-calls-expensive why system calls are expensive
https://blog.codingconfessions.com/p/linux-context-switching-internals: context switching internals
https://dev.to/bhushitha_hashan/a-deep-dive-into-kernel-context-switching-2g55 Use of gs register, rsp swap problem, and context switching q/a style overview.
https://www.youtube.com/watch?v=fEnWqibCwo0&list=PLDW872573QAb4bj0URobvQTD41IV6gRkx&index=25 xv6, which is a simple Unix operating system developed under the MIT license for educational purposes: how context switching happens under this os.
https://read.seas.harvard.edu/cs161/2024/exercises/syscall/ register leakage in syscall.