Q: which linux c++thread is stuck #CSY

This is a typical “c++ecosystem question”. It’s not about c++ or C; it’s about linux instrumentation tools.

Q1: Given a multi-threaded server, you see some telltale signs that process is stuck and you suspect only one of the threads is stuck while the other threads are fine. How do you verify?

Q2: What if it’s a production environment?
A: I guess all my solution should be usable on production, since the entire machine is non-functioning. We can’t make it any worse.  If the machine is still doing useful work, then we should probably wait till end of day to investigate.

–Method: thread dump? Not popular for c++ processes. I have reason to believe it’s a JVM feature, since java threads are always jvm constructs, usually based on operating system threads [1]. JVM has full visibility into all threads and provides comprehensive instrumentation interface.

https://www.thoughtspot.com/codex/threadstacks-library-inspect-stacktraces-live-c-processes shows a custom c++ thread dumper but you need custom hooks in your c++ source code.

[1] Note “kernel-thread” has an unrelated meaning in the linux context

–Method: gdb

thread apply all bt – prints a stack trace of every thread, allowing you to somewhat easily find the stuck one

I think in gdb you can release each thread one by one and suspend only one suspect thread, allowing the good threads to continue

–Method: /proc — the dynamic pseudo file system

For each process, a lot of information is available in /proc/12345 . Information on each thread is available in /proc/12345/task/67890 where 67890 is the kernel thread ID. This is where ps, top and other tools get thread information.

 

by default, interrupts==hardware interrupts

In operation system jargon, “interrupts” means hardware interrupts by default. Timer hardware, NIC, keyboard/mouse, and possibly hard disk … can all generate hardware interrupts.

“Interrupt signal” often refers to the electrical signal, never the higher-level SIG* signals in unix (unsupported in Windows AFAIK).

There is also a well-known concept of “software interrupt”. I see them as fake interrupts, simulated in software. My book [[linux kernel]] says software interrupts are employed for 1) system calls 2) debugger breakpoints.

A key difference between software vs hardware interrupts — hardware interrupts come from devices with their independent power supply and clock signal, so hardware interrupts can hit CPU any time, even in the middle of a CPU clock cycle. Software interrupts won’t. Software interrupts are simulated by code running in the same CPU, driven by the same clock signal.

In other words, hardware interrupts are unsynchronized with CPU, whereas software interrupts are synchronized with CPU.

process(!!thr)context-switch overhead~10 millisec

This blogpost is about process switch. No comment about thread context switch.

2003 edition of [[LinuxKernel]] P351 gave a typical number of 10 millisec. Bear in mind that timer hardware interrupt interval is usually longer than 1 millisec.

## linux kernel concurrency primitives #to elaborate

I feel the concurrency primitives in kernel are conceptually similar to those in thread libraries but fundamentally different in implementation. This topic is deep and time consuming so I will only memorize some jargons for show-off

I should read the OReilly Linux book + Josh’s book

• spinlock — addressed in both books
• read-write spinlock
• binary and counting semaphores
• read-write semaphores
• RCU?

 

kernThread ] linux^Unix

Here are a few things I can recall, from [[UnderstandingLinuxKernel]].

P 104 lists some minor Linux kernel threads, but here are the important ones:

• process 0 (swapper) is a kernel thread. This process gets scheduled when there’s no other process to run
• process 1 (init) is a kernel thread. This process launches and monitors all other processes except process 0. It’s also the adopted parents of all orphaned zombies
• both of them run forever. Most other linux kernel threads run for a short while and exit.

Linux kernThr is completely unrelated concept to traditional unix kernThr. More difference than similarities (zero?)

• Unix kernel threads aka “native threads” concept was first known to me in jvm.  kernel scheduler can see native threads but not java green threads.
• native threads often run user mode but linux kernel threads only run in kernel mode and very limited in usage. I think they are mostly set up to access hardware

Linux kernel threads are less fundamental than kernel routines including sysCall handlers + interrupt handlers.

1. every Linux user process enter kernel mode via kernel routines
2. not every user process interacts with kernel threads

minimum hardware requirement to run linux #historical observation

1. Bell labs Unix, in its early days, were able to run on hardware considered “underpowered” even by the standard of that day — P33 [[art of unix programming]]. I believe contemporary kernels were unable to run on those low-end machines.
2. Linux (P77) has a similar characteristic. In the 1990’s the big commercial Unixes targeted enterprise-class hardware but Linux emphasized doing more with less. Today, Linux powers 99% of the world’s most powerful supercomputers but Linux also runs on low-end or obsolete hardware.

In both cases, I feel that an OS designed with very low minimum hardware requirement turned out to be actually more efficient, more adaptable, more versatile, more powerful than their conventional competitors.

At what juncture would kernel scheduler run@@

Kernel scheduler has an algorithm and therefore implemented as a sequence of instructions. You can think of it as some black-box function/routine.

I think it is Not really a long-running background process. In Linux, I believe it is an on-demand routine, but not run on behalf of any process.

Background — Many familiar on-demand kernel routines do run on behalf of an “owner process” —

• accessing a file or socket
• accessing some device such as NIC
• accessing memory

However, other on-demand kernel routines (often interrupt handlers) do not have an “owner process”. Here are some routines —

• reacting to timer interrupts
• reacting to low-level emergency hardware interrupts like …. ?

So the scheduler is a classic example. I think scheduler can get triggered by timer interrupts. See P 350 [[linux kernel]]

interrupt+signal ] kernel

See hardware-interrupt^signals

There are precise answers in the kernel code, but here are my high-level, imprecise pointers, based on [[linux kernel]]:

• Based on my “adapted Intel terminology”, interrupts from cpu are known as cpu Exceptions, whereas interrupts from other devices are known as hardware interrupts
  • interrupting hardware includes hardware clocks, keyboard, NIC and other I/O devices.
  • cpu exceptions are generated by problematic/erroneous instructions
• LG2 — “software interrupts” concept is not so well-defined, and used primarily for 1) notifying debugger 2) implement system calls

Both hardware interrupts and cpu exceptions can generate SIG* signals. Signals are much higher-level constructs than the hardware-level interrupts. Signal as a concept is 50% kernel 50% UserMode:

• A signal is always addressed to a user process.
• signal delivery is kernel’s job; signal handling is user process’s job

 

Liunx(!!Unix)kernel thread can! run user program

–“Liunx kernel thread cannot run user programs”, as explained in [[UnderstandingLinuxKernel]] P3.

Removing ‘kernel‘… Linux userland threads (based on LWP, explained below) do run user programs.

Removing ‘thread‘… Linux kernel processes or interrupt routines could possibly run under a user process pid.

Removing ‘Linux‘ … Other Unix variants do use kernel threads to run both 1) user programs and 2) kernel routines. This is my understanding from reading about JVM…

• unix  — multi-threaded user apps =use=> LWP =use=> kernel threads
• linux — multi-threaded user apps =use=> LWP. LWP is the basic unit of concurrency in user apps. Linux LWP is unrelated to Unix LWP.

kernel scheduler: triggered by timer+I/O interrupts

keyboard interrupt can also trigger scheduler

frequency@(hardware)timer interrupts #KHz

In the 2003 edition of [[LinuxKernel]] P195, 1200 interrupts/second was the highest frequency, above 1KHz.

The CPU must process these timer interrupts. If it were easy to “nullify” these interrupts and eliminate the CPU overhead, then we would have millions of time interrupts per second but I doubt it. The overhead is probably unavoidable.

Scheduler is a kernel component. I think scheduler divides cpu time into slots. I guess the smallest quantum is limited by this frequency, among other factors.

P350 of The same book says that scheduler runs upon timer interrupts + keyboard interrupts.

Note many people get confused by software timer vs hardware timer. Hardware timer is my focus.

Many people get confused by software interrupt vs hardware interrupts. I have a separate blogpost (interrupts=hardware interrupts #by default), but basically most interrupts are hardware interrupts. Software interrupts are simulated.

 

hardware-Interrupt^signal #unfinished

Let’s compare in terms of delays, socket, timer devices, software timers…

I feel interrupts^signals are somewhat like cells^molecules — at different levels!

— socket

signals are important tools for socket programmers at application and library level.  Kernel support for sockets probably use signals too.

Interrupt (handling) is in kernel (codebase) and are probably too low-level so application developers are a few abstraction levels higher.

SIGIO is mostly used for socket and other I/O devices.

— timers
timer interrupts actually refer to hardware-timers, a few abstraction levels below application developers.

software timers are a very different animal. Software timers integrate with SIGALRM.

linux hardware interrupt handler #phrasebook

I think some interrupts are generated by software, but here I focus on hardware interrupt handlers.

• pseudo-function — Each handler is like a pseudo-function containing a series of instructions that will run on a cpu core
• top priority — interrupt context is higher priority than kernel context or process context. You can say “interrupt” means “emergency”. Emergency vehicles don’t obey traffic rules.
  • However, an interrupt handler “function” can get interrupted by another [1]. The kernel somehow remembers the “stack”
• not preemptible — except the [1] scenario, kernel can’t suspend a hardware interrupt handler in mid-stream and put in another series of instructions in the “driver’s seat”
• no PID — since there’s no preemption, we don’t need a pid associated with this series of instructions.

How is debugger breakpoint implemented@@ brief notes #CSY

This is a once-only obscure interview question. I said up-front that CPU interrupts were needed. I still think so.

I believe CPU support is needed to debug assembly programs, where kernel may not exist.

For regular C program I still believe special CPU instructions are needed.

https://eli.thegreenplace.net/2011/01/27/how-debuggers-work-part-2-breakpoints seems to agree.

It says Interrupt #3 is designed for debugging. It also says SIGTRAP is used in linux, but windows supports no signals.

 

killing a stuck thread: cancellation points #CSY

CSY shared this interview question:

Q: once you know one of many threads is stuck in a production process, what can you do? Can you kill a single thread?
A: there will Not be a standard construct provided by OS or thread library because killing a thread is inherently unsafe.. Look at java Thread.stop()
A: yes if I have a builtin kill-hook in the binary

https://www.thoughtspot.com/codex/threadstacks-library-inspect-stacktraces-live-c-processes describes a readonly custom hook. It is conceivable to add a kill feature —

• Each thread runs a main loop to check an exit-condition periodically.
• This exit-condition would be similar to pthreads “cancellation points”

https://stackoverflow.com/questions/10961714/how-to-properly-stop-the-thread-in-java shows two common kill hooks — interrupt and Boolean flag

Note java had a deprecated thread.stop().

 

sysV IPC objects live till linux shutdown #Viswa

Hi Viswa,

“An IPC resource is persistent: unless explicitly removed by a process it is kept in memory and remains available until system shutdown.”

I just found this sentence in [[understandingLinuxKernel]] section on “System V IPC”.

“IPC resource” includes shared mem and semaphore.

components@kernel beside deviceDrivers #signals,interrupts

Q: exactly what constitute the linux kernel?

[[OpSys by William Stallings]] P99 has a nice diagram showing most of the components in linux kernel are device drivers and similar software

• NIC driver
• char device driver — for keyboard
• block device driver — for disk
• physical memory interface
• device interrupt handlers

What are the other components of kernel?

1. system calls — the highest level in the kernel
1. Signals — the Signal definition and handlers are at the same level as syscalls, but at a higher level than the device interrupt module.
2. TCP/IP module —-  is a level between syscalls .. and .. NIC driver
2. virtual memory —– is a level between syscalls .. and .. physical memory interface
2. filesystem module – is a level between syscalls .. and .. block device driver

Lastly, scheduler — is a highly visible component of kernel. Scheduler relies critically on the device interrupt module.

unix family tree #MacOS

This is academic knowledge for the self-respected techie.

https://upload.wikimedia.org/wikipedia/commons/c/cd/Unix_timeline.en.svg  and https://en.wikipedia.org/wiki/UNIX_System_V#/media/File:Unix_history-simple.svg show

• MacOS is based on BSD
• iOS and MacOS are  based on Darwin
  • Darwin is based on BSD
• linux contains no BSD or Unix codebase
• most commercial Unix versions are based on sysV

unix signal Lesson 1 #Trex IV

Q: why did you say the signals sent are pending and there’s a delay in response?
%%A: if the target process is executing some special instructions the kernel won’t deliver the signal.
AA: For example waiting for network/disk IO — such a process may be almost unkillable. https://major.io/2010/03/18/sigterm-vs-sigkill/ says a reboot is required.

%%A: It’s possible that when target process gets a time slice it then gets a chance to check the “signal table”.
AA: A Unix signal is picked up only at start of a time slice by the recipient “cpu-driver”. See P126[[art of debugging]].

So I gave two correct answers to Olgun of Trexquant but he didn’t acknowledge. I also described the (common) scenario of a one-core machine. The signaling process need to give up the CPU before the receiving process can react to it.

Postman drops a letter on your doorstep when you are out. This is the best way to deliver the signal. It’s uncommon (impossible?) to interrupt a process /midstream/ a sequence of instructions. I guess the reason is efficiency — no efficient implementation of such a “real-time” interrupt mechanism. In Unix, what is common is the signal mechanism.

Pending signals for a parent process is not inherited by a forked child process.

Unix signals can be generated from interactive user, from any other process, from kernel or from hardware, but they all have a target PID.

Unix Signal is at a lower level than threading. Thread preemption often depends on Unix signals

Unix Signal is a kernel-level mechanism.

Unix Signals target a process, not a thread.

A Unix Signal is an event. A Signal handler is an event callback function whose address is associated with one specific signal.

## 10 unix signal scenarios

A signal can originate from outside the process or from within.

The precise meaning of signal generation requires a clear understanding of signal handlers. See P125 [[art of debugging]] and P280 [[linux sys programming]]

— External —
# (SIGKILL/SIGTERM) q(kill) commands
# (SIGINT) ctrl-C
# (SIGHUP) we can send this signal to Apache, to trigger a configuration reload.

— internal, i.e. some kernel code module “sends” this signal to the process committing the “crime” —
# (SIGFPE) divide by zero; arithmetic overflow,
# (SIGSEGV) memory access violation
# (SIGABRT) assertion failure can cause this signal to be generated
# (SIGTRAP) target process hitting a breakpoint. Except debuggers, every process ignores this signal.

3 types of system services ] linux

Hi Pravin,

You asked about services written in c++. I googled and found that “service” has a specific meaning in linux/unix. http://www.comptechdoc.org/os/linux/howlinuxworks/linux_hlservices.html describes 3 types of services —

• A one time only program run at boot-up to provide a function to the system such as kudzu, or keytable.
• A program run as a local daemon upon startup that provides system services such as gpm, autofs, cron, and atd.
• A program run as a network-daemon upon startup that provides networking services such as dhcpd, bootparamd, arpwatch, gated, and httpd.

The way I see it (no expert here), the standard *nix network daemons provide a Generic functionality at the System level – like DHCP, DNS, http, FTP, mail etc.

In contrast, a custom c++ engine used in finance has site-specific business logic. It often binds to a standard network port. Alternatively it may use database, RPC, (Corba?), MOM, or some proprietary protocol for network communication. In some cases the communication is entirely inter-process — no networking. Just IPC like shared memory, named pipes or unix domain sockets. I don’t know which communication mode is dominant. I guess shared memory and MOM are popular.