MIC Scheduler & Intel Thread Director

MIC Scheduler & Intel Thread Director

Overview

Starting in MidnightBSD 4.2-CURRENT, the amd64 GENERIC kernel uses the sched_mic(4) (Michaelangelo) scheduler by default and includes the hfi(4) driver for Intel Hardware Feedback Interface (HFI) and Intel Thread Director (ITD).

These components work together through the cpucap(9) CPU capability layer:

On homogeneous processors (AMD non-hybrid, non-hybrid Intel, or virtual machines without exposed HFI), sched_mic(4) behaves identically to sched_ule(4) and hfi(4) does not attach. The only visible difference is that kern.sched.name reports MIC.

Core classes and the MIC scheduler

On heterogeneous (hybrid) x86 processors, sched_mic(4) recognizes four priority tiers when placing threads:

  1. Performance cores (class 1): Intel P-cores and AMD 3D V-Cache CCD cores.
  2. Efficiency cores (class 2): Intel E-cores, AMD compute-CCD cores, and AMD mobile “C” cores.
  3. Secondary SMT threads: the second thread of a core whose sibling SMT thread is already busy.
  4. Low-power efficiency cores (class 4): Intel LP-E cores, which are scheduled last.

Core preference is a soft, tunable bias: real CPU load always decides first, and core class orders CPUs whose load is otherwise acceptable. When an idle CPU looks for queued work to steal, a CPU of the same or better class steals immediately once kern.sched.steal_thresh allows, whereas a worse-class CPU (or an idle SMT sibling of a busy core) waits until the thread has been queued for kern.sched.steal_delay ticks (about 4 ms by default). This prevents brief interrupt preemptions from bouncing threads off performance cores while allowing real backlogs to spill onto efficiency cores.

By default, kern.sched.prefer_compute is 1 (preferring Intel P-cores or AMD frequency/compute cores), except on AMD X3D parts where it defaults to 0 so threads prefer the large-cache 3D V-Cache CCD. Setting kern.sched.prefer_compute=0 on a hybrid Intel system instructs the scheduler to prefer efficiency cores using their efficiency scores.

Intel HFI and Thread Director (hfi(4))

Intel hybrid processors starting with the 12th generation Core (Alder Lake) provide the Hardware Feedback Interface. The hfi(4) driver allocates one table per physical package, and the processor writes per-core performance and energy-efficiency scores (0–255, where 0 indicates the processor recommends scheduling nothing on that core) into the table and raises a local APIC thermal interrupt when thermal or power limits change.

When kern.sched.use_scores is 1 (the default), sched_mic(4) derives placement costs directly from those scores using cost = (m - s) * 384 / m (where s is the CPU's score and m is the best score across all CPUs). CPUs within 1/16 of the best score (cost at most 24) rank alongside performance cores for work stealing and class swapping, so slightly lower-scoring P-cores remain peers while throttled P-cores rank with efficiency cores.

On processors that also support Intel Thread Director (CPUID.06H:EAX[23]), hfi(4) enables per-thread hardware classification (4 classes on current processors) by default:

Note that once hfi(4) hands table memory to the processor, those pages must remain allocated for the rest of the boot. As a result, hfi(4) cannot be detached or unloaded once attached (returning EBUSY).

Checking status and monitoring

Use sysctl(8) to verify that the MIC scheduler, hfi(4), and Thread Director are active and inspect live scores and classification counters.

Check the active scheduler and score mode:

sysctl kern.sched.name kern.sched.use_scores kern.sched.scores_active

Check hardware Thread Director support and hfi(4) attachment:

sysctl hw.itd.supported hw.itd.hreset hw.itd.classes
sysctl dev.hfi.0.enabled dev.hfi.0.itd dev.hfi.0.classes

View the per-CPU HFI table summary (each entry shows cpuN:row:perf/eff, followed by additional ,perf/eff pairs for each Thread Director class):

sysctl dev.hfi.0.summary

Inspect cpucap(9) providers, detected core classes (1=P, 2=E, 4=LP-E), and computed scheduler costs and ranks:

sysctl kern.cpucap.providers kern.cpucap.class
sysctl kern.sched.cpu_cost kern.sched.cpu_rank kern.sched.class_cost

Monitor Thread Director classification and swap activity:

sysctl kern.sched.itd_active kern.sched.itd_classes
sysctl kern.sched.itd_reads kern.sched.itd_invalid kern.sched.itd_changes kern.sched.itd_hist
sysctl kern.sched.itd_swaps kern.sched.itd_swap_follows kern.sched.itd_swap_skipped

On some processors (such as 12th-gen Alder Lake mobile parts), the hardware writes the HFI table once at boot and may report invalid class reads while a thread is on an E-core; in that case sched_mic(4) preserves the thread's last valid class. Newer parts (such as Core Ultra processors) update the table dynamically at run time, which can be observed via rising dev.hfi.0.pkg.N.interrupts and dev.hfi.0.pkg.N.updates counters.

Tuning and configuration

The table below summarizes the primary loader tunables (set in /boot/loader.conf) and runtime sysctl variables (set with sysctl(8) or /etc/sysctl.conf):

Variable Type Default Description
hw.hfi.itd Loader tunable 1 Enable Intel Thread Director at hfi(4) attach when supported by hardware.
hint.hfi.0.disabled Loader tunable 0 Set to 1 to prevent hfi(4) from attaching at boot.
dev.hfi.0.delay_ms Tunable / sysctl 100 Debounce delay in milliseconds between an HFI table update and publishing to cpucap(9).
kern.sched.use_scores Tunable / sysctl 1 Derive CPU placement costs from dynamic HFI scores instead of fixed class weights.
kern.sched.itd Sysctl 1 Use per-thread Thread Director class tables when hw.hfi.itd and use_scores are active.
kern.sched.itd_swap Sysctl 1 Allow the periodic balancer to swap running threads between core classes.
kern.sched.prefer_compute Sysctl 1 (0 on AMD X3D) Set to 1 to prefer performance/compute cores, or 0 to prefer efficiency/3D V-Cache cores.
kern.sched.steal_delay Sysctl ~4 ms (in ticks) Ticks a thread must wait in a queue before an idle worse-class CPU may steal it.
kern.sched.smt_busy_penalty Sysctl 192 Penalty for an idle SMT sibling of a busy core. Set to 0 to restore sched_ule(4) SMT behavior.

Opting out or reverting to sched_ule(4)

If you want to disable Thread Director, score-based placement, or hfi(4) without rebuilding the kernel, add the appropriate option to /boot/loader.conf and reboot:

# Disable Thread Director only (keep HFI base scores)
hw.hfi.itd="0"

# Or disable score-based placement (fall back to static class weights)
kern.sched.use_scores="0"

# Or prevent hfi(4) from attaching entirely
hint.hfi.0.disabled="1"

At runtime, you can also toggle Thread Director placement or swapping without rebooting:

doas sysctl kern.sched.itd=0
doas sysctl kern.sched.itd_swap=0

To build a custom kernel using the traditional sched_ule(4) scheduler instead of sched_mic(4) and hfi(4), create a kernel configuration file that includes GENERIC and removes both:

include GENERIC
ident MYKERNEL

nooptions SCHED_MIC
nodevice hfi

Testing on homogeneous hardware or VMs

Developers and testers can exercise hybrid placement and score tables on homogeneous systems or virtual machines using cpucap(9) and sched_mic(4) overrides:

Further reading