Self-Defending OS

Why it matters

Security today is a stack of isolated external tools layered on top of an operating system. An OS that treats security as an intrinsic capability could respond at kernel speed to behavior it has learned to reason about. Behavior-Adaptive OS Security asks whether a system can model its own process, memory, network, filesystem, privilege, and syscall behavior, and turn that model into adaptive defense — not a collection of external products.

Research gap

The research challenge is whether an OS can build and maintain a trustworthy runtime model of its own behavior across multiple domains and then act on deviations quickly enough to matter. Static policy engines and signature-based tools are mature, but a closed loop where learned behavioral risk directly drives changing defensive policy remains a largely open systems and AI problem.

Research questions

1. 01 How do we construct a behavioral security model of a running OS that is expressive yet fast enough for in-kernel evaluation?
2. 02 What is the minimum telemetry set needed to detect meaningful behavioral deviation without flooding the model with noise?
3. 03 How can defensive policies adapt at runtime without hurting legitimate workloads or enabling self-inflicted denial of service?
4. 04 How do we validate that a learned risk state corresponds to real adversary behavior rather than a spurious correlation?

Suggested technologies

RusteBPFseccompXGBoostLinux Kernel

Required skills

Operating SystemsSystems ProgrammingMachine LearningThreat Modeling

Team and supervisor

3-5 researchers, supervised by Muhammad Qasim.

Suitable for: OS, Cybersecurity, Systems, AI.

Expected deliverables

• Design and prototype of a behavior-adaptive OS security layer
• Open-source evaluation harness for adaptive defense under attack scenarios
• Research paper on Behavior-Adaptive OS Security