Foreword xix Acknowledgments xxi Introduction xxiii 1 Concepts and Tools 1 Windows Operating System Versions 1 Foundation Concepts and Terms 2 Windows API 2 Services, Functions, and Routines 4 Processes, Threads, and Jobs 5 Virtual Memory 14 Kernel Mode vs User Mode 16 Terminal Services and Multiple Sessions 19 Objects and Handles 21 Security 22 Registry 23 Unicode 23 Digging into Windows Internals 24 Reliability and Performance Monitor 25 Kernel Debugging 26 Windows Software Development Kit 31 Windows Driver Kit 31 Sysinternals Tools 32 Conclusion 32 Microsoft is interested in hearing your feedback so we can continually improve our books and learning resources for you. To participate in a brief online survey, please visit: www.microsoft.com/learning/booksurvey/ What do you think of this book? We want to hear from you! vi Table of Contents 2 System Architecture 33 Requirements and Design Goals 33 Operating System Model 34 Architecture Overview 35 Portability 38 Symmetric Multiprocessing 39 Scalability 43 Differences Between Client and Server Versions 43 Checked Build 47 Key System Components 49 Environment Subsystems and Subsystem DLLs 50 Ntdll dll 57 Executive 58 Kernel 61 Hardware Abstraction Layer 65 Device Drivers 68 System Processes 74 Conclusion 83 3 System Mechanisms 85 Trap Dispatching 85 Interrupt Dispatching 87 Exception Dispatching 114 System Service Dispatching 125 Object Manager 133 Executive Objects 136 Object Structure 138 Synchronization 170 High-IRQL Synchronization 172 Low-IRQL Synchronization 177 System Worker Threads 198 Windows Global Flags 200 Advanced Local Procedure Calls (ALPCs) 202 Kernel Event Tracing 207 Wow64 211 Wow64 Process Address Space Layout 211 System Calls 212 Exception Dispatching 212 Table of Contents vii User Callbacks 212 File System Redirection 212 Registry Redirection and Reflection 213 I/O Control Requests 214 16-Bit Installer Applications 215 Printing 215 Restrictions 215 User-Mode Debugging 216 Kernel Support 216 Native Support 217 Windows Subsystem Support 219 Image Loader 220 Early Process Initialization 222 Loaded Module Database 223 Import Parsing 226 Post Import Process Initialization 227 Hypervisor (Hyper-V) 228 Partitions 230 Root Partition 230 Child Partitions 232 Hardware Emulation and Support 234 Kernel Transaction Manager 240 Hotpatch Support 242 Kernel Patch Protection 244 Code Integrity 246 Conclusion 248 4 Management Mechanisms 249 The Registry 249 Viewing and Changing the Registry 249 Registry Usage 250 Registry Data Types 251 Registry Logical Structure 252 Transactional Registry (TxR) 260 Monitoring Registry Activity 262 Registry Internals 266 Services 281 Service Applications 282 The Service Control Manager 300 viii Table of Contents Service Startup 303 Startup Errors 307 Accepting the Boot and Last Known Good 308 Service Failures 310 Service Shutdown 311 Shared Service Processes 313 Service Tags 316 Service Control Programs 317 Windows Management Instrumentation 318 Providers 319 The Common Information Model and the Managed Object Format Language 320 Class Association 325 WMI Implementation 327 WMI Security 329 Windows Diagnostic Infrastructure 329 WDI Instrumentation 330 Diagnostic Policy Service 330 Diagnostic Functionality 332 Conclusion 333 5 Processes, Threads, and Jobs 335 Process Internals 335 Data Structures 335 Kernel Variables 342 Performance Counters 343 Relevant Functions 344 Protected Processes 346 Flow of CreateProcess 348 Stage 1: Converting and Validating Parameters and Flags 350 Stage 2: Opening the Image to Be Executed 351 Stage 3: Creating the Windows Executive Process Object (PspAllocateProcess) 354 Stage 4: Creating the Initial Thread and Its Stack and Context 359 Stage 5: Performing Windows Subsystem–Specific Post-Initialization 360 Stage 6: Starting Execution of the Initial Thread 362 Stage 7: Performing Process Initialization in the Context of the New Process 363 Table of Contents ix Thread Internals 370 Data Structures 370 Kernel Variables 379 Performance Counters 379 Relevant Functions 380 Birth of a Thread 380 Examining Thread Activity 381 Limitations on Protected Process Threads 384 Worker Factories (Thread Pools) 386 Thread Scheduling 391 Overview of Windows Scheduling 391 Priority Levels 393 Windows Scheduling APIs 395 Relevant Tools 396 Real-Time Priorities 399 Thread States 400 Dispatcher Database 404 Quantum 406 Scheduling Scenarios 413 Context Switching 418 Idle Thread 418 Priority Boosts 419 Multiprocessor Systems 434 Multiprocessor Thread-Scheduling Algorithms 442 CPU Rate Limits 444 Job Objects 445 Conclusion 450 6 Security 451 Security Ratings 451 Trusted Computer System Evaluation Criteria 451 The Common Criteria 453 Security System Components 454 Protecting Objects 458 Access Checks 459 Security Descriptors and Access Control 484 Account Rights and Privileges 501 Account Rights 502 x Table of Contents Privileges 503 Super Privileges 509 Security Auditing 511 Logon 513 Winlogon Initialization 515 User Logon Steps 516 User Account Control 520 Virtualization 521 Elevation 528 Software Restriction Policies 533 Conclusion 535 7 I/O System 537 I/O System Components 537 The I/O Manager 539 Typical I/O Processing 540 Device Drivers 541 Types of Device Drivers 541 Structure of a Driver 547 Driver Objects and Device Objects 550 Opening Devices 555 I/O Processing 562 Types of I/O 563 I/O Request to a Single-Layered Driver 572 I/O Requests to Layered Drivers 578 I/O Cancellation 587 I/O Completion Ports 592 I/O Prioritization 598 Driver Verifier 604 Kernel-Mode Driver Framework (KMDF) 606 Structure and Operation of a KMDF Driver 607 KMDF Data Model 608 KMDF I/O Model 612 User-Mode Driver Framework (UMDF) 616 The Plug and Play (PnP) Manager 619 Level of Plug and Play Support 620 Driver Support for Plug and Play 621 Table of Contents xi Driver Loading, Initialization, and Installation 623 Driver Installation 632 The Power Manager 636 Power Manager Operation 638 Driver Power Operation 639 Driver and Application Control of Device Power 643 Conclusion 644 8 Storage Management 645 Storage Terminology 645 Disk Drivers 646 Winload 646 Disk Class, Port, and Miniport Drivers ..647 Disk Device Objects 650 Partition Manager 651 Volume Management 652 Basic Disks 653 Dynamic Disks 656 Multipartition Volume Management 661 The Volume Namespace 667 Volume I/O Operations 674 Virtual Disk Service 675 BitLocker Drive Encryption 677 BitLocker Architecture 677 Encryption Keys 679 Trusted Platform Module (TPM) 681 BitLocker Boot Process 683 BitLocker Key Recovery 684 Full Volume Encryption Driver 686 BitLocker Management 687 Volume Shadow Copy Service 688 Shadow Copies 688 VSS Architecture 688 VSS Operation 689 Uses in Windows 692 Conclusion 698 xii Table of Contents 9 Memory Management 699 Introduction to the Memory Manager 699 Memory Manager Components 700 Internal Synchronization 701 Examining Memory Usage 701 Services the Memory Manager Provides 704 Large and Small Pages 705 Reserving and Committing Pages 706 Locking Memory 707 Allocation Granularity 708 Shared Memory and Mapped Files 709 Protecting Memory 711 No Execute Page Protection 713 Copy-on-Write 718 Address Windowing Extensions 719 Kernel-Mode Heaps (System Memory Pools) 721 Pool Sizes 722 Monitoring Pool Usage 724 Look-Aside Lists 728 Heap Manager 729 Types of Heaps 730 Heap Manager Structure 731 Heap Synchronization 732 The Low Fragmentation Heap 732 Heap Security Features 733 Heap Debugging Features 734 Pageheap 735 Virtual Address Space Layouts 736 x86 Address Space Layouts 737 x86 System Address Space Layout 740 x86 Session Space 740 System Page Table Entries 744 64-Bit Address Space Layouts 745 64-Bit Virtual Addressing Limitations 749 Dynamic System Virtual Address Space Management 751 System Virtual Address Space Quotas 756 User Address Space Layout 757 Table of Contents xiii Address Translation 761 x86 Virtual Address Translation 762 Translation Look-Aside Buffer 768 Physical Address Extension (PAE) 769 IA64 Virtual Address Translation 772 x64 Virtual Address Translation 773 Page Fault Handling 774 Invalid PTEs 775 Prototype PTEs 776 In-Paging I/O 778 Collided Page Faults 779 Clustered Page Faults 779 Page Files 780 Stacks 784 User Stacks 785 Kernel Stacks 786 DPC Stack 787 Virtual Address Descriptors 787 Process VADs 788 Rotate VADs 790 NUMA 791 Section Objects 792 Driver Verifier 799 Page Frame Number Database 803 Page List Dynamics 807 Page Priority 809 Modified Page Writer 812 PFN Data Structures 814 Physical Memory Limits 818 Windows Client Memory Limits 819 Working Sets 822 Demand Paging 823 Logical Prefetcher 823 Placement Policy 827 Working Set Management 828 Balance Set Manager and Swapper 831 System Working Set 832 Memory Notification Events 833 xiv Table of Contents Proactive Memory Management (SuperFetch) 836 Components 836 Tracing and Logging 838 Scenarios 840 Page Priority and Rebalancing 840 Robust Performance 843 ReadyBoost 844 ReadyDrive 845 Conclusion 847 10 Cache Manager 849 Key Features of the Cache Manager 849 Single, Centralized System Cache 850 The Memory Manager 850 Cache Coherency 850 Virtual Block Caching 852 Stream-Based Caching 852 Recoverable File System Support 853 Cache Virtual Memory Management 854 Cache Size 855 Cache Virtual Size 855 Cache Working Set Size 856 Cache Physical Size 858 Cache Data Structures 859 Systemwide Cache Data Structures 860 Per-File Cache Data Structures 862 File System Interfaces 868 Copying to and from the Cache 869 Caching with the Mapping and Pinning Interfaces 870 Caching with the Direct Memory Access Interfaces 872 Fast I/O 873 Read Ahead and Write Behind 875 Intelligent Read-Ahead 875 Write-Back Caching and Lazy Writing 877 Write Throttling 885 System Threads 886 Conclusion 887 Table of Contents xv 11 File Systems 889 Windows File System Formats 890 CDFS 890 UDF 891 FAT12, FAT16, and FAT32 891 exFAT 894 NTFS 895 File System Driver Architecture 895 Local FSDs 896 Remote FSDs 897 File System Operation 901 File System Filter Drivers 907 Troubleshooting File System Problems 908 Process Monitor Basic vs Advanced Modes 908 Process Monitor Troubleshooting Techniques 909 Common Log File System 910 NTFS Design Goals and Features 918 High-End File System Requirements 918 Advanced Features of NTFS 920 NTFS File System Driver 934 NTFS On-Disk Structure 937 Volumes 937 Clusters 937 Master File Table 938 File Reference Numbers 942 File Records 942 File Names 945 Resident and Nonresident Attributes 948 Data Compression and Sparse Files 951 The Change Journal File 956 Indexing 960 Object IDs 961 Quota Tracking 962 Consolidated Security 963 Reparse Points 965 Transaction Support 965 xvi Table of Contents NTFS Recovery Support 974 Design 975 Metadata Logging 976 Recovery 981 NTFS Bad-Cluster Recovery 985 Self-Healing 989 Encrypting File System Security 990 Encrypting a File for the First Time 993 The Decryption Process 998 Backing Up Encrypted Files 999 Conclusion 1000 12 Networking 1001 Windows Networking Architecture 1001 The OSI Reference Model 1001 Windows Networking Components 1003 Networking APIs 1006 Windows Sockets 1006 Winsock Kernel (WSK) 1012 Remote Procedure Call 1014 Web Access APIs 1018 Named Pipes and Mailslots 1021 NetBIOS 1027 Other Networking APIs 1030 Multiple Redirector Support 1033 Multiple Provider Router 1034 Multiple UNC Provider 1037 Name Resolution 1039 Domain Name System 1039 Windows Internet Name Service 1039 Peer Name Resolution Protocol 1039 Location and Topology 1042 Network Location Awareness (NLA) 1042 Link-Layer Topology Discovery (LLTD) 1043 Protocol Drivers 1044 Windows Filtering Platform (WFP) 1047 NDIS Drivers 1053 Variations on the NDIS Miniport 1057 Connection-Oriented NDIS 1057 Table of Contents xvii Remote NDIS 1060 QoS 1062 Binding 1064 Layered Network Services 1066 Remote Access 1066 Active Directory 1066 Network Load Balancing 1068 Distributed File System and DFS Replication 1069 Conclusion 1071 13 Startup and Shutdown 1073 Boot Process 1073 BIOS Preboot 1073 The BIOS Boot Sector and Bootmgr 1077 The EFI Boot Process 1086 Initializing the Kernel and Executive Subsystems 1088 Smss, Csrss, and Wininit 1094 ReadyBoot 1099 Images That Start Automatically 1100 Troubleshooting Boot and Startup Problems 1101 Last Known Good 1101 Safe Mode 1101 Windows Recovery Environment (WinRE) 1106 Solving Common Boot Problems 1109 Shutdown 1115 Conclusion 1118 14 Crash Dump Analysis 1119 Why Does Windows Crash? 1119 The Blue Screen 1120 Troubleshooting Crashes 1124 Crash Dump Files 1125 Crash Dump Generation 1130 Windows Error Reporting 1131 Online Crash Analysis 1133 Basic Crash Dump Analysis 1134 Notmyfault 1134 Basic Crash Dump Analysis 1135 Verbose Analysis 1137 xviii Table of Contents Using Crash Troubleshooting Tools 1139 Buffer Overrun, Memory Corruptions, and Special Pool 1140 Code Overwrite and System Code Write Protection 1143 Advanced Crash Dump Analysis 1144 Stack Trashes 1145 Hung or Unresponsive Systems 1147 When There Is No Crash Dump 1150 Conclusion 1152 Glossary 1153 Index 1185 Microsoft is interested in hearing your feedback so we can continually improve our books and learning resources for you. To participate in a brief online survey, please visit: www.microsoft.com/learning/booksurvey/ What do you think of this book? We want to hear from you!