Archive for April 2015

introduction

Most of the time when there is memory pressure on the system it’s easy to find the offending process by simply checking the Task Manager. However, there is one kind of memory usage that can consume Commit Charge but won’t show up on any Task Manager (including Process Explorer, Process Hacker and even opening the correct process with VMMap), it is Pagefile-Backed Sections.

Pagefile-backed sections

Sections are the underlying objects of Memory-Mapped Files and pagefile-backed sections mean that the backing store behind the memory is the Pagefile, and thus they consume Commit Charge. It makes sense to use those objects when you want to store temporary data that you don’t want constantly in memory (by creating the sections, filling them with data and then unmapping them from the process address space, that’s why VMMap doesn’t help here as well – if the sections are unmapped), an advantage of this method over simple file I/O is that you don’t leave unneeded files in case the process or the system crashes.

An example of extensive pagefile-backed sections usage is Process Monitor, that tool can capture a very large amount (many millions) of system events without pressuring physical memory using pagefile-backed sections as the event store. I once forgot that I left it open and ran out of Commit Charge and got the “your computer is low on memory. close programs to prevent information loss” error. Process Explorer couldn’t help finding the guilty process and that’s how this post was born.

Looking for pagefile-backed sections

In this great post (suggested reading for more details about Windows virtual memory management), Sysinternals’ handle.exe with the –l flag is suggested to display the size of the pagefile-backed sections that all processes has handles to, however currently that flag seems broken (crashes), so I’ll suggest alternative methods for tracking pagefile-backed sections.

Both Process Explorer and Process Hacker are capable of displaying the size of the pagefile-backed sections that a process has handles to, with this method you would have to go over suspect processes one by one.

Process explorer

First we should check Show Unnamed Handles and Mappings:

Now in the handles pane we can see unnamed pagefile-backed sections and by double-clicking them we can see their size (in hex bytes):
 

Process hacker

After choosing and double-clicking on a process, in the handles tab uncheck hide unnamed handles and similarly to Process Explorer you can view the section size:

Note that in both applications in this case you can see that the type of the section is “Reserve”  which means that the memory specified is reserved and not necessarily committed (if nothing was committed there is zero impact on Commit Charge).

Checking the Committed memory

In order to see the committed memory by the sections we can do local kernel debugging with LiveKD.
We can take the address of the Section object from either application and dump its _SECTION_OBJECT structure:

0: kd> dt nt!_SECTION_OBJECT 0xFFFFC00065CB95C0
   +0x000 StartingVa       : 0xffffd000`2213d8e8 Void
   +0x008 EndingVa         : 0xfffff802`4e4a8f75 Void
   +0x010 Parent           : 0x00000000`00000001 Void
   +0x018 LeftChild        : (null) 
   +0x020 RightChild       : (null) 
   +0x028 Segment          : 0xffffc000`5c0f66b0 _SEGMENT_OBJECT

The Segment structure contains the committed memory size:

0: kd> dt nt!_SEGMENT 0xffffc000`5c0f66b0
   +0x000 ControlArea      : 0xffffe001`43f15880 _CONTROL_AREA
   +0x008 TotalNumberOfPtes : 0x17d79
   +0x00c SegmentFlags     : _SEGMENT_FLAGS
   +0x010 NumberOfCommittedPages : 0x9c00
   +0x018 SizeOfSegment    : 0x17d79000
   +0x020 ExtendInfo       : (null) 
   +0x020 BasedAddress     : (null) 
   +0x028 SegmentLock      : _EX_PUSH_LOCK
   +0x030 u1               : <unnamed-tag>
   +0x038 u2               : <unnamed-tag>
   +0x040 PrototypePte     : (null) 

I Used _SEGMENT instead of _SEGMENT_OBJECT (that is contained in _SECTION_OBJECT’s symbols) because _SEGMENT_OBJECT’s output didn’t make sense (for example NonExtendedPtes can’t be the same size as the section in bytes, as a PTE is used for each page, which is 0x1000 bytes of memory): 

0: kd> dt nt!_SEGMENT_OBJECT 0xffffc000`5c0f66b0
   +0x000 BaseAddress      : 0xffffe001`43f15880 Void
   +0x008 TotalNumberOfPtes : 0x17d79
   +0x010 SizeOfSegment    : _LARGE_INTEGER 0x9c00
   +0x018 NonExtendedPtes  : 0x17d79000
   +0x01c ImageCommitment  : 0
   +0x020 ControlArea      : (null) 
   +0x028 Subsection       : (null) 
   +0x030 MmSectionFlags   : 0xffffe001`45e7a8c0 _MMSECTION_FLAGS
   +0x038 MmSubSectionFlags : 0x00000000`3fde0000 _MMSUBSECTION_FLAGS

So the committed memory is NumberOfCommittedPages * 0x1000 bytes, which in this case is 0x9c00000 bytes, or 156MBs.

other methods

Possible alternative methods that would not require checking each process are based on kernel debugger commands:

1. !ca 0 21 or !ca 0 28 – that command displays all system control areas related to pagefile-backed sections, control areas are structures related to file mappings, and they are pointing to Segment structures that we’ve seen above. However the 0 parameter seems currently broken (checked both with LiveKD on Windows 8.1 and live debugging on Windows 10).


2. !handle 0 1 Section – that command displays all handles in all processes pointing to section objects and it can be scripted to dump the size of the committed memory using the structures displayed above.

introduction

Using this great dotPeek feature we can debug any third party managed assembly we want (it gets even better when we prevent JIT optimizations), unless it was built without debug information. This can be seen during debug in the Modules window:
Such assemblies were created without a PDB file using that setting:
The debugger sees that the assembly was created without a PDB file, so it will refuse loading the “fake” PDB that dotPeek creates.

Note that there is no reason to build assemblies with this setting, the fact that the assembly points to a PDB file is mostly relevant during debug time and has no impact on performance. Thus it’s rare to find such assemblies (all Microsoft’s .NET Framework assemblies and even Windows’ native DLLs were built with PDBs for example) but they exist.

Where is that information stored?

This information is stored in the PE header, and can be viewed with dumbin /headers:

C:\> dumpbin /headers Test.dll
...
  Debug Directories

        Time Type       Size      RVA  Pointer
    -------- ------ -------- -------- --------
    55324F94 cv           24 00125D90   123F90    Format: RSDS, {C2B3677B-EE83-4
1DA-8CA2-C16D74BB6C87}, 1, Test.pdb
...

The output above contains the GUID that is matched by the debugger in the loaded PDB file along with few other things. When Debug Info is set to none no Debug directories are emitted and thus no PDB file can be matched.

What can we do?

Theoretically it is possible to directly add Debug Directories to the assembly but no common tool does it, so we’ll use ildasm and ilasm:

C:\> ildasm NoDebugInfo.dll /out=Test.il
C:\> ilasm /dll /pdb /Out:Test.dll Test.il

Those pair of commands create an assembly with a PDB file, thus the assembly has Debug Directories that point to the PDB. You now may be asking the following:

1. What if the assembly is strong name signed?
   This will break the signature, but for some cases it isn’t validated and when it is you can disable validation for this assembly using sn –Vr <AssemblyFile> (don’t forget to restore the setting later using sn –Vu or sn -Vx). Note that you must run the correct edition (32 bit or 64 bit) for the program you are running, the Visual Studio Developer Command Prompt on my machine references the 32 bit one so for 64 bit application I must run the one located in C:\Program Files (x86)\Microsoft SDKs\Windows\v8.1A\bin\NETFX 4.5.1 Tools\x64 on my machine.


2. What does the created PDB contain?
   .NET PDBs point between IL offsets and source code lines, in this case instead of C# our source is IL, which isn’t really useful because the same IL instructions are embedded in the assembly, However for our purpose it doesn’t matter, we just wanted to add Debug Directories and we’ll use the dotPeek provided PDB that is “faking” the PDB signature to actually debug C# code.

Introduction

A question that one may ask is "Is the Windows kernel aware of the .Net Framework?".
Assuming "No" here is very sensible because the Windows kernel should expose all of the necessary services for any software framework in the Windows API, and any kernel changes made for a software framework would hurt encapsulation.
Indeed, the entire .NET CLR is made of DLL’s that are using the Windows API, However there are few features in .NET that involve kernel modifications and I will discuss them in this post.

How .NET executables launch

.NET executables (and .NET DLLs as well) are like other exe files on Windows, they use the Portable Executable (PE) file format, and are usually launched with a CreateProcess Windows API call.
When Windows launches a native process the last step of the process startup is passing the execution control to the entry point of the executable (can be viewed with dumpbin /headers command). With .NET executables, since Windows XP, the system passes the execution to _CorExeMain function of the mscoree.dll (ignoring the entry point), which initializes the CLR and passes the control to it (On older versions of Windows the system did pass execution to the executables’s entry point which contained short code to jump to _CorExeMain).

The component that runs at the start of the process, and is responsible for initialization tasks like calling _CorExeMain, is called the NTDLL Loader, you can note that its functions start with ntdll!Ldr*.

How does windows recognize .NET executables? They contain .NET specific metadata, the part that is relevant to this post can be viewed (and modified) with the CorFlags utility.
This is the first demonstration of Windows' knowledge of .NET, but the next features are bigger.

AnyCPU Executables

The interesting feature of .NET executables that run as 64-bit  on 64-bit Windows, and as 32-bit on 32-bit windows.

The newer Prefer 32-bit option isn’t very interesting for this post, as on 64-bit windows it makes the executable behave like a 32-bit only executable when launched, thus we will refer to AnyCPU as an executable built without this flag.

How does it work?

The documentation of mscoree’s _CorValidateImage refers to changing 32-bit (PE32) executable image to 64-bit (PE32+) executable image in memory (an executable in memory is called an Image) on 64-bit Windows, and you might think that it's all to it, However that's a user mode function that is supposed to be called in the newly created process by the NTDLL Loader, the decision about the architecture of a process happens earlier, and in kernel mode (kernel structures like EPROCESS depend on it), so there has to be some kernel mode code knowing about .NET which decides which architecture to use.

Note: I debugged and _CorValidateImage isn't getting called on Windows 8.1 (on Windows 7 it does), its responsibility seems to have moved to the Loader's LdrpCorValidateImage and LdrpCorFixupImage functions.

Entering kernel mode

I attached a kernel debugger to a Windows 10 VM, I wanted to inspect the data structures related to executables in memory.

I started with finding the process (I chose ILSpy, an AnyCPU executable):

0: kd> !process 0 1 ILSpy.exe

PROCESS ffffe0001e7e57c0
    SessionId: 1  Cid: 533c    Peb: 7ff5ffb86000  ParentCid: 1808
FreezeCount 1
    DirBase: 23540f000  ObjectTable: ffffc00158aded80  HandleCount: <Data Not Accessible>
    Image: ILSpy.exe
    VadRoot ffffe0001d03dac0 Vads 12 Clone 0 Private 44. Modified 5. Locked 0.
    DeviceMap 0000000000000000
    Token                             ffffc0015c62c930
    ElapsedTime                       09:02:01.350
    UserTime                          00:00:00.000
    KernelTime                        00:00:00.000
    QuotaPoolUsage[PagedPool]         17352
    QuotaPoolUsage[NonPagedPool]      1408
    Working Set Sizes (now,min,max)  (161, 50, 345) (644KB, 200KB, 1380KB)
    PeakWorkingSetSize                148
    VirtualSize                       3 Mb
    PeakVirtualSize                   3 Mb
    PageFaultCount                    199
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      70
    DebugPort                         ffffe0000e433cf0
    Job                               ffffe0001f724430

VAD's are structures that windows uses to track the virtual address space of a process, each process has a VAD root (highlighted in the output above) and it points to other VAD's (and so on) forming a tree structure that represents the virtual address space of a process. The !VAD debugger extenstion command helps analyze VAD's. By passing the VAD Root of the process, the command displays the virtual memory layout of the process:

0: kd> !vad ffffe0001d03dac0
VAD             level      start      end    commit
ffffe000127bd430 ( 3)         b90       c15         1 Mapped  Exe  EXECUTE_WRITECOPY  \path\ILSpy.exe
ffffe00012993010 ( 2)         c20       c3f        32 Private      READWRITE          
ffffe0001e92e320 ( 3)         c40       c4e         0 Mapped       READONLY           Pagefile-backed section
ffffe00018be7dc0 ( 1)         c50       d4f         5 Private      READWRITE          
ffffe00019f154c0 ( 3)         d50       d53         0 Mapped       READONLY           Pagefile-backed section
ffffe0001dc9a090 ( 2)         d60       d60         0 Mapped       READONLY           Pagefile-backed section
ffffe00019057120 ( 3)         d70       d71         2 Private      READWRITE          
ffffe0001d03dac0 ( 0)       7ffe0     7ffef        -1 Private      READONLY           
ffffe0000f121ca0 ( 2)   7ff5ffb60 7ff5ffb82         0 Mapped       READONLY           Pagefile-backed section
ffffe0000db3a5a0 ( 1)   7ff5ffb86 7ff5ffb86         1 Private      READWRITE          
ffffe00019a990f0 ( 3)   7ff5ffb8e 7ff5ffb8f         2 Private      READWRITE          
ffffe00019c043d0 ( 2)   7ffb3c5e0 7ffb3c78b        17 Mapped  Exe  EXECUTE_WRITECOPY  \Windows\System32\ntdll.dll

Total VADs: 12, average level: 3, maximum depth: 3

We can see above the VAD describing our executable image (highlighted), now when we pass it to the !VAD command with the extra parameter 1, we’ll get extra details about that specific image:

0: kd> !vad ffffe000127bd430 1

VAD @ ffffe000127bd430
  Start VPN                  b90  End VPN              c15  Control Area  ffffe000196e5d80
  FirstProtoPte ffffc00146cbfbd0  LastPte ffffc00146cbfff8  Commit Charge                1 (1.)
  Secured.Flink                0  Blink                  0  Banked/Extend                0
  File Offset                  0  
      ImageMap ViewShare EXECUTE_WRITECOPY 

ControlArea  @ ffffe000196e5d80
  Segment      ffffc001584de880  Flink      ffffe000127bd490  Blink        ffffe000126fe150
  Section Ref                 2  Pfn Ref                  83  Mapped Views                2
  User Ref                    4  WaitForDel                0  Flush Count              5ed8
  File Object  ffffe0000f1f3c80  ModWriteCount             0  System Views             ffff
  WritableRefs                0  
  Flags (a0) Image File 

      \path\ILSpy.exe

Segment @ ffffc001584de880
  ControlArea     ffffe000196e5d80  BasedAddress  0000000000400000
  Total Ptes                    86
  Segment Size               86000  Committed                    0
  Image Commit                   0  Image Info    ffffc001584de8c8
  ProtoPtes       ffffc00146cbfbd0
  Flags (1c20000) ProtectionMask 

The output contains a property named Image Info (highlighted), that seems to contain the address of some kind of a structure (addresses starting with ffff are kernel space). Now the goal is to figure out which one, unfortunately the debug symbols of the object containing it - _SEGMENT, give no info about its type:

0: kd> dt nt!_SEGMENT ffffc001584de880
   +0x000 ControlArea      : 0xffffe000`196e5d80 _CONTROL_AREA
   +0x008 TotalNumberOfPtes : 0x86
   +0x00c SegmentFlags     : _SEGMENT_FLAGS
   +0x010 NumberOfCommittedPages : 0
   +0x018 SizeOfSegment    : 0x86000
   +0x020 ExtendInfo       : 0x00000000`00400000 _MMEXTEND_INFO
   +0x020 BasedAddress     : 0x00000000`00400000 Void
   +0x028 SegmentLock      : _EX_PUSH_LOCK
   +0x030 u1               : <unnamed-tag>
   +0x038 u2               : <unnamed-tag>
   +0x040 PrototypePte     : 0xffffc001`46cbfbd0 _MMPTE
0: kd> dps ffffc001584de880
ffffc001`584de880  ffffe000`196e5d80
ffffc001`584de888  01c20000`00000086
ffffc001`584de890  00000000`00000000
ffffc001`584de898  00000000`00086000
ffffc001`584de8a0  00000000`00400000
ffffc001`584de8a8  00000000`00000000
ffffc001`584de8b0  00000000`00000000
ffffc001`584de8b8  ffffc001`584de8c8
ffffc001`584de8c0  ffffc001`46cbfbd0

The above commands show the missing symbol information (first command) and the actual raw memory (second command) containing our pointer at the same offset (0x38).

The next option is guessing the type by searching for existing symbols:

0: kd> dt nt!*image*info*
          ntkrnlmp!_MI_SECTION_IMAGE_INFORMATION
          ntkrnlmp!_SECTION_IMAGE_INFORMATION
          ntkrnlmp!_MI_EXTRA_IMAGE_INFORMATION

Trying dt with _SECTION_IMAGE_INFORMATION, and bingo, it fits (data like stack sizes makes sense):

0: kd> dt nt!_SECTION_IMAGE_INFORMATION ffffc001584de8c8
   +0x000 TransferAddress  : 0x00000000`004734be Void
   +0x008 ZeroBits         : 0
   +0x010 MaximumStackSize : 0x100000
   +0x018 CommittedStackSize : 0x1000
   +0x020 SubSystemType    : 2
   +0x024 SubSystemMinorVersion : 0
   +0x026 SubSystemMajorVersion : 4
   +0x024 SubSystemVersion : 0x40000
   +0x028 GpValue          : 0
   +0x02c ImageCharacteristics : 0x102
   +0x02e DllCharacteristics : 0x8540
   +0x030 Machine          : 0x14c
   +0x032 ImageContainsCode : 0x1 ''
   +0x033 ImageFlags       : 0x3 ''
   +0x033 ComPlusNativeReady : 0y1
   +0x033 ComPlusILOnly    : 0y1
   +0x033 ImageDynamicallyRelocated : 0y0
   +0x033 ImageMappedFlat  : 0y0
   +0x033 BaseBelow4gb     : 0y0
   +0x033 Reserved         : 0y000
   +0x034 LoaderFlags      : 1
   +0x038 ImageFileSize    : 0x83000
   +0x03c CheckSum         : 0x8859c

Immediatly it looks like it contains 2 interesting flags: ComPlusILOnly and ComPlusNativeReady.
ComPlusILOnly seems helpful, and it actually is: it simply tells us if the PE contains only IL instructions (CorFlags ILONLY flag) or it's a mixed mode (or native) executable. Obviously an executable that contains any amount of machine code (mixed or native) can't be AnyCPU, thus this flag must be set for it to work.

I couldn't really make sense of the name of ComPlusNativeReady flag but from testing it with different executables it is turned on (value is 1) when the executable is AnyCPU, the relevant CorFlags flags for that - 32BITREQ (only 32-bit) and 32BITPREF (prefer 32-bit) should be turned off, and the ILONLY flag must be turned on.

By changing these value in the debugger with eb <ImageFlagsAddres> <0,1,2,3> I observed how new processes of the same executable spawn differently (it worked because the PE headers aren't parsed each time a new process is launched, they are cached).

As for the kernel execution flow, the parsing of the executable seem to happen at the nt!MiParseImageSectionHeaders and nt!MiParseComImage functions, and the branching based on the Image Flags at nt!PspAllocateProcess and nt!PspDetectComplusILImage (checked using memory access breakpoints and some tracing).

Repositioning the image in memory

For increased security, since Windows Vista, a feature called ASLR is repositioning images in memory between boots, the reason that it happens only between boots and not for every process launch is because images should be shared between processes for best performance (think of images like ntdll.dll that every process contains and the memory that would have been wasted if they weren’t shared), and because of the fact that they contain pointers to absolute addresses, those addresses should be the same for every process. Thus all images are relocated only once per boot and otherwise every image has its own constant base address for all processes.

However, IL Only images contain no native code and thus no pointers to absolute addresses, so unlike native images there is no reason not to relocate them per process without compromising the sharing, and it actually is implemented in the kernel. Based on the ComPlusILOnly flag, the kernel decides whether to relocate the image every time the image is loaded to a process.
Below is a screen capture of Process Explorer’s DLL’s view of two processes of the same executable, note how the Base value changes. Similar relocations happen with managed IL only referenced DLL’s, unlike the Base of ntdll which stays the same.
 

Conclusion

In this post we have seen how few features in .NET required Windows kernel modifications and inspected their high level implementation with some debugging techniques, This list of features wasn’t meant to be exhaustive but those are the features that I’m currently aware of, I’ll be happy to find more such features in the feature.