WoW64 internals: Tale of GetSystemFileCacheSize

Few days ago someone asked me if I can somehow add GetSystemFileCacheSize to wow64ext library. I’ve researched this topic a bit and the final answer is no, because it is not necessary. In today post I’ll try to describe internals of GetSystemFileCacheSize function and its limitations, I’ll also show the different way of obtaining the same information as original GetSystemFileCacheSize.

GetSystemFileCacheSize internals

Body of the function can be found inside kernel32 library, it is pretty simple:

BOOL WINAPI GetSystemFileCacheSize
(
	PSIZE_T lpMinimumFileCacheSize, 
	PSIZE_T lpMaximumFileCacheSize, 
	PDWORD lpFlags
)
{
	BYTE* _teb32 = (BYTE*)__readfsdword(0x18);	// mov     eax, large fs:18h
	BYTE* _teb64 = *(BYTE**)(_teb32 + 0xF70);	// mov     eax, [eax+0F70h]
	DWORD unk_v = **(DWORD**)(_teb64 + 0x14D0);	// mov     eax, [eax+14D0h]
 
	SYSTEM_FILECACHE_INFORMATION sfi;
	//SystemFileCacheInformationEx = 0x51
	NTSTATUS ret = NtQuerySystemInformation((SYSTEM_INFORMATION_CLASS)0x51, &sfi, sizeof(sfi), 0);
	if (ret < 0)
	{
		BaseSetLastNTError(ret);
		return FALSE;
	}
 
	if ((unsigned int)sfi.MinimumWorkingSet * (unsigned __int64)unk_v > 0xFFFFFFFF ||
	    (unsigned int)sfi.MaximumWorkingSet * (unsigned __int64)unk_v > 0xFFFFFFFF )
	{
		BaseSetLastNTError(STATUS_INTEGER_OVERFLOW);
		return FALSE;
	}
 
	*lpMinimumFileCacheSize = unk_v * sfi.MinimumWorkingSet;
	*lpMaximumFileCacheSize = unk_v * sfi.MaximumWorkingSet;
	*lpFlags = 0;
 
	if (sfi.Flags & FILE_CACHE_MIN_HARD_ENABLE)
		*lpFlags = FILE_CACHE_MIN_HARD_ENABLE;
 
	if (sfi.Flags & FILE_CACHE_MAX_HARD_ENABLE)
		*lpFlags |= FILE_CACHE_MAX_HARD_ENABLE;
 
	return TRUE;
}

So, let’s study it step by step. At first it gets some magic value from the TEB, figuring out where this value came from is the key to understand the whole function. FS:0x18 contains TEB32 pointer, next instruction gets another pointer from TEB32+0xF70, according to PDB symbols, this field is called GdiBatchCount:

TEB32:
...
   +0xf6c WinSockData           : Ptr32 Void
   +0xf70 GdiBatchCount         : Uint4B
   +0xf74 CurrentIdealProcessor : _PROCESSOR_NUMBER
...

First surprise, in WoW64 this field contains pointer to x64 TEB (yes, in WoW64 processes there are two TEBs, x86 and x64, as well as two PEBs, but regular readers of this blog probably already knew it :) ). In third step, it gets value from TEB64+0x14D0, according to PDB symbols it is one of the entries inside TlsSlots:

TEB64:
...
   +0x1478 DeallocationStack : Ptr64 Void
   +0x1480 TlsSlots          : [64] Ptr64 Void
   +0x1680 TlsLinks          : _LIST_ENTRY
...

Precisely speaking it is TEB64.TlsSlot[0x0A]. It took me some time to track the code responsible for writing to this particular TlsSlot, but I’ve succeeded. There is only one place inside wow64.dll that writes at this address:

;wow64.dll::Wow64LdrpInitialize:
...
.text:0000000078BDC15C         mov     rax, gs:30h
.text:0000000078BDC165         mov     rdi, rcx
.text:0000000078BDC168         xor     r13d, r13d
.text:0000000078BDC16B         mov     ecx, [rax+2030h]
.text:0000000078BDC171         or      r15, 0FFFFFFFFFFFFFFFFh
.text:0000000078BDC175         mov     r14d, 1
.text:0000000078BDC17B         add     rcx, 248h
.text:0000000078BDC182         mov     cs:Wow64InfoPtr, rcx
.text:0000000078BDC189         mov     rcx, gs:30h
.text:0000000078BDC192         mov     rax, cs:Wow64InfoPtr
.text:0000000078BDC199         mov     [rcx+14D0h], rax       ;<- write to TlsSlot
...

Above snippet is a part of the Wow64LdrpInitialize function (it is exported from wow64.dll) and there are a few interesting things. At first it gets TEB64 address from GS:0x30 (standard procedure on x64 windows), then it gets address from TEB64+0x2030, adds 0x248 to this address and stores this value in TEB64.TlsSlot[0x0A]. It stores it also in a variable called Wow64InfoPtr. Looking at this code under debugger reveals more interesting details, apparently TEB64+0x2000 points to TEB32, TEB32+0x30 contains pointer to PEB32:

TEB32:
...
   +0x02c ThreadLocalStoragePointer : 0x7efdd02c 
   +0x030 ProcessEnvironmentBlock   : 0x7efde000 _PEB
   +0x034 LastErrorValue            : 0
...

So, there is some mystical Wow64InfoPtr structure at PEB32+0x248 address. On Windows 7, 0x248 is the exact size of PEB32 structure (aligned to 8, on windows Vista it is 0x238, probably on Windows 8 it will be also different), so this new structure follows PEB32. Querying google for Wow64InfoPtr returns 0 results. Judging from the references, this structure contains at least three fields, I was interested only in the first one:

;wow64.dll::ProcessInit:
...
.text:0000000078BD8F70         mov     rax, cs:Wow64InfoPtr
.text:0000000078BD8F77         mov     edx, 1
.text:0000000078BD8F7C         and     dword ptr [rax+8], 0
.text:0000000078BD8F80         mov     dword ptr [rax], 1000h
...

Above code is part of the wow64.dll::ProcessInit function, and it has hardcoded 0x1000 value that is assigned to the first field of the Wow64InfoPtr structure. Summing things up, those first three operations at the begining of GetSystemFileCacheSize are just getting hardcoded 0x1000 value. I can guess, that this value represents size of the memory page inside WoW64 process, I can also confirm this guess by looking at the x64 version of GetSystemFileCacheSize:

;kernel32.dll::GetSystemFileCacheSize:
...
.text:0000000078D67CF6         mov     eax, cs:SysInfo.uPageSize
.text:0000000078D67CFC         mov     ecx, [rsp+68h+var_48.Flags]
.text:0000000078D67D00         imul    rax, [rsp+68h+var_48.MinimumWorkingSet]
.text:0000000078D67D06         mov     [rsi], rax
.text:0000000078D67D09         mov     eax, cs:SysInfo.uPageSize
.text:0000000078D67D0F         imul    rax, [rsp+68h+var_48.MaximumWorkingSet]
...

Further part of the function is clear, it calls NtQuerySystemInformation with SYSTEM_INFORMATION_CLASS set to SystemFileCacheInformationEx (0x51). After this call SYSTEM_FILECACHE_INFORMATION structure is filled with desired information:

typedef struct _SYSTEM_FILECACHE_INFORMATION {
	SIZE_T CurrentSize;
	SIZE_T PeakSize;
	ULONG PageFaultCount;
	SIZE_T MinimumWorkingSet;
	SIZE_T MaximumWorkingSet;
	SIZE_T CurrentSizeIncludingTransitionInPages;
	SIZE_T PeakSizeIncludingTransitionInPages;
	ULONG TransitionRePurposeCount;
	ULONG Flags;
} SYSTEM_FILECACHE_INFORMATION, *PSYSTEM_FILECACHE_INFORMATION;

At this point function verifies if output values (lpMinimumFileCacheSize and lpMaximumFileCacheSize) fits in 32bits:

if ((unsigned int)sfi.MinimumWorkingSet * (unsigned __int64)unk_v > 0xFFFFFFFF ||
    (unsigned int)sfi.MaximumWorkingSet * (unsigned __int64)unk_v > 0xFFFFFFFF )
{ ... }

With the default (?) system settings lpMaximumFileCacheSize will exceed 32bits, because sfi.MaximumWorkingSet is set to 0x10000000 and unk_v (page size) is 0x1000, so multiplication of those values wouldn’t fit in 32bits. In that case function will return FALSE and set last Win32 error to ERROR_ARITHMETIC_OVERFLOW (0x00000216).

GetSystemFileCacheSize replacement

I’m not really sure if there is some proper (and documented) way to replace this function, but one can use NtQuerySystemInformation with SystemFileCacheInformationEx (0x51) or SystemFileCacheInformation (0x15) classes, both classes are using the same SYSTEM_FILECACHE_INFORMATION structure. SystemFileCacheInformation (0x15) is used by CacheSet sysinternals tool. To get exactly the same values as from GetSystemFileCacheSize, results returned by NtQuerySystemInformation should be adjusted by below code snippet:

	lpMinimumFileCacheSize = PAGE_SIZE * sfi.MinimumWorkingSet;
	lpMaximumFileCacheSize = PAGE_SIZE * sfi.MaximumWorkingSet;
	lpFlags = 0;
 
	if (sfi.Flags & FILE_CACHE_MIN_HARD_ENABLE)
		lpFlags = FILE_CACHE_MIN_HARD_ENABLE;
 
	if (sfi.Flags & FILE_CACHE_MAX_HARD_ENABLE)
		lpFlags |= FILE_CACHE_MAX_HARD_ENABLE;

Thanks for reading.

11 Comments

bw June 28, 2013 at 20:25

It seems like you’re wasting a lot of your time in Wow64 :)

1. ReWolf June 28, 2013 at 20:28
  
  All roads lead to WoW64 ]:->
  
waliedassar July 1, 2013 at 10:59

FS:0×18 contains TEB32 pointer, next instruction gets another pointer from TEB32+0xF70

Also, in Wow64 processes, the 64Bit TEB has a pointer to corresponding 32Bit TEB at offset 0x0, which is _NT_TIB::ExceptionList. They have chosen this field since the 64Bit code, as you know, has no stack-based exception handling.

1. ReWolf July 1, 2013 at 11:41
  
  Thanks for the nice explanation.
  
2. master131 March 8, 2014 at 05:14
  
  I was looking around to confirm that this was the default behavior on XP/Server 2003 64-bit and came across the WRK (Windows Research Kernel) with this nice line of code:
  TebBase->NtTib.ExceptionList = (PVOID)Teb32Base;
  
  PS – Reference is here: http:// [ link removed ] /WRK-v1.2/base/ntos/mm/procsup.c

Thanks for the solution. I have tried the NtSetSystemInformation to replace SetSystemFileCacheSize, but unlike the NtQuerySystemInformation, NtSetSystemInformation uses Bytes rather than 4K Pages and has the same 32 bit limitation with WoW64. The older 32 bit CacheSet will display the maximum cache over 4GB, but also fails to set maximum cache > 4GB. The current CacheSet is OK. Is there any alternative solution to set the Cache over 4GB with WoW64.

In this case you’ll need to call x64 version of NtSetSystemInformation. You can do it with wow64ext library, just look at the sources and you’ll figure it out (GetModuleHandle64, GetProcAddress64, X64Call). Information class will be the same as for NtQuerySystemInformation, just keep in mind that you’ll need to define custom version of SYSTEM_FILECACHE_INFORMATION. On x64 platform SIZE_T is 64bit value so you can replace it with ULONGLONG:

typedef struct _SYSTEM_FILECACHE_INFORMATION64 {
	ULONGLONG CurrentSize;
	ULONGLONG PeakSize;
	ULONG PageFaultCount;
	ULONGLONG MinimumWorkingSet;
	ULONGLONG MaximumWorkingSet;
	ULONGLONG CurrentSizeIncludingTransitionInPages;
	ULONGLONG PeakSizeIncludingTransitionInPages;
	ULONG TransitionRePurposeCount;
	ULONG Flags;
} SYSTEM_FILECACHE_INFORMATION64, *PSYSTEM_FILECACHE_INFORMATION64;

I’m not sure about the alignment, but size of this structure on x64 should be 0x40, so it seems that it has default x64 (8-bytes) alignment, in that case PageFaultCount will be probably padded up to 8 bytes. Remember also about setting SeIncreaseQuotaPrivilege.

@ReWolf
I can confirm the structure works with PageFaultCount padded with 8 bytes, and the results are identical to the 32bit function when sizes don’t extend beyond 32 bits. The length is 64 bytes.
However I have had an “access violation” error . Using a timer I am reading the size of the file cache every second, but at some point this crashed. At the same time the utility continues to run on WHS2011 every second updating a display of the file cache usage.
Windows 7 gave the option to launch the debugger and I extracted the code which listed the breakpoint just after the retf call in X64Call:

52F58512   mov         dword ptr [esp+4],23h
52F5851A   add         dword ptr [esp],0Dh
52F5851E   retf
52F5851F   mov         esp,dword ptr [ebp-98h]      ; ! access violation !
52F58525   mov         eax,dword ptr [ebp-5Ch]
52F58528   mov         edx,dword ptr [ebp-58h]

The breakpoint is at 52F5851F where ebp = 18F86Ch, esp = 18F6F8h
and the matching line in X64CALL is

mov    esp, back_esp]

The contents of ebp-98h (18F7D4h) is 0018F6FCh
I have started another instance and this seems to be working fine for a few hours at least.
What can I do to minimise further access violation errors?

ReWolf July 12, 2013 at 08:08

Unfortunately I’ve no idea what could cause access violation at that point ;/

hors July 4, 2013 at 13:29

Good job!

//apparently TEB64+0×2000 points to TEB32
yep.
http://redplait.blogspot.de/2012/12/teb32-of-wow64-process.html

//TEB32+0×30 contains pointer to PEB32:

its invariable

Awk October 19, 2013 at 10:06

You sir, are doing a great job at this.
I love this blog. Keep it up.

GetSystemFileCacheSize internals

GetSystemFileCacheSize replacement

11 Comments

Leave a Reply to ReWolf Cancel reply