Few days ago someone asked me if I can somehow add GetSystemFileCacheSize to wow64ext library. I’ve researched this topic a bit and the final answer is no, because it is not necessary. In today post I’ll try to describe internals of GetSystemFileCacheSize function and its limitations, I’ll also show the different way of obtaining the same information as original GetSystemFileCacheSize.
GetSystemFileCacheSize internals
Body of the function can be found inside kernel32 library, it is pretty simple:
BOOL WINAPI GetSystemFileCacheSize ( PSIZE_T lpMinimumFileCacheSize, PSIZE_T lpMaximumFileCacheSize, PDWORD lpFlags ) { BYTE* _teb32 = (BYTE*)__readfsdword(0x18); // mov eax, large fs:18h BYTE* _teb64 = *(BYTE**)(_teb32 + 0xF70); // mov eax, [eax+0F70h] DWORD unk_v = **(DWORD**)(_teb64 + 0x14D0); // mov eax, [eax+14D0h] SYSTEM_FILECACHE_INFORMATION sfi; //SystemFileCacheInformationEx = 0x51 NTSTATUS ret = NtQuerySystemInformation((SYSTEM_INFORMATION_CLASS)0x51, &sfi, sizeof(sfi), 0); if (ret < 0) { BaseSetLastNTError(ret); return FALSE; } if ((unsigned int)sfi.MinimumWorkingSet * (unsigned __int64)unk_v > 0xFFFFFFFF || (unsigned int)sfi.MaximumWorkingSet * (unsigned __int64)unk_v > 0xFFFFFFFF ) { BaseSetLastNTError(STATUS_INTEGER_OVERFLOW); return FALSE; } *lpMinimumFileCacheSize = unk_v * sfi.MinimumWorkingSet; *lpMaximumFileCacheSize = unk_v * sfi.MaximumWorkingSet; *lpFlags = 0; if (sfi.Flags & FILE_CACHE_MIN_HARD_ENABLE) *lpFlags = FILE_CACHE_MIN_HARD_ENABLE; if (sfi.Flags & FILE_CACHE_MAX_HARD_ENABLE) *lpFlags |= FILE_CACHE_MAX_HARD_ENABLE; return TRUE; } |
So, let’s study it step by step. At first it gets some magic value from the TEB, figuring out where this value came from is the key to understand the whole function. FS:0x18 contains TEB32 pointer, next instruction gets another pointer from TEB32+0xF70, according to PDB symbols, this field is called GdiBatchCount:
TEB32: ... +0xf6c WinSockData : Ptr32 Void +0xf70 GdiBatchCount : Uint4B +0xf74 CurrentIdealProcessor : _PROCESSOR_NUMBER ...
First surprise, in WoW64 this field contains pointer to x64 TEB (yes, in WoW64 processes there are two TEBs, x86 and x64, as well as two PEBs, but regular readers of this blog probably already knew it :) ). In third step, it gets value from TEB64+0x14D0, according to PDB symbols it is one of the entries inside TlsSlots:
TEB64: ... +0x1478 DeallocationStack : Ptr64 Void +0x1480 TlsSlots : [64] Ptr64 Void +0x1680 TlsLinks : _LIST_ENTRY ...
Precisely speaking it is TEB64.TlsSlot[0x0A]. It took me some time to track the code responsible for writing to this particular TlsSlot, but I’ve succeeded. There is only one place inside wow64.dll that writes at this address:
;wow64.dll::Wow64LdrpInitialize: ... .text:0000000078BDC15C mov rax, gs:30h .text:0000000078BDC165 mov rdi, rcx .text:0000000078BDC168 xor r13d, r13d .text:0000000078BDC16B mov ecx, [rax+2030h] .text:0000000078BDC171 or r15, 0FFFFFFFFFFFFFFFFh .text:0000000078BDC175 mov r14d, 1 .text:0000000078BDC17B add rcx, 248h .text:0000000078BDC182 mov cs:Wow64InfoPtr, rcx .text:0000000078BDC189 mov rcx, gs:30h .text:0000000078BDC192 mov rax, cs:Wow64InfoPtr .text:0000000078BDC199 mov [rcx+14D0h], rax ;<- write to TlsSlot ... |
Above snippet is a part of the Wow64LdrpInitialize function (it is exported from wow64.dll) and there are a few interesting things. At first it gets TEB64 address from GS:0x30 (standard procedure on x64 windows), then it gets address from TEB64+0x2030, adds 0x248 to this address and stores this value in TEB64.TlsSlot[0x0A]. It stores it also in a variable called Wow64InfoPtr. Looking at this code under debugger reveals more interesting details, apparently TEB64+0x2000 points to TEB32, TEB32+0x30 contains pointer to PEB32:
TEB32: ... +0x02c ThreadLocalStoragePointer : 0x7efdd02c +0x030 ProcessEnvironmentBlock : 0x7efde000 _PEB +0x034 LastErrorValue : 0 ...
So, there is some mystical Wow64InfoPtr structure at PEB32+0x248 address. On Windows 7, 0x248 is the exact size of PEB32 structure (aligned to 8, on windows Vista it is 0x238, probably on Windows 8 it will be also different), so this new structure follows PEB32. Querying google for Wow64InfoPtr returns 0 results. Judging from the references, this structure contains at least three fields, I was interested only in the first one:
;wow64.dll::ProcessInit: ... .text:0000000078BD8F70 mov rax, cs:Wow64InfoPtr .text:0000000078BD8F77 mov edx, 1 .text:0000000078BD8F7C and dword ptr [rax+8], 0 .text:0000000078BD8F80 mov dword ptr [rax], 1000h ... |
Above code is part of the wow64.dll::ProcessInit function, and it has hardcoded 0x1000 value that is assigned to the first field of the Wow64InfoPtr structure. Summing things up, those first three operations at the begining of GetSystemFileCacheSize are just getting hardcoded 0x1000 value. I can guess, that this value represents size of the memory page inside WoW64 process, I can also confirm this guess by looking at the x64 version of GetSystemFileCacheSize:
;kernel32.dll::GetSystemFileCacheSize: ... .text:0000000078D67CF6 mov eax, cs:SysInfo.uPageSize .text:0000000078D67CFC mov ecx, [rsp+68h+var_48.Flags] .text:0000000078D67D00 imul rax, [rsp+68h+var_48.MinimumWorkingSet] .text:0000000078D67D06 mov [rsi], rax .text:0000000078D67D09 mov eax, cs:SysInfo.uPageSize .text:0000000078D67D0F imul rax, [rsp+68h+var_48.MaximumWorkingSet] ... |
Further part of the function is clear, it calls NtQuerySystemInformation with SYSTEM_INFORMATION_CLASS set to SystemFileCacheInformationEx (0x51). After this call SYSTEM_FILECACHE_INFORMATION structure is filled with desired information:
typedef struct _SYSTEM_FILECACHE_INFORMATION { SIZE_T CurrentSize; SIZE_T PeakSize; ULONG PageFaultCount; SIZE_T MinimumWorkingSet; SIZE_T MaximumWorkingSet; SIZE_T CurrentSizeIncludingTransitionInPages; SIZE_T PeakSizeIncludingTransitionInPages; ULONG TransitionRePurposeCount; ULONG Flags; } SYSTEM_FILECACHE_INFORMATION, *PSYSTEM_FILECACHE_INFORMATION; |
At this point function verifies if output values (lpMinimumFileCacheSize and lpMaximumFileCacheSize) fits in 32bits:
if ((unsigned int)sfi.MinimumWorkingSet * (unsigned __int64)unk_v > 0xFFFFFFFF || (unsigned int)sfi.MaximumWorkingSet * (unsigned __int64)unk_v > 0xFFFFFFFF ) { ... } |
With the default (?) system settings lpMaximumFileCacheSize will exceed 32bits, because sfi.MaximumWorkingSet is set to 0x10000000 and unk_v (page size) is 0x1000, so multiplication of those values wouldn’t fit in 32bits. In that case function will return FALSE and set last Win32 error to ERROR_ARITHMETIC_OVERFLOW (0x00000216).
GetSystemFileCacheSize replacement
I’m not really sure if there is some proper (and documented) way to replace this function, but one can use NtQuerySystemInformation with SystemFileCacheInformationEx (0x51) or SystemFileCacheInformation (0x15) classes, both classes are using the same SYSTEM_FILECACHE_INFORMATION structure. SystemFileCacheInformation (0x15) is used by CacheSet sysinternals tool. To get exactly the same values as from GetSystemFileCacheSize, results returned by NtQuerySystemInformation should be adjusted by below code snippet:
lpMinimumFileCacheSize = PAGE_SIZE * sfi.MinimumWorkingSet; lpMaximumFileCacheSize = PAGE_SIZE * sfi.MaximumWorkingSet; lpFlags = 0; if (sfi.Flags & FILE_CACHE_MIN_HARD_ENABLE) lpFlags = FILE_CACHE_MIN_HARD_ENABLE; if (sfi.Flags & FILE_CACHE_MAX_HARD_ENABLE) lpFlags |= FILE_CACHE_MAX_HARD_ENABLE; |
Thanks for reading.
It seems like you’re wasting a lot of your time in Wow64 :)
All roads lead to WoW64 ]:->
Also, in Wow64 processes, the 64Bit TEB has a pointer to corresponding 32Bit TEB at offset 0x0, which is _NT_TIB::ExceptionList. They have chosen this field since the 64Bit code, as you know, has no stack-based exception handling.
Thanks for the nice explanation.
I was looking around to confirm that this was the default behavior on XP/Server 2003 64-bit and came across the WRK (Windows Research Kernel) with this nice line of code:
TebBase->NtTib.ExceptionList = (PVOID)Teb32Base;
PS – Reference is here: http:// [ link removed ] /WRK-v1.2/base/ntos/mm/procsup.c
Thanks for the solution. I have tried the NtSetSystemInformation to replace SetSystemFileCacheSize, but unlike the NtQuerySystemInformation, NtSetSystemInformation uses Bytes rather than 4K Pages and has the same 32 bit limitation with WoW64. The older 32 bit CacheSet will display the maximum cache over 4GB, but also fails to set maximum cache > 4GB. The current CacheSet is OK. Is there any alternative solution to set the Cache over 4GB with WoW64.
In this case you’ll need to call x64 version of NtSetSystemInformation. You can do it with wow64ext library, just look at the sources and you’ll figure it out (GetModuleHandle64, GetProcAddress64, X64Call). Information class will be the same as for NtQuerySystemInformation, just keep in mind that you’ll need to define custom version of SYSTEM_FILECACHE_INFORMATION. On x64 platform SIZE_T is 64bit value so you can replace it with ULONGLONG:
I’m not sure about the alignment, but size of this structure on x64 should be 0x40, so it seems that it has default x64 (8-bytes) alignment, in that case PageFaultCount will be probably padded up to 8 bytes. Remember also about setting SeIncreaseQuotaPrivilege.
@ReWolf
I can confirm the structure works with PageFaultCount padded with 8 bytes, and the results are identical to the 32bit function when sizes don’t extend beyond 32 bits. The length is 64 bytes.
However I have had an “access violation” error . Using a timer I am reading the size of the file cache every second, but at some point this crashed. At the same time the utility continues to run on WHS2011 every second updating a display of the file cache usage.
Windows 7 gave the option to launch the debugger and I extracted the code which listed the breakpoint just after the retf call in X64Call:
The breakpoint is at 52F5851F where ebp = 18F86Ch, esp = 18F6F8h
and the matching line in X64CALL is
The contents of ebp-98h (18F7D4h) is 0018F6FCh
I have started another instance and this seems to be working fine for a few hours at least.
What can I do to minimise further access violation errors?
Unfortunately I’ve no idea what could cause access violation at that point ;/
Good job!
//apparently TEB64+0×2000 points to TEB32
yep.
http://redplait.blogspot.de/2012/12/teb32-of-wow64-process.html
//TEB32+0×30 contains pointer to PEB32:
its invariable
You sir, are doing a great job at this.
I love this blog. Keep it up.