Debugging the Memory Allocation on Terminal Server


When you have some weird issues with the users on Terminal Services server which can be related somehow to the incorrect memory allocation it is always a good idea to take a look into the kernel memory.

Below you can see an example of a short investigation and a solution for the issue I faced.

The problem was that the server was working slowly sometimes and could hang for a while for some users. The tuning I described here was already applied.

So, let’s dive into the kernel

lkd> !vm 1

*** Virtual Memory Usage ***
Physical Memory:     2096892 (   8387568 Kb)
Page File: \??\C:\Pagefiles\1\pagefile.sys
Current:   4177920 Kb  Free Space:   3923976 Kb
Minimum:   4177920 Kb  Maximum:      4194304 Kb
Page File: \??\C:\Pagefiles\2\pagefile.sys
Current:   4177920 Kb  Free Space:   3972784 Kb
Minimum:   4177920 Kb  Maximum:      4194304 Kb
Page File: \??\C:\Pagefiles\3\pagefile.sys
Current:   4177920 Kb  Free Space:   3979812 Kb
Minimum:   4177920 Kb  Maximum:      4194304 Kb

Available Pages:     1631654 (   6526616 Kb)
ResAvail Pages:      1976670 (   7906680 Kb)
Locked IO Pages:         145 (       580 Kb)
Free System PTEs:     147688 (    590752 Kb)
Free NP PTEs:          32766 (    131064 Kb)
Free Special NP:           0 (         0 Kb)
Modified Pages:          200 (       800 Kb)
Modified PF Pages:       197 (       788 Kb)
NonPagedPool Usage:    12396 (     49584 Kb)
NonPagedPool Max:      65215 (    260860 Kb)

PagedPool 0 Usage:     30147 (    120588 Kb)
PagedPool 1 Usage:      2811 (     11244 Kb)
PagedPool 2 Usage:      2857 (     11428 Kb)
PagedPool 3 Usage:      2847 (     11388 Kb)
PagedPool 4 Usage:      2815 (     11260 Kb)
PagedPool Usage:       41477 (    165908 Kb)
PagedPool Maximum:    134144 (    536576 Kb)

********** 5 pool allocations have failed **********

Session Commit:        29540 (    118160 Kb)
Shared Commit:         31761 (    127044 Kb)
Special Pool:              0 (         0 Kb)
Shared Process:        24695 (     98780 Kb)
PagedPool Commit:      41541 (    166164 Kb)
Driver Commit:         13126 (     52504 Kb)
Committed pages:      580007 (   2320028 Kb)
Commit limit:        5180416 (  20721664 Kb)

As you can see, no problem at all with the pools; they have plenty of memory to use. However, 5 pool allocations have failed. Let’s check out the session pools

lkd> !vm 4

*** Virtual Memory Usage ***
Physical Memory:     2096892 (   8387568 Kb)
Page File: \??\C:\Pagefiles\1\pagefile.sys
Current:   4177920 Kb  Free Space:   3924108 Kb
Minimum:   4177920 Kb  Maximum:      4194304 Kb
Page File: \??\C:\Pagefiles\2\pagefile.sys
Current:   4177920 Kb  Free Space:   3972848 Kb
Minimum:   4177920 Kb  Maximum:      4194304 Kb
Page File: \??\C:\Pagefiles\3\pagefile.sys
Current:   4177920 Kb  Free Space:   3979900 Kb
Minimum:   4177920 Kb  Maximum:      4194304 Kb

Available Pages:     1630874 (   6523496 Kb)
ResAvail Pages:      1976665 (   7906660 Kb)
Locked IO Pages:         140 (       560 Kb)
Free System PTEs:     147688 (    590752 Kb)
Free NP PTEs:          32766 (    131064 Kb)
Free Special NP:           0 (         0 Kb)
Modified Pages:          235 (       940 Kb)
Modified PF Pages:       213 (       852 Kb)
NonPagedPool Usage:    12396 (     49584 Kb)
NonPagedPool Max:      65215 (    260860 Kb)
PagedPool 0 Usage:     30129 (    120516 Kb)
PagedPool 1 Usage:      2811 (     11244 Kb)
PagedPool 2 Usage:      2857 (     11428 Kb)
PagedPool 3 Usage:      2847 (     11388 Kb)
PagedPool 4 Usage:      2815 (     11260 Kb)
PagedPool Usage:       41459 (    165836 Kb)
PagedPool Maximum:    134144 (    536576 Kb)

********** 5 pool allocations have failed **********

Shared Commit:         31761 (    127044 Kb)
Special Pool:              0 (         0 Kb)
Shared Process:        24722 (     98888 Kb)
PagedPool Commit:      41502 (    166008 Kb)
Driver Commit:         13126 (     52504 Kb)
Committed pages:      580209 (   2320836 Kb)
Commit limit:        5180416 (  20721664 Kb)
Total Private:        440505 (   1762020 Kb)

36c0 M2M.EXE          20167 (     80668 Kb)
2bdc spoolsv.exe      12713 (     50852 Kb)
0c94 POWERPNT.EXE     10951 (     43804 Kb)
0748 Rtvscan.exe      10653 (     42612 Kb)
0e20 iexplore.exe      9573 (     38292 Kb)

………………………………..

0ae8 csrss.exe            0 (         0 Kb)
06d4 EModelViewer.ex      0 (         0 Kb)
064c rdpclip.exe          0 (         0 Kb)
05ac SmcGui.exe           0 (         0 Kb)

Terminal Server Memory Usage By Session:

Session Paged Pool Maximum is 32768K
Session View Space Maximum is 20480K

Session ID 0 @ f79b7000:
Paged Pool Usage:        3488K
Commit Usage:            8264K

Session ID c @ b691b000:
Paged Pool Usage:        6632K
Commit Usage:            6996K

Session ID 13 @ b68f9000:
Paged Pool Usage:        5020K

*** 10 Pool Allocation Failures ***

Commit Usage:            5440K

Session ID 19 @ b6837000:
Paged Pool Usage:        6168K
Commit Usage:            6536K

I removed some output that is not needed now. The session #13 has the problem. Use TSAdmin to find the user name based on session number. Be careful, the session number is hexdecimal in WinDbg and decimal in TSAdmin. So, the session number for TSAdmin is 19.

Let’s find what is the pool that has a problem. Get the addresses for MM Failure symbols.

lkd> x nt!mm*failure*
808a6a60 nt!MmPoolFailures = <no type information>
808a6a20 nt!MmPoolFailureReasons = <no type information>
808a6ca0 nt!MmPteFailures = <no type information>
808a5c20 nt!MmSessionFailureCauses = <no type information>

The PoolFailureReasons is the one we need. The structure is

typedef enum _MM_POOL_FAILURE_REASONS {
MmNonPagedNoPtes,
MmPriorityTooLow,
MmNonPagedNoPagesAvailable,
MmPagedNoPtes,
MmSessionPagedNoPtes,
MmPagedNoPagesAvailable,
MmSessionPagedNoPagesAvailable,
MmPagedNoCommit,
MmSessionPagedNoCommit,
MmNonPagedNoResidentAvailable,
MmNonPagedNoCommit,
MmMaximumFailureReason
} MM_POOL_FAILURE_REASONS;

It contains 12 items, so we need to get 12-byte (0xC) memory dump starting with the 808a6a20

lkd> dc 808a6a20 lc
808a6a20  00000000 00000000 00000000 00000000  …………….
808a6a30  0000000a 00000000 00000000 00000000  …………….
808a6a40  00000000 00000000 00000000 00000000  …………….

The non-zero value is located in slot 4 (starting with 0, of course). This is same 10 (0xA) failed pool allocations

The fourth slot in the _MM_POOL_FAILURE_REASONS structure is MmSessionPagedNoPtes. It is not very difficult to guess that there are no enough PTEs for the session.

The http://support.microsoft.com/kb/840342 link has the information related to the session memory allocated to GUI object. However, it also mentions about the SessionPoolSize parameter. The long story short: you need to increase two values in the registry. Actually you need to create these values, because they do not exist in the registry by default. The values are:

Key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\MemoryManagement\
Value Name: SessionPoolSize
Value Type: DWORD
Value: 64

Key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\MemoryManagement\
Value Name: SessionViewSize
Value Type: DWORD
Value: 48

Note: The default values for SessionViewSize and SessionPoolSize are 20 (megabytes) and 16 (megabytes) respectively

Note: Usually the SessionViewSize value is greater to the SessionPoolSize value. There is no dependency between theses two values.

Advertisements

2 Responses to Debugging the Memory Allocation on Terminal Server

  1. secure payment systems…

    […]Debugging the Memory Allocation on Terminal Server « Notes of Windows Admin[…]…

  2. personal online payments…

    […]Debugging the Memory Allocation on Terminal Server « Notes of Windows Admin[…]…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: