Resolving VMware Workstation 10.0.6 crash

Two days ago VMware published new update for Workstation version 10 (release notes). According to the changelog it should fix some security issues reported in OpenSSL. That’s nice, however there is a small problem with this update. VMware.exe (the GUI part of VMmare) started crashing immediately after update. This was quite a learning experience, do not update critical software if you have something important to do, as the new version can be worse than the one you are using. Since I didn’t have the previous installer at hand, I had to somehow resolve this issue differently (yeah sure, I just wanted to debug it and see, why it is crashing).

The crash on my machine is 100% repeatable, and it crashes on security cookie check inside libcurl.dll. As most of you know libcurl is an open source library, so the bug should be hidden somewhere inside libcurl source code, or in the additional/changed code that guys from VMware possibly touched. In the latter case, there should be available modified source code of the libcurl on the VMware page. It is not the case this time, so (in theory) it is fault of the libcurl itself. Crash happens inside the function that seems to enumerate DNS servers. Libcurl can be compiled with many different options and can use many different third party libraries. VMware’s libcurl version is compiled with static version of zlib and c-ares, it is also dynamically linked with OpenSSL. It is worth to note which versions of mentioned libraries are used. Most of open source libraries includes version number inside compiled binaries so it is rather easy to gather those information:

VMWare 10.0.5 VMWare 10.0.6 Current version
libcurl 7.19.5 7.24.0 7.42.1
zlib 1.2.3 1.2.3 1.2.8
c-ares 1.5.1 1.7.5 1.10.0

Particulary interesting for me was libcurl and c-ares, as zlib probably doesn’t enumerate DNSes :) I’ve tried to localize the crashing function inside the source code of mentioned libs, but I was missing one detail (obviously!). In the newest version of c-ares, the function similar to the one that is crashing VMware is named get_DNS_NetworkParams(). This function doesn’t crash and it looks fine. After a few minutes of looking at the code and wondering what’s wrong I realized that I’ve freshly cloned repository with the latest changes and I should probably go back in time and check some older version. Fortunately c-ares repository is properly tagged with release version numbers (at least since version 1.7.0). After checking out proper revision I’ve easily identified crashing function: get_iphlpapi_dns_info(). Simplified code, just to show the problem:

#define INET_ADDRSTRLEN  22
 
int get_iphlpapi_dns_info(char *ret_buf, size_t ret_size)
{
	const size_t  ipv4_size = INET_ADDRSTRLEN + 1;  /* +1 for ',' at end */
	size_t        left = ret_size;
	char         *ret = ret_buf;
	//...
	for (/*...*/)
	{
		//...
		if (left > ipv4_size)
		{
			some_function_that_fills_given_buffer(ret, ipv4_size - 1);
			size_t stringlen = strlen(ret);
			ret[stringlen] = ',';
			ret[stringlen + 1] = '\0';
			ret += stringlen + 1;
			left -= ret - ret_buf;	// <- !!! BUG BUG !!!
		}
		//...
	}
	//...
}
 
//...
	// function call
	char buf[512];
	get_iphlpapi_dns_info(buf,sizeof(buf));
//...

Variable left should be updated on each iteration with the length of the written data, so it will always contain number of bytes left in the ret_buf. As you may already noticed, ret_buf doesn’t change throughout the function, so left is updated with the length of all strings written since the first iteration. It goes below 0 very fast and since it is declared as size_t which is unsigned it will be interpreted as such in the if (left > ipv4_size) comparison and at the end it will overrun ret_buf (only if there is enough records to process by the for loop).

Described bug was present in c-ares library since the very begining (2004-06-10). There were some changes regarding IPv6 (2011-05-17) that were modifying this function, but bug persisted probably due to copy&paste from IPv4 handler. Someone fixed this bug almost a year later (2012-02-25) and the other person rewrote whole code few months later. Unfortunately VMware chose to use version 1.7.5 which was released before this bug was fixed (2011-08-16). Blast from the past.

If you are experiencing crashes in VMware Workstation 10.0.6 just use libcurl.dll from the previous build, at least it will stop crashing.

UPDATE: Someone at VMware forum pointed out that it is sufficient to get libcurl.dll from the \OVFTool\ directory and it will solve the issue. Indeed it is true, as libcurl from mentioned directory uses c-ares version 1.9.1 (libcurl itself is also newer as it has version 7.30.0).

UPDATE 2 (2015-07-10): This issue was fixed with v10.0.7, libcurl was updated to v7.32.0. I’ve also received free copy of VMware Workstation v11. Thanks!:vmware_gift

Comments (31)

  1. 23:53, May 8, 2015Sujit  / Reply

    Hi,
    I’m from the VMware WS team. We’d be happy to take a look at this issue. Could you please provide us more info such as the Windows host version? Could you possibly be able to provide us the Workstation logs, crash dumps etc?
    Thank you.

    • 09:21, May 9, 2015ReWolf  / Reply

      I’ve sent You an e-mail with more details. Thanks for your interest.

  2. 03:06, May 9, 2015FultonTech  / Reply

    Thank you!

  3. 16:32, May 9, 2015byrobin  / Reply

    awesome article~~~~thx for share it~~~

  4. 08:34, May 15, 2015Wijnand  / Reply

    Thanks a million. That did the trick.

  5. 22:19, May 15, 2015platzh1rsch  / Reply

    You just saved my day, thx a lot!

  6. 17:48, May 18, 2015Burak Say  / Reply

    So for windows users, this translates as:
    Copy libcurl.dll from C:\Program Files (x86)\VMware\VMware Workstation\OVFTool\ into C:\Program Files (x86)\VMware\VMware Workstation\

    Thanks for the post!

    • 17:14, June 10, 2015Rand  / Reply

      @Burak Say
      Thanks for the non-programmer translation on where those files are – and thanks to the OP for doing all the hard work!

  7. 14:08, May 20, 2015twenty1forty5  / Reply

    Thanks a lot for solving this problem for me!
    Also thanks Google for pointing me to your blog post, as this issue as of today (of course?) is unknown in the VMware KB!

  8. 20:03, May 20, 2015Dan  / Reply

    Thanks for the great article, fixed my problem right away. Well done!

  9. 14:41, May 22, 2015Andy  / Reply

    Great article, corrected my issue that started 2 days ago where VMWare Workstation will crash after computer reboot. DLL copy corrected it.
    Thanks.

  10. 14:38, May 23, 2015YouJustSavedMyLife  / Reply

    Thanks for the article. I’m literally onsite and had this issue. Copied the file. Fixed. Awesome work.

  11. 06:56, May 24, 2015Shige_z  / Reply

    Thanks for your nice article! I’m troubled with VMware workstation 10.0.6 for a few days.
    I Copied libcurl.dll from \OVFTool\ to \VMware\, then this crash was not occurred.

    My event viewer noticed these bellow log when vmware workstation crashed.

    vmware.exe
    10.0.6.60484
    553febfb
    libcurl.dll
    0.0.0.0
    4fb54a82
    c0000409
    0004243b
    1804
    01d095d98cc4ec7c
    G:\vmware\vmware.exe
    G:\vmware\libcurl.dll

  12. 01:03, May 25, 2015Gav  / Reply

    Thanks!
    Saved me hours!

  13. 06:22, May 29, 2015Bhanu  / Reply

    That helped me too….not sure why VMware would push something that does not work.

  14. 12:08, June 3, 2015Luci  / Reply

    Had the same issue today. This article was really helpful, it saved me a lot of time. Thanks!

  15. 22:10, June 15, 2015uday  / Reply

    Thanks much. Copied libcurl and it worked . saved me from reinstall and what not , lot of troubles

  16. 16:03, June 16, 2015Adam  / Reply

    Thank you for this post. Would be nice if VMWare were have this fixed by now. May 8 they reply saying they will fix. June 16 still not fixed and pushing out the update.

  17. 21:39, June 16, 2015Nico  / Reply

    Thank you! Great information….;-)
    I like it ….

    Greatings Nico

  18. 14:37, June 18, 2015Dan  / Reply

    Thank you. This worked for me this morning.

  19. 21:50, June 19, 2015Vlad  / Reply

    Thank you!!!

  20. 22:38, June 19, 2015J  / Reply

    Perfect, thank you!!!

  21. 22:48, June 23, 2015john  / Reply

    Thanks for this, it solved my problem.

  22. 09:41, June 25, 2015Henrik  / Reply

    Thank you!

  23. 02:55, June 27, 2015Curtis  / Reply

    Thanks for this, it did help me a few months back.

    Today, I experienced an extended version of this due to an extended power outage which dropped the machine. After going through all the motions of re-re-re-replacing libcurl even though it worked in the past, I was getting the same error.

    Turns out I hadn’t noticed that apparently CMOS battery had died and my clock had reset to some default 2099 date of the bios, and I was getting the exact same dialog indicating the BEX error. I’m assuming the underlying crypto requires accurate time, this was choking something up, but once I fixed the time the BEX fault went away.

    I just wanted to make sure this was logged out in the ether somewhere in case anyone else encounters this, comes here and finds your great analysis, and remains “stuck” after trying it like I was.

  24. 18:04, June 30, 2015Philippe Sainte-Marie  / Reply

    Fixed my issue right away. Thanks!

  25. 21:03, July 15, 2015o4  / Reply

    the libcurl.dll copy did the trick!!! well done vmware WS

  26. 12:42, August 14, 2015leon  / Reply

    Having an issue with WS 11, before reboot when moved out of a VM it was no longer possible to focus the window of workstation, had to close it via taskmanager, then restart vmware workstation but then that happend again.

    After a reboot, the vmware.exe is starting, but there’s no window opening at all, the taskmanger shows vmware.exe is running.

    Using v11 on Windows 10 Pro (release).

    • 12:09, August 15, 2015ReWolf  / Reply

      This is rather unrelated topic, so I suggest asking on official VMware forum.

  27. 19:37, February 22, 2016Meng  / Reply

    Thank you so much, copy and replace the libcurl.dll from the program files(x86) to VMWare Workstation works!

  28. 17:29, November 9, 2016Daryl Hernandez  / Reply

    Freaking Awesome! This fixed my issue! Thank you very much!

Leave a Reply

Allowed Tags - You may use these HTML tags and attributes in your comment.

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>