Vmware 3.1.3 modules fail with Linux 2.6.36
Jeff Beard-Shouse — Tue, 12/07/2010 - 16:32
So I was upgrading the kernel on my wife's laptop (its a long story) and I had an issue with vmplayer. The issue was it would not compile the vmware kernel modules. For the record the laptop has KUbuntu 10.04, I had just upgraded the kernel to 2.6.36-1, and I had grabbed the latest vmplayer (3.1.3 at the time). The error message was as follows:
make: Entering directory `/usr/src/linux-headers-2.6.36-1-generic'
CC [M] /tmp/vmware-root/modules/vmmon-only/linux/driver.o
/tmp/vmware-root/modules/vmmon-only/linux/driver.c: In function `init_module':
/tmp/vmware-root/modules/vmmon-only/linux/driver.c:425: error: `struct file_operations' has no member named `ioctl'
make: *** [/tmp/vmware-root/modules/vmmon-only/linux/driver.o] Error 1
make: *** [_module_/tmp/vmware-root/modules/vmmon-only] Error 2
make: Leaving directory `/usr/src/linux-headers-2.6.36-1-generic' make: *** [vmmon.ko] Error 2
make: Leaving directory `/tmp/vmware-root/modules/vmmon-only'
As it turns out this is because vmplayer apparently does not support 2.6.36 yet. According to a Gentoo bug report I found "With the 2.6.36 release it has been removed the ioctl() method from the file_operations structure. All the drivers should now call unlocked_ioctl() which doesn't run under the Big Kernel Lock (BKL)." So I applied the patch as root:
tar xf /usr/lib/vmware/modules/source/vmmon.tar
patch -p0 < /root/18.104.22.168-ioctl.patch
cf /usr/lib/vmware/modules/source/vmmon.tar vmmon-only/
After which vmplayer compiled the modules just fine. Thank you Gentoo community.
Gentoo bug report and patches can be found here.
I hope this will help someone else.