I notice the source code says it supports 8 MB video RAM but the Bahamas64 BIOS is buggy. I noticed that this site: http://www.vgamuseum.info/index.php/com ... vision-864 has a real S3 864 BIOS. I am going to try if this one works correctly. Maybe the Bahamas64 has proprietary differences from the stock 864 (much like the 9fx Vision has differences from the stock/Phoenix 764), hence why it's not working properly with PCem's emulated card.
Edit: Tested it. Basically, VESA stuff now seems to work with 8 MB memory too on PCEM rev. 188 (modified by me and SA1988 to allow choosing 8 MB for the Bahamas64), and 8 MB is correctly returned, and the Windows 3.1 driver seems to recognize that too. In fact, it can load both 1024x768x32bpp and even 1280x1024x32bpp but they render all screwed up. 800x600x32bpp and 640x480x32 bpp render fine but the hardware cursor is a black square with the Bahamas64 BIOS, and a black square with a weird pattern with the Phoenix S3 864 BIOS. Evidently something isn't implemented correctly.
I also tried a Diamond Stealth64 864 BIOS I found somewhere but it seems to be limited to 2 MB memory.
Edit #2: Also, according to SA1988, some Windows version (was it a NT?) reported that it could not switch to 1024x768x32bpp because there 43 Hz refresh rate was not present.
S3 Vision864
Re: S3 Vision864
well 43hz is required for the interlaced 1024x768x32bpp for the S3 Trio32/64 and S3 Vision864.