[Bug] IDE HDD stuttering emulation

Discussion of development and patch submission.
User avatar
James-F
Posts: 69
Joined: Tue 30 May, 2017 10:26 am

[Bug] IDE HDD stuttering emulation

Postby James-F » Sun 04 Jun, 2017 1:29 pm

I have noticed that games runs smoothly if the "CPU time" is below 100%, but there are stutters when the emulated HDD is accessed (not the host HDD).
It happens even with old games that don't use a lot of CPU time, but they still stutter.
That does not happen on a real PC even with a slow 20GB IDE HDD from the 90's.
Nor in DOSBox for that matter.

Is there a way to speed up the HDD access emulation because it is a major bottleneck in the emulation, specifically in PCem.
Last edited by James-F on Sun 04 Jun, 2017 1:43 pm, edited 1 time in total.
Battler
Posts: 793
Joined: Sun 06 Jul, 2014 7:05 pm

Re: [Bug] IDE HDD stuttering emulation

Postby Battler » Sun 04 Jun, 2017 1:32 pm

The IDE emulation needs to be moved into its own thread. For me (on a Pentium Dual-Core), hard disk access can make the difference between 100% and 70%. On a better host CPU it's going to be less severe, but still there.
SarahWalker
Site Admin
Posts: 1333
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Postby SarahWalker » Sun 04 Jun, 2017 1:45 pm

As I've told you several times Battler, the hard disc emulation is not the reason for slowdown. The reason for slowdown is in the CPU emulation not handling OS idling code very well. Moving IDE emulation to another thread would accomplish basically nothing.
User avatar
James-F
Posts: 69
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Postby James-F » Sun 04 Jun, 2017 1:51 pm

Can you do something about it Sarah?
The stuttering is really annoying, and prevent enjoying the emulator like a real machine or even DOSBox.
Battler
Posts: 793
Joined: Sun 06 Jul, 2014 7:05 pm

Re: [Bug] IDE HDD stuttering emulation

Postby Battler » Sun 04 Jun, 2017 2:25 pm

- SarahWalker: No, I'm talking about the slowdown that happens when disk I/O access happens. For example, on Pentium 75, I have Duke Nukem 3D (for DOS) running at solid 100% but when on the loading screen, it goes down to 70% for a while as it's reading from the hard disk.
Now if James-F is not talking about that, then his might indeed be the idle issue, though DOSIDLE.EXE crashes right now (I made a bug report thread about that yesterday or maybe on Friday), so IDLE.COM is the only option.
User avatar
James-F
Posts: 69
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Postby James-F » Sun 04 Jun, 2017 2:36 pm

Moreover, if you actually watch the opening demo in DUke3D (hit escape in the main menu) you will see that the demo stutters only what the HDD light blinks.

The loading screen takes forever because there is much HDD activity during this.
So I assume there is stutter each single HDD access.
Last edited by James-F on Sun 04 Jun, 2017 2:42 pm, edited 1 time in total.
SarahWalker
Site Admin
Posts: 1333
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Postby SarahWalker » Sun 04 Jun, 2017 2:38 pm

Battler wrote:- SarahWalker: No, I'm talking about the slowdown that happens when disk I/O access happens. For example, on Pentium 75, I have Duke Nukem 3D (for DOS) running at solid 100% but when on the loading screen, it goes down to 70% for a while as it's reading from the hard disk.

Yes, because DOS is idling waiting for the IDE transfer to complete. It's not the disk IO that causes the slowdown, it's the CPU emulation of the DOS idle loop that causes the slowdown. Pushing IDE emulation to another thread will do nothing to help that.
Battler
Posts: 793
Joined: Sun 06 Jul, 2014 7:05 pm

Re: [Bug] IDE HDD stuttering emulation

Postby Battler » Sun 04 Jun, 2017 2:42 pm

Ah, I didn't know that. Then the performance idle emulation needs to be improved, unfortunately that's going to be difficult.
User avatar
James-F
Posts: 69
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Postby James-F » Sun 04 Jun, 2017 4:29 pm

SarahWalker please try to fix that as it's a critical bug, these "mini-freezes" are too frequent especially with games.
It might appear as slow emulation, but it's not, the emulation is quite fast but with lots of frequent mini-freezes.
User avatar
James-F
Posts: 69
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Postby James-F » Mon 05 Jun, 2017 5:59 pm

I've tested SCSI HDD and the stuttering is greatly reduced, but still ever so slightly noticeable.
Maybe there should be an option to completely remove the HDD transfer speed limitation, just like the Floppy "turbo timings" option?
Battler
Posts: 793
Joined: Sun 06 Jul, 2014 7:05 pm

Re: [Bug] IDE HDD stuttering emulation

Postby Battler » Mon 05 Jun, 2017 7:41 pm

That won't help. Sarah is right, IDE uses PIO while the SCSI controller I emulated uses DMA, so there's considerably less waiting for transfers.
User avatar
James-F
Posts: 69
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Postby James-F » Tue 06 Jun, 2017 4:36 am

I've created an RAM disk using XMSDSK to remove the HDD emulation completely out of the way, and indeed playing games from this ram disk is now as smooth as DOSBox or my hardware Pentium PC.

It is obviously your choice to completely ignore that issue SarahWalker, but I can't stress enough the importance of this issue and I think you should at least acknowledge how crippled the the emulation is with this 'idle loop' bug present.
Last edited by James-F on Tue 06 Jun, 2017 5:18 am, edited 1 time in total.
User avatar
ppgrainbow
Posts: 402
Joined: Thu 04 Sep, 2014 7:03 am
Contact:

Re: [Bug] IDE HDD stuttering emulation

Postby ppgrainbow » Tue 06 Jun, 2017 5:16 am

James-F wrote:I've created an RAM disk using XMSDSK to remove the HDD emulation completely out of the way, and indeed playing games from this ram disk is now as smooth as DOSBox or my hardware Pentium PC.
It is obviously your choice to completely ignore that issue SarahWalker, but I think you should at least acknowledge how crippled the the emulation is with this 'idle loop' bug present.


So, do you have any idea what does the "idle loop" bug have to do with the IDE hard disk emulation?
User avatar
James-F
Posts: 69
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Postby James-F » Tue 06 Jun, 2017 5:22 am

I don't, but Sarah clearly pointed the issue in her last post.
SarahWalker wrote:Yes, because DOS is idling waiting for the IDE transfer to complete. It's not the disk IO that causes the slowdown, it's the CPU emulation of the DOS idle loop that causes the slowdown.

I can only test and measure from the outside, but no one here knows what is going on in the code better than Sarah.
User avatar
ppgrainbow
Posts: 402
Joined: Thu 04 Sep, 2014 7:03 am
Contact:

Re: [Bug] IDE HDD stuttering emulation

Postby ppgrainbow » Tue 06 Jun, 2017 6:36 am

James-F wrote:I don't, but Sarah clearly pointed the issue in her last post.
SarahWalker wrote:Yes, because DOS is idling waiting for the IDE transfer to complete. It's not the disk IO that causes the slowdown, it's the CPU emulation of the DOS idle loop that causes the slowdown.

I can only test and measure from the outside, but no one here knows what is going on in the code better than Sarah.

I understand what you mean. It's difficult to try to pin down what might be causing the IDE HDD stuttering issue.
Battler
Posts: 793
Joined: Sun 06 Jul, 2014 7:05 pm

Re: [Bug] IDE HDD stuttering emulation

Postby Battler » Tue 06 Jun, 2017 1:58 pm

There's also the fact IDE hard disks generally read one sector a time, though PCem emulates multiple reads but still does a fseeko64/fread on every sector rather than on the whole batch, on the other hand, SCSI hard disks generally read entire blocks of sectors at a time, and the way I wrote the SCSI disk emulation, one single fseeko64/fread is done for the entire batch of sectors requested.
I do however wonder why none of the BIOS'es enable DMA by default, and instead stick to PIO mode 2.
SarahWalker
Site Admin
Posts: 1333
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Postby SarahWalker » Tue 06 Jun, 2017 4:40 pm

James-F wrote:It is obviously your choice to completely ignore that issue SarahWalker, but I can't stress enough the importance of this issue and I think you should at least acknowledge how crippled the the emulation is with this 'idle loop' bug present.

I'm not ignoring the issue, it's just not as easy to fix as you seem to think it is.
User avatar
James-F
Posts: 69
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Postby James-F » Tue 06 Jun, 2017 5:50 pm

I understand.
I am positively sure that you understand that an issue that results in an emulation hindering on this scale outweighs any kind of cosmetic fix.
User avatar
James-F
Posts: 69
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Postby James-F » Sat 17 Jun, 2017 4:31 am

bump

I should say again that this bug doe not happen on a created ram drive or scsi drive.
Maybe it's no a problem of the CPU idle loop after all?

If you can create a virtual IDE disk activity and check what the emulation does when reading from IDE disk.
Please have a look at this, it is real a pain to use PCem for gaming when it stutters like this.
SarahWalker
Site Admin
Posts: 1333
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Postby SarahWalker » Sat 17 Jun, 2017 6:20 am

DOS won't idle on a RAM disk as there's no need to wait for anything.

I have been looking at this, I know what the problems are, I don't have an easy fix. I'm sorry, but this isn't going to be fixed overnight.
SarahWalker
Site Admin
Posts: 1333
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Postby SarahWalker » Sat 17 Jun, 2017 7:19 am

Rev 770 now reads multiple sectors in one go instead of one by one. This may help if your host machine's disc IO system is really bad, otherwise it probably won't gain much.
User avatar
James-F
Posts: 69
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Postby James-F » Wed 02 Aug, 2017 2:41 pm

Any news on this Sarah?
Just a reminder that games are annoying and unplayable with this stuttering.
SarahWalker
Site Admin
Posts: 1333
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Postby SarahWalker » Wed 02 Aug, 2017 4:41 pm

I have been looking at this, I know what the problems are, I don't have an easy fix. I'm sorry, but this isn't going to be fixed overnight.



You could try reducing the emulated CPU as a temporary workaround.
JosepMa
Posts: 70
Joined: Tue 20 Jun, 2017 6:25 pm

Re: [Bug] IDE HDD stuttering emulation

Postby JosepMa » Wed 02 Aug, 2017 7:27 pm

The thing that I don't understand is why does disk I/O affect the emulated speed.
Concretely, the emulated CPU is halted while I/O operations are in place. Once, installing software from a Host's CDROM drive, 10 minutes passed on the Host side while only 2 minutes passed on the emulated side. If it was emulating the I/O wait, the CPU should still advance. (The CPU on the Host OS is not being used, so i don't think it has to do with ide performance, but rather waits and locks).
SarahWalker
Site Admin
Posts: 1333
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Postby SarahWalker » Wed 02 Aug, 2017 7:35 pm

Disk I/O is not multithreaded, hence the emulated CPU effectively halts. This may be worth doing for CD emulation - the current system means you get the levels of slowdown you saw when using a real CD drive.

I don't think it's worthwhile for HDD emulation - unless your system is really really bad, the amount of HDD I/O should not cause noticeable slowdown. As I said earlier in the thread, I don't think the slowdown that's been reported here really has anything to do with I/O, unless James-F is trying to run everything off a USB stick or a Zip drive or something.
JosepMa
Posts: 70
Joined: Tue 20 Jun, 2017 6:25 pm

Re: [Bug] IDE HDD stuttering emulation

Postby JosepMa » Wed 02 Aug, 2017 9:53 pm

I also don't think that the Host's I/O plays any role in this problem (Maybe in the Hosts' CDROM case but not on HDD image). The problem is on the emulated side.
I don't know if you have experienced it, but audio stuttering on Windows 98 startup sound is the norm here, and that's with an i7 6700 @3.4Ghz emulating a 166MMX. HDD image is 8GB stored on a 7200RPM HDD (i could try on an SSD too to completely discard host I/O performance).

Ok, I have to admit that I don't really know if such a real machine would have problems or not in that scenario. I think I mostly used Win98 with UDMA disks (and a Pentium II ), not PIO-only.
User avatar
James-F
Posts: 69
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Postby James-F » Thu 03 Aug, 2017 4:14 pm

The problem is obviously on PCem emulation side, because I compare with real P233MXX ,P100 and DOSBox all of them have no stuttering at all when accessing the HDD.
The CPU emulation in PCem is fast and working properly as far as I can tell, but I experience freezes each time the emulated HDD is accessed, it is in accordance with the emulated HDD led indicator.
This is the reasons why Windows 98 or any other installation that copies (or reads) a lot of files to the HDD is painfully slow in PCem, nothing remotely close to the real machine or even DOSBox.

Emulated SCSI HDD work much faster than IDE, and RAM drive is practically as fast as DOSBox or the real machine.
What is the difference between RAM drive and IDE drive in relation to the emulated CPU? It should tread both the same.
Where "cpu idle loops" come into play when a RAM drive is instantaneous in PCem?

What I don't understand is if the IDE I/O bus is emulated, why not just make it fast or completely remove it's limitations?
At least look at what actually causes the micro freezes in software when the IDE HDD is accessed, it might be completely different to what you assume Sarah.
SarahWalker
Site Admin
Posts: 1333
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Postby SarahWalker » Thu 03 Aug, 2017 5:08 pm

What is the difference between RAM drive and IDE drive in relation to the emulated CPU? It should tread both the same.
Where "cpu idle loops" come into play when a RAM drive is instantaneous in PCem?

DOS/whatever guest OS you're using doesn't have to wait for the RAM drive to respond. It does have to wait for an IDE drive.

At least look at what actually causes the micro freezes in software when the IDE HDD is accessed, it might be completely different to what you assume Sarah.

Why do you assume I haven't looked? I HAVE looked, I DO know for the most part what the issues are, I DO know that they are going to be difficult to fix. I'm not currently fixing it as I don't have any performance work planned for v13 - I do have a significant rework planned for v14, but that's not going to happen for a while, because at the moment I simply do not have the time or energy to do it.

Return to “Development”

Who is online

Users browsing this forum: No registered users and 1 guest