[Bug] IDE HDD stuttering emulation

Discussion of development and patch submission.
User avatar
James-F
Posts: 88
Joined: Tue 30 May, 2017 10:26 am

[Bug] IDE HDD stuttering emulation

Post by James-F »

I have noticed that games runs smoothly if the "CPU time" is below 100%, but there are stutters when the emulated HDD is accessed (not the host HDD).
It happens even with old games that don't use a lot of CPU time, but they still stutter.
That does not happen on a real PC even with a slow 20GB IDE HDD from the 90's.
Nor in DOSBox for that matter.

Is there a way to speed up the HDD access emulation because it is a major bottleneck in the emulation, specifically in PCem.
Last edited by James-F on Sun 04 Jun, 2017 1:43 pm, edited 1 time in total.
Battler
Posts: 793
Joined: Sun 06 Jul, 2014 7:05 pm

Re: [Bug] IDE HDD stuttering emulation

Post by Battler »

The IDE emulation needs to be moved into its own thread. For me (on a Pentium Dual-Core), hard disk access can make the difference between 100% and 70%. On a better host CPU it's going to be less severe, but still there.
User avatar
SarahWalker
Site Admin
Posts: 2054
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Post by SarahWalker »

As I've told you several times Battler, the hard disc emulation is not the reason for slowdown. The reason for slowdown is in the CPU emulation not handling OS idling code very well. Moving IDE emulation to another thread would accomplish basically nothing.
User avatar
James-F
Posts: 88
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Post by James-F »

Can you do something about it Sarah?
The stuttering is really annoying, and prevent enjoying the emulator like a real machine or even DOSBox.
Battler
Posts: 793
Joined: Sun 06 Jul, 2014 7:05 pm

Re: [Bug] IDE HDD stuttering emulation

Post by Battler »

- SarahWalker: No, I'm talking about the slowdown that happens when disk I/O access happens. For example, on Pentium 75, I have Duke Nukem 3D (for DOS) running at solid 100% but when on the loading screen, it goes down to 70% for a while as it's reading from the hard disk.
Now if James-F is not talking about that, then his might indeed be the idle issue, though DOSIDLE.EXE crashes right now (I made a bug report thread about that yesterday or maybe on Friday), so IDLE.COM is the only option.
User avatar
James-F
Posts: 88
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Post by James-F »

Moreover, if you actually watch the opening demo in DUke3D (hit escape in the main menu) you will see that the demo stutters only what the HDD light blinks.

The loading screen takes forever because there is much HDD activity during this.
So I assume there is stutter each single HDD access.
Last edited by James-F on Sun 04 Jun, 2017 2:42 pm, edited 1 time in total.
User avatar
SarahWalker
Site Admin
Posts: 2054
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Post by SarahWalker »

Battler wrote:- SarahWalker: No, I'm talking about the slowdown that happens when disk I/O access happens. For example, on Pentium 75, I have Duke Nukem 3D (for DOS) running at solid 100% but when on the loading screen, it goes down to 70% for a while as it's reading from the hard disk.
Yes, because DOS is idling waiting for the IDE transfer to complete. It's not the disk IO that causes the slowdown, it's the CPU emulation of the DOS idle loop that causes the slowdown. Pushing IDE emulation to another thread will do nothing to help that.
Battler
Posts: 793
Joined: Sun 06 Jul, 2014 7:05 pm

Re: [Bug] IDE HDD stuttering emulation

Post by Battler »

Ah, I didn't know that. Then the performance idle emulation needs to be improved, unfortunately that's going to be difficult.
User avatar
James-F
Posts: 88
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Post by James-F »

SarahWalker please try to fix that as it's a critical bug, these "mini-freezes" are too frequent especially with games.
It might appear as slow emulation, but it's not, the emulation is quite fast but with lots of frequent mini-freezes.
User avatar
James-F
Posts: 88
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Post by James-F »

I've tested SCSI HDD and the stuttering is greatly reduced, but still ever so slightly noticeable.
Maybe there should be an option to completely remove the HDD transfer speed limitation, just like the Floppy "turbo timings" option?
Battler
Posts: 793
Joined: Sun 06 Jul, 2014 7:05 pm

Re: [Bug] IDE HDD stuttering emulation

Post by Battler »

That won't help. Sarah is right, IDE uses PIO while the SCSI controller I emulated uses DMA, so there's considerably less waiting for transfers.
User avatar
James-F
Posts: 88
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Post by James-F »

I've created an RAM disk using XMSDSK to remove the HDD emulation completely out of the way, and indeed playing games from this ram disk is now as smooth as DOSBox or my hardware Pentium PC.

It is obviously your choice to completely ignore that issue SarahWalker, but I can't stress enough the importance of this issue and I think you should at least acknowledge how crippled the the emulation is with this 'idle loop' bug present.
Last edited by James-F on Tue 06 Jun, 2017 5:18 am, edited 1 time in total.
User avatar
ppgrainbow
Posts: 479
Joined: Thu 04 Sep, 2014 7:03 am
Contact:

Re: [Bug] IDE HDD stuttering emulation

Post by ppgrainbow »

James-F wrote:I've created an RAM disk using XMSDSK to remove the HDD emulation completely out of the way, and indeed playing games from this ram disk is now as smooth as DOSBox or my hardware Pentium PC.
It is obviously your choice to completely ignore that issue SarahWalker, but I think you should at least acknowledge how crippled the the emulation is with this 'idle loop' bug present.
So, do you have any idea what does the "idle loop" bug have to do with the IDE hard disk emulation?
User avatar
James-F
Posts: 88
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Post by James-F »

I don't, but Sarah clearly pointed the issue in her last post.
SarahWalker wrote: Yes, because DOS is idling waiting for the IDE transfer to complete. It's not the disk IO that causes the slowdown, it's the CPU emulation of the DOS idle loop that causes the slowdown.
I can only test and measure from the outside, but no one here knows what is going on in the code better than Sarah.
User avatar
ppgrainbow
Posts: 479
Joined: Thu 04 Sep, 2014 7:03 am
Contact:

Re: [Bug] IDE HDD stuttering emulation

Post by ppgrainbow »

James-F wrote:I don't, but Sarah clearly pointed the issue in her last post.
SarahWalker wrote: Yes, because DOS is idling waiting for the IDE transfer to complete. It's not the disk IO that causes the slowdown, it's the CPU emulation of the DOS idle loop that causes the slowdown.
I can only test and measure from the outside, but no one here knows what is going on in the code better than Sarah.
I understand what you mean. It's difficult to try to pin down what might be causing the IDE HDD stuttering issue.
Battler
Posts: 793
Joined: Sun 06 Jul, 2014 7:05 pm

Re: [Bug] IDE HDD stuttering emulation

Post by Battler »

There's also the fact IDE hard disks generally read one sector a time, though PCem emulates multiple reads but still does a fseeko64/fread on every sector rather than on the whole batch, on the other hand, SCSI hard disks generally read entire blocks of sectors at a time, and the way I wrote the SCSI disk emulation, one single fseeko64/fread is done for the entire batch of sectors requested.
I do however wonder why none of the BIOS'es enable DMA by default, and instead stick to PIO mode 2.
User avatar
SarahWalker
Site Admin
Posts: 2054
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Post by SarahWalker »

James-F wrote:It is obviously your choice to completely ignore that issue SarahWalker, but I can't stress enough the importance of this issue and I think you should at least acknowledge how crippled the the emulation is with this 'idle loop' bug present.
I'm not ignoring the issue, it's just not as easy to fix as you seem to think it is.
User avatar
James-F
Posts: 88
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Post by James-F »

I understand.
I am positively sure that you understand that an issue that results in an emulation hindering on this scale outweighs any kind of cosmetic fix.
User avatar
James-F
Posts: 88
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Post by James-F »

bump

I should say again that this bug doe not happen on a created ram drive or scsi drive.
Maybe it's no a problem of the CPU idle loop after all?

If you can create a virtual IDE disk activity and check what the emulation does when reading from IDE disk.
Please have a look at this, it is real a pain to use PCem for gaming when it stutters like this.
User avatar
SarahWalker
Site Admin
Posts: 2054
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Post by SarahWalker »

DOS won't idle on a RAM disk as there's no need to wait for anything.

I have been looking at this, I know what the problems are, I don't have an easy fix. I'm sorry, but this isn't going to be fixed overnight.
User avatar
SarahWalker
Site Admin
Posts: 2054
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Post by SarahWalker »

Rev 770 now reads multiple sectors in one go instead of one by one. This may help if your host machine's disc IO system is really bad, otherwise it probably won't gain much.
User avatar
James-F
Posts: 88
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Post by James-F »

Any news on this Sarah?
Just a reminder that games are annoying and unplayable with this stuttering.
User avatar
SarahWalker
Site Admin
Posts: 2054
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Post by SarahWalker »

I have been looking at this, I know what the problems are, I don't have an easy fix. I'm sorry, but this isn't going to be fixed overnight.

You could try reducing the emulated CPU as a temporary workaround.
JosepMa
Posts: 202
Joined: Tue 20 Jun, 2017 6:25 pm

Re: [Bug] IDE HDD stuttering emulation

Post by JosepMa »

The thing that I don't understand is why does disk I/O affect the emulated speed.
Concretely, the emulated CPU is halted while I/O operations are in place. Once, installing software from a Host's CDROM drive, 10 minutes passed on the Host side while only 2 minutes passed on the emulated side. If it was emulating the I/O wait, the CPU should still advance. (The CPU on the Host OS is not being used, so i don't think it has to do with ide performance, but rather waits and locks).
User avatar
SarahWalker
Site Admin
Posts: 2054
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Post by SarahWalker »

Disk I/O is not multithreaded, hence the emulated CPU effectively halts. This may be worth doing for CD emulation - the current system means you get the levels of slowdown you saw when using a real CD drive.

I don't think it's worthwhile for HDD emulation - unless your system is really really bad, the amount of HDD I/O should not cause noticeable slowdown. As I said earlier in the thread, I don't think the slowdown that's been reported here really has anything to do with I/O, unless James-F is trying to run everything off a USB stick or a Zip drive or something.
JosepMa
Posts: 202
Joined: Tue 20 Jun, 2017 6:25 pm

Re: [Bug] IDE HDD stuttering emulation

Post by JosepMa »

I also don't think that the Host's I/O plays any role in this problem (Maybe in the Hosts' CDROM case but not on HDD image). The problem is on the emulated side.
I don't know if you have experienced it, but audio stuttering on Windows 98 startup sound is the norm here, and that's with an i7 6700 @3.4Ghz emulating a 166MMX. HDD image is 8GB stored on a 7200RPM HDD (i could try on an SSD too to completely discard host I/O performance).

Ok, I have to admit that I don't really know if such a real machine would have problems or not in that scenario. I think I mostly used Win98 with UDMA disks (and a Pentium II ), not PIO-only.
User avatar
James-F
Posts: 88
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Post by James-F »

The problem is obviously on PCem emulation side, because I compare with real P233MXX ,P100 and DOSBox all of them have no stuttering at all when accessing the HDD.
The CPU emulation in PCem is fast and working properly as far as I can tell, but I experience freezes each time the emulated HDD is accessed, it is in accordance with the emulated HDD led indicator.
This is the reasons why Windows 98 or any other installation that copies (or reads) a lot of files to the HDD is painfully slow in PCem, nothing remotely close to the real machine or even DOSBox.

Emulated SCSI HDD work much faster than IDE, and RAM drive is practically as fast as DOSBox or the real machine.
What is the difference between RAM drive and IDE drive in relation to the emulated CPU? It should tread both the same.
Where "cpu idle loops" come into play when a RAM drive is instantaneous in PCem?

What I don't understand is if the IDE I/O bus is emulated, why not just make it fast or completely remove it's limitations?
At least look at what actually causes the micro freezes in software when the IDE HDD is accessed, it might be completely different to what you assume Sarah.
User avatar
SarahWalker
Site Admin
Posts: 2054
Joined: Thu 24 Apr, 2014 4:18 pm

Re: [Bug] IDE HDD stuttering emulation

Post by SarahWalker »

What is the difference between RAM drive and IDE drive in relation to the emulated CPU? It should tread both the same.
Where "cpu idle loops" come into play when a RAM drive is instantaneous in PCem?
DOS/whatever guest OS you're using doesn't have to wait for the RAM drive to respond. It does have to wait for an IDE drive.
At least look at what actually causes the micro freezes in software when the IDE HDD is accessed, it might be completely different to what you assume Sarah.
Why do you assume I haven't looked? I HAVE looked, I DO know for the most part what the issues are, I DO know that they are going to be difficult to fix. I'm not currently fixing it as I don't have any performance work planned for v13 - I do have a significant rework planned for v14, but that's not going to happen for a while, because at the moment I simply do not have the time or energy to do it.
User avatar
James-F
Posts: 88
Joined: Tue 30 May, 2017 10:26 am

Re: [Bug] IDE HDD stuttering emulation

Post by James-F »

Hi.

It's been a long time since I've posted or tried PCem, and now I'm running v15. All is superb except this issue.
It seems like the IDE stuttering issue is still present and it's a really big annoyance for gaming with PCem,, all games have stuttering every time the IDE HDD is accessed.
I've probably said that already some years ago, but the IDE emulation is a huge bottleneck for PCem.
I'm still hopeful to see some hack/fix for this after years.
User avatar
gen_angry
Posts: 127
Joined: Sat 25 Feb, 2017 6:48 am

Re: [Bug] IDE HDD stuttering emulation

Post by gen_angry »

James-F wrote: Wed 07 Aug, 2019 8:38 am Hi.

It's been a long time since I've posted or tried PCem, and now I'm running v15. All is superb except this issue.
It seems like the IDE stuttering issue is still present and it's a really big annoyance for gaming with PCem,, all games have stuttering every time the IDE HDD is accessed.
I've probably said that already some years ago, but the IDE emulation is a huge bottleneck for PCem.
I'm still hopeful to see some hack/fix for this after years.
What settings are you using or is there another way to reproduce the issue?

I just tried Duke 3D with a P75, Shuttle HOT-557 board. 32MB RAM, ATi Mach64, SB16, 2GB IDE HD + 8x CDROM. Disk image is on a 7200RPM drive. It was pegged at 100% speed for me the entire time, loading, game demo, playing, etc. Not a single stutter and was very quick.

Tried it on a 486/100 + SIS 496/497 board (same other settings as above), same result.
Post Reply