That's not a bad idea, I had plans for this and there's a bit of code in there for that, but used it. I've mostly been using the tracelog mechanism.
I'm kinda close to finding "the bug" anyway. Or at least I think so.
It's one of these, I suspect RTE is a false positive. that JMP to a4c00118 is wrong, it should be a0 not a4. Of these DIVU is the most likely culprit.
66627-3/a02206d2 DIVU.W D1,D0
66649-3/a0c04a7a DIVU.W #$000a,D5
6507-3/002e007c RTE
6550-3/a4c00118 JMP.L {PDIS:a4c03f12}=(PC+#$3dfa) src:a4c03f12 dst:00f80006
6587-3/a0221166 TST.W $e800(A7)
16271-3/0002289c MOVE.W 0(A0+D0.W),0(A1+D0.W)
The current challenge is that to allow the high byte to work properly...
So two things seem to come up over and over again, DIVU.W has a mismatch in the "N" flag, which is interesting because this is unsigned division, so the N flag should be zeroed, except it does seem that real hardware sets it sometimes, but I'm unsure under what conditions exactly. The other bigger issue is that the high byte of the PC is being set to a0.
To allow for PC=a0xxxxxx, I increased the memory space allowed from 16MB to 4GB, but this is causing the mmu translation cache array to rise to 640MB from 2MB! This is an array of pointers to 512 byte pages that then connect to the MMU translation table and then to Instruction Parameter Caches (i.e. the operands passed to opcodes). Having done this successfully.
I'm looking at another possible way of solving this, by patching just those instructions that support addressing modes off of PC (Pidx + Pdis), which are going to be branch, jmp, jsr, lea, pea, dbra, and possibly a few others. I could alternately find a sparse array C library (can't use C++ here), but it's likely not worth the trouble and the general slowdown for everything else.
DIVU.W is kind of interesting because a few years back when Tom re-released his LisaMandelbrot app, we spent a bunch of time chasing DIVS, which didn't solve anything. Perhaps it was actually DIVU and not DIVS that was broken there... But the main goal of this is to get the LPW linker working properly which currently throws "Bad Intrin patch"...
Another possibility is to give up on libGenerator and switch to another CPU core, but this is a much harder task and requires finding a portable 68000 core that's GPL licensed, and also MMU friendly. This last part shouldn't be too difficult because luckily the 68000 requires memory space for I/O. But the license compatibility is likely an issue. and ofc, switching cores would likely introduce lots of new bugs and incompatibilities.
This:
http://gendev.spritesmind.net/forum/viewtopic.php?f=2&t=3023#p35357 seems very interesting, but skimming the code of Nemesis's exodus emulator, it seems to relies on LLVM which would basically shut out PPC as a platform (unless someone has backported it - maybe darwn ports?, I don't know) and certainly the announcement about the micro/nanocode emulation might be for another emulator core entirely. The license likely is also incompatible, though if it's worthwhile I may choose to change licenses.
UAE's core is certainly an option as it's GPL'ed and likely by now pretty accurate and might bring JIT features with it. I think it's also used for minivMac.