Dear Don,

you know I am working on a modified MMIX simulator,
based on your mmix-sim.w, teaching it new tricks
like dynamic TRAPS and hardware access through a 
simulated bus. In this context, I have implemented
RAM as a separate simulator, connected to the CPU
via the bus. Consequently, I had to add support for
page tables and virtual address translation.
Its all working nicely. Concurrently, I am writing
a small MMIX BIOS, located in ROM, to be able to
boot the MMIX computer and as a proof of concept
for all the mechanisms needed for an operating system.

Now, I started to implement the page fault mechanism,
and I hit a solid obstacle.

Page faults occur when the processor translates
virtual addresses to physical addresses and discovers
that the page tables do not provide a translation.
This causes a forced trap, the instruction dies,
and a jump to rT occurs. Works nicely! 
But what now? I can not use registers, all of them,
locals and globals, might be in use by the application.
Nor can I use the special registers; the page fault
might be inside a TRIP, and the TRAP registers just
got fresh values through the page fault.
So we use marginal registers, best by using a PUSHJ $255,YZ.

But what if the page fault was caused by moving a value
from the register stack to M8[rS]? This is a write operation
to the stack segment and if the RAM used in the stack segment
grows past the allocated page boundary---page fault.

In this case, the PUSHJ $255,YZ will try to write rL to the
register stack and since there were no available marginal
registers in the first place, this will again cause a spill
of registers to memory, to a virtual location that still has
no physical counterpart...page fault again.

The operating system goes into an infinite loop. 

The problem is really bad. Almost any instruction, e.g.
ADD $10,$1,$0,  might cause a page fault, if it promotes
a marginal register to a local one and causes the register stack
to spill to memory. Even the instructions inside a trap handler
are not protected against this. Nested traps, however, require
the saving of the special registers rWW, rXX,... 
This raises the question how to save these registers
without risking a page fault. Or how to switch to a
operating system owned stack (by changing rS) that
is sure not to cause a page fault because it is kept big enough.

So far no good solution came to my mind. 
If I had an idea, I started to dislike it after some hours
or a day (the most). Do I just miss the simple solution?

Sincerely Yours
   Martin


Don's answer was quite inconclusive...

So further considerations are needed.

Problem 1: 
At the beginning of a forced trap, interrupts are disabled.
Hence, we have to avoid page faults because we can not handle
them. What instructions are possible?
All instructions that do not involve memory access.
Unfortunately due to the register stack, simple instructions
like SET $3,5 can involve memory access since this
migth turn a marginal register $3 into a local register $3 and
this might cause some hidden registers to spill to memory.
We can use only registers if we are sure that the register
stack will not spill to memory, or if the memory is sure to
be mapped, and access will not cause a page fault.

Historically, there was a free register, $255, which was saved
to rB or rBB and could be used in the TRAP handler, but
then it was clear that rJ had to be saved before MMIX could
do a PUSHJ, and hence Don redesigned the TRIP and TRAP moving
rJ to $255. And now no register can be used any more!

The 16 Byte TRIP Hanlder, given by Don

PUSHJ 255,Handler; PUT rJ,$255; GET $255,rB; RESUME
 
would be nicer, if rJ would go to rJJ or rJJJ
and back in case of a resume. this would again make
$255 available to use in the trap handler.

Alternatively the PUSHJ instruction would need to be replaced
by a JMP instruction. There is, after all, no need for the
TRAP instruction to return to the calling site of the PUSHJ.
The TRAP can return directly to the interrupted program using
a RESUME.

The $255 can be used to store away registers
by Loading it with a physical address in RAM  and storing
registers $0 to $31 there, then using them, and restoring 
afterwards. If the TRAP Handler needs a PUSHJ, rJ should be
stored in a local register and restored as usual.
We should however be aware that a PUSHJ is dangerous, because
it creates new locals and might spill registers to the stack segment
hence before using PUSHJ 255,XXX we should check rS and make
sure that either the PUSHJ and successive use of registers 
will not cause a page fault, or save away the trap registers
and enable traps again.

This is my last idea (nonsense!):

rS is the memory address of $0. So if we make an
address translation for rS each time we change it,
and generate a TRAP interrupt if we can not write to rS,
we can savely disable interrupts and still use all the
physically avaliable local registers $0 up to $254
since the hidden registers can be swaped to memory
without problems. The operating system must make sure
that the stack segemnt of a running process is completely
maped and loaded up to rS before resuming or starting a process.

This idea is nonsense. We can not use the registers $0 $1, ...
because they might be in use by the interrupted program.
if we make a PUSHJ $255, we hide these registers, but then
we can not use $0, $1, ... because this might cause
additional hidden register to need swaping to memory.
The idea might work with rS+8*256 instead. But this is not nice.

I dislike the waste of global registers, since we have 256 of
them and rarely use most of them. Its a waste.