Print

[rayarachelian]

Now FTP is where it starts to get tricky, and this is a protocol which pre-dates TCP/IP and as a result has a lot of baggage. If your goal is to back up a ProFile, I think you'd be better off inventing your own protocol. Or just start reading and sending each 512-byte data block in sequence as soon as a client connects to the Lisa's listening TCP port.

Perhaps implementing just a minimum binary transfer ftp is enough... but... I mean, we already have ways to transfer a whole ProFile at the block level via BLU.

I'm thinking more along the lines of a mechanism for getting individual files in/out of a volume, but also being able to send/receive a lot of files at once would be super useful too. Perhaps in the long run this is less of a case for an ftp app, than for a VirtualBox Guest Addition/Tools like thing for LisaEm - ideally I'd like to give LPW a lot of puppet strings and automate running it from an external IDE.

But an ftp app would be a good starting point for it.
The LOS Manual documents the low level kernel functions such as opening a file and so on that would be needed, but I haven't seen (not that I've looked very deeply either) how to access those syscalls from assembly directly bypassing the pascal libraries.

(Nod to your notice that passing large parameter sets on the stack via LINK/UNLINK is very inefficient and requires a lot of copying. That's a big thing I saw in LOS code when looking at the trace logs, a ton of inefficient shoveling of bits back and forth between caller and callee, it's even worse when the Pascal lib stuff hands off to the assembly side. It's on the order of several dozen instructions wasted just shoveling parameters back and forth through the stack.

As an OT aside, this is one reason why my favorite ISA is SPARC as it has register windows with in/out/global register sets, too bad Oracle killed it and Sun didn't make it. SPARC 4v was very very yummy. I imagine an alternate universe where this is the common ISA instead of x86_64 and how much better it would be....)
But passing pointers to records (structures in C) instead is a good work-around idea the Lisa pascal/Mac guys had.

I have an idea about how to implement a Guest Tools thing in LisaEm without adding any software to LOS itself. At first I was planning on writing a native Lisa app, but I've a better idea. Basically I can wait in the emulator until it goes to supervisor mode (or a specific context), save the registers and then replace a specific MMU segment with a block of code and transfer buffer, then set the PC to point there. The code in that special MMU segment could make calls into the kernel to open file handles, read or write to a file, get/set file metadata (or directory info), and close the file handle. Then when it's done, it could call an F-Line trap much like LisaEm's ROMless facility (i.e. F-Line traps) after it has data to return.
Once the transfer is complete, it can then reset the registers and the MMU segment back to what it was before taking action, and then continue as if nothing changed. So it would be transparent from the LOS side. With some extra work it could even tie into a FUSE filesystem.

What I'm missing is how to call those kernel pascal procedures from assembly. I can tell what the parameters are from the pascal structures and such, but what I'm missing is some sort of mapping to the syscalls, are they A-Line traps? are they fixed addresses? How do I find that out? This is pretty easy stuff in Xenix/UniPlus as syscalls are usually a single trap with a parameter. Here, I suspect they're specific addresses using that A-Line trap mechanism, but... need more info. And is it the same across LOS 1.0, 2.0, 3.0, 3.1, or does it change?

I suppose I could write a small it of code that makes a single call to a single thing and then invokes the debugger so I can see what happened, for example, make a single pascal call to say, open a file, and then log everything that happens from there end to end until it comes back to the next line to see what the kernel syscalls actually look like, but then, to quote Sal from Futurama "Whoas! What do I look like, a guy whose not lazy? Don't gets me started" - I suppose if I have to I'll find a way to automate that.

Sorry, it's a bit off topic to this conversation, mostly thinking out-loud for future things.

There's a (predictable) additional issue that the Lisa FS has metadata that you won't find on other operating systems....
my solution was to package everything on the fly into a tar archive with three "files" inside....

The Lisa's format for text files is a little weird --- it's faintly block oriented (files are always sized in multiples of 1K, IIRC), and it uses run-length encoding for runs of spaces. I'm still trying to track down where I saw the format documented (the documentation calls it the "One World Text File Format" or something like that), but here's a bit of Python that can decode it.

Lastly, there is the issue of protected files, which the Lisa OS will refuse to let you open. These files are interesting ones like Pascal compiler (Pascal.obj) and the various tools. It's possible that this more recently-discovered mechanism for jumping into supervisor mode can offer a workaround.

Yup, the Lisa does have a lot of metadata associated with it and it's in several layers (including the actual file name presented to the user from the Desktop manager database - the thing that allows duplicate file names). tar is absolutely a good container for this (as would be something like cpio, or even macbinary though it would have to be a variant of it due to the resource fork thing on the mac.)

I think I recall AEK mentioning that text file RLE compression format for the Lisa Sources. It's used to compress the indentations in Pascal code for example. Interestingly they did have a PackBits implementation, but didn't use it for this and rather they wrote a different one. Oh well.

I wouldn't worry too much about the bozo/protected bit, we have ways to remove that now. It may be possible that a call to unprotect (or if not it can be done through the deserialize tool.)

[D.Finni]

There's a (predictable) additional issue that the Lisa FS has metadata that you won't find on other operating systems. Besides the usual stuff you see in a file listing (creation time, modification time, and so on), there's also "type" and "subtype" fields (reminds me of filetype and creator for Macintosh)

Good to know. Maybe we can figure out a protocol for transporting this metadata too. The Macintosh has MacBinary and BinHex formats.

I'm still trying to track down where I saw the format documented (the documentation calls it the "One World Text File Format" or something like that)

I have that text file, and actually just glanced at its contents earlier this morning. I already wrote a tool to read the Lisa text files a couple years ago.

(Since we were talking about it earlier: the system software manual has the programmer's documentation for the QuickPort library, PDF pages 336-392.)

Yeah, I was reading that documentation earlier this morning too. I think QuickPort is the way to go. I also saw that Lisa's register conventions are identical to Macintosh. A5 is used as an application globals pointer, same as on Macintosh.

My concern since reading the Memory Management chapter of Lisa OS is how to allocate blocks of memory in a heap. On the Macintosh, you call NewHandle or NewPtr to allocate a relocatable or non-relocatable block in the application heap.

[D.Finni]

What I'm missing is how to call those kernel pascal procedures from assembly. I can tell what the parameters are from the pascal structures and such, but what I'm missing is some sort of mapping to the syscalls, are they A-Line traps?

They're A-line traps so far as I know. The Macintosh team borrowed all the good stuff from Lisa. I'm continually surprised to discover new similarities.

One of the Workshop or OS manuals has an Assembly language chapter and it shows how to call Pascal routines from assembly and vice-versa. I was reading it this morning.

How do I find that out?

I would think disassembly would give an answer pretty darn quick. Or even quicker, just isolate a .OBJ on any Lisa .dc42 disk image and check for frequency and spacing of $Axxx words.

Or if your Lisa emulator had a fancy debugger ( ) you could check the contents of the Line 1010 emulator vector at low memory location $0028 and then disassemble from the address stored there.

[rayarachelian]

My concern since reading the Memory Management chapter of Lisa OS is how to allocate blocks of memory in a heap. On the Macintosh, you call NewHandle or NewPtr to allocate a relocatable or non-relocatable block in the application heap.

Might be wrong on this, but since the Lisa has an MMU, it doesn't have to reorder memory to get rid of gaps, my guess is that it's not necessary to use handles.

One of the Workshop or OS manuals has an Assembly language chapter and it shows how to call Pascal routines from assembly and vice-versa. I was reading it this morning.
...
Or if your Lisa emulator had a fancy debugger ( ) you could check the contents of the Line 1010 emulator vector at low memory location $0028 and then disassemble from the address stored there.

Will dig again, the question isn't around calling conventions, that's easy enough to sort out. It's about mapping the routines in the LOS Manual to the exact A-Line routines, which are actually addresses in kernel-space, that's what that A-Line trampoline thing really is: a generic jump mechanism. ie. which A-Line is open, which is create, which is close, which is read, etc. And are those stable across LOS releases?

It's actually pretty easy to capture them all by just using the tracelog facility rather than a debugger, and it would capture them all in one shot, but what's difficult is reading the gigabytes of output.


Hmmm, perhaps I can insert assembly code that doesn't do anything but can let LisaEm know something about what's being called. That might work. Maybe I can insert a set of F-Line traps with a number to note what section of code is being called. i.e. F001 could mean open, F002 could mean close, etc. Then the code could be written in pascal, and then when the next A-Line trap is called, I can map it to that meaning.
Or maybe the F-Line trap could pass a pointer to a string that says "open", etc. and that string can be used to record it.
There we go.

Having a debugger would turn that process manual which would make it painful. But yeah, yeah, I can take a hint. I'll aim for a fancypants debugger in 1.2.8.
I'll even randomly and colloquially refer to it as "The David Finnegan Memory-all Debugger" 

[stepleton]

Good to know. Maybe we can figure out a protocol for transporting this metadata too. The Macintosh has MacBinary and BinHex formats.

For what it's worth (and for your consideration) there's been a single-file container format for a while

It's that tar-based format I mentioned just above. I call it a ".lar" file in the README for my file download program, and there are no prizes for guessing what the L stands for.

As mentioned, there are three files inside the .lar archive. The first is a text file with many of the file attributes written out in a format that serves me but would be fine to change. The second is a file containing the label data. The third is the file data itself.

tar might seem like a heavyweight choice, but the specific file ordering I use simplifies access for homemade implementations (in other words, you don't really need to implement tar yourself). Since the first two files are small, file contents always start at fixed locations: $200 (file attributes), $600 (label contents), and $A00 (the file itself).

Of course any modern Mac or Linux machine will have tar installed already, and so will Lisa Xenix and UniPlus for that matter . It's easy to come by for Windows too. Choosing a text format for the attributes inside comes from the same desire for easy accessibility.

About the only thing that's not so great is the overhead, at least $780 bytes of it: $200 for each file header, the $180 bytes of tar file block that the label file doesn't use, and probably several dozen more bytes not used by the attributes text file. But: the Lisa doesn't have to store that --- a transfer program can pack and unpack .lar data on the fly, tar is a streaming format after all --- and any modern computer on the other end will probably not be bothered by the extra space.

Sure, all those extra bytes will take time coming over the serial cable, but the amount of my time that this wastes is probably smaller than what would be required by writing decoders for a custom format on a range of plaforms

[D.Finni]

Good to know. Maybe we can figure out a protocol for transporting this metadata too. The Macintosh has MacBinary and BinHex formats.

For what it's worth (and for your consideration) there's been a single-file container format for a while

It's that tar-based format I mentioned just above. I call it a ".lar" file in the README for my file download program, and there are no prizes for guessing what the L stands for.

Unless anyone has any objections, I suppose this is the way to go. I certainly don't want to invent another format if there's already a good one around.

Will dig again, the question isn't around calling conventions, that's easy enough to sort out. It's about mapping the routines in the LOS Manual to the exact A-Line routines, which are actually addresses in kernel-space, that's what that A-Line trampoline thing really is: a generic jump mechanism. ie. which A-Line is open, which is create, which is close, which is read, etc. And are those stable across LOS releases?

Ah, ok now I understand what you're after. The Lisa documentation is awfully tight-lipped about it, from my review of it last evening. I'm not totally sure that LOS uses A-line traps. I should admit that it's just my assumption based on so many similarities between Macintosh and Lisa architecture. On Macintosh, the trap dispatch table is located at $400 in RAM.

[D.Finni]

The Lisa's format for text files is a little weird --- it's faintly block oriented (files are always sized in multiples of 1K, IIRC), and it uses run-length encoding for runs of spaces. I'm still trying to track down where I saw the format documented (the documentation calls it the "One World Text File Format" or something like that),

I attached that text file to this post.

[rayarachelian]

Ah, ok now I understand what you're after. The Lisa documentation is awfully tight-lipped about it, from my review of it last evening. I'm not totally sure that LOS uses A-line traps. I should admit that it's just my assumption based on so many similarities between Macintosh and Lisa architecture. On Macintosh, the trap dispatch table is located at $400 in RAM.

Yeah the LOS manual just shows you the pascal procedure names, so I'd need to compile code with the pascal compiler, even though I'm just after the assembly side of things, and then analyze the generated code and actual A-Line calls.

Yeah, they're not the same as in A-Line traps in Classic Mac OS (<10.0). On the Mac, the A-Line traps aren't addresses and they have fixed meanings. When you call a specific one in System 0.9, say A0000, it has the the same function as in System 6.0.8 and even in 7.5 or 9.2. And this is what allows for portable code.

In LOS, the A-Line trap is an address into the kernel code, and you can potentially pass anything there, even things that don't belong at all - hence, it's a supervisor trampoline. So it has the potential of changing between versions, further, it's undocumented.

For example, in Solaris 2.6 on SPARC, the open syscall is exactly the same trap as in Solaris 10 SPARC, so the ABI (Application Binary Interface) makes it 100% portable. (Linux on i386 didn't do this, and so you'd need a bevy of libc binaries to have backwards compatibility with previous releases, not sure about x86_64.)

Back to LOS, as long as you can figure out the address of your code in context 0 (and AFAIK context zero maps all RAM), your process can run in supervisor mode.

I suspect that if you're after this, you can ask the MMU mapping for your running segment by using something like this: (this is just a sample off the top of my head, not an actual tested implementation, probably is very wrong in some ways, but it's a skeleton)

Code: [Select]

getsupervisor:
      LEA shellcode,A0
      MOVE.L A0,D0
      AND #00ffffff,D0
      MOVE.L D0,(saveaddress); save this address for later
      LSR D0,#17  ; d0 now has the segment number of the shellcode
      ; some call here to get mmu register for physical address
      MOVE.L (saveaddress),D0
      ADD A0,D0 ; assuming A0 contains the physical address offset of the segment, add our saved value to it
      AND.L #$00FFFFFF,D0
      ORI.L #$A00000,D0
      LEA trampoline
      MOVE.L D0,(trampoline)
      NOP ; insert some NOPs since we're doing self modifying code here and don't want whatever
      NOP ; pipeline cache our CPU may have from preventing this from working properly
      NOP
      NOP
trampoline: DB.L #$A0000000
      RTS
...
shellcode: evil code to "Same thing we do every night Pinky, try and take over the world!"
... and now we're in supervisor mode.
     RTE

Some shifting, and then a read of the MMU register for your current code by calling the right call to give you back the physical address. (Basically shifting it by 9 would give you the segment number and to this you'd add the physical address, and then the offset you got from LEA PC+X,A0).

Then, since context zero (which runs as supervisor) has a very fixed, predictable memory map, you can calculate which address it would match there for your shellcode, and then invoke the A-Line trap for that address (with some self-modifying code to write that address in the A-Line opcode) and you've got supervisor access to the whole of LOS.

Also, on the Mac the A-Line traps are only 16 bits, where on the Lisa they're 32 bits because they're actually addresses (and the high by of 'A?' is a throwaway, so actually 24 bits of address range, but stored as 32 bits worth of opcodes).

[D.Finni]

Beta testers are continuing to find new bugs. I think it's logical to hold off a native Lisa port until the code is debugged.