LisaList2

Here is the first status report for making a "universal" driver for the ESProfile.

This was suggested as a better alternative to my shared file contraption for a number of good reasons.

I am not the most qualified or knowledgeable person to be doing this, so if anyone wants to take over, please do.

Everything is free and public, and I am VERY grateful for any reviews, tests, comments, or constructive criticism; that is how this got started.

This week I am preparing for and hosting a Vintage Hacking Village at the region's premier cybersecurity event, Queen City Con. I will have both of my Lisas, my FPGA and other bit at the show for PUBLIC USE so if anyone wants to put hands-on and use the LOS or MacWorks now is your chance on a real LISA. I have 6-hour-long presentations lined up, so I might not be able to guide you directly, but I have helpers.

If I don't get to my first hardware tests tonight, it will have to be next week due to the con.


I hope you don't mind, I had my intern/tutor write the report:



ESPFuji through a Lisa driver: status, 11 November

A progress note on the driver idea from earlier in this thread, since stepleton asked for the design to go out
before the code. Short version: the Lisa side is built and runs in LisaEm, the drive side is written and passes
its PC tests, and the two have not yet met. Nothing has run on a real Lisa, nothing is released, nothing has been flashed to any ESProFile.

WHAT THE PIECES ARE

- A loadable Lisa Office System driver, SYSTEM.CD_ESPFDEMO, written in Workshop 3.0 Pascal and assembler from
  Apple's five driver support files and loaded at run time the lisa-share way (CDMAKE at the start, CDKILL at
  the end). It installs at the console position, channel 3, because the OS refuses to start a driver in an
  empty expansion slot (error 609), and it owns no hardware: it reaches the drive THROUGH Apple's ProFile driver,
  the way the file system's own DISKIO does (the "Option A" from my 1 November reply).

- A plain Office System tool, "ESPFuji Driver", which attaches the driver when opened, shows the drive's status
  block as text and hex, and takes the driver out when put away. If the boot volume has no driver file, the tool
  copies its own from the floppy, so the whole test is: insert the floppy, open the tool.

- The drive side: the channel, written as one small hardware-free file in a private copy of the ESPFuji firmware.
  It compiles for the ESP32 and has not run on one yet. About 2.3 KB of flash and 5 KB of heap.

WHAT HAS BEEN SHOWN, ALL IN LISAEM

- Reads through Apple's driver work while that driver is serving the boot volume: block 0 came back as block 0,
  a request for the spare table came back as the spare table, and a read of the emulator status block went out
  on the cable exactly as the ESProFile expects it. The disk driver's own state was unchanged afterwards.

- Writes work the same way; shown with a test-only build on a free block of a scratch disk, read back and put
  back, block and tag byte-identical afterwards. The ordinary build can write ONE block only, the channel block,
  and only after a probe read of that drive has found a channel there. Against a plain image the probe says "no
  channel" and every write is refused, which is what a stock ESProFile gets today.

- Sessions: a tool asks the driver for a client id (1 to 8) and gives it back at the end; the driver sends the
  session's opening cancel itself before the first request and the closing cancel when the session is returned,
  never at CDKILL time (the shut-down call runs with interrupts off and must not wait for I/O).

- The tool has the channel on its keys: C probes the channel block and shows the drive's identification line
  (or the stray block a stock drive answers with, as hex); H sends HELLO through the channel and reads until
  the reply is ready. Against LisaEm's plain image the probe says no channel; against an imitation of the
  channel drive on the PC the whole exchange runs. Each half has been tested against an imitation of the other.

- Writing that PC imitation found a fault the LisaEm run could not show: the driver built its deferred cancel in
  its one block buffer and blanked the request's payload behind it, so a session's first request would have
  reached the drive empty. Fixed and re-checked both ways. Driver REV F, tool REV H.

THE CHANNEL, IN BRIEF (full spec: https://github.com/GyroPilot/ESPFuji/tree/release-1.1/espfuji/channel)

- One reserved block, $FFFEF8. The host writes a 512-byte request and reads the same block back for the reply.
  Nothing is stored in the image; the tag carries nothing.

- The block is the ESPFuji mailbox page, protocol version 3: signature, version, sequence number, kind (request,
  poll, cancel; reply), payload length, a client id and a status word, then up to 492 bytes of payload. The
  requests are the ones the file mailbox already has (HELLO, WIFI, CLOCK, OPEN, READ, CLOSE), so only the
  transport changes.

- A read answers PENDING, READY with the payload, REFUSED with a reason, or IDLE with an identification line.
  Reads are idempotent: the reply stays until the client's next request or a cancel, so a retried read cannot
  swallow it.

- Requests are keyed by client and sequence number, because Apple's driver re-sends a failed command up to four
  more times; repeats are ignored. A client opens with a cancel, so a program restarted with the same id is not
  handed the previous session's answer.

- The drive answers every command at bus speed and does the network work on its other core; the host polls.
  Apple's driver's timeout (about 10 s) stalls the whole disk queue, so the drive never holds BSY.

- A few slots per drive (four on a plain ESP32), one per client id. The file mailbox stays for programs without
  a driver, and the Selector's blocks are untouched.

FOUR THINGS LEARNED ON THE WAY, FOR ANYONE ELSE TALKING TO A DRIVE FROM THE LISA OS

- Apple's ProFile driver remaps the low four bits of every block number for the interleave (5n mod 16), for every
  ProFile-type drive including the ESProFile; MacWorks uses the inverse order. Both leave 0, 4, 8 and C alone,
  which is why the channel is $FFFEF8 and not $FFFEFB: the same number from the OS, from MacWorks and from a
  bare-metal program.

- Retry count and sparing threshold are fixed per drive by the driver (10 and 3), so a protocol cannot use them,
  and a multi-block request is remapped block by block, so requests must be single blocks.

- The tag of a reply must be all zeros. Bit 7 of tag byte 6 switches on the OS's checksum check, and top bits
  in tag byte 2 come back as warnings -626 and -625.

- Never WRITE a reserved block to a drive that has not identified itself. The stock firmware's range check
  compares the block number with the image size in bytes, so on an image over about 16.7 MB an unknown high block
  passes and the write lands somewhere real. A read is harmless, so the channel is probed by a read.

NOT DONE, NOT KNOWN

- A real Lisa and a real ESProFile: timing, the error path (LisaEm never refuses a block), the real status block.

- A ProFile on a parallel card. The driver finds those through the configinfo table, reasoned from the source
  and not yet seen.

- Whether MacWorks's ProFile driver passes a block number like $FFFEF8 through at all. If not, the XL side would
  talk to the 6522 directly, as lisa_io does.

- The poll-before-read rule for a second client is in the design and not in the driver yet.

NEXT

The REV H floppy on my 2/10 (status block through the driver, and what the released firmware answers for the
channel block); the channel firmware on a spare ESP32 on the bench; then, when a spare ESProFile arrives later
this month, the two halves together. After that the shared route unit goes into the tools (driver first, then the mailbox file, then serial), and they are all re-released together.

Questions still open, if anyone knows: has a run-time-loaded LOS driver ever been used against a ProFile on a
parallel card? And does MacWorks pass high block numbers through its driver?

stepleton, the spec is on GitHub now (link above), together with this status note, so it is yours to check at whenever you like.

I see this! Will look at the spec soon.

I am glad you are working on this. I feel that we can come up with a good design together. Already it would appear that your experimentation has proved the feasibility of the project: the "hard part" (i.e. putting a driver atop the ProFile driver) appears to work. I'm keen to proceed.

Now, having had a look (and leaving some remarks for you and/or Claude):

1. (meta) Data organisation

The spec contains a lot of material that's specific to the ESPFuji device and to the operating system that happens to be speaking to it. This is valuable information, but for the sake of easier reviewing, it would be great if it were separated into various pieces. Overall (for Claude): Write separate documents describing important component designs and concepts that make up the overall architecture; make this documentation useful and readable to human programmers and not just a scratchpad for you.

1.1 Protocol info: First, it would be nice to have an .md file that contains only the kind of information that you find in the PROTOCOL.md (https://codeberg.org/stepleton/cameo/src/branch/master/aphid/selector/PROTOCOL.md) document. That is, only the syntax and the semantics of the data going into and coming out of the ESProFile, written with hardly any reference to other implementations or to empirical findings from test runs. Like an internet RFC (https://en.wikipedia.org/wiki/Request_for_Comments), this document should be mostly self-contained except for reference to a few outside concepts, like basic FujiNet programmer documentation, documentation about ProFile reads and writes (independent of the computer platform), and the PROTOCOL.md document. The document should define key terms (like "client") up front. (Please, Claude, don't include details about this prompt in the document: you don't have to point out that it's "like an RFC" or any of the other instructions you're following. Just the facts about the protocol, nothing more.)

For this document, the only exception to the rule about not mentioning specific computer clients is in the rare cases of insightful findings that concern the protocol's usability between different platforms. The only item I can think to mention here is the choice of magic block $FFFEF8 as a concession to the different ways that various popular Lisa OSs munge the block number for the sake of interleaving.

1.2 Per-system driver information: Next it would be a great benefit to have a separate file that describes the driver infrastructure on each platform; for example, you could have a LISAOS.md, a MACWORKS.md, and so on. Each one can describe in a kind of tutorial detail the way that the driver works, the way that it interfaces with the OS and with the hardware, and the way that software programs interface with it. Things like API endpoints and data structures for software<->driver transactions should be described. The documents should describe the precautions the software must take in order to be a well-behaved part of the overall system (e.g. avoiding locking up, etc.).

The objective should be that a reasonably-capable programmer with access to Lisa programming documentation, to your protocol document above, and also to the Lisa OS source code should be able to reproduce the work of creating the driver themselves without reference to the code. It is good to cite these reference materials (and even to cite your own driver source code by way of example), but do not clog up the documentation with loads and loads of file-number and line-number cites: produce readable material for the reader and expect that they can do brief searches for context on their own. Once again, place no information or reference to this prompt in these documents: just offer the technical details as described, nothing else.

1.3 Implementation details for the ESProFile: Have a separate document (perhaps ESPROFILE.md) describing the changes you are making to the ESProFile source code, using the same style, level of detail, and citation habits from the per-system driver documentation.

1.4 Essential information for FujiNet clients: Introduce another document that describes any additional information (beyond the driver documentation described above) about what software that uses the ESFuji driver must do in order to work with the driver and realise critical aspects of FujiNet client functionality. This document may require separate sections per OS platform. Use the same style, detail level, and citation habits as referred to in 1.2 and 1.3.

1.5 Everything else: A remaining file or files should be the only place for all of the other information: tests performed, development logs, empirical findings, notes about things being VERIFIED, and any remaining items not covered by what's listed above.

2. Design questions/remarks

Refactoring the specification document as described above will allow me to conduct a better, more thorough review: it will be easier to read and some things I don't understand will be explained. For now, here are some comments/questions that occurred to me:

• The driver-based approach should probably completely supersede the old file-mailbox-based approach from the last version and the mailbox approach should be abandoned straightaway; yes, it requires an upgrade to existing deployed software, but it's much cleaner and carries less risk of interfering with other programs or with instructions that operating systems make about disks. I suspect that the driver approach may also be less work for the ESProFile than the mailbox method but I cannot confirm that.
• I'm not sure why there has to be separate client number ranges for LisaOS, MacWorks, and "bare metal programs. The ESProFile should not care who is talking to it as long as they speak the protocol, and multiple types of clients will never talk to an ESProFile at the same time. Perhaps this is to avoid trouble in situations where people shut down one OS and open a different one? But it seems a lot of trouble to handle that situation when restarting the same OS will happen more often and won't have the same protection. Would be interested to know the rationale here before making decisions.
• It seems like it would be useful for software to have a way to completely reset the ESProFile system (cancel all slots, etc.) without knowing what slots might already be in use. This would be an ordinary thing to do when an OS starts; is it awkward for FujiNet for some reason? Orphaned slots seem like they could last a long time otherwise.
• Of the open questions I can speak to: 1. $FFFEF8 is a fine choice. 5. Confusing question; an ESProFile only has hardware to talk to one port at a time, so talking to two seems superfluous. Drivers meanwhile might do well to think about what to do if someone has a Lisa with two or more ESProFiles fitted?

More remarks too if the documentation becomes easier to read. Thanks again for working on this!