LisaList2
General Category => LisaList2 => Building LOS From Source => Topic started by: sigma7 on July 20, 2025, 02:24:09 pm
-
Generously, Alex has provided a ProFile image with the Workshop and his build scripts installed via GitHub: https://github.com/alexthecat123/LisaSourceCompilation (https://github.com/alexthecat123/LisaSourceCompilation)
However, since the License to download the Source files prohibits re-sharing them, posting a ProFile image that already contains the source files isn't allowed.
So, to actually build LOS from Source on a real Lisa, each individual has to move the many source files to the Lisa...
While we figure out the best way to do that, please post your ideas and progress here to minimize duplication of effort.
-
for future use... I'll put links to the top choice(s) here
-
Trying to brainstorm the possibilities, I've collected the following suggestions:
- Using the Workshop's scripting to automate multiple transfers via Serial
- Writing a specific program to transfer via Serial
- Transfer via Floppy using the MacCom function
Alex reported in his VCF presentation that floppy transfers introduced corruption; perhaps solving this problem is a possibility.
It has been suggestion that a method of packing all the necessary files into one, then unpacking on the Lisa is likely to minimize effort.
-
Additional thoughts
- Create a ProFile image with all the source files installed, over-write the contents of the source files in the image keeping the file structure otherwise intact, share the image along with some software that will repopulate using the source files on a local machine
- Remap the console to a serial port and simulate typing each of the source files
- Use the file manager to copy from a serial port to a file (same as the Transfer program but maybe easier to script?)
-
An option to consider is to do serial port automation for text file transfer by means of a custom program that runs on the Lisa. The program should listen on a designated serial port configured to use 8 bit bytes and whatever speed we think we can get away with. It begins in "idle mode", where if it receives a string like this:
~~~~~~ NEW FILE: APLW/UNITDSP.TEXT ~ LINES: 1397
then it opens the text file "APLW/UNITDSP.TEXT" and fills it with the next 1,397 lines that come over the wire. Once 1397 lines are received, it closes the text file, returns to idle mode, and awaits the next file. This continues until it receives the string ~~~~~~ THAT'S ALL FOLKS, where it quits. LIBPORT.TEXT (https://github.com/stepleton-xx/lisabbs/blob/master/LIBPORT.TEXT) from my old LisaBBS project might be a help, though it doesn't really have a concept of line-based I/O.
On the other end of the serial cable, a modern computer. Either it uses a modern program which scans the Lisa source code directories and sends them off one by one between the NEW FILE delimiters, or, even simpler: write a program that concatenates all of the source code files with these delimiters into one very, very large single file and send that out the serial port. These programs should both be pretty short, and probably writing it in Python would be quick work for your favourite AI, although no shade if that kind of thing doesn't appeal.
(If it doesn't bother you, here is Gemini's attempt, which looks pretty close to me: https://g.co/gemini/share/b412e38848e1 . It makes use of the pyserial (https://pyserial.readthedocs.io/en/latest/pyserial.html) library.)
...
While I was typing this, the idea you (sigma7) shared of having the modern computer just drive the console-remapped Workshop over the serial port seems mighty appealing, requiring no programming on the Lisa side, and it could make use of a lightly-modified version of the AI-made program I linked above. Is there a way on the Workshop side to write data from the console directly into a file?
[note: some edits to fix grammar etc.]
-
Is there a way on the Workshop side to write data from the console directly into a file?
One is the "DumpPatch" utility, which (if I'm reading the workshop manual accurately) permits entering data in hexadecimal form, then saving to a specified filename.
"Copy" is another option suitable for ordinary text... console can be the input and output can be pre-directed to a file.
IIRC, one can also "Copy" from a serial port to a file (or vice-versa as one might do to dump a file to a printer), dunno whether this works concurrently with mapping Console to the same serial port, or if there would be any advantage to doing so.
-
I really like the simplicity of just driving the Workshop directly, if we can get that to work! And I think the direct Copy option from the File-Mgr is probably simpler to get running than DumpPatch. Although DumpPatch could be handy if some of the special characters can't be directly typed over serial and would need to be entered in hex instead.
Alex reported in his VCF presentation that floppy transfers introduced corruption; perhaps solving this problem is a possibility.
Just to provide a bit more detail, MacCom would always cut off the start of each file and append some garbage (or sometimes text from another file) onto the end of each file. So it seemed to me like the pointer to the start of the file was getting offset slightly ahead of where the file actually began on disk. Not that this matters anyway, because I think that fully automating the process is absolutely the way to go!
-
Some discoveries in working on uploads...
Aside from > and < used for redirecting to/from a filename or KEYBOARD/CONSOLE, the < and > characters shown below are not typed.
To configure the serial port, starting at the main Workshop menu:
rportconfig<return>
then type return to go through the settings.
To switch the console to RS-232 port B, starting at the main Workshop menu, ending in the System-Mgr menu:
scrby
To switch the console back to main when finished using the serial port, starting at the main Workshop menu, ending in the System-Mgr menu:
scmy
To start uploading a file, to be named "filename", starting at the main Workshop menu:
Note: ".text" is appended to the filename unless it already includes it
with echo to the screen
rcopy<return>
>filename<return>
without echo to the screen
rcopy<return>
<-KEYBOARD,>filename<return>
To end uploading (sometimes one less <ctrl-m> is needed, perhaps patience makes a difference?)
<ctrl-n><ctrl-m><ctrl-m>
To run the charcount function on a file named filename, starting at the main Workshop menu:
rcharcount<return>
<filename,>-CONSOLE<return>
To interrupt an operation (equivalent of Apple-. or ctrl-c)
<ctrl-[>
Text File Compression
The Workshop manual mentions something about compressing the leading spaces on each line in a text file, using a DLE character and a space count character. Presumably this is to not punish indentation styles. I believe line lengths are limited to 255 characters so a single count character is always sufficient.
In terms of the size of a file as reported by charcount vs the size of a file prior to upload, it seems that you can get the numbers to match when:
- the line ending is considered a single character
- the compression reduces the size by the number of leading spaces greater than 2 on each line
- the compressed size is 2 greater than that derived from the two preceding rules
I haven't investigated how the data is actually stored on the disk (eg. is a line with 2 spaces DLE 0x02 or ' '? etc., and why/where are 2 characters added... a 16 bit file length perhaps?)
DLE (aka "Data Link Escape") is 0x10 in the ASCII table.
A line starting with 6 spaces is encoded as 0x10 0x26
I'm closing in on an AppleScript solution using Roger Meier's CoolTerm https://www.freeware.the-meiers.org (https://www.freeware.the-meiers.org)
CoolTerm claims to be python scriptable too and is available for Windows and Linux. The pacing option that waits for a character's echo seems particularly well suited to this task. YMMV
edit: for clarity... english is hard
edit2: added charcount and copy without echo
edit3: added thoughts about the leading spaces compression in text files
-
That's some great progress! I wonder how high we're going to be able to go with the baud rate? I started experiencing corruption with anything over 1200, but maybe other people will have better results.
-
If the proposed method doesn't display text on the screen during the transfer, then it may be possible to use a higher baud rate. I suspect a major reason for the LisaTerminal slowness is because (if I recall correctly) it draws the text to the display while it receives it. If I'm wrong about that, then disregard. If I'm right, then that surely steals a lot of cycles.
The Lisa serial port has hardware flow control capability, which may be a bit more dependable than XON/XOFF flow control. Some USB/serial adapters may not be as good at it, though. I wonder if it would help.
-
If the proposed method doesn't display text on the screen during the transfer, then it may be possible to use a higher baud rate.
...
The Lisa serial port has hardware flow control capability, which may be a bit more dependable.
Using the per character echo pacing, I was able to achieve about 39 CPS with what I suspect is high reliability. That would make about 6 days to upload the OS and Apps, but at least it might run unattended.
Instead of using -stdin aka -CONSOLE for the input, one can use -KEYBOARD as the input which doesn't echo the characters and so goes much faster as you say.
Then one has the problem of establishing reliability or at least confidence. Using the leisurely charcount function has potential but is complicated by the fact that apparently the leading spaces on each line in a file are compressed using a DLE character followed by the space count (or something like that). So the character count of a file as uploaded doesn't match the charcount result (which counts the compressed bytes) even when 100% successful.
I do see the hardware handshake toggling when I'm overloading the transfer; so am investigating what the limits are with that.
disclaimer: I'm still pfaffing about, so much of that may be incorrect
-
I know I said that I was taking a break from Lisa coding for now, but the more I thought about this, the more I realized that it would actually be a pretty fun project. So I've started work on a Python program using pyserial that will hopefully be pretty nice too.
It's currently capable of transferring a single file at a time, and logs any transmission errors to a text file for future reference. Now I just need to add the ability to do multi-file transfers and polish it up a bit, and it should be done.
The more solutions we have, the better, and this along with James' AppleScript program should be really effective!
-
...really effective!
I've only been able to get a consistent rate of 101 CPS, so I'm thinking a different approach is warranted.
Hopefully you/python can do better!
That's with a prolific chipset USB adapter, I'll look for an FTDI one to try, but I think the bottleneck is the other end.
Still easier than using Transfer and typing filenames in one at a time... respect dude.
-
Looking through the Alex's directions for source file corrections, it looks to me that these files will not need editing if uploaded with 8 bit characters (Aside from dropping the last character in each file):
APHP/T12ALERT.TEXT
APLP/T8ALERT.TEXT
APLL/DBCONVERT.TEXT
APLL/T5LM.TEXT
APLT/CONVERT.TEXT
APLT/INIT.TEXT
APLW/T1MENUS.TEXT
APLW/T1ALERT.TEXT
APLD/T4ALERT_MENUS.TEXT
APLC/T3ALERT.TEXT
APLC/LCFILER.TEXT
APLC/APPDIBOX.TEXT
APBG/T2ALERT.TEXT
LIBPR/DWBTN.TEXT
LIBPR/PRALERT.TEXT
LIBQP/UVT100.TEXT
LIBTK/UOBJECT4.TEXT
APHP/HP.TEXT does need editing to fix the following characters; I think this can be done pre-upload by using these values...
Diamond: ◊ 0xD7 #215
Divide: ÷ 0xD6 #214
Slash: / 0x2F #47
-
I just updated the repo with a Python script in the scripts directory called lisa_serial_transfer.py and some some new instructions in the Getting the Code Onto Your Lisa section of the readme explaining how to use it. I'm able to get about 60 characters per second out of it, so not as good as James (101 is really impressive!), but still better than what I was initially hoping for. With this script, it'll take a little less than 100 hours to send over all the code.
To make up for the less than optimal transmission rate, it provides detailed status info throughout the transfer, as well as error checking and recovery abilities if the Lisa suddenly stops responding, which seems to happen occasionally if you have a cheap USB to serial adapter.
Let me know if you guys have any problems with it!
Looking through the Alex's directions for source file corrections, it looks to me that these files will not need editing if uploaded with 8 bit characters (Aside from dropping the last character in each file):
APHP/T12ALERT.TEXT
APLP/T8ALERT.TEXT
APLL/DBCONVERT.TEXT
APLL/T5LM.TEXT
APLT/CONVERT.TEXT
APLT/INIT.TEXT
APLW/T1MENUS.TEXT
APLW/T1ALERT.TEXT
APLD/T4ALERT_MENUS.TEXT
APLC/T3ALERT.TEXT
APLC/LCFILER.TEXT
APLC/APPDIBOX.TEXT
APBG/T2ALERT.TEXT
LIBPR/DWBTN.TEXT
LIBPR/PRALERT.TEXT
LIBQP/UVT100.TEXT
LIBTK/UOBJECT4.TEXT
APHP/HP.TEXT does need editing to fix the following characters; I think this can be done pre-upload by using these values...
Diamond: ◊ 0xD7 #215
Divide: ÷ 0xD6 #214
Slash: / 0x2F #47
For whatever reason, my program (and any terminal software that I've used for testing) refuses to send 8-bit bytes. Although I'm wondering if it's something on the Lisa end where it just strips the top byte without consulting you at all. Have you experienced the same problem in your testing?
-
While I'm at it, I think I'm going to try and write another Python script that automatically applies all the necessary patches to the source code as well. So stay tuned for any progress on that...
-
it looks to me that these files will not need editing if uploaded with 8 bit characters (Aside from dropping the last character in each file) ...
my program (and any terminal software that I've used for testing) refuses to send 8-bit bytes. Although I'm wondering if it's something on the Lisa end where it just strips the top byte without consulting you at all. Have you experienced the same problem in your testing?
If I upload with -CONSOLE (so characters are echo'd back), I can see that the extended character set is being returned, so it looks like the high bit is making it through the serial driver, but then stripped when saved to disk.
Potential ways to deal with this:
- Fix it in post (on the Lisa end) as originally suggested
- Change the constants from characters in a string to hex values not in a string
- Create an exec file to automate the fixes on the Lisa end
- Patch the runtime software
Thoughts?
-
Hail Mary: is there a chance that the Concat utility is usable here and better-behaved than Copy?
-
I tried "translit" to see if it might yield a speed improvement (it didn't), but haven't checked if it or concat is more tolerant of high ascii.
Good news (I think): I've seen a substantial speed boost by setting a larger buffer in portconfig.
It wouldn't accept a value larger than 32767 as valid, and claimed there wasn't enough memory to do that or 16k when asked to commit the new values, but it would do 8192.
I left the low threshold at the default and set the high threshold to 8000. I presume that's the point it toggles the handshake line.
With those settings (and 9600 baud) I saw 600 CPS for small files, but files larger than the buffer still end around 100 CPS.
-
If I upload with -CONSOLE (so characters are echo'd back), I can see that the extended character set is being returned, so it looks like the high bit is making it through the serial driver, but then stripped when saved to disk.
Same here. I'm not super familiar with the inner workings of the 8530 SCC, so I might be completely wrong here, but perhaps the echo is handled in hardware, which is why the characters are echoed just fine, whereas all the software is written to ignore the high bit? Not that the exact reason really matters though.
Hail Mary: is there a chance that the Concat utility is usable here and better-behaved than Copy?
Good idea. I'll try that as soon as I can!
Good news (I think): I've seen a substantial speed boost by setting a larger buffer in portconfig.
Oh wow, that's awesome! I always figured that choosing that option would just cause it to automatically adjust the buffer size, but being able to set it manually is really great. This also led me to discover that handshaking wasn't working properly on my USB to serial adapter, so I think I might be able to get a bit more speed out of my Python program once I get that figured out.
-
Okay, I've made some changes to my code, and I'm now able to get about 120 CPS (average, including the breaks when the buffer gets full) for large files and 400-ish for smaller ones that don't fill up the buffer. It also turns out that automatic handshaking is completely broken in the Pyserial library, so I have to manually poll DSR and stop transmitting when it goes low. But that's not a big deal at all now that I've figured it out.
The new version of the program isn't quite ready to be put on Github yet, but I'm hoping to have it up by the end of the day!
Also, I just tried out CONCAT, and unfortunately it doesn't preserve the high bit either. So we'll need to find another solution there.
Progress is coming along pretty well on the auto-patch program too, so hopefully I'll be able to release that by the end of the weekend. That way, the only patches that would need to be done manually (unless we find a way to automate them too) are the corrupt character ones on the Lisa end.
-
I wonder if it may be time to DIY a transfer program on the Lisa side. It is at least possible to send 8-bit data from a program running on Lisa OS over the serial port; the YMODEM implementation I made requires it, so the problem does not lie in the Lisa OS serial driver or in the OS libraries. But setting YMODEM aside, I've mooted a wire "protocol" for text file transmission upthread, which Alex's python program could reproduce with minor modification; the major work of course is on the Lisa side.
I barely have time to tie my shoes these days (after-market elastic drawstring shoelaces for the win), but if I were doing this I would start from LIBPORT.TEXT (https://github.com/stepleton-xx/lisabbs/blob/master/LIBPORT.TEXT) and write a function called TimedReadLine that reads "\r"-terminated lines from the serial port (the few string library functions starting from PDF page 142 here (https://bitsavers.org/pdf/apple/lisa/workshop_3.0/Lisa_Pascal_3.0_Reference_Manual_1984.pdf) may help).
From there on it's a matter of reading lines and then writing them to files, I should think.
The largest source file (APDM-DESK.TEXT.unix.txt) is under 256k, so we might even consider loading all incoming files directly into RAM, provided it's not difficult for Workshop programs to allocate so much memory. Storing file data in RAM should be about as fast as it gets, and that would maximise throughput. But this would be something to fix only after a simpler approach was working, I think.
-
it may be time to DIY a transfer program on the Lisa side
A transfer program could certainly help a lot with the speed.
From looking at source code and dumping some bytes from text files, I think the issue is that the system is designed around an "international text" concept so input text gets filtered before being stored in a file in what looks vaguely like a precursor to UTF-16.
It may be possible to pre-code the highbit characters or perhaps implement a base64 decoder or something like that since there aren't a lot of files that need that support.
Still, it may be easier to ( have Alex :) ) change the high bit characters to hex values instead of string constants.
-
I have a suspicion that maybe PORTCONFIG is the problem, and that it always sets the port up in 7-bit mode given that it never asks you what you want. So I think the next thing I'm going to do is to write a simple Pascal program that forces the port into 8-bit mode. Hopefully that fixes things. If not, then I guess it's got to be something with COPY.
And by the way, the new faster version of the transfer tool is on Github now.
Edit: It looks like my suspicion was unfounded. After looking at some documentation, it seems like the driver automatically chooses 7 or 8 bits depending on whether you enable or disable parity, which PORTCONFIG most certainly allows you to do. I think I'm going to try and write a simple program to replace COPY that actually supports 8-bit bytes, and hopefully that'll fix the problem.
-
Good news, I figured it out!
It turns out that it has nothing to do with the OS, the driver, PORTCONFIG, or COPY being weird and intentionally deciding to strip the high bit, but instead it's an annoying quirk of LIBPL (the Pascal runtime library). Whenever you use the LIBPL routines to open the console for reading and it detects that the console has been mapped to the serial port, it automatically assumes that you're using the high bit for parity and discards it. The solution here is to use the OS routines (look in LIBOS/SYSCALL) to open and read the console, since these just give you the data untouched.
I've written a replacement for the COPY program that uses these discoveries to preserve the high bit, so we should be able to send all the Lisa characters over properly now. I'll put the new version of the disk image and transfer program on Github either tonight or tomorrow.
Now it's just a matter of finishing the program that patches all the source files (I'm pretty close at this point), and then people should be able to copy everything over to their Lisas and start compiling without having to make any changes to stuff!
-
Okay, the new version of the transmitter program and the disk image that preserves the high bit is now on Github, along with a script called patch_files.py that applies all the necessary patches to the source files. So after running patch_files.py on your copy of the code and running the transfer script to copy it all over to the Lisa, it should theoretically build without any additional effort.
I've manually verified that all the patches get applied properly, but I haven't had the chance to do a full transfer of all the source files to make sure that no bugs pop up during the transfer or build. But I just started a full transfer about 10 minutes ago, so hopefully I'll have some answers in about 30-35 hours.
-
... the disk image that preserves the high bit is now on Github ...
For those that want to use the disk image with an X/ProFile, see Topic: XProFile convert ProFile disk image Tool (https://lisalist2.com/index.php/topic,663.0.html)
For those that want to use the disk image with LisaEm, I think someone reported that libdc42 was able to convert it to a DC42 image for use with LisaEM (can't find the post... am looking).
Other emulators may just need the image file moved to their media; (if other instructions are needed, let me know and I'll add a pointer here).
-
I think someone reported that libdc42 was able to convert it to a DC42 image for use with LisaEM (can't find the post... am looking)
That was probably me.
You can use the LisaEm raw-to-dc42 command-line tool to convert a raw "profile.image" file to dc42 format, for use in LisaEm. Once you compile LisaEm, the executable is in bin/tools/raw-to-dc42, and the source code is here: https://github.com/arcanebyte/lisaem/blob/master/src/tools/src/raw-to-dc42.c
-
For those that want to use the disk image with LisaEm, I think someone reported that libdc42 was able to convert it to a DC42 image for use with LisaEM (can't find the post... am looking).
Other emulators may just need the image file moved to their media; (if other instructions are needed, let me know and I'll add a pointer here).
Note that I've never been able to get the serial ports working in LisaEm (things always hang for me whenever I try to access them), so unless I'm just doing something wrong, then LisaEm might only be an option after you've used an actual Lisa to transfer all the code over. But I can confirm that LisaEm is completely capable of compiling everything without issue; I actually used it for some really tricky debugging of the OS at one point, and then moved back over to actual hardware once that was done.
The disk image that I provided will work on ESProFile, Cameo/Aphid, and ArduinoFile simply by copying it over to your SD card, and it looks like we have a good solution for X/ProFile now too. Unless I'm forgetting one, I guess the only emulator that's left is IDEFile, although I'm not sure about what sort of disk format it uses or whether it supports images of arbitrary sizes.
My test transfer of all the source code is still going strong (with one minor bug where it incorrectly named LIBQD/TEXT.TEXT as LIBQD.TEXT.TEXT), and I'll be able to try and compile all the newly-copied and patched stuff and report back on the success or failure of that at some point tomorrow. It's looking like our average transfer speed is going to be about 171 CPS, which puts the total transfer time at 34 hours. A decent bit of time, but not too bad given that it's completely automatic!
-
Alright, the transfer ended up taking 34 hours as predicted, but I've had a few issues now that I've tried to compile everything. Mostly just minor stuff relating to my build scripts not taking into account a few dependencies from really early on in my efforts (which I didn't notice until trying to do a clean build now), but I'm fixing those and recompiling as I type this. The biggest problem is that I somehow forgot to put the patches for LisaGuide into the patch program, so I had to spend a few hours putting those in there.
Since I'm still working out these couple issues, I'd suggest not trying to do a transfer of your own until I get them resolved! Which shouldn't be any later than tomorrow.
Sorry for the slight delay!
-
I've finished working out all the issues at this point, except for one really weird one where the newly-compiled SYSTEM.OS fails to boot the Lisa with a 10730 (bad SYSTEM.OS) error. Not sure what's happening here because I'm pretty sure everything is identical to my original working build image, but I'm trying to get it figured out.
I'm at the beach for the weekend, so I might be a bit slower than usual to get this done, but I brought a Lisa with me to keep working on everything. Sorry for the delay; I promise I'll get the final working version of everything onto Github as fast as I possibly can!
-
It turns out that my disk image was corrupted in some way, and that's what was causing the 10730. It compiles and boots just fine on a second image. So I've put all the fully tested and working scripts on Github now.
If you've already downloaded the LOS Compilation Base disk image in the past, be sure to redownload the new version before continuing!
Detailed instructions are in the readme, but here's the gist of it.
1. Run the patch_files.py script with your Lisa_Source directory as the argument, and it'll apply all the needed patches to your source files. Make sure it says it patched them all!
2. Connect a serial cable between your modern computer and your Lisa. If you're using a USB to serial adapter, make sure it supports DTR/DSR handshaking!
3. Run the ALEX/TRANSFER.TEXT EXEC script on your Lisa (hit R and then type "<ALEX/TRANSFER") and wait for the Lisa to switch to the alt console.
4. Run the lisa_serial_transfer.py script with your serial port and Lisa_Source directory as the arguments, and hit enter when it asks if you've run ALEX/TRANSFER.TEXT.
5. Wait for the transfer to complete. It'll take about 34 hours, but it's completely automated, so it doesn't feel too bad.
6. Once it's done, shut down your Lisa and MAKE A BACKUP OF YOUR DISK IMAGE!!! Then boot back up and run the ALEX/MAKE/ALL_NOFLOP EXEC script to build everything.
7. Building everything will take about 8 hours. Times might vary depending on the speed of your ProFile emulator. Make sure to reboot immediately after the build has finished!
Please let me know if you encounter any problems (you shouldn't, but you never know) and enjoy!
-
Please let me know if you encounter any problems (you shouldn't, but you never know) and enjoy!
Thanks for making this available, Alex!
-
+1 Thank you, Alex. Outstanding. 8)
-
Hello,
I'm currently trying to transfer the LisaOS source files to my Apple Lisa using the serial transfer script from this GitHub repo:
https://github.com/alexthecat123/LisaSourceCompilation/blob/main/scripts/lisa_serial_transfer.py
The machine is a Lisa 2/5 with 1MB of RAM.
However, during the transfer, the Lisa crashes with a Level 7 Interrupt and a CHK RANGE ERROR. I've attached a photo of the screen at the time of the crash (see below).
(http://)
I'm wondering if anyone here has seen a similar issue or can help me interpret the crash and register dump. It seems to be failing during a MOVE.W instruction involving A5, possibly trying to read an invalid address.
Any insight or suggestions would be greatly appreciated!
Thanks, Ales
-
So it looks like you're getting an error 643 when trying to read some data from the serial port. I haven't been able to figure out what 643 means exactly, but I do know that the 640 range is reserved for serial port errors, so clearly something's going wrong with the port. And then the failed read is causing something (array index or variable value, not sure) to go out of bounds, leading to the CHK RANGE ERROR.
So we just need to figure out what's causing this read error on your serial port. Have you tried the transfer multiple times, and if so, does it give the error every time? Does it always occur when transferring the first file, or are you sometimes able to make some progress before it errors out?
-
643 is "Unexpected RS-232 interrupt", not to be confused with 647 a.k.a. "Unexpected RS-232 timer interrupt". I wonder if the sending computer or the serial adaptor is doing something funny with e.g. hardware handshaking lines.
-
643 is "Unexpected RS-232 interrupt"
Where did you find that? It is missing from a few versions of the various workshop manuals that I examined.
Am testing now with 1MB of RAM, seems ok; but I suppose APLD/PRT.TEXT is reached after quite a few transfers.
-
Where did you find that? It is missing from a few versions of the various workshop manuals that I examined.
Yeah, that's what I'm wondering too! I looked everywhere, including in the source code, and couldn't find it.
Am testing now with 1MB of RAM, seems ok; but I suppose APLD/PRT.TEXT is reached after quite a few transfers.
I've noticed that there can be some variance in how Python walks the directory structure on different people's computers, so it's possible that APLD/PRT.TEXT comes earlier or later in @snua12's transfer than in yours and in mine. In fact, looking at the log output to the alt console, it appears as if it might be their very first file, unless the LisaBug output covered up previous logs.
Assuming I'm right about it being the very first file, then I bet this points to some sort of weird serial configuration issue like @stepleton said. Not sure about Windows, but on macOS and Linux, I've noticed that many of my USB to serial adapters show up as 2 different devices and only one of them supports hardware handshaking. So you just have to try both and pick the one that doesn't break things partways into the transfer.
-
I really appreciate all your advice so far!
I’m running the transfer script on Linux with a PL2303 USB–RS232 adapter. In the past, the internal battery leaked and I had to repair a damaged trace with a bodge wire. I also replaced the AM26LS32ACN line receiver because Port B RxD was completely dead before, so there’s a chance the serial hardware might still have marginal issues.
The failure (error 643: “Unexpected RS-232 interrupt”) happens mostly on certain LisaOS source files. Lowering the transfer speed from 28800 bps to 14400 bps doesn’t help. For testing, I use a directory with only a single file. When I replace the file’s contents with repeated test123 strings to make it large, it transfers fine every time. This suggests the crash might be triggered by specific byte patterns in the data stream rather than file size or total length.
Has anyone seen RS-232 errors caused by certain data patterns on the Lisa, especially when hardware might be slightly out of spec?
-
Maybe I was playing fast and loose :)
To get directly to one of those lists of error codes, I Googled my favourite Lisa OS error code:
1162 "Power is already off"
(I would love to know the circumstances that could raise an 1162; in any case, I could not find an instance of the Lisa OS using it, though it does seem to define a constant called "pwr_already_off" in a few places.)
Anyway, the first Google hit was this 1982 document (http://www.bitsavers.org/pdf/apple/lisa/os/Guide_to_the_Operating_System_5.1_Jun82.pdf), and scrolling up to PDF page 44 brings us to a list of RS-232 errors. I suppose there is a chance that the list is out-of-date by version 3 of the OS, of course --- it's funny that the 643 code is undocumented in later resources like this one (https://bitsavers.org/pdf/apple/lisa/service/072-0085_Level_1_Lisa_Technical_Procedures_Mar85.pdf).
We can also do some sleuthing. The file LISA_OS/OS/source-rs232.text.unix.txt defines a constant called errbase with the value 640, and raises errors offset from that constant in several locations. The two places where the offset is 3 seem to involve copying data out of the "typeahead" buffer, though I find the code pretty difficult to understand at a glance if I'm being completely honest! There is mention of interrupts near the first offset-3 place.
I'll have to stop my investigations there. But if I can suggest a diagnostic experiment to @snua12, what if you do the following. Instead of having repeated "test123" strings, can you try having strings of varying lengths and with runs of spaces of varying lengths in them? I say this because the Lisa's text file format handles these things in interesting ways. Runs of spaces are "compressed" to save space, and the Lisa may need to do extra processing to do that kind of text encoding. Additionally, Lisa text files have a block structure, and a line of an inconvenient length at the right time may force the Lisa to close the current text file block and start a new one --- meaning it might be getting distracted at a critical moment while data continues to stream in. (Citation: PDF pages 37-38 here (http://www.bitsavers.org/pdf/apple/lisa/toolkit_3.0/Package_2_Examples/17_Lisa_Development_System_Internals_Documentation_Feb84.pdf))
If a more interesting "artificial" file can cause this kind of trouble, we might get a clue from it, particularly if the text data (outside of the spaces and newlines) continues to be something boring like "test123".
-
Where did you find that? It is missing from a few versions of the various workshop manuals that I examined.
Yeah, that's what I'm wondering too! I looked everywhere, including in the source code, and couldn't find it.
APLT/REC.TEXT contains this:
if RecUartLen<>0 then {Any UART receives?}
CheckReceive:=True
else
if OnLine then begin
Read_Data (eCode, RefNum, Ord (@RecUartSeq), PaocSize, NrRead, Sequential, 0);
if eCode <> 0 then begin
if eCode = 643 then begin {input missed}
RecUartSeq [NrRead+1] := Chr(ParityChar);
NrRead := NrRead + 1;
end else begin
Close_Object (eCode, RefNum);
Unmount (eCode, UartPath [PortA]);
UartOpen:=False;
OnLine:=False;
GiveAlert (HTerm, AlrtNote, ALUart, SerialName [PortA], TheTitle, '');
NrRead:=0;
end;
end;
RecUartLen:=NrRead;
if NrRead>0 then
CheckReceive:=True;
end;
Is {input missed} a useful comment? I'm not sure.
edit: seeing as it then adds a filler character, I'm supposing this means receive buffer overrun.
Perhaps Ales's computer or python implementation is particularly fast?
-
seeing as it then adds a filler character, I'm supposing this means receive buffer overrun.
Perhaps Ales's computer or python implementation is particularly fast?
The Python program checks the state of DTR before sending each byte, so we shouldn't have any buffer overruns here unless handshaking isn't functioning properly. An easy way for @snua12 to check: does the Python program ever pause transmitting in the middle of a file and start flashing the "Waiting for Lisa to catch up" message at the top of the terminal window? We expect it to pause at the end of a file regardless of whether or not handshaking is set up, but pauses in the middle of larger files (anything greater than about 8K) will indicate that handshaking is working properly.
Runs of spaces are "compressed" to save space, and the Lisa may need to do extra processing to do that kind of text encoding. Additionally, Lisa text files have a block structure, and a line of an inconvenient length at the right time may force the Lisa to close the current text file block and start a new one --- meaning it might be getting distracted at a critical moment while data continues to stream in.
We shouldn't have to worry about the space compression here; don't 100% quote me on this, but I believe that the Pascal reference manual states somewhere that it doesn't automatically do this compression when you write data to a text file. It can read compressed text from a file and convert it to a regular string, but can't take a string and compress it into a file. I guess you have to write your own text file interface routines in order to take advantage of the compression.
The closing and opening of a block messing up timing might be a valid concern, although I'd assume that the SCC would keep us safe in this situation too. I'm guessing it would just keep receiving until its internal buffer filled up (not sure how big that is; I've never read the 8530 datasheet) and then deassert DTR to pause things until the OS finishes opening the new block and comes in to clear the buffer.
My best guess right now is that this is a problem with handshaking. If you're not getting the pauses in the middle of file transfers, then it might be a good idea to probe the DTR pin on Serial B and make sure that it's toggling. If it toggles after about 8K of data and then toggles again 30-45 seconds later, then your Lisa should be asserting it properly. If you're getting the toggling but not getting pauses in the Python program, then something's probably up with your USB to serial adapter.
-
checks the state of DTR
On my setup (not using Python to send data), I observed that toggling CTS was what worked for handshaking, DTR & DSR didn't throttle the data being sent to the Lisa. So I wired DSR from the Lisa to CTS on the USB-Serial adapter, and then handshaking worked in the driver. ie. I no longer needed to watch DTR or DSR, just turn on the driver's hardware handshake.
Runs of spaces are "compressed" to save space, and the Lisa may need to do extra processing to do that kind of text encoding. Additionally, Lisa text files have a block structure, and a line of an inconvenient length at the right time may force the Lisa to close the current text file block and start a new one --- meaning it might be getting distracted at a critical moment while data continues to stream in.
My observation is that:
- the compression of spaces occurs only at the beginning of a line of text (not elsewhere in a line/paragraph)
- uploading using 'copy' from -keyboard or -console, implements that compression, however using Alex's Receive program, the compression does not occur
- the compression (and slowdown) when uploading using 'copy' does not occur while collecting it into the receive buffer, it happens later when when saving the file. I think this is also the point at which text blocks would be arranged so that lines are not split across blocks (assuming that is still the case in Workshop 3.0).
- charcount seems much faster when the file has not experienced compression (such as when saved by Alex's Receive program)
it would just keep receiving until its internal buffer filled up (not sure how big that is)
The 8530 SCC has a 3 byte receive buffer. The pin compatible 85230 ESCC has a 5 byte buffer.
then deassert DTR to pause things
To be clear, the SCC doesn't do the handshaking automatically, the 68k software needs to observe that it is running low on it's own RAM receive buffer and do the handshake. The size, low and high threshold settings are for the block of RAM the Lisa driver sets aside for the receive buffer.
Since the SCC buffer is barely large enough for an interrupt handler to get the bytes before losing some, once in the interrupt handler, it will keep checking for and collecting data, essentially locking out other processing.
-
Reading back over my post, I realized that I said DTR a few times when I meant DSR. Sorry about that! So when you're checking the signal coming out of the Lisa, make sure that you're checking DSR not DTR!
On my setup (not using Python to send data), I observed that toggling CTS was what worked for handshaking, DTR & DSR didn't throttle the data being sent to the Lisa. So I wired DSR from the Lisa to CTS on the USB-Serial adapter, and then handshaking worked in the driver. ie. I no longer needed to watch DTR or DSR, just turn on the driver's hardware handshake.
I noticed the same thing; the serial driver wouldn't pay any attention to DTR and DSR no matter what I did. But I decided to use DSR anyway by manually polling it instead of connecting it to CTS so that people wouldn't have to make custom cables to do their transfers.
To be clear, the SCC doesn't do the handshaking automatically, the 68k software needs to observe that it is running low on it's own RAM receive buffer and do the handshake. The size, low and high threshold settings are for the block of RAM the Lisa driver sets aside for the receive buffer.
Right, I understand that the big RAM buffer is completely managed by the serial driver and not the SCC itself. I just figured that, if the SCC's internal 3-byte buffer filled up and the 68K didn't come along to clear it, that it would automatically deassert DSR to tell the other computer to stop sending data until the 68K has the time to empty the SCC buffer. But I guess you're saying it's not that intelligent!
-
Has anyone seen RS-232 errors...
I've encountered one error 643 in transferring prolly 1.5 times the whole set of files (the set as recommended by Alex); it was not accompanied by the level 7 interrupt.
Since the problem you've experienced is only semi-repeatable (AFAICT), I wonder if it is an interference issue rather than a configuration or data pattern issue.
I'm using quite long cables, maybe 30' overall composed of a couple long ones strung together, a re-wired 3" null-modem adapter, and a DB25-DE9 adapter; not much for 80's RS-232, but for a USB adapter, perhaps not typical usage.
All the shields are connected, and I have all the equipment power connected to a common ground, as well.
If you have a voltmeter, you might disconnect the serial cable and check the AC voltage between the shields, and/or the grounds (pin 7 of the DB25) to see if the two ends of the link are fighting a ground differential.
-
I finally managed to transfer all the files successfully!
The issue turned out to be hardware-related: due to past battery leakage, the G (enable) pin of the AM26LS32ACN was not grounded correctly. After fixing that, I also built a short cable according to the pinout http://sigmasevensystems.com/blumanual.html (http://sigmasevensystems.com/blumanual.html)
Lisa (DB25M) PC (DE9F)
2 Tx > Rx 2
3 Rx < Tx 3
7 Gnd 5
6 DSR < RTS 7
20 DTR > CTS 8
Finally, I had to tweak the Python transfer script to use CTS/RTS flow control instead of DSR/DTR.
diff --git a/scripts/lisa_serial_transfer.py b/scripts/lisa_serial_transfer.py
index 6a2b177..53fb4c1 100644
--- a/scripts/lisa_serial_transfer.py
+++ b/scripts/lisa_serial_transfer.py
@@ -99,7 +99,7 @@ def send_single_file(file_path, filename):
while (byte := source_file.read(1)): # Read the file byte by byte.
state_start = time.time()
# Flow control in Pyserial is broken, so we have to check DSR manually. If it's low, we need to block until the Lisa's done processing data.
- while not lisa.dsr:
+ while not lisa.cts:
# So say that we're waiting for the Lisa in the progress bar.
print_progress_bar(filename, total_files - len(path_list), total_files, source_file.tell(), os.path.getsize(file_path), size, start_time, True)
if time.time() - state_start > buffer_timeout: # And if it's not done processing in buffer_timeout seconds, print a warning.
@@ -138,7 +138,7 @@ def send_single_file(file_path, filename):
print_progress_bar(filename, total_files - len(path_list), total_files, 0, 1, size, time.time(), True)
time.sleep(2)
state_start = time.time()
- while not lisa.dsr: # Wait for the Lisa to be ready for our 'end of transfer' commands, just like above.
+ while not lisa.cts: # Wait for the Lisa to be ready for our 'end of transfer' commands, just like above.
if time.time() - state_start > buffer_timeout:
print('WARNING: Lisa is taking forever to empty its buffer, probably hung!')
log_file.write('\nWARNING: Lisa is taking forever to empty its buffer, probably hung! ')
@@ -181,7 +181,7 @@ def send_single_file(file_path, filename):
# This code executes at the end of a file transfer; it's very similar to the code from the Except above.
# lisa.flush()
state_start = time.time()
- while not lisa.dsr: # Wait for the Lisa to finish processing its data buffer.
+ while not lisa.cts: # Wait for the Lisa to finish processing its data buffer.
print_progress_bar(filename, total_files - len(path_list), total_files, os.path.getsize(file_path), os.path.getsize(file_path), size, start_time, True)
if time.time() - state_start > buffer_timeout:
print('\nWARNING: Lisa is taking forever to empty its buffer, probably hung!')
@@ -259,7 +259,7 @@ sys.stdout.flush()
input('Run the EXEC file ALEX/TRANSFER.TEXT on your Lisa, and hit RETURN on this computer once the Lisa screen goes blank...')
try: # Attempt to connect to the Lisa over serial, and exit with an error if we fail.
- lisa = serial.Serial(port=sys.argv[1], baudrate=28800, bytesize=8, dsrdtr=True, timeout=1)
+ lisa = serial.Serial(port=sys.argv[1], baudrate=28800, bytesize=8, rtscts=True, timeout=1)
With these changes in place, the transfers seem to be now reliable.
Thanks again to everyone here for the advice and pointers — it definitely set me on the right track.
-
Glad to hear it! Be sure to make a backup of your disk now before going any further, and let us know if you have any issues with actually building everything.