Print

[sigma7]

Generously, Alex has provided a ProFile image with the Workshop and his build scripts installed via GitHub: https://github.com/alexthecat123/LisaSourceCompilation

However, since the License to download the Source files prohibits re-sharing them, posting a ProFile image that already contains the source files isn't allowed.

So, to actually build LOS from Source on a real Lisa, each individual has to move the many source files to the Lisa...

While we figure out the best way to do that, please post your ideas and progress here to minimize duplication of effort.

[sigma7]

for future use... I'll put links to the top choice(s) here

[sigma7]

Trying to brainstorm the possibilities, I've collected the following suggestions:

• Using the Workshop's scripting to automate multiple transfers via Serial
• Writing a specific program to transfer via Serial
• Transfer via Floppy using the MacCom function

Alex reported in his VCF presentation that floppy transfers introduced corruption; perhaps solving this problem is a possibility.

It has been suggestion that a method of packing all the necessary files into one, then unpacking on the Lisa is likely to minimize effort.

[sigma7]

Additional thoughts

• Create a ProFile image with all the source files installed, over-write the contents of the source files in the image keeping the file structure otherwise intact, share the image along with some software that will repopulate using the source files on a local machine
• Remap the console to a serial port and simulate typing each of the source files
• Use the file manager to copy from a serial port to a file (same as the Transfer program but maybe easier to script?)

[stepleton]

An option to consider is to do serial port automation for text file transfer by means of a custom program that runs on the Lisa. The program should listen on a designated serial port configured to use 8 bit bytes and whatever speed we think we can get away with. It begins in "idle mode", where if it receives a string like this:

~~~~~~ NEW FILE: APLW/UNITDSP.TEXT ~ LINES: 1397

then it opens the text file "APLW/UNITDSP.TEXT" and fills it with the next 1,397 lines that come over the wire. Once 1397 lines are received, it closes the text file, returns to idle mode, and awaits the next file. This continues until it receives the string ~~~~~~ THAT'S ALL FOLKS, where it quits. LIBPORT.TEXT from my old LisaBBS project might be a help, though it doesn't really have a concept of line-based I/O.

On the other end of the serial cable, a modern computer. Either it uses a modern program which scans the Lisa source code directories and sends them off one by one between the NEW FILE delimiters, or, even simpler: write a program that concatenates all of the source code files with these delimiters into one very, very large single file and send that out the serial port. These programs should both be pretty short, and probably writing it in Python would be quick work for your favourite AI, although no shade if that kind of thing doesn't appeal.

(If it doesn't bother you, here is Gemini's attempt, which looks pretty close to me: https://g.co/gemini/share/b412e38848e1 . It makes use of the pyserial library.)

...

While I was typing this, the idea you (sigma7) shared of having the modern computer just drive the console-remapped Workshop over the serial port seems mighty appealing, requiring no programming on the Lisa side, and it could make use of a lightly-modified version of the AI-made program I linked above. Is there a way on the Workshop side to write data from the console directly into a file?

[note: some edits to fix grammar etc.]

[sigma7]

Is there a way on the Workshop side to write data from the console directly into a file?

One is the "DumpPatch" utility, which (if I'm reading the workshop manual accurately) permits entering data in hexadecimal form, then saving to a specified filename.

"Copy" is another option suitable for ordinary text... console can be the input and output can be pre-directed to a file.

IIRC, one can also "Copy" from a serial port to a file (or vice-versa as one might do to dump a file to a printer), dunno whether this works concurrently with mapping Console to the same serial port, or if there would be any advantage to doing so.

[AlexTheCat123]

I really like the simplicity of just driving the Workshop directly, if we can get that to work! And I think the direct Copy option from the File-Mgr is probably simpler to get running than DumpPatch. Although DumpPatch could be handy if some of the special characters can't be directly typed over serial and would need to be entered in hex instead.

Alex reported in his VCF presentation that floppy transfers introduced corruption; perhaps solving this problem is a possibility.

Just to provide a bit more detail, MacCom would always cut off the start of each file and append some garbage (or sometimes text from another file) onto the end of each file. So it seemed to me like the pointer to the start of the file was getting offset slightly ahead of where the file actually began on disk. Not that this matters anyway, because I think that fully automating the process is absolutely the way to go!

[sigma7]

Some discoveries in working on uploads...

Aside from > and < used for redirecting to/from a filename or KEYBOARD/CONSOLE, the < and > characters shown below are not typed.

To configure the serial port, starting at the main Workshop menu:

Code: [Select]

rportconfig<return>
then type return to go through the settings.

To switch the console to RS-232 port B, starting at the main Workshop menu, ending in the System-Mgr menu:

Code: [Select]

scrby
To switch the console back to main when finished using the serial port, starting at the main Workshop menu, ending in the System-Mgr menu:

Code: [Select]

scmy
To start uploading a file, to be named "filename", starting at the main Workshop menu:
Note: ".text" is appended to the filename unless it already includes it

with echo to the screen

Code: [Select]

rcopy<return>
>filename<return>
without echo to the screen

Code: [Select]

rcopy<return>
<-KEYBOARD,>filename<return>
To end uploading (sometimes one less <ctrl-m> is needed, perhaps patience makes a difference?)

Code: [Select]

<ctrl-n><ctrl-m><ctrl-m>
To run the charcount function on a file named filename, starting at the main Workshop menu:

Code: [Select]

rcharcount<return>
<filename,>-CONSOLE<return>
To interrupt an operation (equivalent of Apple-. or ctrl-c)

Code: [Select]

<ctrl-[>

Text File Compression

The Workshop manual mentions something about compressing the leading spaces on each line in a text file, using a DLE character and a space count character. Presumably this is to not punish indentation styles. I believe line lengths are limited to 255 characters so a single count character is always sufficient.

In terms of the size of a file as reported by charcount vs the size of a file prior to upload, it seems that you can get the numbers to match when:
- the line ending is considered a single character
- the compression reduces the size by the number of leading spaces greater than 2 on each line
- the compressed size is 2 greater than that derived from the two preceding rules

I haven't investigated how the data is actually stored on the disk (eg. is a line with 2 spaces DLE 0x02 or '  '? etc., and why/where are 2 characters added... a 16 bit file length perhaps?)

DLE (aka "Data Link Escape") is 0x10 in the ASCII table.
A line starting with 6 spaces is encoded as 0x10 0x26


I'm closing in on an AppleScript solution using Roger Meier's CoolTerm https://www.freeware.the-meiers.org

CoolTerm claims to be python scriptable too and is available for Windows and Linux. The pacing option that waits for a character's echo seems particularly well suited to this task. YMMV

edit: for clarity... english is hard
edit2: added charcount and copy without echo
edit3: added thoughts about the leading spaces compression in text files

[AlexTheCat123]

That's some great progress! I wonder how high we're going to be able to go with the baud rate? I started experiencing corruption with anything over 1200, but maybe other people will have better results.

[stepleton]

If the proposed method doesn't display text on the screen during the transfer, then it may be possible to use a higher baud rate. I suspect a major reason for the LisaTerminal slowness is because (if I recall correctly) it draws the text to the display while it receives it. If I'm wrong about that, then disregard. If I'm right, then that surely steals a lot of cycles.

The Lisa serial port has hardware flow control capability, which may be a bit more dependable than XON/XOFF flow control. Some USB/serial adapters may not be as good at it, though. I wonder if it would help.

[sigma7]

If the proposed method doesn't display text on the screen during the transfer, then it may be possible to use a higher baud rate.
...
The Lisa serial port has hardware flow control capability, which may be a bit more dependable.

Using the per character echo pacing, I was able to achieve about 39 CPS with what I suspect is high reliability. That would make about 6 days to upload the OS and Apps, but at least it might run unattended.

Instead of using -stdin aka -CONSOLE for the input, one can use -KEYBOARD as the input which doesn't echo the characters and so goes much faster as you say.

Then one has the problem of establishing reliability or at least confidence. Using the leisurely charcount function has potential but is complicated by the fact that apparently the leading spaces on each line in a file are compressed using a DLE character followed by the space count (or something like that). So the character count of a file as uploaded doesn't match the charcount result (which counts the compressed bytes) even when 100% successful.

I do see the hardware handshake toggling when I'm overloading the transfer; so am investigating what the limits are with that.

disclaimer: I'm still pfaffing about, so much of that may be incorrect

[AlexTheCat123]

I know I said that I was taking a break from Lisa coding for now, but the more I thought about this, the more I realized that it would actually be a pretty fun project. So I've started work on a Python program using pyserial that will hopefully be pretty nice too.

It's currently capable of transferring a single file at a time, and logs any transmission errors to a text file for future reference. Now I just need to add the ability to do multi-file transfers and polish it up a bit, and it should be done.

The more solutions we have, the better, and this along with James' AppleScript program should be really effective!

[sigma7]

...really effective!

I've only been able to get a consistent rate of 101 CPS, so I'm thinking a different approach is warranted.

Hopefully you/python can do better!

That's with a prolific chipset USB adapter, I'll look for an FTDI one to try, but I think the bottleneck is the other end.

Still easier than using Transfer and typing filenames in one at a time... respect dude.

[sigma7]

Looking through the Alex's directions for source file corrections, it looks to me that these files will not need editing if uploaded with 8 bit characters (Aside from dropping the last character in each file):

Code: [Select]

APHP/T12ALERT.TEXT 
APLP/T8ALERT.TEXT
APLL/DBCONVERT.TEXT
APLL/T5LM.TEXT
APLT/CONVERT.TEXT
APLT/INIT.TEXT
APLW/T1MENUS.TEXT
APLW/T1ALERT.TEXT
APLD/T4ALERT_MENUS.TEXT
APLC/T3ALERT.TEXT
APLC/LCFILER.TEXT
APLC/APPDIBOX.TEXT
APBG/T2ALERT.TEXT
LIBPR/DWBTN.TEXT
LIBPR/PRALERT.TEXT
LIBQP/UVT100.TEXT
LIBTK/UOBJECT4.TEXT
APHP/HP.TEXT does need editing to fix the following characters; I think this can be done pre-upload by using these values...

Code: [Select]

Diamond: ◊ 0xD7 #215
Divide: ÷ 0xD6 #214
Slash: / 0x2F #47


[AlexTheCat123]

I just updated the repo with a Python script in the scripts directory called lisa_serial_transfer.py and some some new instructions in the Getting the Code Onto Your Lisa section of the readme explaining how to use it. I'm able to get about 60 characters per second out of it, so not as good as James (101 is really impressive!), but still better than what I was initially hoping for. With this script, it'll take a little less than 100 hours to send over all the code.

To make up for the less than optimal transmission rate, it provides detailed status info throughout the transfer, as well as error checking and recovery abilities if the Lisa suddenly stops responding, which seems to happen occasionally if you have a cheap USB to serial adapter.

Let me know if you guys have any problems with it!

Looking through the Alex's directions for source file corrections, it looks to me that these files will not need editing if uploaded with 8 bit characters (Aside from dropping the last character in each file):

Code: [Select]

APHP/T12ALERT.TEXT 
APLP/T8ALERT.TEXT
APLL/DBCONVERT.TEXT
APLL/T5LM.TEXT
APLT/CONVERT.TEXT
APLT/INIT.TEXT
APLW/T1MENUS.TEXT
APLW/T1ALERT.TEXT
APLD/T4ALERT_MENUS.TEXT
APLC/T3ALERT.TEXT
APLC/LCFILER.TEXT
APLC/APPDIBOX.TEXT
APBG/T2ALERT.TEXT
LIBPR/DWBTN.TEXT
LIBPR/PRALERT.TEXT
LIBQP/UVT100.TEXT
LIBTK/UOBJECT4.TEXT
APHP/HP.TEXT does need editing to fix the following characters; I think this can be done pre-upload by using these values...

Code: [Select]

Diamond: ◊ 0xD7 #215
Divide: ÷ 0xD6 #214
Slash: / 0x2F #47


For whatever reason, my program (and any terminal software that I've used for testing) refuses to send 8-bit bytes. Although I'm wondering if it's something on the Lisa end where it just strips the top byte without consulting you at all. Have you experienced the same problem in your testing?