Print

[AlexTheCat123]

While I'm at it, I think I'm going to try and write another Python script that automatically applies all the necessary patches to the source code as well. So stay tuned for any progress on that...

[sigma7]

it looks to me that these files will not need editing if uploaded with 8 bit characters (Aside from dropping the last character in each file) ...

my program (and any terminal software that I've used for testing) refuses to send 8-bit bytes. Although I'm wondering if it's something on the Lisa end where it just strips the top byte without consulting you at all. Have you experienced the same problem in your testing?

If I upload with -CONSOLE (so characters are echo'd back), I can see that the extended character set is being returned, so it looks like the high bit is making it through the serial driver, but then stripped when saved to disk.

Potential ways to deal with this:

• Fix it in post (on the Lisa end) as originally suggested
• Change the constants from characters in a string to hex values not in a string
• Create an exec file to automate the fixes on the Lisa end
• Patch the runtime software

Thoughts?

[stepleton]

Hail Mary: is there a chance that the Concat utility is usable here and better-behaved than Copy?

[sigma7]

I tried "translit" to see if it might yield a speed improvement (it didn't), but haven't checked if it or concat is more tolerant of high ascii.

Good news (I think): I've seen a substantial speed boost by setting a larger buffer in portconfig.

It wouldn't accept a value larger than 32767 as valid, and claimed there wasn't enough memory to do that or 16k when asked to commit the new values, but it would do 8192.

I left the low threshold at the default and set the high threshold to 8000. I presume that's the point it toggles the handshake line.

With those settings (and 9600 baud) I saw 600 CPS for small files, but files larger than the buffer still end around 100 CPS.

[AlexTheCat123]

If I upload with -CONSOLE (so characters are echo'd back), I can see that the extended character set is being returned, so it looks like the high bit is making it through the serial driver, but then stripped when saved to disk.

Same here. I'm not super familiar with the inner workings of the 8530 SCC, so I might be completely wrong here, but perhaps the echo is handled in hardware, which is why the characters are echoed just fine, whereas all the software is written to ignore the high bit? Not that the exact reason really matters though.

Hail Mary: is there a chance that the Concat utility is usable here and better-behaved than Copy?

Good idea. I'll try that as soon as I can!

Good news (I think): I've seen a substantial speed boost by setting a larger buffer in portconfig.

Oh wow, that's awesome! I always figured that choosing that option would just cause it to automatically adjust the buffer size, but being able to set it manually is really great. This also led me to discover that handshaking wasn't working properly on my USB to serial adapter, so I think I might be able to get a bit more speed out of my Python program once I get that figured out.

[AlexTheCat123]

Okay, I've made some changes to my code, and I'm now able to get about 120 CPS (average, including the breaks when the buffer gets full) for large files and 400-ish for smaller ones that don't fill up the buffer. It also turns out that automatic handshaking is completely broken in the Pyserial library, so I have to manually poll DSR and stop transmitting when it goes low. But that's not a big deal at all now that I've figured it out.

The new version of the program isn't quite ready to be put on Github yet, but I'm hoping to have it up by the end of the day!

Also, I just tried out CONCAT, and unfortunately it doesn't preserve the high bit either. So we'll need to find another solution there.

Progress is coming along pretty well on the auto-patch program too, so hopefully I'll be able to release that by the end of the weekend. That way, the only patches that would need to be done manually (unless we find a way to automate them too) are the corrupt character ones on the Lisa end.

[stepleton]

I wonder if it may be time to DIY a transfer program on the Lisa side. It is at least possible to send 8-bit data from a program running on Lisa OS over the serial port; the YMODEM implementation I made requires it, so the problem does not lie in the Lisa OS serial driver or in the OS libraries. But setting YMODEM aside, I've mooted a wire "protocol" for text file transmission upthread, which Alex's python program could reproduce with minor modification; the major work of course is on the Lisa side.

I barely have time to tie my shoes these days (after-market elastic drawstring shoelaces for the win), but if I were doing this I would start from LIBPORT.TEXT and write a function called TimedReadLine that reads "\r"-terminated lines from the serial port (the few string library functions starting from PDF page 142 here may help).

From there on it's a matter of reading lines and then writing them to files, I should think.

The largest source file (APDM-DESK.TEXT.unix.txt) is under 256k, so we might even consider loading all incoming files directly into RAM, provided it's not difficult for Workshop programs to allocate so much memory. Storing file data in RAM should be about as fast as it gets, and that would maximise throughput. But this would be something to fix only after a simpler approach was working, I think.

[sigma7]

it may be time to DIY a transfer program on the Lisa side

A transfer program could certainly help a lot with the speed.

From looking at source code and dumping some bytes from text files, I think the issue is that the system is designed around an "international text" concept so input text gets filtered before being stored in a file in what looks vaguely like a precursor to UTF-16.

It may be possible to pre-code the highbit characters or perhaps implement a base64 decoder or something like that since there aren't a lot of files that need that support.

Still, it may be easier to ( have Alex ) change the high bit characters to hex values instead of string constants.

[AlexTheCat123]

I have a suspicion that maybe PORTCONFIG is the problem, and that it always sets the port up in 7-bit mode given that it never asks you what you want. So I think the next thing I'm going to do is to write a simple Pascal program that forces the port into 8-bit mode. Hopefully that fixes things. If not, then I guess it's got to be something with COPY.

And by the way, the new faster version of the transfer tool is on Github now.

Edit: It looks like my suspicion was unfounded. After looking at some documentation, it seems like the driver automatically chooses 7 or 8 bits depending on whether you enable or disable parity, which PORTCONFIG most certainly allows you to do. I think I'm going to try and write a simple program to replace COPY that actually supports 8-bit bytes, and hopefully that'll fix the problem.

[AlexTheCat123]

Good news, I figured it out!

It turns out that it has nothing to do with the OS, the driver, PORTCONFIG, or COPY being weird and intentionally deciding to strip the high bit, but instead it's an annoying quirk of LIBPL (the Pascal runtime library). Whenever you use the LIBPL routines to open the console for reading and it detects that the console has been mapped to the serial port, it automatically assumes that you're using the high bit for parity and discards it. The solution here is to use the OS routines (look in LIBOS/SYSCALL) to open and read the console, since these just give you the data untouched.

I've written a replacement for the COPY program that uses these discoveries to preserve the high bit, so we should be able to send all the Lisa characters over properly now. I'll put the new version of the disk image and transfer program on Github either tonight or tomorrow.

Now it's just a matter of finishing the program that patches all the source files (I'm pretty close at this point), and then people should be able to copy everything over to their Lisas and start compiling without having to make any changes to stuff!

[AlexTheCat123]

Okay, the new version of the transmitter program and the disk image that preserves the high bit is now on Github, along with a script called patch_files.py that applies all the necessary patches to the source files. So after running patch_files.py on your copy of the code and running the transfer script to copy it all over to the Lisa, it should theoretically build without any additional effort.

I've manually verified that all the patches get applied properly, but I haven't had the chance to do a full transfer of all the source files to make sure that no bugs pop up during the transfer or build. But I just started a full transfer about 10 minutes ago, so hopefully I'll have some answers in about 30-35 hours.

[sigma7]

... the disk image that preserves the high bit is now on Github ...

For those that want to use the disk image with an X/ProFile, see Topic: XProFile convert ProFile disk image Tool

For those that want to use the disk image with LisaEm, I think someone reported that libdc42 was able to convert it to a DC42 image for use with LisaEM (can't find the post... am looking).

Other emulators may just need the image file moved to their media; (if other instructions are needed, let me know and I'll add a pointer here).

[TorZidan]

I think someone reported that libdc42 was able to convert it to a DC42 image for use with LisaEM (can't find the post... am looking)

That was probably me.
You can use the LisaEm raw-to-dc42 command-line tool to convert a raw "profile.image" file to dc42 format, for use in LisaEm. Once you compile LisaEm, the executable is in bin/tools/raw-to-dc42, and the source code is here: https://github.com/arcanebyte/lisaem/blob/master/src/tools/src/raw-to-dc42.c

[AlexTheCat123]

For those that want to use the disk image with LisaEm, I think someone reported that libdc42 was able to convert it to a DC42 image for use with LisaEM (can't find the post... am looking).

Other emulators may just need the image file moved to their media; (if other instructions are needed, let me know and I'll add a pointer here).

Note that I've never been able to get the serial ports working in LisaEm (things always hang for me whenever I try to access them), so unless I'm just doing something wrong, then LisaEm might only be an option after you've used an actual Lisa to transfer all the code over. But I can confirm that LisaEm is completely capable of compiling everything without issue; I actually used it for some really tricky debugging of the OS at one point, and then moved back over to actual hardware once that was done.

The disk image that I provided will work on ESProFile, Cameo/Aphid, and ArduinoFile simply by copying it over to your SD card, and it looks like we have a good solution for X/ProFile now too. Unless I'm forgetting one, I guess the only emulator that's left is IDEFile, although I'm not sure about what sort of disk format it uses or whether it supports images of arbitrary sizes.

My test transfer of all the source code is still going strong (with one minor bug where it incorrectly named LIBQD/TEXT.TEXT as LIBQD.TEXT.TEXT), and I'll be able to try and compile all the newly-copied and patched stuff and report back on the success or failure of that at some point tomorrow. It's looking like our average transfer speed is going to be about 171 CPS, which puts the total transfer time at 34 hours. A decent bit of time, but not too bad given that it's completely automatic!

[AlexTheCat123]

Alright, the transfer ended up taking 34 hours as predicted, but I've had a few issues now that I've tried to compile everything. Mostly just minor stuff relating to my build scripts not taking into account a few dependencies from really early on in my efforts (which I didn't notice until trying to do a clean build now), but I'm fixing those and recompiling as I type this. The biggest problem is that I somehow forgot to put the patches for LisaGuide into the patch program, so I had to spend a few hours putting those in there.

Since I'm still working out these couple issues, I'd suggest not trying to do a transfer of your own until I get them resolved! Which shouldn't be any later than tomorrow.

Sorry for the slight delay!