The LOS and Workshop file systems have been "decoded"

[TorZidan]

I caught the "LOS software developer" fever from the community, and decided to complete a personal project that has been hanging unfinished for many years.

Here it is: https://github.com/TorZidan/LisaFileSystemTool/

I used elbow grease and AI to decode the LOS & Workshop file system format, by peeking into the LOS open source code, and via many trials and errors on many different disk image files.

Unlike previous attempts by others, everything is complete and well documented. Note: the "add file to a disk image" feature is experimental, works only with LOS 1.0  and 2.0 disk images, for now.

Section https://github.com/TorZidan/LisaFileSystemTool/#59-protected-files-theft-protection-and-removing-the-protection-deserialization (protected files) is particularly interesting.

UPDATE:  the "add file to a disk image" feature is fully implemented and works on all versions of the file system. It can add both binary and text files properly. Needs more testing....

I am really impressed with the PI coding harness and the Qwen 3.8 AI model running locally on my video card. It decoded the B-tree "add file" implementation from the Lisa OS sources (Pascal and 68k Assembly code) , came up on its own with ways to test the "addfile" functionality, by finding disk images on my PC if the necessary file system version, copying it into /tmp, adding a file to it, then dumping the file and comparing it with the original. It even found somehow the LisaEm binary on my PC and launched it, but got stuck at the boot screen.

Enjoy!

[bmwcyclist]

FANTASTIC! Thank you very much!

[stepleton]

That is impressive.

Would there be any chance of organising this code into a few separate modules? All in one file is convenient, but it looks like there are some handy routines here that would be nice to use from other programs. You could have a module for disk image I/O, a module for filesystem metadata, routines for traversing the FS, routines for retrieving and creating files, etc.

Either way, one thing that seems like it could be fun would be some visualisation of the sector bitmap. I'm imagining something that resembles the diagram of occupied sectors in a disk defrag tool...

Finally, now that you've got a way to write to files: can you write a script that deposits a compilable set of Lisa Office System source code files onto a hard drive image? Would be a real stress test!

Cheers,
--T

[TorZidan]

Would there be any chance of organising this code into a few separate modules?

The code is split into two python files: the "add file" related code is in the 2nd file, and uses (imports) routines from the 1st file). If you have something else in mind, send me a pull request, I will accept your changes.

Finally, now that you've got a way to write to files: can you write a script that deposits a compilable set of Lisa Office System source code files onto a hard drive image? Would be a real stress test!

Yes, this has been my end goal. The script is ready at https://github.com/TorZidan/LisaFileSystemTool/blob/main/LisaSourceCompilation/upload_files.sh . It works great and adds successfully 858 lisa source files.
The files look great in Workshop File Manager and the Workshop Editor. However, compiling them fails "midway" (after hundreds of files are being compiled), and LisaEm crashes and exits into the "debugger". I will figure it out eventually. It could be a LisaEm bug...

[sigma7]

everything is complete and well documented

UPDATE:  the "add file to a disk image" feature is fully implemented and works on all versions of the file

Wow, this is really tremendous development - well done!

... AI model running locally on my video card.

Was your use of AI entirely self-hosted, or did you need to use some big/cloud AI too?

If self-hosted AI is sufficient for LOS development, I'd like to do something similar. Do you have advice on what hardware & software I should collect to set it up?

[TorZidan]

Was your use of AI entirely self-hosted, or did you need to use some big/cloud AI too?

Yes, my AI setup is completely self-hosted.
Here is my PC hardware:
- CPU:AMD Ryzen™ 5 7600X × 12
- RAM: 64 GB
- Most importantly, the GPU: NVIDIA GeForce RTX™ 4090 with 24 GB of video RAM  <<< This is most important, the more the better.
- well
- 1000W power supply (the video card uses ~400w when "thinking")
- a PC case with plenty of fans to dissipate that heat.

And my software setup:
- The latest Linux ubuntu
- llama.cpp is the "inference" engine: it's at https://github.com/ggml-org/llama.cpp , compiled from sources with flags for "use the Nvida GPU", find details online.
- The Qwen3.8-27B AI model by AliBaba is "open weight" which means "free to download as use", most likely.
- I run it like this:
  export LLAMA_CACHE="~/ai-models/unsloth/Qwen3.8-27B-GGUF"
  ~/github/llama.cpp/build/bin/llama serve \
  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL \
  --host 0.0.0.0 \
  --port 8081 \
  --ctx-size 129000 \
  --parallel 1 \
  --gpu-layers all \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --batch-size 2048 \
  --ubatch-size 512 \
  --no-mmproj \
  --jinja \
  --spec-type draft-mtp \
  --spec-draft-n-max 2

- the "PI" coding harness from https://pi.dev/ . It's basically a command line chat tool where you tell it what to do and it communicates with the AI model.

- Visual studio code (vscode) as the text "editor". The pi coding harness does not integrate in it (it's a separate command line tool), but I am used to that. There are many other harnesses that run as vscdode "extensions", so you can talk to the AI from a side bar in vscode.

My recommendation: buy an Nvidia (not AMD) GPU, either the "4" series like mine, or the latest "5" series. 24 GB is the minimum video RAM. Things are expensive with the recent RAM shortages...

And there is another route which I can't comment on as I have no experience with: a MacBook Pro with e.g. 64GB of RAM; the RAM is shared between the CPU and GPU (a unique feature), which makes it usable for AI work.

How fast is it?
It shows its thinking, and I cannot keep up reading it. So it is acceptably fast, and completes small tasks in under a minute.

How bad is it? The "context" size of 129000 (see it in the command above) is the maximum it can use for my 24 GB of video RAM. It is not sufficient for huge coding tasks: it may "run out of context", and before that it will frequently "compact the context", which slows it down considerably. Splitting a large task into smaller ones and given them one at a time, and telling it exactly you want it to do helps here.

Qwen3.8-27B is versed in all programming language, including Pascal.
It is fun to watch it "think", write and run temporary tools that it uses to verify things, etc.

For example, today I was investigating why the Lisa OS compilation corrupts my SYSTEM.OS file (which results in a boot error 10730 that Alex also observed when compiling the sources). It  found a bug in the Lisa OS source files that sometimes results in file corruption when one file is being overwritten with a longer-size file. Excerpt:

```
The bug:                                                                                   
 The disk (non-VM) branch of the middle-pages loop (lines ~2261–2294) has no such patch at all. It just does the contiguous/non-contiguous bookkeeping, nextviamap, and sets the new request's pflink := redlight — but nothing ever goes back and       
 patches the old leof page's forward link. So when all middle pages go through the disk path and the write ends exactly on a page boundary (lastpage = 0, patch site 3 is skipped), update_link is set but never consumed, and the old leof page's tag   
 keeps fwd = redlight (END). The tag chain is silently truncated at the old EOF — while the smallmap, bitmap, and MDDF are all updated correctly by other code paths, so the volume looks consistent.         
The fix is in place. /tmp/source-fsprim_fixed.text.unix.txt is byte-identical to the original except for one 7-line insertion in fileio's middle-pages loop, at the top of the disk (non-VM) branch (line 2260):                                                                
pascal                  
      end  (* of vm buffer access *)                          
      else       
      begin                               
  +    (* in case other middle contig pgs. need to link to this pg *)                              
  +    if (currelpage > lastrelpage) and update_link then    (* connect lastrelpage to new pages *)
  +    begin
  +      nextioreqp^.pflink := curabspage; (* writing a new pflink *)
  +      update_link := false;            (* link has now been updated *)
  +    end;
  +    
        if curabspage = nextcontig then
        ...
```

Find more resources at https://www.reddit.com/r/Qwen_AI/

[sigma7]

NVIDIA GeForce RTX™ 4090 with 24 GB of video RAM  <<< This is most important, the more the better.

Looked at today's pricing of GPUs with even more VRAM... could be the worst sticker-shock I've ever experienced!

Many thanks for your list, very helpful. I'll probably ask again for an update when I win the lottery or have otherwise recovered my (bank) balance.