Print

[drybones99]

I recently asked this question at 68kmla.org, but I was told to come here. I was recently able to transfer, using BLU (Basic Lisa Utility), a raw disk image of the Widget hard drive in my Lisa 2/10. However, I have been unable to get this disk image to be bootable in either LisaEm or IDLE. It is at this point that I turn to this forum for any suggestions on how I could possibly get this raw disk image to work in any Lisa emulator.

More information about my attempts can be found here in this thread:
https://68kmla.org/forums/index.php?/topic/59047-is-there-any-way-to-use-a-blu-widget-disk-image-with-an-emulator/

[rayarachelian]

Hi, and welcome!

I've downloaded the widget BLU image from the 68kmla thread, and will take a look at the conversion output. BLU has its own format, which isn't the same as raw, but close.

Currently the two candidates are that I suspect, is it either has to do with interleave or perhaps the widget boot loader is different than the profile one. It's also possible that there's some horrible bug in the blu-to-dc42 tool.

[rayarachelian]

So at first glance the conversion failed, there's not supposed to be this many MDDF's (this is like the superblock).
Pls do me a favor and upload the BLU file and not the dc42 either here if it will let you, or over on 68kmla.

thanks.

Code: [Select]


File size:     10350676
Filename:      3_-_BUILT-IN_PARALLEL_PORT_(10_MB_Widget_Hard_Disk).dc42
Sectors:       19456
Tag size each: 20
Sec size each: 512
Sector bytes:  9961472
Tag bytes:     389120
Image Type:    0 (0=twig, 1=sony400k, 2=sony800k)
Tagstart @     9961556
Sectorstart @  84
Maxtrack:      0
Maxsec/track:  0
Max heads:     0
Calc DataChks: db0b5b93
Img DataChks:  db0b5b93
Calc TagChks:  1406b0aa
Calc TagChk0:  17bd45c2
Img TagChks:   1406b0aa
Sorting tags
Found allocation bitmap block at sector #0027(39)
Found allocation bitmap block at sector #0028(40)
Found allocation bitmap block at sector #0029(41)
MDDF (Superblock) found at sector #0026(  38) fsversion:11
MDDF (Superblock) found at sector #0134( 308) fsversion:11
MDDF (Superblock) found at sector #258a(9610) fsversion:00
MDDF (Superblock) found at sector #258b(9611) fsversion:ffff
MDDF (Superblock) found at sector #258c(9612) fsversion:ffaa
MDDF (Superblock) found at sector #258d(9613) fsversion:ffff
MDDF (Superblock) found at sector #258e(9614) fsversion:ffff
MDDF (Superblock) found at sector #258f(9615) fsversion:9009
MDDF (Superblock) found at sector #2590(9616) fsversion:9009
MDDF (Superblock) found at sector #2591(9617) fsversion:4d4c
MDDF (Superblock) found at sector #2932(10546) fsversion:3900
MDDF (Superblock) found at sector #2933(10547) fsversion:ff3d
MDDF (Superblock) found at sector #2934(10548) fsversion:ffff
MDDF (Superblock) found at sector #2935(10549) fsversion:f3f
MDDF (Superblock) found at sector #2936(10550) fsversion:2002
MDDF (Superblock) found at sector #2937(10551) fsversion:ffff
MDDF (Superblock) found at sector #2938(10552) fsversion:ffff
MDDF (Superblock) found at sector #2939(10553) fsversion:fff0
MDDF (Superblock) found at sector #293a(10554) fsversion:7273
MDDF (Superblock) found at sector #293b(10555) fsversion:fffe
MDDF (Superblock) found at sector #293c(10556) fsversion:4d50
MDDF (Superblock) found at sector #293d(10557) fsversion:fff8
MDDF (Superblock) found at sector #293e(10558) fsversion:08
MDDF (Superblock) found at sector #293f(10559) fsversion:fff8
MDDF (Superblock) found at sector #2940(10560) fsversion:fff0


[drybones99]

I have been unable to attach the file here, getting a "413 Request Entity Too Large" error, despite the file size being well under 16,384 KB, the stated maximum, at 10,109 KB, but my latest post in the thread at 68kmla.org has the original BLU file attached, unmodified from when it was originally created by BLU. It is not in the .dc42 format.

[rayarachelian]

Good news, I found some bugs in the code, it's now producing an image where the tags are correct, however, bad news, attempting to boot from it throws a 10726 error.

I found a couple of bugs in the converter program, and now, looking with lisafsh-tool, I see that the blocks that have the boot loader on the profile match the converted widget drive. I then tried to repair the Widget image using the LOS install disk #1 and it failed to do so with error 200/663, but not sure what that means.

Also tried to boot off a good profile image and attach the converted image on a dual port parallel slot (you can do that by changing the preferences, telling LOS there's a 2 port card in one of the slots, and that, say, the upper port is a profile) but the disk isn't recognized.
 
So likely there's some more stuff needed to get this working.

[rayarachelian]

Hi @drybones99, some better news, I tweaked the tool a bit more and now both tags and data is a lot better, I'm able to run the "dir" command in Lisafsh tool and it's also able to recognize the MDDF, likely this is now working, though I don't have time today to test this in LisaEm due to work obligations.

I'll test this later and if all is good and I manage to fix other bugs I'll push the code to github, and cut a binary release as well.

I've uploaded the converted, xz compressed image to the thread here: https://68kmla.org/forums/index.php?/topic/59047-is-there-any-way-to-use-a-blu-widget-disk-image-with-an-emulator/&do=findComment&comment=631222

[rayarachelian]

To add a bit more to this thread, it does seem that BLU has a slightly different tag format for Widget drives.

I have been able to update blu-to-dc42 so it works fine with ProFiles and was even able to boot LisaEm from a BLU image made of one my Profile drives.

The reason blu-to-dc42 wasn't working was because I started working with a BLU image made from a Widget, as I was attempting to add multi-sector read/write support to LisaEm to fully emulate the Widget specific protocol.

(As an aside, Widgets also emulate ProFiles which is why they're compatible, but LOS knows the difference and will use a Widget fully with the version 2 protocol if the ffffff block returns that version. Simply naming a drive "Widget-10" is insufficient - the protocol version must be changed to signify that multi block transfers are available.)

About a year back Tom Stepleton sent me a Widget image whose tags were off my two bytes when compared to a Profile image.

The only 2/10 machine that I have that has a Widget was behaving very weirdly with an early Floppy Emu, I don't recall if it's version B or A but it has an SD slot rather than an SDHC slot. I can look it up next time I turn it on. It's possible it needs a firmware update, but it was behaving very weirdly and I didn't know whether the Widget was bad or the floppy emu wasn't working, because when I tried to install LOS to that Widget by using the floppy emu, it would always fail during the copy.

However, BLU v0.9 boots up just fine and I was able to read images of the Widget over Xmodem and they transferred just fine. So that was puzzling as the drive works fine in BLU, but not in the LOS installer.

On a whim, last night I restored that weird image I got from Tom with the tags in the wrong place. And I was able to boot the Lisa 2/10 just fine off the Widget. This confirms the image wasn't damaged in the transfer and that the FILE ID tag is indeed supposed to be off by two bytes in the BLU format for Widgets. (And also there's an issue with the floppy emu.)

The floppyemu works just fine with two of my 2/5 machines, so that likely indicates some slight incompatibility with the 2/10. It does seem to boot the installer fine off of Disk 1, but when it tries to copy files to the Widget it fails either on disk 1, 2 or 3. I never got further than that. Likely the floppy boot loader works fine with floppy emu, but once the LOS kernel loads it behaves slightly differently than the floppyemu expects.

The next push of the blu-to-dc42 tool should work for both ProFiles and Widgets, though you wouldn't be able to boot off converted images originating from Widget drives since the boot blocks (and possibly loader) is different. (I would suggest adding them as a 2nd hard drive via a 2-port expansion slot in LisaEm if you need to access their files.)
I'm not 100% sure right now that I've correctly fixed the tags in the blu-to-dc42 when dealing with Widgets yet. I'll need to experiment some more, but likely by then I'll sort this out fully.

There may be a way to copy the boot block+loader blocks off a Profile, I'm not sure, I've not tested that. If that works, I might write a tool for that. I do plan to add Widget support to LisaEm in future releases, but that's a far lower priority right now so it won't be in 1.2.7.

So from this experiment, I infer that:

1. the specific Floppy emu version + firmware has issues on 2/10's (or at least it does on mine, so maybe I/O board issues, but it does work with a real floppy drive) and, more importantly,

2. BLU does deal with Widgets differently.

It's possible, but highly unlikely that Widgets use tags differently than ProFiles, as looking at the ROM source code it expects that the boot AA AA signature is always in tag bytes 4,5.

Edit: So I've used NeoWidEx to read the boot block off the Widget and I notice that indeed bytes 4,5 of the tag bytes are AA AA as expected, so this is confirmed as a BLU feature/bug.


I don't see anything in the PROREAD ROM call that behaves differently if the drive is a Widget or a ProFile.
The only mention of Widget in the ROM source is a change to the timeouts, but that affects all drives.

Code: [Select]

0000| 0000 0004             FILEID          .EQU    4               ; OFFSET TO FILEID
0000| 0000 AAAA             BOOTPAT         .EQU    $AAAA           ;FILEID FOR BOOT PATTERN
...
1ECE|                       ;-----------------------------------------------------------------------------
1ECE|                       ;  Routine to boot from Profile hard disk attached to parallel port
1ECE|                       ;  Sets up input parameters and then calls READ routine.  If no error
1ECE|                       ;  on return (carry not set), a jump to the loaded boot code is done.
1ECE|                       ;-----------------------------------------------------------------------------
1ECE|
1ECE|                       PROBOOT
...
1ED0| 227C 0001 FFEC                MOVE.L  #HDRBUFR,A1             ; set up ptr to save header
1ED6| 247C 0002 0000                MOVE.L  #HDRBUFR+20,A2          ; ptr for data
...
1EEE|                       ;  Now verify header and if OK, jump to startup program
1EEE|
1EEE| 3029 0004                     MOVE    FILEID(A1),D0           ; get file id
1EF2| 0C40 AAAA                     CMP     #BOOTPAT,D0             ; is it a boot block?
1EF6| 6704                          BEQ.S   PBOOT                   ; continue if OK
1EF8| 7054                          MOVEQ   #BADHDR,D0              ; else set error code
1EFA| 6014                          BRA.S   HDSKERR                 ; and exit
...


[sigma7]

I don't see anything in the PROREAD ROM call that behaves differently if the drive is a Widget or a ProFile.

The ROMs load the boot block of the Widget in ProFile fashion, ie. assuming 20 tag bytes then 512 data bytes.

Thereafter the RAM based code treats the Widget blocks as having 512 data bytes followed by 20 tag bytes.

Consequently, the boot block code in the ProFile is different from the Widget.

[rayarachelian]

I don't see anything in the PROREAD ROM call that behaves differently if the drive is a Widget or a ProFile.

The ROMs load the boot block of the Widget in ProFile fashion, ie. assuming 20 tag bytes then 512 data bytes.

Thereafter the RAM based code treats the Widget blocks as having 512 data bytes followed by 20 tag bytes.

Consequently, the boot block code in the ProFile is different from the Widget.

Oh, I see, I didn't realize that this was what was going on with the Widget images, thank you.

I saw something unexpected at the hexdump level tag wise with BLU images, but then somehow downloading a BLU image and then uploading it back always worked. Somehow I saw the tag bytes as being off by two bytes - that is AAAA happening at offset 0x512+2 rather than 0x512+4, but really it's in a totally different location after block zero.

So in reality BLU is treating the blocks as singular 532 byte blocks and doesn't care where the tags are, but it's only because dc42 separates them into two different streams that there are issues.

I'll modify the code to pick up tags differently between block 0 and the rest instead. I was offsetting the tags by two and inserting zeros at the front which is incorrect.

Thanks James.

[sigma7]

So in reality BLU is treating the blocks as singular 532 byte blocks and doesn't care where the tags are, but it's only because dc42 separates them into two different streams that there are issues.

Sorry no, BLU does know (or thinks it knows) where the tags are.

If you directly access the parallel port to read data from a ProFile, for each block you'll find the first 20 bytes are tags followed by 512 bytes data, with the blocks interleaved.

If you do the same with a Widget using the ProFile compatible commands (dunno about the Widget specific commands), for each block you'll find the first 512 bytes are data followed by 20 bytes of tags, with the blocks not interleaved (ie. interleaving is typically done transparently at the drive level).

However (IIRC), when BLU reads/writes hard disk images, it:

• deinterleaves/reinterleaves blocks when the device is a ProFile
• puts/expects tags at the end in each 532 bytes of the image (ie. re-orders tags and data when a ProFile is involved)

The consequence of this is (supposed to be) that the image file always has the blocks in the OS level order, and the tags are always in the same place.

[patrick]

Neither ProFile nor Widget distinguish between tags and data. They just take 532 bytes in consecutive order and store them as one block. Data transfer inside the drive is controlled by a DMA counter. Any meaning behind these bytes comes from the OS. I have no idea why Apple reversed the order for Widget,  but there is definitely no technical reason.

Same applies to IDEfile, but here you can store up to 1024 bytes in a block (two IDE sectors).

[sigma7]

Any meaning behind these bytes comes from the OS. I have no idea why Apple reversed the order for Widget

I suspect the reason was intended to be performance: calculating a checksum to be put in the tags can be done concurrently while bytes are written when the tags come last, but requires an extra scan of the block prior to writing when the tags come first.

... but that's a guess.

[rayarachelian]

So here's why it was confusing.
Hex dump of a profile via blu:

Code: [Select]

00000000  50 52 4f 46 49 4c 45 20  20 20 20 20 20 00 00 00  |PROFILE      ...|
00000010  03 98 00 26 00 02 14 20  00 00 ff ff ff ff ff ff  |...&... ........|
00000020  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
...
000001c0  fd fa 00 00 00 00 00 00  00 00 db 6d db 00 50 52  |...........m..PR|
000001d0  4f 46 49 4c 45 20 20 20  20 20 ff ff ff ff ff ff  |OFILE     ......|
000001e0  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
000001f0  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
00000200  4c 69 73 61 20 48 44 20  49 6d 67 20 42 4c 55 56  |Lisa HD Img BLUV|
00000210  30 2e 39 30]4e fa 00 16  aa aa 08 50 22 4a 2c f8  |0.90N......P"J,.|  <- start of sector after the bracket at 0x214
00000220  28 76 00 1e 00 00 00 00  00 00 00 00 20 38 02 a8  |(v.......... 8..|
...

000003f0  6a 08 10 10 66 04 10 bc  00 02 41 fa fe 18 d0 fc  |j...f.....A.....|
00000400  00 0e 31 d0 02 10 42 a7  42 a7 42 a7 41 fa fe 06  |..1...B.B.B.A...|
00000410  d0 fc 00 08]00 00 00 21  aa aa 82 00 ff ff ff 56  |.......!.......V| <- sector 0 ends at 0x413, tags start at 0x414
00000420  00 00 ff ff ff ff ff ff] 81 a4 00 01 00 03 00 03  |................| < tags end at bracket 0x427, sector 1 starts.
00000430  00 00 00 5f 00 01 ff 00  00 00 00 00 00 00 00 00  |..._............|

But here's the Widget.

Code: [Select]

00000000  57 69 64 67 65 74 2d 31  30 20 20 20 20 00 01 00  |Widget-10    ...|
00000010  1a 43 00 4c 00 02 14 02  02 02 13 00 00 4c 00 00  |.C.L.........L..|
00000020  00 00 00 00 00 01 80 80  00 00 00 00 00 00 00 00  |................|
...
00000200  4c 69 73 61 20 48 44 20  49 6d 67 20 42 4c 55 56  |Lisa HD Img BLUV|
00000210  30 2e 39 30]4e fa 00 16  aa aa 08 50 22 4a 2d 0c  |0.90N......P"J-.| <- sector starts at  0x214 same as ProFile
00000220  28 76 00 1e 00 00 00 00  4e fa 00 16 aa aa 08 50  |(v......N......P|
...
000003f0  02 07 0c 01 06 0b 20 7c  00 00 01 b3 22 7c 00 fc  |...... |...."|..|
00000400  c0 31 10 11 6a 08 10 10  66 04 10 bc 00 02 41 fa  |.1..j...f.....A.|
00000410  fe 18 d0 fc]00 00 aa aa  82 00 ff ff ff 00 00 00  |................| < sector ends at 0x413 tags start at 414 but aa,aa is now in the wrong place
00000420  ff ff ff ff ff ff 21 2f  00 00 00 84 00 00 80 85  |......!/........|

So this is what I was seeing and getting confused by. But yeah, if I flip the sector start and tags around then, it makes more sense. And fun thing here. That's a hack. The tags now start at the right place. But that 4efa is a jmp:

https://onlinedisassembler.com/odaweb/ provides this disassembly for the tag!

The first opcode is a JMP to the start of the sector, that's how and why this stupid thing works. Wow. I mean, why.

Code: [Select]

.data :00000000 1600fa4e jmp %pc@(0x00000018)
.data :00000004 aaaa .short 0xaaaa         # required AA,AA tags
.data :00000006 4a225008 bchg #74,%a0@
.data :0000000a 0c2d movel %a4,%fp@-         
But beyond that, it's still the case that tags for Widget BLU images, but not sectors are shifted two bytes to the left, where the fileid is at bytes 2,3 rather than 4,5 in the tags, so I have to shift them over.
Replacing the boot sector and boot loader with those from a ProFile results in 10726 (boot device not found) when booting, so more is needed to getting booting. I think there's some file somewhere that controls the boot process, and possibly it's different between widgets and profiles. I've seen this before when I've tried to move a ProFile from the motherboard parallel port to a dual parallel card and tried to boot off it, or perhaps it's got a different device path.
I guess a hint to this is that the Preferences panel has a list of device drivers and devices.

Edit: so I extracted both the source profile and the modified widget and found these files are different, gotta be one of these:

Code: [Select]

Binary files CIBTNDATA.bin and ../widget-test.d/CIBTNDATA.bin differ  # has to do with printers
Binary files DWBTNDATA.bin and ../widget-test.d/DWBTNDATA.bin differ   # has to do with printers
Binary files FONT.HEUR.bin and ../widget-test.d/FONT.HEUR.bin differ # fonts I presume
Binary files INTRINSIC.LIB.bin and ../widget-test.d/INTRINSIC.LIB.bin differ # not sure why
Binary files LCORBGLIB.OBJ.bin and ../widget-test.d/LCORBGLIB.OBJ.bin differ
Binary files PARBTNDATA.bin and ../widget-test.d/PARBTNDATA.bin differ # seems to be about device connections - maybe this one
Binary files shell.Office_System.bin and ../widget-test.d/shell.Office_System.bin differ # not sure why
Binary files SYS2LIB.OBJ.bin and ../widget-test.d/SYS2LIB.OBJ.bin differThough some of these might be false positives - Running strings against them, I see shell.Office_System has a slightly different version, one is 3.1 the other 3.0, so that might be another issue here.

[sigma7]

it's still the case that tags for Widget BLU images, but not sectors are shifted two bytes to the left

I suppose that could be a BLU bug  but funny it would take this long to show up.

However, block 0 is special, so don't use it as a template.

When writing a boot block to a Widget using a Widget driver, one sets the first part of the data field (to pretend) to be the tags the ROM would expect from a ProFile, as block 0 will be loaded using ProFile code. The block 0 Tag field will be loaded as the last 20 bytes of the data field, so if the boot code was more than 492 bytes, the excess code would need to be put in the Tag field before writing to the Widget. If the code is not more than 492 bytes, then the Tag field might contain garbage.

The first opcode is a JMP to the start of the sector... why

My guess is that the boot ROM was developed before the ProFile format/tag arrangement was finalized, and someone decided that execution might as well start at the first byte loaded from the disk, then inertia prevented any changes. Or perhaps it was left as an easter egg to annoy Ray.

when I've tried to move a ProFile from the motherboard parallel port to a dual parallel card and tried to boot

I believe (aka imagine) that:

• most environments' boot blocks will only work with the built-in port
• in MacWorks Plus 1.x, Chuck added a boot block version that would work from a dual parallel card*; however it was not interchangeable with the internal port boot block (the version installed depended on where the drive was attached; it is possible this was slot dependent as well)
• in MacWorks Plus II, an attempt was made to have boot blocks that work from dual parallel cards and internal ports, but it was still device dependent (ProFile/Widget)
• in BLU, an attempt was made to have boot blocks that work from dual parallel cards and internal ports, and be device independent (ProFile/Widget)

*It is possible that the boot block for the dual parallel card predated Chuck and was originally written for MacWorks XL

[rayarachelian]

The first opcode is a JMP to the start of the sector... why

My guess is that the boot ROM was developed before the ProFile format/tag arrangement was finalized, and someone decided that execution might as well start at the first byte loaded from the disk, then inertia prevented any changes. Or perhaps it was left as an easter egg to annoy Ray.

Yeah, so, that was a rhetorical why.

The technical why of the JMP opcode is obvious, they why of why swap the location of the tags is more of the question. Most likely it was Macintosh related since Mac OS didn't use the tags*, and there was a push to move to the XL, and doing so would avoid reading an extra 20 bytes per block, ending the transfer a little bit early, thus you can get a 4% speedup and if the Lisa runs at 5MHz, but the Mac runs at 8MHz because it uses a better memory vs display interleave (and possibly faster DRAM), then yeah, you'll want any increase in access you can get.
* it's possible the very early versions of the early Mac OS did use tags, but they were planning on removing the need for them and this drove the decision to move the tags to the end.

What's really interesting to me is that if you've ever really seen a Mac 128 in action, you'll realize how insanely slow it was vs a Lisa. This isn't because of the CPU speed, but rather because of the lack of memory and lack of a hard drive. The Lisa had a huge advantage over the 128, 512k, and 512KE. Only when the Plus with it's 1MB of RAM and SCSI port was released could the Mac really compete against the Lisa.
The early Mac 128 was a complete shit show of floppy swapping, even for normal operations, and those drives were very very slow. You'd need at least a second external floppy to make use of the 128 without a having to swap 5-10x each time you did anything interesting. The 512K could potentially use the HD20, but that hard drive was introduced on September 17, 1985, quite a bit after the April 29, 1985 discontinuation of the Mac XL.
Because of this, a Lisa with a ProFile or Widget was far more advantageous than the highest end Mac available. At least until the Plus with an external SCSI drive. And getting a 4% speedup in disk access would certainly help. It's certainly funny to call the Lisa a speed demon, but a Mac XL 2/10, when compared to the Mac 128, even with the addition of an external floppy drive, it certainly is a lot more like a "Maserati for the Mind", than the Mac could possibly be.