meh, been setting breakpoints and trying to break this since Monday night, but not getting very far. Got several stack traces, and found a function call that has a parameter with the "INFORMIX Master Menu" logo string, with it but it's taking too long to do this on an actual Lisa.
Instead, I'm going back to trying to get Xenix working on LisaEm, that way I can use the tracelog to look at the entire run instead. I have some notes in a notebook I took of what subroutines when called will cause the program to quit and what they in turn call and where they succeed vs eventually quit.
The most annoying thing is that :s doesn't step in Xenix adb, rather it's equivalent to :c and continues the run of the program, so at first I was disassembling basic blocks and whenever it was about to do a JSR or BSR or a branch, I'd set two breakpoints and see where it stopped and noted it down on paper. Then repeated again and again and again.
Eventually I found that :e does step properly, and have been using that, only it doesn't output anything at all, so I also have to do ?i and then $r to see the instruction and the register changes between instructions, so it's cumbersome and worse than setting breakpoints. (This is why running it under LisaEm with it's tracelog mechanism in Xenix's single user mode would be easier, and would also be a much better use of my time to get working.)
I also tried to play with simple binaries such as true, false, yes, echo, and cat, and find that some binaries seem to have a base (that is starting address) of 0x100000 and others 0x800000, but it's confusing to find out from adb ahead of time before running what the entry point address is so it's hard to know where to set breakpoints before doing :r (and you can't do :e to step before :r is started). $m doesn't seem to be consistent there but I think the b1 is the base where the code is loaded.
Weirdly, I would have thought that true/false would be less than a dozen bytes, but this isn't the case. Rather they seem to be a few hundred bytes in size where a lot of the startup code is linked in that's not used for anything productive. (You'd only need true/false to return a zero or a one and quit the program.) So these were compiled from C and linked to stdlib. (Xenix uses statically linked libraries, but the linker is a bit smart in that it only links the library functions used by the program. LisaOS on the other hand seems to have all the shared code in memory already and uses various A-line and other traps as well as A5 for iopaslib/graf/os calls.)
While the text segment is sometimes at 0x800000 (for informix it is), strings and other static data are at offset to address 0:
0x888 - serial number string in binary SCOSNHXL000000SBP0
0x921 - "Invalid Serial Number" <<<-- this is the thing we want to set a watchpoint for via :b
0x1b3 - "INFORMIX Master Menu"... <<< - found this in a stack trace. called like so: 0x801cd4(0x1b3)
Instead this excess startup code deals with the argument count, arguments, environ, setup, etc. (I was looking for what traps/syscalls quit a xenix program vs what trap print a string to stdout, etc.) I suspect trap #0 with d0=0 outputs a character, but haven't quite found what terminates a program yet, but likely some other value passed to trap #0. I think what happens is that d0 or a0 are passed as argc, *argv[], and either something else passes *environ right after that or it calculates it and pushes them all on the stack as args to main(). then the __start function eventually calls _main at 0x8000050. It might do some other stuff like setup the stack/heap memory areas before it calls __main... but __main shows up in the stack trace, and __start does not (shows up as ? ? ? ?)
(The linker takes functions like main(int argc, char *argv[], char *env[]) and renames them to _main, while the start function I think is named __start, or maybe it remains just start, but start is part of stdlib and is written in assembler.)
__start basically is a wrapper around main. So after main is called, it takes the return value off the stack and then calls _exit with it, and then calls trap 0 with d0=1 which is the syscall for exit. Since the binary is statically linked stdlib is at the end of the code. So with stripped binaries you could do some pattern analysis and recognize which stdlib functions are linked in by their code.
I haven't done this, but should be theoretically possible to go through the documentation, create a program that calls all the stdlib functions, not strip that binary and get the opcodes for all of them, then use those to find what stdlib functions are linked into a stripped binary, so you can recognize what a stripped binary calls that way, and thus build an strace/truss equivalent to help with reverse engineering.
(This might be project for a future date ofc.) A better way would be to figure out the ar .a and .o formats (possibly using nm) and analyze all of the libraries that way to build code signatures.
A side effect of static linking is that since each program will be of different size, stdlib (and other libs) will be relocated so in one binary printf will have one address, in another it'll be totally different - which is why such an analyzer would be useful and which is why a stripped informix binary is much more annoying. I don't expect the authors to have left helpful function names such as "serial_number_check" - rather they would have obfuscated the names to something else, but knowing what printf is would be very helpful.
As a heuristic you can also assume any functions at the beginning of the binary are program code and towards the end are linked in libraries.
However, having looked at _start, I know that _exit() is at 0x80784a (as that's called immediately after JSR 0x800050 - which I know is _main() ), and can use _exit()'s address to set a breakpoint when it's called. I'm assuming that normal startup in informix will print the banner, and then do a serial number check, and then when that fails immediately call exit rather than return from main, and so I can use $C to see the stack right when exit is called, so I can then disassemble the caller functions and figure out from there what decisions were made to reach exit() and thus be able to find the serial number check. So I'll give this one more shot before switching back to trying to get Xenix to boot on LisaEm.
It went a lot faster with multiplan because that test was very early on in the start up and if I remember correctly, and all the shifting/rotating and multiplying was pretty obvious; and also brand, I think had the symbols linked in so it was easy to tell one function from another and what it does.
So tl;dr going to first try tracing the stack from a breakpoint set on _exit as a last ditch attempt before pivotting back to trying to get Xenix to boot on LisaEm, and then will do a single tracelog of this under LisaEm and see the whole run that way vs the notes I took.
Author
Topic: Re: Xenix adb reverse engineering (informix brand/serial) (Read 16384 times)