How to run Magic Lantern into QEMU?!...

[nikfreak]

Well done and cool stuff. I'll have a closer look once solved some 70D issues. The detailed sd command states from your posted log re-activated my interest in researching sd card related stuff. I would also like to test if I can re-produce the ML menu timeout bug now in QEMU.

[dmilligan]

Thanks for updating to qemu 2.3.0, I can get it to run now on OSX 10.10 (older qemu versions won't compile), well sort of. Things seem to boot:

Code: [Select]

K287 ICU Firmware Version 1.1.1 ( 3.3.7 )
[DebugMsg] (139,5)
ICU Release DateTime 2012.05.14 13:18:25
[DebugMsg] (0,3) [SEQ] CreateSequencer (Startup, Num = 6)
[DebugMsg] (0,2) [SEQ] NotifyComplete (Startup, Flag = 0x10000)
[DebugMsg] (0,3) [SEQ] NotifyComplete (Cur = 0, 0x10000, Flag = 0x10000)
[DebugMsg] (50,3) Magic Lantern Nightly.2015May30.60D111 (91e1ed550ec7 (qemu))
[DebugMsg] (50,3) Built on 2015-05-30 14:50:55 UTC by david@Artemis.local
[*****] Starting task ff02b61c(3d2494) Startup
[DebugMsg] (0,5) [SEQ] seqEventDispatch (Startup, 0)
[GPIO]     at 0xFF012604     [0xC0222004] <- 0x12      : ???
[DebugMsg] (139,5) startupEntry
[*****] Starting task ff012584(0) Startup


Code: [Select]

[*****] Starting task 1fe07970(0) menu_task
[*****] Starting task 1fe09e24(0) menu_redraw_task
[*****] Starting task 1fe1ba88(0) bitrate_task
[*****] Starting task 1fe28180(0) focus_task
[*****] Starting task 1fe291a0(0) notifybox_task
[*****] Starting task 1fe2c124(0) fps_task
[*****] Starting task 1fe34244(0) shoot_task
[*****] Starting task 1fe2cd8c(0) clock_task
[*****] Starting task 1fe3b1bc(0) audio_common_task
[*****] Starting task 1fe43770(0) livev_hiprio_task
[*****] Starting task 1fe4245c(0) cls_task
[*****] Starting task 1fe452f0(0) beep_task
[*****] Starting task 1fe4d6d0(0) console_task
[*****] Starting task 1fe544dc(0) qemu_key_poll
[*****] Starting task 1fe0d7fc(0) debug_task
[*****] Starting task 1fe1d1e0(0) tweak_task


Code: [Select]

Load modules...
Linking..
Register modules...
Load configs...
Init modules...
Updating symbols...
  [i] 404: dual_iso_get_recovery_iso 1fe1fe00
  [i] 404: dual_iso_is_active 1fe1fe00
  [i] 404: auto_ettr_intervalometer_wait 1fe2cba0
  [i] 404: auto_ettr_intervalometer_warning 1fe2cba0
  [i] 404: auto_ettr_export_correction 1fe389d0
  [i] 404: mlv_snd_is_enabled 1fe3a8f0
  [i] 404: dual_iso_get_dr_improvement 1fe48da0
  [i] 404: dual_iso_get_recovery_iso 1fe48da0
Modules loaded

But I never get a screen image (the qemu window comes up, but it's just black), just this over and over again:

Code: [Select]

[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT

I can send the whole log if it will help

I haven't tried booting Canon firmware yet, I need to get a hold of an SD card small enough I can make an image of it...

[wolf]

@dmilligan
I tried to load the lua module, but nothing did appear in the menu.
Anyway, the QEMU progress is impressive.

[dmilligan]

Did you have any scripts? There is no menu for lua itself, scripts can create or add their own menu items wherever they see fit. You can check the console for script loading/status/errors.

[wolf]

Did you have any scripts? There is no menu for lua itself, scripts can create or add their own menu items wherever they see fit. You can check the console for script loading/status/errors.

Yes.
/qemu/sdcard/ML/SCRIPTS/{CALC.LUA,EDITOR.LUA,SOKOBAN.LUA}
It seems to me that QEMU doesn't "look" for the script folder.
I did run your lua module on my camera and it did work very well.

$grep SCRIPTS log.txt
[FIO wrapper] readdir() => SCRIPTS size=4096 mode=10 time=556420ee

[dmilligan]

IDK, perhaps some FIO routines not emulated, or incomplete

[wolf]

@ dmilligan
How did you compile QEMU version 2.3.0?
I've got 1.6.0.

[dmilligan]

The latest in the qemu branch uses 2.3.0. Just run the install script.

[nkls]

Hello,

I'm trying to emulate 100D with qemu 2.3.0 and I'm running into some issues. I've got it to run quite far but there seems to be multiple issues and I'm not sure which one to try to fix next...

I've attached the log file for the longest I've got it running, and the major changes from what's in the repo is:

- Added addresses for 100D
- No boot flag, just canon stuff
- Asserts aren't fatal, they just print to serial and return to caller (GDB hack)
- DebugMsg prints to log (with color output, GDB/QEMU magic)
- Sequencer events are delayed if being propagated too early, to avoid errors (again some GDB magic)
- Added some SPI EEPROM code
- Grepped out the USB/HDMI/MIC clobber


So some questions:

Is there an SPI EEPROM in 60D?
The 100D firmware tries to load some properties from there and isn't happy with getting zeros. Not sure whether real data is necessary for the emulation, but it seems to be the reason why some asserts are raised.

There are some "unknown spells", should I try to figure out what they do? For the 60D, are the spell responses dumped from the camera or reversed from firmware?

It also complains about I2C and RTC at some points, but I don't think those are fatal errors.

Somehow it crashes to a $pc=4 state which I 'm trying to track down atm, but any thoughts on what to do to make it run further would be very appreciated.

This is the qemu log:
https://www.dropbox.com/s/y92sf7babtf3t2w/100D.log?dl=0

[nikfreak]

a1ex if you read this:
May we expect an update to the qemu branch anytime soon? Sort of having issues like dmilligan encountered since you last updated qemu to 2.3.

Thanks for updating to qemu 2.3.0, I can get it to run now on OSX 10.10 (older qemu versions won't compile), well sort of. Things seem to boot:
....
But I never get a screen image (the qemu window comes up, but it's just black), just this over and over again:

For me it just looks like this:


Qemu 1.6 worked great for 70D but with 2.3 I just get the black graphics shown as posted above whether I try 70D or 100D. For my use case it does the job for porting ROMs to see if they boot up but ofc I am hesitated to see the canon gui emulation which you presented on forums, too. The actual qemu branch also gives me a compile error in file "eos.c" at "eos_tio_handle" where I have to bypass "msg_arg1". So I just hope an updated branch might fix my problems or I simply am going to setup a new virtual machine with an older ubuntu release for qemu only. No rush...

[nikfreak]

Major progress: I'm able to launch Canon GUI under QEMU

A1ex, how about an update to qemu? At it's actual state from bitbucket I always have to uncomment some lines to get qemu compiled. I just set-up a fresh 12.04 Ubuntu but had the exact same error on 14.04 Ubuntu. So quoting your post from May above I assume there's some local work on your build machine which needs to be merged:

Code: [Select]

/home/ml/qemu/qemu/qemu-2.3.0/hw/arm/eos.c:2716:9: error: variable ‘msg_arg1’ set but not used [-Werror=unused-but-set-variable]
cc1: all warnings being treated as errors
make[1]: *** [hw/arm/eos.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [subdir-arm-softmmu] Error 2

And once you read this: I would welcome the dmspy-experiments branch being merged into unified,too.

[a1ex]

Updated to run the Lua module on 60D.



@nikfreak: I'm not getting this error about msg_arg1 (not even a warning), probably different cflags (I'm always (re-)installing it from contrib/qemu/install.sh script, but on an older Ubuntu). I've fixed that one anyway (it was, indeed, a bug).

@nkls: unfortunately, the MPU spells are valid only for 60D for now.

[nikfreak]

Really strange. As I didn't have the graphical glitch shown below in early days of 70D QEMU porting

The only thing I can think of atm is a wrong conflicting merge of my 70D branch into qemu's as I already tried a new ubuntu installation and experience the same. Any clue when 70D might become part of unified and merged into qemu by you? Porting began Dec 2015 and I feel it's just about time to get it into unified...

[a1ex]

Yeah, the stuff in the QEMU branch expects Canon GUI to do all the display initializations (including color palette).

But that only works on 60D atm...

[a1ex]

Small progress: CF card emulation appears to work on 5D2 bootloader, enough to load autoexec.bin from a CF image file.

The 5D2 GUI isn't working yet though...

[nkls]

I've pushed my changes to qemu to my bitbucket now, it's located at:

https://bitbucket.org/niklastisk/qemu (branches: eos-develop and 100D-testing)

It was a while since I touched the code but I hope parts of it can be useful to someone at least.

Things I recall I've done:

• updated qemu to 2.5.50, I haven't tried to merge with master in a while but it may work with the latest version
• moved all eos related code to /hw/eos and moved sections of eos.c to their own files
• DebugMsg printing, see eos_workspace for how to use it
• tried to get 100D serial flash to work without much success, but the code is there
• probably some other stuff, look at the commits...

[a1ex]

Very nice work! You seem pretty familiar with qemu internals (where I'm struggling).

I'll try to integrate it with the latest branch.

Regarding your questions (sorry, I somehow missed your post, so you probably got the answers meanwhile):

Is there an SPI EEPROM in 60D?

There is an EEPROM that you can dump with this module.

I didn't use this file in my emulation though.

There are some "unknown spells", should I try to figure out what they do? For the 60D, are the spell responses dumped from the camera or reversed from firmware?

They are taken from a startup log, from the dm-spy-experiments branch (CONFIG_DEBUG_INTERCEPT_STARTUP=y in Makefile.user), that contains calls to mpu_send/mpu_recv branch. The code is generated with this script.

They are quite incomplete - just enough to boot the main dialog. I know the buttons are also sent from MPU in the same way, and it's easy to get the pattern for them. I also figured out how to decode some properties, but but most other events are still a mystery to me.

[g3gg0]

@nikls:
i am impressed and happy to see such rev.eng. skills
guessing right that you are using IDA?

[nkls]

Thanks guys, and thanks for the info! I was trying to to get 100D working and I guess I learned a bit about the qemu internals on the way. I wouldn't call myself an expert on the system but I know my way around bigger C/linux projects like this.

I never realized the dm-spy-experiments dumped the mpu calls, I guess I have to make one of 100D when I get the time!

Yeah, I'm using IDA for the decompiling and code analysis.


About that serial flash though, I'm under the impression that it's only available in newer cameras (100D and 70D?), and that it is necessary to emulate them to get to the main dialog in qemu. But correct me if I'm wrong.

In 100D there are references to two flash chips PC28F128M29EWHF and K5C262GATMAF50 which are both 16MB iirc. There is a function "IsAddressSerialFlash" which is used by the property manager (or some flash access abstraction) to determine if the data is stored in flash rom (same as firmware) or serial flash rom (controlled over serial interface). I think I had it setup with DMA, interrupts and whatnot but I never made it to the main dialog. I'll give it another shot sometime but atm I'm too busy with other projects.

Here are some extracts from the factory menu, note the S and V options in FACTADJ and the SIO6 menu for accessing the serial rom.

Code: [Select]

FACTADJ

********** FACTORY ADJUSTMENT MENU VER 0.01 **********
0. Exit from Factory Adjustment
1. Leak Check
2. SDRAM Check
3. ROM Check
4. HDMI Check
5. Video Adjustment
6. ICU Version Check
7. Audio Check
8. Adjustment Data Display and Change
9. Check Flag Display and Initialization
A. ALL Check
B. ALLP Check
F. FROM ID Check
P. Power Domain Check
S. Serial Flash Check
V. Serial Flash Version Check
Z. Input_Device_Unique

S
Serial Flash Check.
346
Select OK or NG

V
SROM Version Check.
4.2.1
Select OK or NG


FROMUTILITY

************ FROMUTILITY MENU Ver 0.09 ************
[Type:346 Body:DC Revision:0.00]
0.Exit from FROM Menu
1.Erase Sector(0x20000)
2.Erase Chip
4.Write from card
5.Write from DRAM
6.Firm   flag 0xF8000000 0x00000000 ON
7.Boot   flag 0xF8000004 0x00000000 OFF
8.UpDate flag 0xF800000C 0xFFFFFFFF OFF
9.Create Boot Disk
A.Exec Program from card
B.ALL Block Unprotect
C.Connect card
G.Memory Dump
I.Write Data
J.Direct Jump
P.Power Domain
S.SROM Menu
U.Firm update
Z.RCbind.bin update

S

**** SROM(SIO6) Menu ****
0.Exit from SROM Menu
1.Erase Chip   0x01000000
2.Erase Block  0x0000F000
3.Erase Sector 0x00001000
4.Write Data
5.Write from Card
6.SROM Dump(SIO Read)
7.SROM Dump(QUAD Read)
8.Get Info
9.Get Version

8
8.Get Info 0x80000346

9
9.Get Version 4.2.1


[a1ex]

Yes, experience with Linux definitely helps. I don't have that one either.

I tried to run your serial flash dumper on 5D3, but the serial flash appears unused here. There are no mentions of it in the startup log, and calling SF_CreateFlash locks up. Didn't try to debug further. The 5D3 firmware has these strings: PC28F128M29EWHF and MBM29LV640EBXEI70G.

There are plenty of serial flash messages on the 70D startup log, and from the errors, I think they are needed for emulation. Fortunately, the communication protocol doesn't seem too complicated.

70D startup log says:

Code: [Select]

[SF] InstallSerialFlash 4 0xc022002c 0x0 0x800000 1

and these strings are present: PC28F128M29EWHF and 64Mt8Kx8m64Kx126b8Kx8.


Quick question: when emulating the factory menu, how are you entering the input selection?

[nkls]

The chip install code is run quite early in the init task, this is how it looks in IDA:


As you can see there is a flag (derived from FROM) which determines what chip to install. On a second thought, those routines must be for setting up writing to main flash.

The function call before "SerialFlash Initialize" is definitely for SROM though, I guess there would be some similar call in 70D but not in 5D3? Maybe the 5D3 has more space in main flash memory and thus no need for it?
In 100D the output is

Code: [Select]

[SF] InstallSerialFlash 6 0xc022c0d4 0xf0000000 0x1000000 1
so it's similar.

I think I've got most of the protocol working, take a look in this file in 100D-testing. It gets messy once you try to emulate DMA since it is shared with the SD memory transfer.


Quick answer: (too much of a hack to push to bitbucket)

Code: [Select]

unsigned int eos_handle_tio ( unsigned int parm, EOSState *s, unsigned int address, unsigned char type, unsigned int value )
{
    unsigned int ret = 1;
    const char * msg = 0;
    int msg_arg1 = 0;

    switch(address & 0xFF)
    {
        case 0x00:
            if(type & MODE_WRITE)
            {
                if((value == 0x08 || value == 0x0A || value == 0x0D || (value >= 0x20 && value <= 0x7F)))
                {
                    printf("\x1B[31m%c\x1B[0m", value);
                    return 0;
                }
            }
            else
            {
                return 0;
            }
            break;

        case 0x04:
            msg = "Read byte: 0x%02X";
            msg_arg1 = s->tio_rxbyte & 0xFF;
            ret = s->tio_rxbyte & 0xFF;
            while (ret == 0 || ret == EOF) {
                ret = getchar();
            }
            return ret;
            break;
       
        case 0x08:
            /* quiet */
            return 0;

        case 0x14:
            if(type & MODE_WRITE)
            {
                if(value & 1)
                {
                    msg = "Reset RX indicator";
                    s->tio_rxbyte |= 0x100;
                    return 0; /* nkls: quiet */
                }
                else
                {
                    /* quiet */
                    return 0;
                }
            }
            else
            {
                if((s->tio_rxbyte & 0x100) == 0)
                {
                    msg = "Signalling RX indicator";
                    ret = 3;
                }
                else
                {
                    /* quiet */
                    return 3; /* nkls: signal something on the line */
                    //return 2;
                }
            }
            break;
    }

    io_log("TIO", s, address, type, value, ret, msg, msg_arg1, 0);
    return ret;
}
is there a hidden menu option in other firmwares as well?

[g3gg0]

one problem of qemu was, you couldnt emulate real flashes. (not SPI)
to write properties, the firmware is issuing flash state machine commands (writes to the flash base address)

QEMU cannot handle (passing writes to a driver) this. at least the version of 1.5 years ago wasn't able.
IIRC this has to do with the fact that the ARM instruction set isn't interpreted, but somewhat translated.

so it might be troublesome to get the whole camera hardware behave correctly using QEMU, if there is no workaround.

[a1ex]

@nkls: I think I've managed to integrate your code with my latest changes. It was a little difficult, and I'm not yet sure I did the merges right, so may I ask you to check whether it's still working fine on your side?

The DebugMsg hack is pretty cool - I just managed to boot the 60D GUI with full debug messages, without any additional guest code (no autoexec.bin loaded).

Didn't try the other stuff yet.

[a1ex]

Now that @nkls introduced me to the world of GDB scripting, here's a quick tip for tracing various functions around the firmware:

Code: [Select]

# task_create
br *0x98CC
commands
  silent
  printf "task_create(%s, prio=%x, stack=%x, entry=%x, arg=%x)\n", $r0, $r1, $r2, $r3, *(int*)$sp
  c
end

br *0xFF12CB14
commands
  silent
  printf "SearchFromProperty(%x,%x) from %x\n", $r0, $r1, $pc
  c
end

I also tried to implement the DebugMsg hook with pure gdb scripting, but I got stuck because gdb wants the exact number of arguments to the format string. For example, this works for a DebugMsg call with a single % in it, but fails otherwise.

Code: [Select]

  eval "printf \"[DebugMsg] (%%02x,%%02x) %s\n\", $r0, $r1, $r3", $r2

Any hints would be welcome.

[nkls]

Nice to see things pushed to the main repo and that you've sorted out the open ends I left in there! It seems to work and looks correct, so I think it's all good.

I remember trying to get the debug messages showing through pure gdb for a while but then gave up and hacked it together in C instead. It might be possible to some extent but I bet it'll be ugly. Maybe count %-signs and branch to different cases?

The gdb-qemu link isn't ideal either, the current:

Code: [Select]

set *0xCF999001 = *0xCF999001triggers the IO-handle when it reads from *0xCF999001 but not when it writes to it and I can't figure out why. Not that it matters much, but it would be nice to be able to send values to qemu through a single address.