But what I don't get is how video mode/liveview can suck up so much memory on the 5d3 and 6d, there's more then 100MB taken by liveview.
You have to remember ML is operating in the margins. Why would Canon give the camera extra memory that it didn't need? It's a miracle there's any memory for ML to use at all. There are many reasons LV would need memory, and lots of it (want specific reasons, ask a Canon engineer, or do some reverse engineering). Also, just because memory is allocated, doesn't mean it's actually being used, it just means its reserved (and we can't use it). So LV is 'reserving' this memory not necessarily using it (but who knows, it might be using it).
So can anyone tell where the "memory hack" function in the raw video menu get's it's extra memory from ?
It just gets it from shoot_malloc, like the rest of the memory it uses, it just calls shoot_malloc under different circumstances. Here's basically how the hack works (to the best of my understanding from looking at the code). Before allocating memory with shoot_malloc (which is some function similar to
malloc that's in the Canon firmware), LV is paused. Then we call shoot_malloc, then LV is resumed. Because LV is paused there is more memory available to shoot_malloc (why? ask a Canon engineer), we take it, and then LV doesn't seem to mind that it's gone.
I specifically put the srm_malloc call outside of the memory hack pause in my changes to raw_rec and mlv_rec (i.e. it doesn't try to malloc while LV is paused) for safety reasons -> I didn't want srm_malloc to steal memory that LV actually needed.
What would happen if you did? It might not make any difference, you might get more srm, or you might just crash the camera or err70. If you're feeling brave, move the srm_malloc_suite(0); call next to the shoot_malloc_suite(0); and find out.