Magic Lantern Forum

Status: experimental, need your testing!

Short description:
io_crypt is a module which automatically encrypts .CR2 and .JPG while you shoot them.
The original file content is never written to card, so there is no way to restore the image content by reading the raw sectors etc.
You can choose between different modes and security levels.
This was formerly discussed there (http://www.magiclantern.fm/forum/index.php?topic=9963.0) and was requested already a few times.

Detailed description:
This module hooks the file-io operations for your SD and CF card and places custom read/write routines instead.
These custom r/w operations encrypt your file content before the card's real write handler is being called.
For you there is no additional task to do after you shot the image - just shoot as usual and your files are encrypted.

There are two possible modes:
- Password
    Before you shoot images, you have to enter a password which is being used for all images
    The password gets fed into a LFSR (Linear Feedback Shift Register) to shuffle the bits and get a 64 bit file key.
    advantage: you can enter different keys, one per "session" or "access level" and share them accordingly
    disadvantage: you have to enter the key every time you power on the camera (storing is insecure of course)

- RSA
    Before you start your shooting, you create a RSA public/private key pair via menu.
    (edit: this takes up to 10 minutes with a 4096 bit key!!)
    Then you copy the private key from your card (ML/DATA/IO_CRYPT.KEY), store it at a safe place and delete it from your card (!!).
    You need the private key only for decrypting (on computer), the public key only for encrypting (on camera)
    With the internal PRNG for every image a separate file key is being generated and encrypted using RSA.
    advantage: no password must be entered, power on and shoot. every image has a different, random "password"
    disadvantage: you have to prepare yourself a bit by copying and deleting the encryption keys correctly

In both modes, the file content is being encrypted using a XOR operation with the output of a 64-bit LFSR that was pre-loaded with the file key and the current block numver.
To make random access feasible and the encryption fast enough, the keys are used blockwise.
This obviously weakens encryption a lot and makes it possible to recover the 64 bit block encryption key using known plaintext attacks.
The good thing - know plaintext attacks are only suitable for file content that has a predictable pattern, like the file header.

Still the encryption i implemented is *not* military grade. Although it is (imho) safe enough for a normal individual.

Options:
    Blocksize
    The block size that is being encrypted with the same 64 bit key.
    larger is faster, but insecure. smaller values slow down saving. choose.
   
    Ask for password on startup
    If you are in Password mode, camera will ask for password right after poweron.
    When disabled, you have to enter the menu manually and set the key - else no pictures will be encrypted.

    RSA Keysize
    Choose the largest value that you can tolerate. The larger the size, the longer generating will take (up to 10 minutes...).
    Also saving will slow down a bit with larger keys


Image review:
Canon caches the images you have shot until you poweroff the camera or the memory gets full (5-10 images).
As long the images are in cache, you can review it without any problem, even if you change the key.

In RSA-Mode you currently *not* review images other than those in cache. Not sure if i will implement it at all.
In Password mode, you can view images when you set the correct password.

Decryption:
After you copied the files onto your computer, you can decrypt it with io_decrypt which is (not yet) avaiable precompiled, but you can get from the repository.

./io_decrypt <in_file> [out_file] [password]

If you want to decrypt password protected files (LFSR64), you have to supply the encryption password on commandline.
For RSA encrypted files, the private key ML/DATA/IO_CRYPT.KEY must be in the current directory.

Compatibility:
The module contains some camera specific memory addresses, so it has to be ported for every model.
Cameras that are supported: 7D, 5D3, 60D, 600D, 650D
Next cameras being added: 5D2, 6D
If you have a different model and want to use/test the module, please post it here.

Disclaimer:
1. Do not do any illegal stuff with it.
2. It is meant for e.g. reporters whose security depends on the footage not being revealed or for securing sensible information
3. Dont rely on it. It will for sure somewhen fail and your footage is gone.
4. Dont cry when something goes badly wrong.


Download:
You can always download my latest build there (http://ml.g3gg0.de/modules/io_crypt.mo/io_crypt.mo)
here (http://ml.g3gg0.de/pub_files/dcf5bed018f76de8e45f624e93d22699/io_decrypt.exe) is the windows console decrypter.


ToDo:
- Show fake images instead of the standard canon error screen
- background encryption for unsupported models. will scan, encrypt and save the images in background while your camera is idle.

g33gg0

I would like to test and use this module. I only have a 650D, and I see that it is not supported. Is there a specific reason ( hardware limitations) for this?

Thanx

its no technical reason - i will look up the addresses and you can try if it works ;)

cool! looking forward

updated, can you d/l and try?

I have no interest in this, but since you're getting no feedback, if you update it to 5D3.123 I'll run it through it paces.

I can do the 5d2 if you can find the addresses. Thanks!!

Hello, World!


My setup:

Cam: 650D
shooting mode: P
IQ: Jpeg + RAW

ML: 06.02.14 nightly
io_crypt: 05.feb. 18:07
ime_base & ime_std:   09.12.13 (ime_base.zip von g3gg0)

blocksize: 128k (most of the time)


in short:

RSA does not work
Password works but also crashes the cam sometimes (but maybe its caused by the ime-modules i also tried for the first time)


in long:

When I'm on "Set password" and press SET or PLAY a failure message appears:
"IME error, Crypto disabled"
(sometimes this warning flickers)
--» needed some more time to realize the need for ime-modules :)

"Creating RSA key (2048 bits)
this may take a while"
(flickers)

Test: Speed      "Test done"
Test: RSA         "finished successfully"

kreating Key: ok
but menu says: "NoKeys!"

in playback after taking a shot:
"No key entered, not encrypting!"
and it really doesn't ;)

when i shut the cam down, it kept displaying sensor cleaning
couldn't hear if it was actually vibrating (noisy surrounding)
Power off & power on
nothing
power off
no display but card-LED kept lighting
removed Battery


battery in & next power on:
everything seems ok
file Manager shows two extra files:
IO_CRYPT.CR2 & IO_CRYPT.DAT (both 20.0 MB)

At this piont i installed the ime-modules.
Password mode worked fine! I had pictures on card encrypted by two different passwords and could also display them after restarting the cam and typing the respective passwords.



Another time i started the cam this happened:
asked me to type a password
i typed it (same password i still have encrypted pics from on the card) and klicked "OK"
nothing happens
i klick OK for another time
IME-screen flickers one time
i klick OK for the third time
camera freezed


Another time:
after some time being idle the display shuts down
thought it would be normal standby but it doesn't react to any button and card-led is constant on.
took battery out
I can't remember in which Menu it has been.


back to the keys:
I tested keygeneration with different keysizes, sometimes in RSA-mode and sometimes in password mode.
I could not find a IO_CRYPT.KEY in ML/DATA/ on any of the two tested cards.
But it always said: "Creating RSA key (1024 bits) this may take a while"
Has there to be a confirmation message when the key is generated successfully?



@all devs:
Thank you for all the impressive features making our cams to entire new systems!

Quote from: g3gg0 on February 02, 2014, 12:36:25 AM
If you have a different model and want to use/test the module, please post it here.

... 6d here, is there any procedure on how to find the addresses? If I'm too dumb for this, I can still forward it to 1% for help :-p

Quote from: g3gg0 on February 02, 2014, 12:36:25 AM
1. Do not do any illegal stuff with it.
2. It is meant for e.g. reporters whose security depends on the footage not being revealed or for securing sensible information

I'd like to note that these two items are often mutually exclusive.

Quote from: eos on February 06, 2014, 07:38:11 PM
@all devs:
Thank you for all the impressive features making our cams to entire new systems![/b]

thanks for testing.

with ask-for-pw enabled, starting in LV/video mode can cause trouble.
so start in photo mode only. dont like that feature anyway and planning to remove it.

about RSA: yeah, found a bug that will most likely cause that problem.

will fix those.

okay can you test RSA on 650D again?

Quote from: g3gg0 on February 06, 2014, 08:26:05 PM
with ask-for-pw enabled, starting in LV/video mode can cause trouble.
so start in photo mode only. dont like that feature anyway and planning to remove it.

Fortunately i had no time to test it in lv or movie. ;)

I think ask-for-password-on-startup is a very helpful feature for forgetful people.
And it isn't anoying if you use one card exclusively for private things.
I don't need it today but i would really miss it in future!
What about leaving it in the comments for times with less features on the workbench?

[650D in M; with 07.febr. nightly & iocrypt from 06.feb 21:00]

disabled sensor cleaning (& ask-for-pw most of the time), formatted card and took a new ML installation (not copying files from zip over existing ML with all its configurations)
Since then i had no stability problems. There was only one display failure (disappearing after next button press; made a photo with my cellphone; can try to reproduce it).

but again RSA:
mode: rsa
test rsa: successful
create rsakey: files for private and public keys are there
                        but rest like yersterday (NoKey! in menu; no key while shooting)

fixed RSA for non-5D3 now hopefully ;)

you now also can drag n drop files to the io_decrypt.exe, it will scan for the "io_crypt.key" from the generation process in camera.
dont forget to delete it from camera.

next step is adding support for the other cameras.

How do I find these addresses... I looked via ram and its somewhere near CF/SD stuff. Nothing seems to link to it directly in FW and when I use memory browser the first one is a pointer to an FF address. The other area is empty.

short:
in cam nice
RSA on pc not working
pw not testet on pc
decypt.exe does not work

long:

[650D; M; 8.feb nightly; iocrypt 8.feb 2:33; io_decrypt.exe 08.02; blocksize 128k]

ask-for-pw-on-startup: on
mode: RSA
keysize: 512
create rsa key
menu says: INSECURE!
key files exist
encryption passed!!!

but on pc:
DEcrypted files not readable (neither jpg nor raw) (only tried drag&drop)

removed io_crypt.key  from card
menu: "Active"
made a photo but could see it after power off/on
did not wait long enough for shutdown/clear RAM
wanted to take pics until RAM full
shot one
seemed to be encrypting
didn't wait
shot another one
error 70 (have a crashlog; but no dump)
battery out/in and extra off/on for modules to load
everything encrypted and fine

create 1024bit key
moved private key to B:/ML (foolish for sure but its only a test)
"Insecure!"
power off/on
"Active"
i dunno: is it me outsmarting the module or is it "The cleverer give in?" (<-- would be a nice message somewhere between/around "Insecure" and "active", but "foolish" would fit better into junkie menu. ;))

created 2048bit key
blocksize 256k
file manager doesn't show a new key

didn't wait long enough for kegeneration.
i think the "Creating RSA key... ...may take a while" didnt show up for a feeled minute. So i thougt it would be done already and i would have missed the confirmation message poppin up for a second.
Can u please make "Creating RSA key... ...take a while" showing up permanent or in equal time slots?
Additional imho the best would be a permanent "done"-message until a half-pressed shutter button.

btw: standby altough without trouble (tested a couple of times)

interesting for fools:
this time private key got moved to B:/. Menu kept showing "insecure" (Even after long shutdown).

.exe crashed with some files
many were 0Byte after decryption
not any working picture
one jpg is 16MB! (before & after decryption)
But didn't try the pw-protected ones per command line (simply tried everyone per drag&drop to save time).

Quote from: 1% on February 08, 2014, 07:34:10 AM
How do I find these addresses... I looked via ram and its somewhere near CF/SD stuff. Nothing seems to link to it directly in FW and when I use memory browser the first one is a pointer to an FF address. The other area is empty.

i answered via mobile phone but it seems the browser didnt submit my post. argh...
ok then again.
(example 7D v2.0.3, slave)

finding iodev_table
1. starting point FIO_ReadFile at 0xFF1FC434
2. if file handle id is < 100, it enters 0xFF3279F4
3. there the first call is 0xFF08CFA8
4. this function references to 0x2D3B0 (pointers to tables of functions) and calls a function from there using BX
5. as we (always) use the third entry in that table, our iodev_table = 0x2D3B8 (it points to 0xFF58B350)

the function tables have these functions:
  0xFF58B350:
  DCD iodev_OpenFile
  DCD iodev_CloseFile
  DCD iodev_unsupported
  DCD iodev_ReadFile
  DCD iodev_WriteFile
  DCD iodev_unsupported
  DCD iodev_unsupported
  DCD iodev_unsupported_2
  DCD iodev_unsupported

finding iodev_ctx
1. go to iodev_OpenFile, which is at 0xFF458BBC (referenced in table found above)
2. enter the 3rd function 0xFF3E0D50. its the one with return value being checked (SUBS R5, R0, #0)
3. the first function (0xFF3E0958) being entered is allocating an fd.
4. this function references to 0x85510, so iodev_ctx = 0x85510

finding iodev_ctx_size
1. the same function as you used above (0xFF3E0958) goes through all entries
2. it adds 0x18 bytes per iteration, so iodev_ctx_size = 0x18

voila

on 6D, traced back FIO_READFILE



0x9DEF8 -> ffB6DF3C
0x9DEFC -> FF9CC7AC
0x9DF00 -> 0x7380C
0x9DF04 -> FFC38890
0x9DF08  -> FFC38800



Unfortunately the list doesn't seem to match:


off_7380C       DCD FIO_unsupported     ; DATA XREF: sub_5558C+4o
RAM:0007380C                                         ; RAM:off_556F8o ...
RAM:00073810                 DCD sub_55344
RAM:00073814                 DCD FIO_unsupported
RAM:00073818                 DCD sub_54A4C
RAM:0007381C                 DCD sub_54C80
RAM:00073820                 DCD FIO_unsupported
RAM:00073824                 DCD FIO_unsupported
RAM:00073828                 DCD sub_55430
RAM:0007382C                 DCD sub_55418


0x9DF08  -> FFC38800

Looks Closer


off_FFC38800    DCD sub_FF8353F4        ; DATA XREF: EFAT_Match+4o
ROM:FFC38800                                         ; ROM:off_FF796A28o
ROM:FFC38804                 DCD sub_FF835250
ROM:FFC38808                 DCD FIO_unsupported
ROM:FFC3880C                 DCD sub_FF8355AC
ROM:FFC38810                 DCD sub_FF83563C
ROM:FFC38814                 DCD FIO_unsupported
ROM:FFC38818                 DCD FIO_unsupported
ROM:FFC3881C                 DCD sub_FF1515AC
ROM:FFC38820                 DCD FIO_unsupported


First function -> first call leads to:


STMFD   SP!, {R4-R8,LR}
ROM:FF7967CC                 LDR     R7, =dword_75E34
ROM:FF7967D0                 LDR     R4, [R0,#0x1C]
ROM:FF7967D4                 MOV     R6, R0


So maybe that is ctx?

50D

0x1F200 -> FFC11F8C
0x1F204 -> FFC0DDEC

0x1F208 -> FFC8168C


Last one seems to make sense:

ROM:FFC8168C off_FFC8168C    DCD sub_FFB075E0        ; DATA XREF: sub_FFB0791C+4Co
ROM:FFC8168C                                         ; ROM:off_FFB07DACo
ROM:FFC81690                 DCD sub_FFB0762C
ROM:FFC81694                 DCD FIO_unsupported_0
ROM:FFC81698                 DCD sub_FFB07664
ROM:FFC8169C                 DCD sub_FFB076AC
ROM:FFC816A0                 DCD FIO_unsupported_0
ROM:FFC816A4                 DCD FIO_unsupported_0
ROM:FFC816A8                 DCD sub_FF81B6B4
ROM:FFC816AC                 DCD sub_FFB076F4
ROM:FFC816B0                 DCD sub_FFB077C0
ROM:FFC816B4                 DCD sub_FFB0780C
ROM:FFC816B8                 DCD FIO_unsupported_0
ROM:FFC816BC                 DCD sub_FFB07844
ROM:FFC816C0                 DCD sub_FFB0788C
ROM:FFC816C4                 DCD FIO_unsupported_0
ROM:FFC816C8                 DCD FIO_unsupported_0
ROM:FFC816CC                 DCD sub_FF81B6B4
ROM:FFC816D0                 DCD sub_FFB078D4


But FFB699D8 function which should be third call of sub_FFB075E0  doesn't seem to operate any variables. Its like 50D is missing div function, I never see it subtracting.

hmm.. the 6D version is doing something, I can't read the photos but keys were generated.. they are 1KB (i picked smallest key size)

https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/ee19913b23023260ad87a33dea0edb56e2e48565

get some overflow errors from the backend however. also can't read back any pics... can't try the RSA because I can't compile the .exe

[650D; M; 8.feb nightly; iocrypt 8.feb 2:33; io_decrypt.exe 08.feb]

To see, or not to see, that is the question:
While testing different settings i went to playback mode after pic 9. Then all encrypted pics schowed "Cannot playback image". But pic 5 & 6 showed the original photo in the same size as the question mark placeholder normally seen with "Cannot playback image".

I'd like to know how this works but my coding skills are way to crappy to grasp it by myself. Are thumbnail and 'big photo' at differend areas in ram before encryption (not packed in one file until written to card or a question of memory management)?

maybe linked:
By chance i saw an overflow about 10 minutes later (have dump).

i disabled scratch memory as i suspect it to cause trouble.
is it any better?

(get it from first post as always)

No more overflows

Quote0x9DF04 -> FFC38890
0x9DF08  -> FFC38800

Both of these produce encrypted files you can't decrypt... maybe the CTX is wrong?

hmm when encrypting works, then all is fine there.
you have the latest version without scratch?

Yep, it works on 7D. The password version doesn't even work. Files just come out black when decrypted.

QuoteSTMFD   SP!, {R4-R10,LR}
ROM:FF796594                 LDR     R4, =0xCC110
ROM:FF796598                 LDR     R6, =dword_75E2C
ROM:FF79659C                 

Found this and tryied both CC110 and 75E2c, still doesn't decrypt.

Maybe see if I'm doing something wrong in 6D?

Quote from: 1% on February 09, 2014, 09:24:11 PMFiles just come out black when decrypted.

thats weird, because this means its decrypted fine, just the image content is black.
are you sure??

Yea, I didn't take a black pic... either way on providing the correct pass I see a mini version of the thumbnail when I reboot the camera. Running it through IO_Crypt the file is blank and can't be opened by photoshop.

Password is AAA
http://ge.tt/5hq0NfJ1/v/0?c

        iodev_table = 0x9DF08;
        iodev_ctx = 0xCC110;
        iodev_ctx_size = 0x20;

can you:
- load and enable trace.mo
- start camera
- set password using password encrypion
- shoot one image
- poweroff

and send me both log and image please?

ha, didn't think to log it.

shit, I'm not getting the log... even if I comment out the if(1) in iocrypt_init

comment out the -DTRACE_DISABLED in Makefile ;)

Log1:

http://paste2.org/Oy3mb7y1

log 2:

http://paste2.org/wNfpZY1s

File:
http://ge.tt/8nZBEgJ1/v/0?c

[650D; M; 9.feb nightly; iocrypt 9.feb 19:44; io_decrypt.exe 09.feb]

decoded .cr2 & .jpg still not readable (neither pw nor rsa).

Do the log like I did.... maybe g3gg0 can compare the 2

fixed a bug that happened rarely on the 5D3 and sometimes on the 7D.
reason was an odd address offset that i didnt handle cleanly on block size borders.

should fix the issue you had, 1%

Ok, will give it a go.

It appears to be working... the files now decrypt in camera and when you don't set the right password they do not. Can't test the RSA unless you updated the exe


c:\mingw\include\io.h:302:39: error: unknown type name 'off64_t'
__CRT_INLINE off64_t lseek64 (int fd, off64_t offset, int whence) {
                                       ^
In file included from crypt_rsa.c:24:0:
c:\mingw\include\unistd.h:65:20: error: unknown type name 'off_t'
int ftruncate(int, off_t);


still get an error 70 during series shooting, which i am currently tracking down.

Wonder if the other iodev_ctx I had will work too since behavior was the same with both.. previous was pointer to somewhere, current goes to 0.

Sorry, I've been very busy in the last days.
I have no toolchain. So can someone give me a download to a compiled trace.mo?

Ok, EOSM is working:


    else if(streq(camera_model_short, "EOSM"))
    {
        iodev_table = 0x6F518;
        iodev_ctx = 0x9C4A8;
        iodev_ctx_size = 0x20;
    }


Only tested password... did jpegs

Cool. Thanks.
did you have any issues with series shooting longer than 5 seconds?


ASSERT: 0
at ./Capture/Capture.c:526, task EventMgr
lv:0 mode:3


Magic Lantern version : Tragic.2014Feb13.6D113
Mercurial changeset   : 9706fb780627+ (unified) tip
Built on 2014-02-14 04:46:32 UTC by User@PC.
Free Memory  : 341K + 1661K


http://paste2.org/LVVd7x4e

Yea, It dies

it is some weird error.
either some memory areas are corrupted when the module is loaded, or ... hmm i dont know.
point is, even if i remove the whole encryption stuff and just cycle through a few bytes without modifying, it crashes.
making the code smaller, causes the error to disappear.

caller stack cannot be the issue. i reserve a few bytes only in a function that is calling complex FIO_ stuff which potentially needs hundreds of bytes.
also tskmon isnt ringing.

Can you prepare a minimal working example with these few bytes?

yep, can prepare one.
first i wanted to check if the memory backend changes made things worse or better.

basically it was commenting out all update_key in encrypt crypt_lfsr64_encrypt and commenting out the assignments in crypt_lfsr64_xor_uint*
so the loops were iterating, but not changing anything. the assembly code was sane and just counting up registers, depending on optimization level.

alternatively running the first update_key, doing a msleep(20) and returning caused the same crash.

disabling trace module didnt help

[650D; P; 14.feb nightly; iocrypt 9.feb 19:44]

In review most pictures are only shown as a thumbnail with
"Cannot playback image".

shooting series (only tried with pw):
mostly i get an error 80 and sometimes error 70.

Then 650D must be wrong/still not decrypting.


    else if(streq(camera_model_short, "50D"))
    {
        trace_write(iocrypt_trace_ctx, "io_crypt: Detected 50");
        iodev_table = 0x1F208;
        iodev_ctx = 0x49A64;
        iodev_ctx_size = 0x18;
    }


Not sure on the CTX but images are taken and encrypted... just not decrypted.

will disable encryption when series shooting is active.

When shoot mode is not single image mode, encryption is disabled.
this will catch the cases where shooting will fail during IO encryption.

this will also disable it in silent image mode that 5d3 and 6d (iirc) have.

of course it not the root reason, but if someone can find out where the crashes come from, please tell me.

Did it also error on 600D and 7D in burst?

I'll have to log 50D and post here :) Same should be done for 650D

tried on 5d3 and 7d, both crash.
my (private) 600d is currently at a friend

Safe to say then it crashes on all.

Quote from: g3gg0 on February 15, 2014, 03:17:49 PM
this will also disable it in silent image mode that 5d3 and 6d (iirc) have.

Is this a side effect or intentional? As far as I can see, shooting in single image mode but silent shouldn't be different from vanilla non-burst mode - or am I missing some internals?

I think it crashes from many writes so a burst of silent pics might do the same thing but didn't check.

Quote from: 1% on February 15, 2014, 07:23:43 PM
I think it crashes from many writes so a burst of silent pics might do the same thing but didn't check.

In that case, silent pix are *less* likely to crash than quick single normal pics as the lag on shutter lag on silent pics is longer.

Try the older module and see what it does.

this is a module limitation.
if you look at the headers, then you will find #ifdefs for some models.
but from the module view you can only compile for one model.

so i would have to use numbers and check them according to model_name.
quite a lot work for some dirty workaround.

fixed that crashing issues finally :)
you can download the module in the main post.

Quote from: g3gg0 on February 15, 2014, 11:33:47 PM
so i would have to use numbers and check them according to model_name.
quite a lot work for some dirty workaround.

I have the same problem, and I think some model-detection framework should go into the core for modules to use so not every author needs to add them again and again - like (model_is_60d() || model_is_6d()) that work like the #ifdef in the core... and can be just replaced when code code is refactored into modules.

As for silent shooting imho it's worth to do a check, photo journalist as one the intended groups for this modules are bound to use it to get inconspicuous that need to be safeguarded by encryption.

i already removed the restrictions.
all problems gone now, so no need to restrict to single image anymore.

about model stuff.
well, i recommend to use the values defined in propvalues.h along with property PROP_CAM_MODEL content.
to be done.

the most annoying thing is that canon relies on priority inversion, which is.. quite... "impressing" :)
i hope they documented that internally at least

Quote from: g3gg0 on February 16, 2014, 02:14:21 AM
the most annoying thing is that canon relies on priority inversion, which is.. quite... "impressing" :)
i hope they documented that internally at least

Ugh? Are we simple mod authors supposed to understand what you're talking about :-p?

Feel free to document the priority/thread stuff for the rest of us, for example I don't understand at all when offloading some metering loop in my auto_iso module from the general shoot task to a dedicated thread is a good idea - either for being friendly to the rest of the ml tasks or to increase performance.

http://en.wikipedia.org/wiki/Priority_inversion
explained quite well there.

in this case the shooting threads need memory.
this memory is allocated by a manager which has a lower priority than the shooting threads.
so for the time the shooting threads were executed, the memory manager didnt get time for processing.
this was no problem, as the FIO_Write function waited for a semaphore released by interrupts and thus handed CPU over to other threads.

now i am (well, i was) blocking the shooting threads while they are writing.
blocking with executing real code, not by sleeping or waiting for a semaphore. (sleep also gives away CPU time to the next lower threads)

well, this caused the shooting thread to delay the FIO_Write() call a bit longer, which didnt hurt the task itself.
but i didnt give away cpu time in this call, but i used it to calculate stuff (encrypt).

this lead to the memory manager task, which unfortunately has a lower priority, not being executed fast enough.
some other high priority shooting tasks also didnt give away CPU time and so the memory manager starved.

in the current approach, i just put the encryption into a separate crypt thread, which has a "normal" priority.
so the shooting tasks give away their CPU time by waiting for a semaphore and the memory manager can run.
and all is fine.

funny, eh?

lol, I had to look it up and I was going to post the wiki link :)

i just saw that in some cases the image is non-decryptable when using RSA. (encrypted_size: 0 when decrypting)
but thats no serious bug i guess. maybe just not handling the asynchronous RSA encryption 100% correct.
will check that tomorrow.

Quote from: 1% on February 15, 2014, 12:20:35 AM
Then 650D must be wrong/still not decrypting.

Code Select Expand


    else if(streq(camera_model_short, "50D"))
    {
        trace_write(iocrypt_trace_ctx, "io_crypt: Detected 50");
        iodev_table = 0x1F208;
        iodev_ctx = 0x49A64;
        iodev_ctx_size = 0x18;
    }


Not sure on the CTX but images are taken and encrypted... just not decrypted.

according to https://bitbucket.org/hudson/magic-lantern/commits/e769afc111947416d7c9e45d72f353a08f01a04b
there must be smth wrong. can you provide me a RAM/ROM dump?

650D should have:
        iodev_table = 0x54060;
        iodev_ctx = 0x7C278;
        iodev_ctx_size = 0x20;

and this was already set correctly.
can you check with the latest version and describe what is happening when:
- shooting an image with encryption password
- decrypting on pc
- setting password as before and view image in camera (after reboot)

please do that with trace enabled and upload the traces (two, one for saving, one for playback)

maybe you have a ram dump also?

[650D; P; 16.feb nightly; iocrypt 16.feb 1:27]

The changes in menu are nice.
Series shooting works.

on pc:
cannot open decrypted pw (neither jpg nor cr2)
cannot decrypt rsa (jpg&raw)

I added a log to that commit and uploaded the rom to upload.g3gg0.de

that looks totally fucked up. whats wrong there?
recursively opening files until file descritors run out?!

nanomad's fio, file thing. I had to get rid of card_drive in the module.. so maybe there is a bug with that?

can you locally roll back and try?

I will try that now and make another log.

Here is another:

http://paste2.org/cksdZGU6

that was encrypting. looks sane.
how about decrypting in camera using playback?

Yea, that part seems right. i'll do the log of it decrypting, where its busted.

Hmm.. saying its not decryptable:

http://paste2.org/d1mPw7D3

BTW, I'm trying: 

// iodev_ctx = 0x49A64;
iodev_ctx = 0x49F18;


Look at both in the firmware... I think the 18 is closer, I tried to xref vs 7D but the functions don't quite match. Neither decrypts.

yep, it looks like the direct access to file offset isnt the best choice for all models.
i could again decide for a footer that is written when closing the file.
that would ease up writing and make that sane.
(might be the best choice for making encrypting available on all models)

alternatively i could use a sidecar file, which could make things complicated.
(calling FIO routines from routines at a lower level)

but reading still requires random access.
the FIO_SeekFile calls iodev_SeekFile somewhere, but i dont have the "correct" handle type at this position.
well, for 5D3 i could easily do that, but for 7D its different again and for 650D even more.



well, now as i use different threads anyway, i could really use sidecar files.
but copying IMGxxx.JPG to PC is much easier than copying some sidecar files too, especially when using a reader.

Any chance at getting a build for the 550D? I'd be happy to try testing it.

Just wondering.

What happens if I open one of these encrypted files in Lightroom or Canon DPP.
Is Lightroom or Canon DPP asking for a password or encryption key file ?

Hi g3gg0,
great to see such a function coming up. However, when it comes to implementing crypto there are a few pitfalls and it seems you didn't manage to stay clear of all of them. I had a quick glance at the code (which is easy to read, good work on that) and with the current state of affairs I'd issue a warning that io_crypt should not be considered as safe unless you just want to hide photos of upcoming christmas presents from your spouse.

The biggest issue I have with your implementation is the selfmade-streamcypher based on the 64-bit LFSR. It just too easy to break. Also using it in the hash function to derive a key from the password (where the password is alphanumeric only and no salt is added) reduces the strength further. Also, you have to add some fresh randomness for every picture or you get the same keystream for different pictures. XORing two encrypted pictures the removes the keystream and returns the difference of the pictures to you.
The next issue is using deterministic RSA. As the symmetric key is only 64 bits (and not 128 as stated in https://bitbucket.org/hudson/magic-lantern/src/703ee626326d7cb310b087a337b9dad2b2d642c4/modules/io_crypt/hash_password.c?at=unified#cl-45) given a series of encrypted pictures an somewhat powerful adversary has a non-negligible chance of guessing a key.
Finally there is no protection against malicious modification of pictures. An attacker may alter some parts of the picture at will. This will likely show up as random noise (instead of some person or object that the attacker wanted removed). But with some sort of message authentication you could at least prove that the manipulation happened after writing the file to the SD-Card.

The great thing is: You showed that it is possible to do on-the-fly encryption on the camera. That is the important part. The next step is putting in good crypto in the right way. Have a look at http://tweetnacl.cr.yp.to/software.html This should compile without much fiddling and can replace your RSA code. (It might be even faster.) For password-based encryption you still will have to find an appropriate function for key derivation. The most common is PBKDF2 and it should not be to hard to adopt e.g. the Code from OpenBSD (http://bxr.su/OpenBSD/lib/libutil/pkcs5_pbkdf2.c).

If you have questions or need any assistance, just drop me a pm. I'd love to see io_crypt mature.

hey msw,

thanks for that constructive feedback!

indeed, some of these issues, especially the obviously weak XOR cipher, i already discussed with alex.
i am confident that this encryption is too weak to save lifes but good enough - as you say - to hide things.
especially as this module is experimental and described that the algorithm has weak points, i hoped to clarify its non-mature state.

main concern is still making the encryption more performant.

PBKDF2:
thanks a lot for pointing me to this algorithm. the (true) reason why i didnt implement it yet is,
that i didnt even look for the "state of the art"-algorithm for this task :)

128 bit:
right, this is a wrong comment. not sure why i left it there.

XOR encryption:
i am totally unstatisfied with the XOR based encryption.
problem is that at the moment i have to encrypt the file content in some random-access file IO routines.
my first approach was building a 64 bit XOR key depending on the key and file offset.
but that CPU is so slow and has to process ~30MiB images with up to 120MiB/s, so i had to play with larger block sizes.
this way i could get from 1MiB/s processing speed to ~20 MiB/s IIRC.
as this is a compromise, i am totally unhappy with it, because compromises are the worst thing when it comes to data security :)

the best thing (as proposed by alex) would be to decouple the random access code from the encryption code.
so that any file IO is buffered plaintext into RAM and on file close, it will get encrypted and written to the card.
same for opening - the file is read in a whole and decrypted into RAM.
so we could use e.g. CBC to get rid of all the possible attacks possible with ECB.


do you want to jump in? which camera model do you have?
the current problem is lack of time (main job consumes more and more time...) and that i tend to focus things that seem to be of interest.
this encryption experiment didnt bring up many voices that shared their experience in terms of stability and workflow from the userbase.

BR,
g3gg0

@Levas:
no, but a LR plugin is possible. just like with dual_iso files.

@g3gg0

Have a 60D, but not running ML and don't want to risk (soft)bricking it. I'm low on time too, but I'll keep it on my agenda. :)

yeah, the 60D is a bit sensible. seems some canon issue :)

I would still love to see this in the unified build some day. Are you still working on this at all?

its a bit too experimental for mainline, sorry

Yeah of course, I'm just wondering if you're still pursuing it or if it's fallen by the wayside.

i think its in state "well, it seems to work".
any help is appreciated, as i personally am very busy with work, house and family.

I tried to compile this module, io_crypt, but I get errors whatever I do. I already has the io_crypt.mo because I found it online. What I need is the io_decrypt for decrypting the RSA-files.

I followed the tutorial from mk11174 http://www.magiclantern.fm/forum/index.php?topic=6425.0 and did everything as he did, but when I wrote Make I got this instead:

magiclantern@magiclantern-VirtualBox:~/magic-lantern/modules/io_crypt$ make
Updated HGVERSION
[ README   ]   module_strings.h
Traceback (most recent call last):
  File "../readme2modulestrings.py", line 113, in <module>
    txt = run('cat README.rst | grep -v -E "^:([^:])+:.+$" | ' + rst2htmlCommand + ' | python ../html2text.py -b 700 | sed "s/\r$//"')
TypeError: cannot concatenate 'str' and 'NoneType' objects
make: *** [module_strings.h] Error 1

Please explain to me as if you would explain to your grand mother or to a child.

hm, tried a make clean already?
this error doesnt tell me anything

Tried make clean.

During the days I have tried different tutorials on Ubuntu, Windows Msys2, Cygwin. One nice thing is that I fail with different things every time and not always the same thing.

Anyway, today I tried http://www.magiclantern.fm/forum/index.php?topic=7579.0 and failed at nr 27. At 27 I got the same error message as I posted above.

I will try http://www.magiclantern.fm/forum/index.php?topic=7579.msg134989#msg134989 later today or tomorrow.

Seems like the script can't find rst2html, are you sure that it's installed?

Those errors come from trying to compile the ML module (you need lots of stuff setup correctly for that, such as the ARM toolchain, python, and various utilities like rst2html). If you don't need the module, and you just want the decrypt utility, try:

make io_decrypt


Quote from: dmilligan on August 26, 2016, 03:37:30 AM
Those errors come from trying to compile the ML module (you need lots of stuff setup correctly for that, such as the ARM toolchain, python, and various utilities like rst2html). If you don't need the module, and you just want the decrypt utility, try:

Code Select Expand


make io_decrypt


That made it. It works in the prepacked Oracle thingy on my Windows but I can't get it to work on my Ubuntu 16.04 (64). What is it that my Ubuntu system is missing? Do I need arm-none-eabi-gcc-4.8.3 even on my Ubuntu machine to run io_decrypt?

bash: ./io_decrypt: The file or folder doesn't exist

Edit: Maybe because of 64 bit. I will check it.

Edit2: It was because of 64 bit. Fixed. Thank you guys. 

Is there an easier way converting 20 raw files? For example:

./io_decrypt {_MG_8457.CR2,_MG_8458.CR2} home/krypterat/a/{_MG_8457.CR2,_MG_8458.CR2}

or
./io_decrypt *.CR2 home/krypterat/a/*CR2



Fixed it with

#!/bin/bash

for i in *.CR2
do
  ./io_decrypt $i /decrypt/$i
done

I'd take a look at stream ciphers which allow random access... Salsa20 might work, I believe you can set the key from any position of 8-byte blocks (and so you can just seek anywhere, get previous block, and ignore the overflow)

Relevant article: https://www.wired.com/2016/12/200-filmmakers-ask-nikon-canon-sell-encrypted-cameras/

Edit: Hacker News discussion here: https://news.ycombinator.com/item?id=13175810

very interesting. probably should polish that feature :D

Absolutely @g3gg0 I'm all for this feature as well and Thanks for sharing the article @chris_overseas!

Absolutely! This is really good stuff, thx for building this. Chacha20 would be really nice ...

So far this is afaik the most promising implementation in the direction of secure picture storage on cameras.

yeah already started with chacha20 (rfc7539) but didn't get far due to lack of time.
if you want to help, please jump in :)

btw the file format is also just experimental.

i tend to do this:
- only encrypt CR2
- keep the original file format (more or less)
- (RSA-mode)embed RSA ciphered keys into comment fields of the file format
- add dummy image as content
- append original, encrypted image at the end/comment field

reasons:
- deniability as all tools see a valid image with uncritical content
- forensics will for sure find the real image though
- no jpeg to simplify development effort

modes:
  RSA-mode: random key for every image, asymmetrically embedded into image. no image review. no chance to "hand out" or "expose" the key
  password mode: enter symmetric key on startup. image review possible.  has to get entered every startup

with ChaCha20 i get ~4.5 - 4.8 MiB/s
with XTEA its ~0.8 MiB/s
with LFSR it was ~20MiB/s (iirc)

Interesting comparisons @g3gg0!

Quote from: g3gg0 on December 31, 2016, 03:09:34 AM
with ChaCha20 i get ~4.5 - 4.8 MiB/s
with XTEA its ~0.8 MiB/s
with LFSR it was ~20MiB/s (iirc)

Could you somewhere share your code with ChaCha20? I would like to check it out, because we (my friend and I) are doing a project where we would like to implement exactly this feature for magic lantern (with Poly1305).

Thanks in advance!
Regards,
Stiefel

well, i don't think i kept it somewhere.
but it was quite simple to integrate.
i used the implementation from insanecoder (http://chacha20.insanecoding.org/)

So we tried to get up and running the io_crypt module, but we encountered some problems with the decryption.

(Our "specs":
Camera: 600D
Platform build: https://builds.magiclantern.fm/jenkins/job/600D.102/387/artifact/platform/600D.102/magiclantern-Nightly.2017Mar12.600D102.zip
io_crypt build: https://builds.magiclantern.fm/jenkins/job/io_crypt/2/artifact/modules/io_crypt/io_crypt.mo)

The encryption seems to be working, however we have some difficulties with the decryption. The camera itself can't decrypt the images in preview mode either. Regardless of what mode (RSA or PWD) we use. We have the same issue on a PC. Either we get a CR2 file which is just green or we get a JPG with various errors. One example: https://drive.google.com/drive/folders/0B6HUB6hIMVcRLWJubVZTNUxhOW8 I also included the Private key, for testing purposes.

Can somebody give us a hint what we are doing wrong? I also include the decrypt-executable which we use. (It was built by ourselves. See the drive link.)

Thanks in advance!

hi stiefel,
what were the exact settings you used?

i found the patch i used to add ChaCha20 support.
the file is a .patch, just renamed to .txt as the forum didnt allow this extension.

Hi g3gg0, thanks for the work.
Have yourself implemented the crypto primitives? I didn't read the thread but, if that's the case, wouldn't be good to use something like libsodium[1] to do it?
Also, deniable encryption is a good idea. For example, you set two passwords, one for the real photos, other for a bunch of meaningless photos. So, if I'm a photojournalist working in some protest and authorities come telling me to show the pictures I can just put the fake password and be safe, kinda like the old Rubberhose file system.
Steganography could be a good project too... just exposing some ideas.



[1] https://libsodium.org

to get proper encryption in camera, it requires *fast* algorithms and seamless integration.
first point was to figure out the capabilities and how to integrate encryption.

the latter point was already mentioned (https://www.magiclantern.fm/forum/index.php?topic=10279.msg176857#msg176857) and is a good addition.

Hi g3gg0,

Quote from: g3gg0 on April 08, 2017, 10:48:50 PM
what were the exact settings you used?

The easiest way to show the exact options, we made a video about it: https://youtu.be/VwA3yAJTVY8 (https://youtu.be/VwA3yAJTVY8) (you might have to wait, depending on when you check it out, because I just uploaded it, and it might take some time for youtube to process it).

Quote from: g3gg0 on April 08, 2017, 10:48:50 PM

i found the patch i used to add ChaCha20 support.
the file is a .patch, just renamed to .txt as the forum didnt allow this extension.

Thanks for it! It is very nice of you!

Hey @g3gg0 , check this out:
https://github.com/boyska/chdkripto

Seems interesting. It seems to use Curve25519 (as it's listed on IANIX page).
Also, for benchmarks:
https://bench.cr.yp.to/supercop.html

The research on in-camera encryption seems really interesting for forensics and could have a real impact as evidence for legal judgements.
Do you know if default Canon CR2 provide a reliable timestamp? If not, would ML silent_pic be able to write a DNG with encrypted timestamps?
Just some ideas.

hi bpv5P,

the encryption causes writing speed to go drastically down.
a simple repeaing XOR key which is not even halway safe, causes a drop in write speed by factor 2 or so?
so any real cipher makes life really hard :(

possible - but practical?

Quote from: g3gg0 on August 08, 2017, 04:58:01 PM
hi bpv5P,

the encryption causes writing speed to go drastically down.
a simple repeaing XOR key which is not even halway safe, causes a drop in write speed by factor 2 or so?
so any real cipher makes life really hard :(

possible - but practical?

Yeah, I don't know the solution. 
Here's another idea: get the camera "MAC address" (I don't know if it's called that way on such devices), and encrypt the DNG metadata using it. It could be useful for legal usage, so you can autheticate the image. RAW image alone is used today as legal evidence, but it can be easily edited with the right tools, preserving metadata.

Although the idea above could be useful for some people, it will not be worth for journalists, for example, since anyone could see and delete the picture.
Another solution could be use homomorphic encryption on Wifi cameras:
https://tfhe.github.io/tfhe/

Or maybe to encrypt the whole filesystem (http://www.magiclantern.fm/forum/index.php?topic=20225.msg188207;topicseen#msg188207) with Chacha20 (https://download.libsodium.org/doc/advanced/chacha20.html), that's a faster algorithm than most of the ciphers...

I don't understand all of these concepts, though, I'm just leaving some links in case you have time and will to research...

this plugin is about 80% of the way there when it comes to solving some big real-world problems... consider the following:

1. The journalist

After taking some photos of a protest, a journalist is detained by the police. They want the journalist's photos so they can prosecute the demonstrators, prosecute the journalist, or (more probably) both.

The journalist gives them his camera, but warns, "the photos are encrypted, so it won't be much use to you."

"What do you mean, encrypted?" the police ask. "What's the password? Tell us the password or we'll lock you up."

"I don't have the password," the journalist replies. "I can't decrypt the photos after I take them. The decryption key is with my editor in Paris. I take the photos, upload the encrypted photos to our server, and then my editor decrypts, edits, and publishes them. Am I free to go?"

Setting this up is easy: the publication's tech department supplies the journalist with a plain-text config file containing the public key and other settings. The journalist copies the config file to all of his memory cards. The camera firmware automatically recognises the config file and encrypts the photos as specified.

(Writing a user-friendly GUI to generate the config and handle decryption of photos should be easy, but can wait until the basic functionality is in place.

Similarly, verifying the key is correct by generating a key-fingerprint-image and displaying it on screen when the camera is turned on may also be worth doing... later.)

2. The forensic investigator

Before taking photos of a crime scene, the investigator generates a new time-stamped public-private key pair using a menu in the camera. The camera stores the private (signing) key in memory -- it's never written to the memory card and can't easily be extracted.

Instead, the camera writes the public (verification) key to the memory card, along with signed metadata about when and how the key-pair was generated. (camera serial number, etc)

Then, every time the camera takes a photo, it writes a cryptographic signature for the entire data file to the memory card. Using the public key (which was written to the card), anyone can verify a given digital image was taken by that camera at that time, and hasn't been modified since.

## Suggestions for implementation

There are a couple of things that would be helpful to make it the last 20% of the way:

- Implement real encryption: For people who need encryption or signing, performance is a secondary concern. Being able to gawk at your photos on the camera is also not so important if being able to do so puts your life and/or freedom at risk. (on-camera decryption is very much of secondary importance and could be implemented later)

- Use established cryptography instead of DIY: Implementing "real" encryption isn't hard. Compact, efficient cryptosystems (like Filo Sottile's `age` /  `rage` ecosystem) are widely available and offer good flexibility. They've also been reviewed for vulnerabilities and errors -- that's not true of anything we might come up with.

- Modularize the encryption: As soon as this sees any real-world use, encryption experts are going to look at it and point out ways we can do it better. Maybe next year someone will discover a crucial flaw in the fabric of reality and we will have to switch to some kind of Super Post-Quantum Isogenic-Singularity algorithm. Either way, the more modular and decoupled the encryption/signing code is from everything else, the easier it will be to implement these inevitable fixes.

- Make the interface conceptually simple and based on established standards... then add newbie-friendly sugar later. Putting an encryption key on the memory card (or reading a verification key from that card) is very basic. Editing a TOML or INI file by hand is also easy. Later on, it's easy to write a user-friendly GUI wrapper which generates the key, writes the config file... and then once the photos are taken, decrypts the photos.

Hello, and welcome to Magic Lantern :)

You are replying to a five year old thread - if you want encryption functionality in the way you've described, you're probably going to have to implement it yourself.  We can help show you how to do that if you'd like.

Actually I thought about this feature :)
I crossed the border recently, and was interrogated. Nothing serious and luckily they didn't check my camera. There were some videos where I say things against our fascist government :) So if you could for example encrypt video in camera, after filming.. even if it is slow. It could be useful.

Maybe even the option to hide videos from camera (like change file extension) so they won't play - can already help in many cases.
Or a password for the camera, with "wrong" password which will trigger a card format, hehe

You're like those PGP etc. users: you just don't get it that using certain technologies is a red flag that will get you into trouble. They (PGP etc, can't say anything against this proposed plugin) are developed precisely for you to advertise the fact that you've something to hide. Pro tip: learn about anti strong encryption laws in whatever 'non-fascist' country you end up in. Will likely save you a lot of legal troubles, too.

P.S. You got screened at the airport and got spooked, right? Now, what would've happened if your camera WAS checked containing encrypted material? You would've been regarded legitimately suspect, detained, and given a proper Israeli-grade (just an example of a country taking its security very seriously) interrogation. A several days-long one, I'd wager.

Maybe.. good point. Then it is better to just hide everything if possible.

I am not that experienced with this whole.. thing. For example in my case, they just took me to the special room like the ones you usually see in the movies.. with fences everywhere and locking doors. Asked a lot of questions for ~30 minutes. I knew it was only for psychological pressure because I heard other guys stories. But the whole bus was waiting for me, and I was afraid that it will leave.. but luckily, the driver understood the situation and other people were also supportive :)