Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]

[REDcineX]

Success!

Last login: Wed Aug 14 00:31:07 on ttys000
Macbook-Pro-i7-26gHz-2012:~ Admin$ cd desktop/untitledfolder
Macbook-Pro-i7-26gHz-2012:untitledfolder Admin$ ls
M12-1656.RAW   _MG_9521.CR2   _MG_9524.CR2   _MG_9527.CR2
_MG_9518.CR2   _MG_9523.CR2   _MG_9526.CR2
Macbook-Pro-i7-26gHz-2012:untitledfolder Admin$ cr2hdr _MG_9523.CR2

Input file     : _MG_9523.CR2
Full size      : 5792 x 3804
Active area    : 5634 x 3753
White level    : 15834
Black borders  : 158 left, 51 top
Black level    : 972
ISO pattern    : BddB GBRG
Estimating ISO difference...
ISO difference : 4.21 EV (1847)
Black delta    : -6.81
Interpolation  : mean23-chroma5x5-contrast
Matching brightness...
Looking for hot pixels...
Hot pixels     : 1438
Full-res reconstruction...
Alias filtering...
Dark chroma filtering...
Building contrast map...
Dilating contrast map...
Smoothing contrast map...
ISO overlap    : 4.8 EV (approx)
Black adjust   : -3
Output file    : _MG_9523.DNG
    1 image files updated


My problem was this:

lrwxr-xr-x     1 root   wheel        49 Aug 13 23:07 cr2hdr -> /Users/Admin/magic-lantern/module/dual_iso/cr2hdr

When I ran the ls -al I saw the folder said "module" not "modules"

[Danne]

Congrats! I, ll check this out more thoroughly later.

[Francis]

I hope it was worth it. 

[jerrykil]

I hope it was worth it. 

very worth it, thanks

[dfort]

When trying to post a question here I got:

"Warning: this topic has not been posted in for at least 120 days.
Unless you're sure you want to reply, please consider starting a new topic."

Not sure if starting a new topic is appropriate so I'll just ask my question.

I followed the instructions for compiling on Mac and it tripped up looking for python.

makefile.user has: PYTHON=python2

For Mac shouldn't it be: PYTHON=python

I made that change and it got a little further but it didn't finish:

[ CC       ]   raw_rec.o
[ MODULE   ]   raw_rec.mo
[ STRIP    ]   raw_rec.mo
[ EXPORTS  ]   raw_rec.sym
00001960 raw_movie_filename
00001964 raw_video_enabled
[ DEPENDS  ]   raw_rec.dep
Not checked (compile ML for these cameras first):
    1100D, 500D, 50D, 550D, 5D2, 5D3, 600D, 60D, 650D, 6D, 700D, 7D, EOSM
make: *** [raw_rec.dep] Error 1

[dmilligan]

It's failing running a python script that checks the dependencies for the modules. I had it set to python2 so that it would use version 2 of python which is apparently the version that works correctly. What happens if you put the PYTHON=python2 back in?

[dfort]

Thanks for the tip dmilligan. This is what happens:

[ DEPENDS  ]   raw_rec.dep
/bin/sh: python2: command not found
make: *** [raw_rec.dep] Error 127

This is a new MacBook Pro running 10.9.2. I think I followed the instructions to install MacPorts and Xcode properly. According to whereis, python is in /usr/bin/python but returns null for python2.

Edit: Looks like this MacBook has python 2.6 and 2.7, the default being 2.7.5. I tried changing python2 to python2.6 and python2.7 and didn't get the command not found error but it always winds up with:

make: *** [raw_rec.dep] Error 1

[dmilligan]

IIRC python2 was on my system by default, though I upgraded to 10.9 from earlier versions, Apple may well be including different versions of things like this by default now, IDK. maybe you can try downloading some different 'official' versions of python and try them (use fink or brew or whatever) rather than the Apple flavored version. I'm not at my mac right now so I can't check what the actual version of python is working and where it came from, but I'll reply again when I am.

[dfort]

No rush. I've got a busy week and can't get back to this until the weekend. So this error is caused by python?

[ DEPENDS  ]   raw_rec.dep
Not checked (compile ML for these cameras first):
    1100D, 500D, 50D, 550D, 5D2, 5D3, 600D, 60D, 650D, 6D, 700D, 7D, EOSM
make: *** [raw_rec.dep] Error 1

[dmilligan]

it's either that or you didn't compile ML itself first before trying to compile whatever module this is, I'm assuming you did that right?

[dfort]

I was just following the instructions at the top of this topic:

In terminal, go to your source ./magic-lantern/modules/raw_rec

Type make. You will get some warnings about not having compiled the source for any model but these instructions are purely for getting the utilities to compile. You'll end up with our first utility, raw2dng, amongst other things like the module files.

Maybe everything went fine? Guess I'll have to wait until Friday night when I get home to find out.

[Audionut]

I updated the OP. 

You must build the autoexec.bin for your camera first, before compiling modules.

[dfort]

Thanks--inching a little closer. I noticed the change in the original post:

First, you must compile the autoexec.bin and associated .sym file for your camera.

I wasn't sure how to do this so I tried compile autoexec.bin for the EOSM this way.

1st try:

Code: [Select]

$ make EOSM
make -C  /Users/rosiefort/magic-lantern/platform/EOSM.202
[ VERSION  ]   ../../platform/EOSM.202/version.bin
make[1]: truncate: No such file or directory
make[1]: *** [../../platform/EOSM.202/version.bin] Error 1
make: *** [EOSM] Error 2

Ah ha--looks like truncate is missing. This should probably be added to the OP:

Code: [Select]

sudo port install truncate
This is what happened next:

Code: [Select]

$ make EOSM
make -C  /Users/rosiefort/magic-lantern/platform/EOSM.202
[ VERSION  ]   ../../platform/EOSM.202/version.bin
truncate: illegal option -- s
Internal error!
make[1]: *** [../../platform/EOSM.202/version.bin] Error 1
make: *** [EOSM] Error 2
Now I'm stuck again. I tried to build other platforms listed in the Makefile with the same results. I searched the wiki and forums for a fix but ran out of time for this weekend. I'm taking off tomorrow for another week of work away from my computer. No rush, but if anyone could give me a hint at what to try next I'd really appreciate it.

BTW--do I really have to compile ML before compiling raw2dng? My goal is simply to get raw2dng_cs2x2 running on a Mac to deal with the pink dots in EOS-M raw footage. Thanks!

[Audionut]

http://www.magiclantern.fm/forum/index.php?topic=7936.msg78404#msg78404

[dfort]

Thanks Audionut for pointing that out but I think I figured it out. The heading on this post is a bit misleading:

Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]

Correct me if I'm wrong but the way to compile raw2dng is to go to the modules/lv_rec directory and do this:

Code: [Select]

$ make raw2dng
[ GCC      ]   raw2dng
[ GCC      ]   raw2dng
raw2dng.c:70:103: warning: format specifies type 'int' but the argument has type
      'unsigned long' [-Wformat]
  ...= %d, should be 192", sizeof(lv_rec_file_footer_t));
       ~~                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
       %lu
raw2dng.c:39:77: note: expanded from macro 'FAIL'
#define FAIL(fmt,...) { fprintf(stderr, "Error: "); fprintf(stderr, fmt, ## __VA_ARGS__)...
                                                                            ^
1 warning generated.
[ GCC      ]   raw2dng

This produced a raw2dng executable. Now I need to figure out how to switch on the options to make this behave like raw2dng_cs2x2.

[kichetof]

There're some problems with the latest update of Xcode 5.1

When I compile cr2hdr I encounter this error:

Code: [Select]

make cr2hdr
REBUILDING
[ README   ]   module_strings.h
[ gcc      ]   cr2hdr
clang: error: unknown argument: '-mno-ms-bitfields' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
make: *** [cr2hdr] Error 1

If I disable -mno-ms-bitfields on Makefile.cr2hdr, it works fine!

CR2HDR_CFLAGS=-m32 -mno-ms-bitfields -O2 -Wall -I$(SRC_DIR) -D_FILE_OFFSET_BITS=64 -fno-strict-aliasing

I tried to bypass this error to set CFLAGS with CFLAGS=-Wunused-command-line-argument-hard-error-in-future and CFLAGS=Wno-error=-Wunused-command-line-argument-hard-error-in-future but nothing to do. Only remove the flag works for the moment

Some infos here, here and here

[dfort]

Success--I got raw2dng compiled and working on my Mac. To get rid of the pink and/or green focus dots it was simply a matter of uncommenting this line.

Code: [Select]

Before:
//~ #define CHROMA_SMOOTH
After:
#define CHROMA_SMOOTH
Developers--excuse me if I'm making this very elementary but I'd like to make this clear to non-programers who have cameras with pink dot issues and are on Macs.

The is my output of "make raw2dng"

Code: [Select]

$ make raw2dng
[ GCC      ]   raw2dng
[ GCC      ]   raw2dng
raw2dng.c:70:103: warning: format specifies type 'int' but the argument has type
      'unsigned long' [-Wformat]
  ...= %d, should be 192", sizeof(lv_rec_file_footer_t));
       ~~                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
       %lu
raw2dng.c:39:77: note: expanded from macro 'FAIL'
  ...{ fprintf(stderr, "Error: "); fprintf(stderr, fmt, ## __VA_ARGS__); fpri...
                                                           ^
raw2dng.c:622:13: warning: unused function 'chroma_smooth_3x3'
      [-Wunused-function]
static void chroma_smooth_3x3(unsigned short * inp, unsigned short * out...
            ^
raw2dng.c:957:13: warning: unused function 'chroma_smooth_5x5'
      [-Wunused-function]
static void chroma_smooth_5x5(unsigned short * inp, unsigned short * out...
            ^
3 warnings generated.
[ GCC      ]   raw2dng
Note that this creates a program that must be run from the command line, drag and drop doesn't work, so you'll need to read up on how to use the command line. Basically, using the Terminal app, navigate into the directory you want to save the .dng frames then--

Code: [Select]

[PATH_TO_raw2dng] [PATH_TO_*.RAW_FILES]
Of course you can put the compiled raw2dng program in a location that can be found in your "PATH" but setting it up depends on your environment and which shell you're using.

For any one else that wants to compile raw2dng on a Mac--please note that the instructions in the OP (original post) won't get you there and developers pointing to other posts are only going to lead you further down the rabbit hole. All that you need is in the magic-lantern/modules/lv-rec directory of the source code. You will need to get the gcc compiler running on your Mac but that can be as simple as installing Apple's Xcode from the app store. It is a free download. You'll also need to install mercurial and dcraw, MacPorts make that pretty easy. What you don't need to do is set up a complete ARM cross-compiling setup if you just want to get raw2dng working.

It would be great if the pre-compiled GUI raw to dng Mac apps could have an option to turn on CHROMA_SMOOTH for users who aren't able to compile ML.

[Hastyr]

I'm working to compile cr2hdr myself - even though I found a MacOS binary online.

It looks like I worked past the following error, though I still cannot build cr2hdr so this workaround may be not be correct.

truncate: illegal option -- s
Internal error!

the problem appears to be in: magic-lanter/src/Makefile.src
two lines have the following:

Code: [Select]

@truncate -s 284 $@and:

Code: [Select]

truncate -s 0 $@ ; \
the macports truncate command doesn't support the -s switch.  So I changed the lines to read:
 

Code: [Select]

@truncate $@ 284and

Code: [Select]

truncate $@ 0 ; \respectively.

now building the magic lantern build dies in 7D_master.203

Code: [Select]

_log truncated to save space_
make -C  /Users/foo/tmptmp/magic-lantern/platform/7D_MASTER.203
[ VERSION  ]   ../../platform/7D_MASTER.203/version.bin
[ CPP      ]   magiclantern.lds
[ AS       ]   entry.o
[ CC       ]   master_boot.o
../../platform/7D_MASTER.203/master_boot.c: In function 'master_callback':
../../platform/7D_MASTER.203/master_boot.c:161:13: error: implicit declaration of function 'ptpbuf_add' [-Werror=implicit-function-declaration]
             ptpbuf_add(0x4005, callstack, strlen(callstack));
             ^
../../platform/7D_MASTER.203/master_boot.c:177:27: warning: initialization makes pointer from integer without a cast [enabled by default]
         uint8_t *buffer = bkpt->ctx[0];
                           ^
cc1: some warnings being treated as errors
make[1]: *** [master_boot.o] Error 1
make: *** [7D_MASTER] Error 2
which I'll work on as I get time...
some cameras do get built.  If I try a make in dual_iso i get the above error

'-mno-ms-bitfields'

disabling it causes a new set of errors unfortunately. 

Code: [Select]

make
[ gcc      ]   cr2hdr
cr2hdr.c:344:34: error: use of undeclared identifier '__module_strings_MODULE_NAME'
    module_strpair_t *strings = &__module_strings_MODULE_NAME[0];
                                 ^
cr2hdr.c:1208:51: warning: if statement has empty body [-Wempty-body]
        if(system("octave --persist iso-curve.m"));
                                                  ^
cr2hdr.c:1208:51: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:1587:55: warning: if statement has empty body [-Wempty-body]
        if(system("octave --persist fullres-curve.m"));
                                                      ^
cr2hdr.c:1587:55: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:1919:56: warning: if statement has empty body [-Wempty-body]
            if(system("dcraw -d -r 1 1 1 1 edges.dng"));
                                                       ^
cr2hdr.c:1919:56: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:1933:59: warning: if statement has empty body [-Wempty-body]
            if(system("dcraw -d -r 1 1 1 1 edge-map.dng"));
                                                          ^
cr2hdr.c:1933:59: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:2264:51: warning: if statement has empty body [-Wempty-body]
        if(system("octave --persist mix-curve.m"));
                                                  ^
cr2hdr.c:2264:51: note: put the semicolon on a separate line to silence this warning
5 warnings and 1 error generated.
exiftool-bridge.c:58:9: warning: variable 'mode' is used uninitialized whenever 'if' condition
      is false [-Wsometimes-uninitialized]
    if (exif_file)
        ^~~~~~~~~
exiftool-bridge.c:70:9: note: uninitialized use occurs here
    if (mode != 0)
        ^~~~
exiftool-bridge.c:58:5: note: remove the 'if' if its condition is always true
    if (exif_file)
    ^~~~~~~~~~~~~~
exiftool-bridge.c:51:13: note: initialize the variable 'mode' to silence this warning
    int mode;
            ^
             = 0
1 warning generated.
make: *** [cr2hdr] Error 1
and no cr2hdr binary.

Next stop is a search of the forums again.  Any timesaving feedback would be appreciated.

Thanks,
Gus

[Hastyr]

and no cr2hdr binary.

Next stop is a search of the forums again.  Any timesaving feedback would be appreciated.

A bit more success!

I found that the following change prevents compiling cr2hdr on MacOS in my configuration..

Code: [Select]

https://bitbucket.org/hudson/magic-lantern/commits/5b78fb409ac81d2bec6b9fecdd352af26d3b27a5
I commented it completely out (the change is self consistent) and despite a slew of errors I am able to develop a binary.  When I use it to post process my photos it work but not as expected. The colors are "off".

Still this is still a work in progress, but getting closer

Thanks,
Gus

[kichetof]

On my side I have this problem since Xcode update to compile cr2hdr.

I need to remove -mno-ms-bitfields to be able to compile

[philmoz]

On my side I have this problem since Xcode update to compile cr2hdr.

I need to remove -mno-ms-bitfields to be able to compile

Or you can install GCC 4.8 or 4.9 (using homebrew or macports).
http://www.ficksworkshop.com/blog/14-coding/65-installing-gcc-on-mac

This will also allow you to build the new cr2hdr-20bit version of cr2hdr - XCode does not support nested functions.

Phil.

[jamespero]

Hey everybody,

I'm new to ML and somewhat new to C programming. I'm trying to get up to speed with ML development so that I can try and get development for the SL1/100D moving a little faster. I'm currently dealing with an error while trying to compile:

Code: [Select]

make -C  /Users/James/documents/repositories/magic-lantern/platform/100D.100
[ VERSION  ]   ../../platform/100D.100/version.bin
[ VERSION  ]   ../../platform/100D.100/version.c
[ CC       ]   version.o
[ CC       ]   boot-hack.o
../../src/boot-hack.c: In function 'my_task_dispatch_hook':
../../src/boot-hack.c:302:5: error: implicit declaration of function 'qprintf' [-Werror=implicit-function-declaration]
     qprintf("dispatch context->pc=0x%08X entry=0x%08X\n", context->pc, task->entry);
     ^
cc1: some warnings being treated as errors
make[1]: *** [boot-hack.o] Error 1
make: *** [100D] Error 2
Based on my experience with programming in php and javascript and a little with arduino, I'm guessing qprintf is not exactly a required element in getting ML to compile but I'm not familiar with qprintf specifically unlike printf and sprintf. Can someone tell me if this is something I need to worry about or how to get this error resolved?

[EDIT]

I'm thinking there's an include that missing from boot-hack.c like stdio.h or the like, but again I'm new to ML and mostly new to C so I'm not entirely sure.

[dmilligan]

it's for QEMU, what branch are you trying to compile? did you do a make clean?

[jamespero]

it's for QEMU, what branch are you trying to compile? did you do a make clean?

I'm trying to compile count's 100D wrk-qemu branch - https://bitbucket.org/berlincount/magic-lantern/commits/all

I've got qemu installed via homebrew, but I think I'm still missing something. I've run a make clean and that seems to complete without error.

I guess at this point I need a little guidance with qemu to make sure I have it installed and configured... :/

[dfort]

I've been having a PM exchange with Danne over compiling raw2dng and mlv_dump command line tools on the Mac and thought I'd share our success and failures with others on the forum. This seemed to be the appropriate forum topic to post this.

Apparently raw2dng, mlv_dump and cr2hdr can be compiled without having to install a cross platform ARM environment in order to build ML.

Starting with a virgin Mac, here's how to compile raw2dng:

1. Install Xcode. This is fairly easy because it is an Apple app.

2. Install the command line tools. This is a part of Xcode but apparently it isn't installed automatically. MacPorts has the instructions on how to get the command line tools working: https://www.macports.org/

3. Install MacPorts. Again, the MacPorts instructions are fairly easy to follow.

4. Install mercurial and dcraw via MacPorts. This is really easy:

Code: [Select]

sudo port install mercurial
sudo port install dcraw
5. Now using the terminal go into a directory where you would like to put the Magic Lantern source code and run this:

Code: [Select]

hg clone -r unified https://bitbucket.org/hudson/magic-lantern
Note that there are lots of other ways to do this that will get you the same results. This works too:

Code: [Select]

hg clone https://bitbucket.org/hudson/magic-lantern
cd magic-lantern/
hg update unified
6. cd into the directory where raw2dng.c and the Makefile for it is located:

Code: [Select]

cd [PATH TO YOUR WORKING DIRECTORY]/magic-lantern/modules/lv_rec
7. Now simply type this in the terminal.

Code: [Select]

make raw2dng
You will get some warning messages but don't worry, in that directory you will find the compiled raw2dng command line tool. It is easy to spot because it doesn't have a file extension.

By the way, what I wanted was a version of raw2dng with chroma smoothing turned on in order to take care of the pink focus pixels on EOS-M raw video. This was very simple to do because it is spelled out very clearly in the raw2dng.c code starting on line 33:

Code: [Select]

/* useful to clean pink dots, may also help with color aliasing, but it's best turned off if you don't have these problems */
//~ #define CHROMA_SMOOTH
I just removed the "//~" and it worked!

Next up, mlv_dump:

Trying to figure this one out on the forum topic was confusing because people were trying all sorts of things but the issue seems to be that the code was written on Linux systems and Apple Xcode sets up the developing environment differently. Long story short:

Code: [Select]

sudo port install gcc47 +universal; sudo port select --set gcc mp-gcc47; hash gcc
It takes a long time to run this, about a couple of hours on my system, so patience! Once it finishes you should be able to compile mlv_dump as well as raw2dng without having to make any source code tweaks.

Finally, cr2hdr:

Running "make cr2hdr" in the magic-lantern/modules/dual_iso directory gave me this error:

Code: [Select]

/bin/sh: rst2html: command not found
It looks like the Mac doesn't have rst2html installed. From what I found out that is a part of docutils but there is no docutils in MacPorts. I tried installing py-docutils. That didn't get me rst2html but it did install /opt/local/bin/rst2html-2.7.py so I made a soft link to it like this:

Code: [Select]

sudo ln -s /opt/local/bin/rst2html-2.7.py /opt/local/bin/rst2html
That did it--whew!

Hope this helps other ML users who want these command line tools running on their Macs. During my many attempts at compiling these tools I ended up installing some unnecessary components--hope I didn't leave out any necessary ones! Maybe one of them is truncate? If so, it can also be installed through MacPorts.