PinkDotRemover tool 650D

[foorgol]

I saw the pink / blue dots in the DNGs as well and I think I have a solution. Or even two (!) solutions...

I) The easy way: use RawTherapee (http://rawtherapee.com)

For those of you who don't know: RawTherapee is a really powerful open source tool for developing RAWs. Runs under Windows, Linux and Mac.

You can directly open a DNG-file and if you then go to the "Raw"-Tab, unfold the "Preprocessing"-section and check "Hot/Dead pixel filter", the disturbing dots will magically disappear. Okay, most of them. Before you do this, you should select a processing profile from the drop down list under the histogram, because selecting a profile resets your "Hot/Dead pixel filter" setting (and all other settings as well) with the profile's default.

RawTherapee can do batch processing, so you can convert a whole image sequence for a movie with just a few clicks.


II) The DIY way: start coding

I hacked a small command line tool that removes the pink pixels directly in the DNG file. It's written in Java, so it should run under Windows and Mac, although I developed it under Linux... but who knows...

Currently the program can only remove pixels in DNGs with 1280x720 resolution, because it uses pre-stored pixel locations. I "calibrated" it to my 650D, but I guess/hope the pixel positions are identical on all 650Ds. Here's an example:

Original DNG:


Modified DNG with dots removed:

(both JPGs developed with RawTherapee and identical settings)

If you want to play with the tool, here it is:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover.zip
Just run it like "java -jar PinkDotRemover.jar YourFile.dng". If you're lucky, the program creates a _YourFile.dng with removed dots. Files will be overwritten without confirmation. Currently the tool is just a quick hack without focus on security or stability... use it at your own risk and cross your fingers...

As you can see above, there are still a very few dots left which the tool doesn't remove yet. But that's only a matter of fine tuning.

I'll clean up the code tomorrow and upload the sources as well. If you're (rightfully!) hesitant to execute a stranger's JAR-files on your machine, you can compile the JAR yourself by then.

Here are a few sample DNGs to compare the original with the modified image:

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M3_before.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M3_after.dng

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_before.dng (DNG for the JPG above)
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_after.dng (DNG for the JPG above)

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M5_before.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M5_after.dng


Let me know what you think... if you think it's useful, I could extend the tool to batch-convert all DNGs in a directory... or to handle resolutions other than 1280x720...

[1ricca]

I saw the pink / blue dots in the DNGs as well and I think I have a solution. Or even two (!) solutions...

I) The easy way: use RawTherapee (http://rawtherapee.com)

For those of you who don't know: RawTherapee is a really powerful open source tool for developing RAWs. Runs under Windows, Linux and Mac.

You can directly open a DNG-file and if you then go to the "Raw"-Tab, unfold the "Preprocessing"-section and check "Hot/Dead pixel filter", the disturbing dots will magically disappear. Okay, most of them. Before you do this, you should select a processing profile from the drop down list under the histogram, because selecting a profile resets your "Hot/Dead pixel filter" setting (and all other settings as well) with the profile's default.

RawTherapee can do batch processing, so you can convert a whole image sequence for a movie with just a few clicks.


II) The DIY way: start coding

I hacked a small command line tool that removes the pink pixels directly in the DNG file. It's written in Java, so it should run under Windows and Mac, although I developed it under Linux... but who knows...

Currently the program can only remove pixels in DNGs with 1280x720 resolution, because it uses pre-stored pixel locations. I "calibrated" it to my 650D, but I guess/hope the pixel positions are identical on all 650Ds. Here's an example:

Original DNG:


Modified DNG with dots removed:

(both JPGs developed with RawTherapee and identical settings)

If you want to play with the tool, here it is:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover.zip
Just run it like "java -jar PinkDotRemover.jar YourFile.dng". If you're lucky, the program creates a _YourFile.dng with removed dots. Files will be overwritten without confirmation. Currently the tool is just a quick hack without focus on security or stability... use it at you own risk and cross your fingers...

As you can see above, there are still a very few dots left which the tool doesn't remove yet. But that's only a matter of fine tuning.

I'll clean up the code tomorrow and upload the sources as well. If you're (rightfully!) hesitant to execute a stranger's JAR-files on your machine, you can compile the JAR yourself by then.

Here are a few sample DNGs to compare the original with the modified image:

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M3_before.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M3_after.dng

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_before.dng (DNG for the JPG above)
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_after.dng (DNG for the JPG above)

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M5_after.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M5_after.dng


Let me know what you think... if you think it's useful, I could extend the tool to batch-convert all DNGs in a directory... or to handle resolutions other than 1280x720...

There's already a superior way than either of these for now. In Lightroom or ACR, turn Noise Removal in both Luminance and Color all the way up and then adjust the detail sliders until just around the point where the dots disappear enough. Finish developing the rest of the settings (exposure, vibrance, brushes, graduated filters, WB, etc) then just select all the photos in LR or Bridge and sync your develope settings. Voila.

[Aborgh]

I saw the pink / blue dots in the DNGs as well and I think I have a solution. Or even two (!) solutions...

I) The easy way: use RawTherapee (http://rawtherapee.com)

For those of you who don't know: RawTherapee is a really powerful open source tool for developing RAWs. Runs under Windows, Linux and Mac.

You can directly open a DNG-file and if you then go to the "Raw"-Tab, unfold the "Preprocessing"-section and check "Hot/Dead pixel filter", the disturbing dots will magically disappear. Okay, most of them. Before you do this, you should select a processing profile from the drop down list under the histogram, because selecting a profile resets your "Hot/Dead pixel filter" setting (and all other settings as well) with the profile's default.

RawTherapee can do batch processing, so you can convert a whole image sequence for a movie with just a few clicks.


II) The DIY way: start coding

I hacked a small command line tool that removes the pink pixels directly in the DNG file. It's written in Java, so it should run under Windows and Mac, although I developed it under Linux... but who knows...

Currently the program can only remove pixels in DNGs with 1280x720 resolution, because it uses pre-stored pixel locations. I "calibrated" it to my 650D, but I guess/hope the pixel positions are identical on all 650Ds. Here's an example:

Original DNG:


Modified DNG with dots removed:

(both JPGs developed with RawTherapee and identical settings)

If you want to play with the tool, here it is:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover.zip
Just run it like "java -jar PinkDotRemover.jar YourFile.dng". If you're lucky, the program creates a _YourFile.dng with removed dots. Files will be overwritten without confirmation. Currently the tool is just a quick hack without focus on security or stability... use it at you own risk and cross your fingers...

As you can see above, there are still a very few dots left which the tool doesn't remove yet. But that's only a matter of fine tuning.

I'll clean up the code tomorrow and upload the sources as well. If you're (rightfully!) hesitant to execute a stranger's JAR-files on your machine, you can compile the JAR yourself by then.

Here are a few sample DNGs to compare the original with the modified image:

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M3_before.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M3_after.dng

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_before.dng (DNG for the JPG above)
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_after.dng (DNG for the JPG above)

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M5_after.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M5_after.dng


Let me know what you think... if you think it's useful, I could extend the tool to batch-convert all DNGs in a directory... or to handle resolutions other than 1280x720...

You should do a batch tool

[foorgol]

Can you find the dot locations? I believe their position is fixed (maybe they change with resolution), but if you know where they are, you can just set them as 0 (bad pixel mark) in raw2dng.

It seems you have two kinds of distorted pixels: first the ones that show up in the center in light areas and second a lot more all over the image in dark areas. For the light ones, I manually picked the start and end coordinates of each "dot grid block" and stored them in my program to iterate over them. For the dark ones, I recorded a completely dark movie frame, pre-processed it a little bit (basically I applied a threshold to suppress noise) and got this one:



I extracted the coordinates of each non-black pixel with a little script and stored them in a text file. During the dot removal procedure I read this file and fix the dots at the stored locations. The coordinate file is here:

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/pixCoord_threshold2068.txt

If you think it's useful I could easily create a combined file with the "light" and the "dark" coordinates. Could you integrate that with raw2dng?

And, as I said: so far I only have the data for 1280x720. Simply scaling them mathematically to other resolutions is unlikely to work well, because you would get rounding errors for non-integer scaling factors. And if you're just one pixel off with the scaled coordinate, the dot removal doesn't work anymore...

[foorgol]

There's already a superior way than either of these for now. In Lightroom or ACR, [...]

You shouldn't forget that not everyone runs Adobe software.

[xanta]

It seems you have two kinds of distorted pixels: first the ones that show up in the center in light areas and second a lot more all over the image in dark areas. For the light ones, I manually picked the start and end coordinates of each "dot grid block" and stored them in my program to iterate over them. For the dark ones, I recorded a completely dark movie frame, pre-processed it a little bit (basically I applied a threshold to suppress noise) and got this one:



I extracted the coordinates of each non-black pixel with a little script and stored them in a text file. During the dot removal procedure I read this file and fix the dots at the stored locations. The coordinate file is here:

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/pixCoord_threshold2068.txt

If you think it's useful I could easily create a combined file with the "light" and the "dark" coordinates. Could you integrate that with raw2dng?

And, as I said: so far I only have the data for 1280x720. Simply scaling them mathematically to other resolutions is unlikely to work well, because you would get rounding errors for non-integer scaling factors. And if you're just one pixel off with the scaled coordinate, the dot removal doesn't work anymore...

What you are doing is incredible!!!!Thanks so much

[foorgol]

I'll clean up the code tomorrow and upload the sources as well. If you're (rightfully!) hesitant to execute a stranger's JAR-files on your machine, you can compile the JAR yourself by then.

Ok, here's a new version! You can now pass a directory name as a command line argument and all files ending in "dng" or "DNG" in this directory will be batch-processed. Or you pass multiple file names as arguments. Or mix it. As you like.

The JAR-file is here:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__001.zip

Just unzip it into a separate folder and execute the JAR file with
java -jar PinkDotRemover.jar <TheFilesAndDirectoriesYouWantToProcess.dng>

The source code for the main application:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__001.src.zip

And the source code for a primitive TIFF-lib to access the DNG-files:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__001.src.zip

You need the lib in order to compile the main application. The source ZIPs contain complete NetBeans-Project trees. If you use a different IDE, just pick the .java-files from the ZIPs. There are no other libs or dependencies for this application.

Please let me know if something doesn't work for you. I've never deployed a java application outside of my little development pond, so maybe I missed something...

And: it's still 1280 x 720 only

[Aborgh]

Ok, here's a new version! You can now pass a directory name as a command line argument and all files ending in "dng" or "DNG" in this directory will be batch-processed. Or you pass multiple file names as arguments. Or mix it. As you like.

The JAR-file is here:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__001.zip

Just unzip it into a separate folder and execute the JAR file with
java -jar PinkDotRemover.jar <TheFilesAndDirectoriesYouWantToProcess.dng>

The source code for the main application:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__001.src.zip

And the source code for a primitive TIFF-lib to access the DNG-files:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__001.src.zip

You need the lib in order to compile the main application. The source ZIPs contain complete NetBeans-Project trees. If you use a different IDE, just pick the .java-files from the ZIPs. There are no other libs or dependencies for this application.

Please let me know if something doesn't work for you. I've never deployed a java application outside of my little development pond, so maybe I missed something...

And: it's still 1280 x 720 only

Sweet! Thanks but I am not sure if it works for me.

I do this:

1. open terminal (using mac)

2. type "cd <and the way to my folder with the jar>

3. type in "java -jar PinkDotRemover.jar 000000.dng"

4. I get this text "Exception in thread "main" java.lang.UnsupportedClassVersionError: org/nodomain/volkerk/PinkDotRemover/PinkDotRemoverMain : Unsupported major.minor version 51.0
   at java.lang.ClassLoader.defineClass1(Native Method)
   at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
   at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
   at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
   at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
   at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
   at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)"

[foorgol]

I get this text "Exception in thread "main" java.lang.UnsupportedClassVersionError: org/nodomain/volkerk/PinkDotRemover/PinkDotRemoverMain : Unsupported major.minor version 51.0

Hmmm... could it be you're running a not-so-much-up-to-date version of java?

The program requires Java 1.7; in Java's weird logic, that the "51.0" in the error message you posted.

Try to run "java -version" to check the java version you're using.


I'll create a new version that runs with the older Java 1.6 later, maybe today. During more testing, found a few ugly pixel distortions in certain regions on certain images, so that I need to create an improved version anyway....

[Aborgh]

Hmmm... could it be you're running a not-so-much-up-to-date version of java?

The program requires Java 1.7; in Java's weird logic, that the "51.0" in the error message you posted.

Try to run "java -version" to check the java version you're using.


I'll create a new version that runs with the older Java 1.6 later, maybe today. During more testing, found a few ugly pixel distortions in certain regions on certain images, so that I need to create an improved version anyway....

I have java 7 update 21 , so the latest but in terminal it shows 1.6 :S

Tried it in windows and it worked...

[foorgol]

I'll create a new version that runs with the older Java 1.6 later, maybe today. During more testing, found a few ugly pixel distortions in certain regions on certain images, so that I need to create an improved version anyway....

Well then, try this one:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__002.zip

* Works with Java 1.5 or later
* Improved dot positions -- I had a few gaps here and there
* Better interpolation of defective pixels along edges -- applying a gradient-based algorithm now instead of plain, static averaging

Enjoy!


Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__002.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__002.src.zip

[xanta]

Well then, try this one:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__002.zip

* Works with Java 1.5 or later
* Improved dot positions -- I had a few gaps here and there
* Better interpolation of defective pixels along edges -- applying a gradient-based algorithm now instead of plain, static averaging

Enjoy!


Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__002.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__002.src.zip

The result is great but some dots still remains in the shadows but its good enough

[foorgol]

Ok guys, time to bother you with a new version of the pink dot remover! This time I added the dot locations for the silent DNGs. So now the program can process both, 1280x720 and 1734x965. Here's a "before" and "after" example of a silent DNG with identical RAW development settings and corrected aspect ratio:




Here's the binary, just unzip it and run java -jar PinkDotRemover.jar <YourFileNameOrDirectory> in the shell:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.zip

Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__003.src.zip

By the way: I was wrong in one of my previous posts; the programm definitely requires Java 1.7... sorry...

[Edgar Matos]

Pretty sweet @foorgol. I would love to try it but, yeah... No, I can't. I don't know how to play with java. I feel like dead weight. The only thing I can do is to wave my hands in the air and send you guys positive energy . Would it be possible to run this using raw2dng?

By the way @Sandsnor. You mentioned that your camera turned off because the battery died, and then you camera died as well.
It happens That my camera turned off automatically. It wouldn't turn on until I used other battery with the memory still inside.
What is ironic, is that the camera was telling me that the battery was full before it turned off, but it wasn't. What I wanted to say is that the code is doing funny things, but i'm sure your camera is alright.       

[demetrisag]

Ok guys, time to bother you with a new version of the pink dot remover! This time I added the dot locations for the silent DNGs. So now the program can process both, 1280x720 and 1734x965. Here's a "before" and "after" example of a silent DNG with identical RAW development settings and corrected aspect ratio:




Here's the binary, just unzip it and run java -jar PinkDotRemover.jar <YourFileNameOrDirectory> in the shell:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.zip

Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__003.src.zip

By the way: I was wrong in one of my previous posts; the programm definitely requires Java 1.7... sorry...

Foorgol I have the latest java installed on my system. So when I go on CMD and CD to the pindotremover folder when I Write the code you mentioned above it says "bad command, unreconisable"

[COMMANDES]

Ok guys, time to bother you with a new version of the pink dot remover! This time I added the dot locations for the silent DNGs. So now the program can process both, 1280x720 and 1734x965. Here's a "before" and "after" example of a silent DNG with identical RAW development settings and corrected aspect ratio:

\\img
\\img

Here's the binary, just unzip it and run java -jar PinkDotRemover.jar <YourFileNameOrDirectory> in the shell:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.zip

Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__003.src.zip

By the way: I was wrong in one of my previous posts; the programm definitely requires Java 1.7... sorry...

Hi foorgol, i get this error

what am I doing wrong? I have java 7 upd 21
For shooting mode used "silent picture"

[donjames150]

Ok guys, time to bother you with a new version of the pink dot remover! This time I added the dot locations for the silent DNGs. So now the program can process both, 1280x720 and 1734x965. Here's a "before" and "after" example of a silent DNG with identical RAW development settings and corrected aspect ratio:

\img
\img

Here's the binary, just unzip it and run java -jar PinkDotRemover.jar <YourFileNameOrDirectory> in the shell:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.zip

Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__003.src.zip

By the way: I was wrong in one of my previous posts; the programm definitely requires Java 1.7... sorry...

thanks so much for your work on this, it works very well, much better than Rawtherapee, although that works relatively well, but still leaves some dots.
If you do use Rawtherapee, use version 3, version 4 kept crashing on me with Win7
It took me awhile to find out I needed to set my environmental variable path so that Windows could locate the java command, but now it works well.

Watch at 720 HD, dots are most noticeable

For Sandsnor,
If you don't have a 2nd battery, I would go to Best Buy or somewhere and get one, that would rule out
that possibility. Radio Shack has them for $22. Lenmar LP-E8

[demetrisag]

Hi foorgol, i get this error

what am I doing wrong? I have java 7 upd 21
For shooting mode used "silent picture"

exact same problem here

[foorgol]

Hi foorgol, i get this error

what am I doing wrong? I have java 7 upd 21
For shooting mode used "silent picture"

exact same problem here

Hmmm... weird... this is most likely an error around reading / writing files. As I am a lazy guy, I don't check for these errors, because I simply assume that I have always the rights and space to read and write

On COMMANDES' screenshot I can also see that you're using special characters in your path name. Maybe that's part of the problem.

So could you please try again and...

  * use path and filenames without special characters. Choose something simple like "C:\temp". No Cyrillic characters or similar, please. Just plain ASCII.

  * make sure that you have permission to read and write in this directory

If this does not solve the problem, please let me know. I would then send you a special debug version with excessive logging to trace down the issue.

[foorgol]

I would love to try it but, yeah... No, I can't. I don't know how to play with java. I feel like dead weight.

*sigh*

That sounds to me as if you're not yet fully in love with all the amenities of the command line.

Well, I also got PMs asking for a GUI. Give me a few days and I'll try to fit the program with a simple GUI, e. g. a file selection dialog. If Java is correctly installed on your machine, a double-click on the jar-file should be sufficient to fire up the GUI.

[Lucas_W]

Next they'll be asking for iOS themed GUI...

</Applerant>

[COMMANDES]

Hmmm... weird... this is most likely an error around reading / writing files. As I am a lazy guy, I don't check for these errors, because I simply assume that I have always the rights and space to read and write

On COMMANDES' screenshot I can also see that you're using special characters in your path name. Maybe that's part of the problem.

So could you please try again and...

  * use path and filenames without special characters. Choose something simple like "C:\temp". No Cyrillic characters or similar, please. Just plain ASCII.

  * make sure that you have permission to read and write in this directory

If this does not solve the problem, please let me know. I would then send you a special debug version with excessive logging to trace down the issue.

So ... Again, this error ... Sad ...

[foorgol]

So ... Again, this error ... Sad ...

For all of you who got that error: could you please download this version

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip

and send me the output via PM?

Please maximize the terminal window, because the debug texts can become quite long...

Thanks a lot!

[demetrisag]

For all of you who got that error: could you please download this version

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip

and send me the output via PM?

Please maximize the terminal window, because the debug texts can become quite long...

Thanks a lot!

There you go mate, I only skipped one line at the very top, which was the command in order to include it all in one screenshot



Thanks a lot for doing that!

And satriani, you are an angel!

[blade]

I have moved this topic out of the 650D alpha topic.  http://www.magiclantern.fm/forum/index.php?topic=6320.250

This should provide a better overview for this nice tool.

Kind regards
Blade