Nouvelles: We are delighted to welcome you to this forum

The version 0.9.9 is available !!!

Auteur Sujet: Use multithread wherever possible.  (Lu 66 fois)

Howking

  • Mercurien
  • *
  • Messages: 4
Use multithread wherever possible.
« le: septembre 01, 2018, 02:15:44 14 »
I'm processing a huge number of files. 10000 etc. So I very much sad that parallel processing is not used everywhere, even if possible. File conversion, preprocesing, .... Future astrofotografie is also in "lucky maging" and processors will be more and more multi-core.

lock042

  • Administrator
  • Terrien
  • *****
  • Messages: 189
Re : Use multithread wherever possible.
« Réponse #1 le: septembre 01, 2018, 02:31:13 14 »
Almost all algorithms are using multithread.
Some are intended to be ported when we will have time. However, in lucky imaging we usually save in SER file, and not in FITS. So the conversion step is skipped.
Preprocessing is not a slow step so that's not a big deal.

Registration and stacking, that can take a long time, are both multithreaded.
« Modifié: septembre 01, 2018, 02:32:53 14 par lock042 »

Howking

  • Mercurien
  • *
  • Messages: 4
Re : Use multithread wherever possible.
« Réponse #2 le: septembre 01, 2018, 05:01:33 17 »
Now I have processed 5400 files. Preprocessing with dark, flat and demosaic. It used one core and took 187 minutes !!!! With 8 cores it could be done in about 30 minutes. That is a big difference.
16:46:48: Filter Pattern: RGGB
16:46:49: Saving FITS: file pp_m75_05397.fit, 3 layer(s), 4032x3024 pixels
16:46:49: Reading FITS: file m75_05398.fit, 1 layer(s), 4032x3024 pixels
16:46:50: Filter Pattern: RGGB
16:46:52: Saving FITS: file pp_m75_05398.fit, 3 layer(s), 4032x3024 pixels
16:46:52: Reading FITS: file m75_05399.fit, 1 layer(s), 4032x3024 pixels
16:46:52: Filter Pattern: RGGB
16:46:54: Saving FITS: file pp_m75_05399.fit, 3 layer(s), 4032x3024 pixels
16:46:54: Reading FITS: file m75_05400.fit, 1 layer(s), 4032x3024 pixels
16:46:54: Filter Pattern: RGGB
16:46:56: Saving FITS: file pp_m75_05400.fit, 3 layer(s), 4032x3024 pixels
16:46:56: Execution time: 187 min 04 s.
16:46:57: Reading FITS: file pp_m75_00001.fit, 3 layer(s), 4032x3024 pixels
16:46:57: Sequence loaded: pp_m75_ (1->5400)
16:46:57: Closing sequence m75_

I usually use 10000 light, 10000 dark and 3000 flat raw files. Multi-threaded processing would greatly help me.

Right now I do this by converting RAW to the dcraw program.
#!/bin/sh
N=0;
I=0;
L=8;
G=2.222;
sudo purge
for F in *
do
N=$(($N + 1))
done
K=$(($(($N /8)) + 1))
for F in *
do
mkdir -p $(($I / $K))
mv $F $(($I / $K))
I=$(($I + 1))
done
dcraw -h -6 -W -b $L -g $G 4.5 -T -t 0 0/*.dng &
dcraw -h -6 -W -b $L -g $G 4.5 -T -t 0 1/*.dng &
dcraw -h -6 -W -b $L -g $G 4.5 -T -t 0 2/*.dng &
dcraw -h -6 -W -b $L -g $G 4.5 -T -t 0 3/*.dng &
dcraw -h -6 -W -b $L -g $G 4.5 -T -t 0 4/*.dng &
dcraw -h -6 -W -b $L -g $G 4.5 -T -t 0 5/*.dng &
dcraw -h -6 -W -b $L -g $G 4.5 -T -t 0 6/*.dng &
dcraw -h -6 -W -b $L -g $G 4.5 -T -t 0 7/*.dng &
wait
rm -rf [0-3]/*.dng
rm -rf [4-7]/*.dng
mv [0-3]/*.tiff $(pwd)
mv [4-7]/*.tiff $(pwd)
rm -rf [0-7]
say done


And then preprocessing dakt, flat, aligning and stackik with Lynkeos.

But I would like to use SiriL, I like it. I just needed to use HW for my file size to the full. I have i7 4.3GHz 4 cores(8 treads) and 4x250GB SSD stripping RAID with write and read 1.8GB / s.

Howking

  • Mercurien
  • *
  • Messages: 4
Re : Use multithread wherever possible.
« Réponse #3 le: septembre 01, 2018, 05:23:39 17 »
Now I've aligned the 5400 files (490GB) I calibrated for 3 hours with 1 core, in just 1m 22s with eight cores.
17:06:45: Registration: processing using method: One Star Registration (deep-sky)
17:06:45: PSF on area: processing...
17:07:33: No star found in the area image 1074 around 2128,1059 (use 'follow star' option?)
17:08:07: Sequence processing partially succeeded, with 1 images that failed and that were temporarily excluded from the sequence.
17:08:07: Execution time: 1 min 22 s.
17:08:07: Registration finished.
17:08:07: Best frame: #5246 with fwhm=2.82.

And he folded in just 5 minutes (490GB) - that's a fantastic performance!!! :-O Only if there not were 3 hours for calibration :-(
17:20:07: Sequence processing succeeded.
17:20:07: Execution time: 5 min 10 s.
17:20:08: Integration of 5400 images:
17:20:08: Pixel combination ......... normalized sum
17:20:08: Normalization ............. none
17:20:08: Pixel rejection ........... none
17:20:08: Rejection parameters ...... none
17:20:08: Background noise value (channel: #0): 88.991 (1.358e-03)
17:20:08: Background noise value (channel: #1): 89.061 (1.359e-03)
17:20:08: Background noise value (channel: #2): 87.387 (1.333e-03)
17:20:08: Saving FITS: file pp_m75_stacked.fit, 3 layer(s), 4032x3024 pixels

The SiriL program is fantastic. That's why I like it. Please do multi-threaded processing everywhere.

lock042

  • Administrator
  • Terrien
  • *****
  • Messages: 189
Re : Use multithread wherever possible.
« Réponse #4 le: septembre 01, 2018, 06:53:21 18 »
Coding takes time, a lot of time. But we are only two on the project.
We are now working on the planetary mode of Siril. So we don't have the time for now but it is on the roadmap : https://free-astro.org/bugs/view.php?id=113
« Modifié: septembre 01, 2018, 07:02:11 19 par lock042 »

Howking

  • Mercurien
  • *
  • Messages: 4
Re : Use multithread wherever possible.
« Réponse #5 le: septembre 01, 2018, 07:20:01 19 »
Well, I know. I've programmed too, but it's been more years back. I would like to join your project, but now it would be easier for me to write the whole program again before i oriented an know in yours SiriL. Meanwhile, I will be forced to use dcraw and lynkeos again. The whole process takes from debayering raw through calibration  and stack 3 hours. (10000 light, 10000 dark and 3000 flat) With SiriL it would be more than 8 hours and that's too much. Damn, SiriL I liked it very much. Thank you for the reaction.

PS:I will monitor the development of SiriL and look forward to the implementation of multi-threads in the future. :-)