Resulting Image（png） Generation Issues

[RADIOMUMM]

Hello,

• What parameter do I use to ensure that the following 4 images are generated in the results folder?

qcpic1="%s/result/01.QC/%s_cleandata/LengthvsQualityScatterPlot_kde.png"%(output,samplename)
qcpic2="%s/result/01.QC/%s_cleandata/Non_WeightedHistogramReadlength.png"%(output,samplename)
qcpic3="%s/result/01.QC/%s_cleandata/Non_WeightedLogTransformed_HistogramReadlength.png"%(output,samplename)
qcpic4="%s/result/01.QC/%s_cleandata/Yield_By_Length.png"%(output,samplename)

• Now the command I use is:
  python3 NanoPlot.py -t 20 --N50 --dpi 300 --fastq test.fq.gz --title test -o %s/result/01.QC/%s

• I've tried adding the --plots and --legacy parameters, but they don't generate the same results every time.

[wdecoster]

they don't generate the same results every time

Could you elaborate on which files you get and which you are missing?

[RADIOMUMM]

I used the command mentioned above and ran it twice in a row, using NanoFilt to intercept the Reads in between.
(NanoPlot --> NanoFilt -->NanoPlot)

First run of NanoPlot (rawdata):

• 1.LengthvsQualityScatterPlot_dot.png
  -2.LengthvsQualityScatterPlot_kde.png
  -3.Non_weightedHistogramReadlength.png
  -4.Non_weightedLogTransformed_HistogramReadlength.png
  -5.WeightedHistogramReadlength.png
  -6.WeightedLogTransformed_HistogramReadlength.png
  -7.Yield_By_Length.png

Second run of NanoPlot (cleandata)：

• 1.LengthvsQualityScatterPlot_dot.png
  -2.LengthvsQualityScatterPlot_kde.png
  -3.Non_weightedHistogramReadlength.png
  -4.Non_weightedLogTransformed_HistogramReadlength.png
  -6.Yield_By_Length.png

This doesn't happen every time; sometimes the results are the same for both runs.

[wdecoster]

I cannot think of any reason for that to happen.

[RADIOMUMM]

No Yield_By_Length.png file was generated this time

(base)[root@418b066513c2 01.QC]# ll test_TB20009174_nocut_cleandata
total 3208
-rw-r--r-- 1 root root 487796 Mar 26 08:22 LengthvsQualityScatterPlot_dot.html
-rw-r--r-- 1 root root 44141 Mar 26 08:22 LengthvsQualityScatterPlot_dot.png
-rw-r--r-- 1 root root 725753 Mar 26 08:22 LengthvsQualityScatterPlot_kde.html
-rw-r--r-- 1 root root 63994 Mar 26 08:22 LengthvsQualityScatterPlot_kde.png
-rw-r--r-- 1 root root 1484310 Mar 26 08:22 NanoPlot-report.html
-rw-r--r-- 1 root root 3778 Mar 26 08:22 NanoPlot_20240326_0811.log
-rw-r--r-- 1 root root 841 Mar 26 08:15 NanoStats.txt
-rw-r--r-- 1 root root 844 Mar 26 08:15 NanoStats_post_filtering.txt
-rw-r--r-- 1 root root 13250 Mar 26 08:16 Non_weightedHistogramReadlength.html
-rw-r--r-- 1 root root 33020 Mar 26 08:16 Non_weightedHistogramReadlength.png
-rw-r--r-- 1 root root 15029 Mar 26 08:16 Non_weightedLogTransformed_HistogramReadlength.html
-rw-r--r-- 1 root root 47257 Mar 26 08:16 Non_weightedLogTransformed_HistogramReadlength.png
-rw-r--r-- 1 root root 14333 Mar 26 08:15 WeightedHistogramReadlength.html
-rw-r--r-- 1 root root 45382 Mar 26 08:15 WeightedHistogramReadlength.png
-rw-r--r-- 1 root root 17818 Mar 26 08:15 WeightedLogTransformed_HistogramReadlength.html
-rw-r--r-- 1 root root 43669 Mar 26 08:16 WeightedLogTransformed_HistogramReadlength.png
-rw-r--r-- 1 root root 198366 Mar 26 08:16 Yield_By_Length.html
-rw-r--r-- 1 root root 705 Mar 26 08:35 cyclone_qc.txt

(base)[root@418b066513c2 01.QC]# ll test_TB20009174_nocut_rawdata
total 3196
-rw-r--r-- 1 root root 483754 Mar 26 08:00 LengthvsQualityScatterPlot_dot.html
-rw-r--r-- 1 root root 48899 Mar 26 08:00 LengthvsQualityScatterPlot_dot.png
-rw-r--r-- 1 root root 719694 Mar 26 08:00 LengthvsQualityScatterPlot_kde.html
-rw-r--r-- 1 root root 84935 Mar 26 08:00 LengthvsQualityScatterPlot_kde.png
-rw-r--r-- 1 root root 1451376 Mar 26 08:00 NanoPlot-report.html
-rw-r--r-- 1 root root 3474 Mar 26 08:00 NanoPlot_20240326_0755.log
-rw-r--r-- 1 root root 845 Mar 26 08:00 NanoStats.txt
-rw-r--r-- 1 root root 13304 Mar 26 08:00 Non_weightedHistogramReadlength.html
-rw-r--r-- 1 root root 32855 Mar 26 08:00 Non_weightedHistogramReadlength.png
-rw-r--r-- 1 root root 15041 Mar 26 08:00 Non_weightedLogTransformed_HistogramReadlength.html
-rw-r--r-- 1 root root 43907 Mar 26 08:00 Non_weightedLogTransformed_HistogramReadlength.png
-rw-r--r-- 1 root root 14379 Mar 26 08:00 WeightedHistogramReadlength.html
-rw-r--r-- 1 root root 41574 Mar 26 08:00 WeightedHistogramReadlength.png
-rw-r--r-- 1 root root 17786 Mar 26 08:00 WeightedLogTransformed_HistogramReadlength.html
-rw-r--r-- 1 root root 39531 Mar 26 08:00 WeightedLogTransformed_HistogramReadlength.png
-rw-r--r-- 1 root root 177708 Mar 26 08:00 Yield_By_Length.html
-rw-r--r-- 1 root root 41155 Mar 26 08:00 Yield_By_Length.png
-rw-r--r-- 1 root root 707 Mar 26 08:35 cyclone_qc.txt

[RADIOMUMM]

I cannot think of any reason for that to happen.
It's so weird.

[wdecoster]

Can you share the log file of a run that failed to generate a png plot?

[RADIOMUMM]

Can you share the log file of a run that failed to generate a png plot?

2024-03-26 08:11:31,873 NanoPlot 1.42.0 started with arguments Namespace(N50=True, alength=False, bam=None, barcoded=False, color='#4CB391', colormap='Greens', cram=None, downsample=None, dpi=300, drop_outliers=False, fasta=None, fastq=['/Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata_trimmed-reads.fastq.gz'], fastq_minimal=None, fastq_rich=None, feather=None, font_scale=1, format=['png'], hide_stats=False, huge=False, info_in_report=False, legacy=None, listcolormaps=False, listcolors=False, loglength=False, maxlength=None, minlength=None, minqual=None, no_N50=False, no_static=False, no_supplementary=False, only_report=False, outdir='/Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata', path='/Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/', percentqual=False, pickle=None, plots=['kde', 'dot'], prefix='', raw=False, readtype='1D', runtime_until=None, store=False, summary=None, threads=20, title='test_TB20009174_nocut_cleandata', tsv_stats=False, ubam=None, verbose=False)
2024-03-26 08:11:31,873 Python version is: 3.8.18 (default, Sep 11 2023, 13:40:15) [GCC 11.2.0]
2024-03-26 08:11:31,885 Nanoget: Starting to collect statistics from plain fastq file.
2024-03-26 08:11:31,885 Nanoget: Decompressing gzipped fastq /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata_trimmed-reads.fastq.gz
2024-03-26 08:15:57,025 Reduced DataFrame memory usage from 4.4492950439453125Mb to 4.4492950439453125Mb
2024-03-26 08:15:57,069 Nanoget: Gathered all metrics of 291581 reads
2024-03-26 08:15:57,213 Calculated statistics
2024-03-26 08:15:57,213 Using sequenced read lengths for plotting.
2024-03-26 08:15:57,222 Removed 29 artefactual reads with very short length and very high quality.
2024-03-26 08:15:57,355 Calculated statistics
2024-03-26 08:15:57,356 NanoPlot: Valid color #4CB391.
2024-03-26 08:15:57,356 NanoPlot: Valid colormap Greens.
2024-03-26 08:15:57,381 NanoPlot: Creating length plots for Read length.
2024-03-26 08:15:57,381 NanoPlot: Using 291552 reads with read length N50 of 30095bp and maximum of 117406bp.
2024-03-26 08:15:59,533 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/WeightedHistogramReadlength as png (or png for --legacy)
2024-03-26 08:16:01,478 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/WeightedLogTransformed_HistogramReadlength as png (or png for --legacy)
2024-03-26 08:16:03,489 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/Non_weightedHistogramReadlength as png (or png for --legacy)
2024-03-26 08:16:05,567 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/Non_weightedLogTransformed_HistogramReadlength as png (or png for --legacy)
2024-03-26 08:22:49,945 No static plots are saved due to some kaleido problem:
2024-03-26 08:22:49,945 Transform failed with error code 1: Failed to serialize document: Uncaught
2024-03-26 08:22:49,946 Created length plots
2024-03-26 08:22:49,952 NanoPlot: Creating Read lengths vs Average read quality plots using 291552 reads.
2024-03-26 08:22:52,388 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/LengthvsQualityScatterPlot_dot as png (or png for --legacy)
2024-03-26 08:22:55,236 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/LengthvsQualityScatterPlot_kde as png (or png for --legacy)
2024-03-26 08:22:55,236 Created LengthvsQual plot
2024-03-26 08:22:55,237 Writing html report.
2024-03-26 08:22:55,258 Finished!

[wdecoster]

This is it:

2024-03-26 08:22:49,945 No static plots are saved due to some kaleido problem:
2024-03-26 08:22:49,945 Transform failed with error code 1: Failed to serialize document: Uncaught

This doesn't help us, but for some reason it failed to create the PNG image for a particular plot. But you still have the plot in HTML format, right?

[RADIOMUMM]

This is it:

2024-03-26 08:22:49,945 No static plots are saved due to some kaleido problem:
2024-03-26 08:22:49,945 Transform failed with error code 1: Failed to serialize document: Uncaught

This doesn't help us, but for some reason it failed to create the PNG image for a particular plot. But you still have the plot in HTML format, right?

right

[hoelzer]

Hey, I have the same problem: no PNGs are produced and the errors are always:

2024-06-04 10:04:00,961 No static plots are saved due to some kaleido problem:
2024-06-04 10:04:00,961 Transform failed with error code 1: Failed to serialize document: Uncaught

I am running on a SLURM cluster and have NanoPlot installed in a Singularity container. NanoPlot version is 1.42.0

[hoelzer]

This might be related to

https://stackoverflow.com/questions/71118601/saving-a-plotly-image-not-working-with-kaleido-even-though-it-is-installed

I am still wondering if this has something to do with executing NanoPlot from a container and/or a HPC environment...

[hoelzer]

Further research revelad that this might be also a problem with internet connection that Kaleido apparently needs to convert HTML to PNG. THis would explain my issue bc I work on a quite restrictive HPC with a lot of ports closed.

See:
plotly/Kaleido#108

Maybe it is possible to also configure Kaleido in NanoPlot to work offline?