justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 14359.2@dunegpschedd02.fnal.gov

Jobsub ID14359.2@dunegpschedd02.fnal.gov
Workflow ID330
Stage ID1
User namemsiden@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-04 16:22:02
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-08-04 22:09:00
From worker nodeHostnamewn-lb-04.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-04 16:24:01
Input fileshd-protodune:np04hd_raw_run027087_0011_dataflow1_datawriter_0_20240614T145109.hdf5
JobscriptExit code0
Real time5h (20645s)
CPU time5h (20560s = 99%)
Max RSS bytes6144450560 (5859 MiB)
Outputting started2025-08-04 22:08:07
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/msiden/channel_map_test/fnal/00330/1/001/pdhd_processing_1_14359_2_1754324657_stage1_stage2.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/msiden/channel_map_test/fnal/00330/1/001/pdhd_processing_1_14359_2_1754324657_stage1_stage2_tfile.root
Finished2025-08-04 22:09:00
Saved logsjustin-logs:14359.2-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

FrameSaver: q=1.93364e+06 n=974095 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 69420 traces, 69420 summary
Retagger: tagging trace set: gauss with 60166 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 60166 traces tagged "gauss"
FrameSaver: q=1.48557e+06 n=784303 tag=gauss
wclsFrameSaver: saving 69420 traces tagged "wiener"
FrameSaver: q=1.60737e+06 n=809563 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 68648 traces, 68648 summary
Retagger: tagging trace set: gauss with 59750 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 59750 traces tagged "gauss"
FrameSaver: q=1.46958e+06 n=795152 tag=gauss
wclsFrameSaver: saving 68648 traces tagged "wiener"
FrameSaver: q=1.59577e+06 n=818475 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 65451 traces, 65451 summary
Retagger: tagging trace set: gauss with 58830 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 58830 traces tagged "gauss"
FrameSaver: q=1.25648e+06 n=772322 tag=gauss
wclsFrameSaver: saving 65451 traces tagged "wiener"
FrameSaver: q=1.34158e+06 n=794738 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 75225 traces, 75225 summary
Retagger: tagging trace set: gauss with 63481 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 63481 traces tagged "gauss"
FrameSaver: q=1.40997e+06 n=748656 tag=gauss
wclsFrameSaver: saving 75225 traces tagged "wiener"
FrameSaver: q=1.61923e+06 n=786151 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 67767 traces, 67767 summary
Retagger: tagging trace set: gauss with 60310 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 60310 traces tagged "gauss"
FrameSaver: q=1.53703e+06 n=820253 tag=gauss
wclsFrameSaver: saving 67767 traces tagged "wiener"
FrameSaver: q=1.58102e+06 n=846765 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 86229 traces, 86229 summary
Retagger: tagging trace set: gauss with 74111 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 74111 traces tagged "gauss"
FrameSaver: q=2.16742e+06 n=1059319 tag=gauss
wclsFrameSaver: saving 86229 traces tagged "wiener"
FrameSaver: q=2.32168e+06 n=1102081 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 94201 traces, 94201 summary
Retagger: tagging trace set: gauss with 80656 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 80656 traces tagged "gauss"
FrameSaver: q=2.26404e+06 n=1150932 tag=gauss
wclsFrameSaver: saving 94201 traces tagged "wiener"
FrameSaver: q=2.44928e+06 n=1163469 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 77145 traces, 77145 summary
Retagger: tagging trace set: gauss with 65954 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 65954 traces tagged "gauss"
FrameSaver: q=1.75083e+06 n=871355 tag=gauss
wclsFrameSaver: saving 77145 traces tagged "wiener"
FrameSaver: q=1.86342e+06 n=878958 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 69126 traces, 69126 summary
Retagger: tagging trace set: gauss with 61038 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 61038 traces tagged "gauss"
FrameSaver: q=1.3939e+06 n=800404 tag=gauss
wclsFrameSaver: saving 69126 traces tagged "wiener"
FrameSaver: q=1.54446e+06 n=826987 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 80930 traces, 80930 summary
Retagger: tagging trace set: gauss with 69370 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 69370 traces tagged "gauss"
FrameSaver: q=1.92922e+06 n=992295 tag=gauss
wclsFrameSaver: saving 80930 traces tagged "wiener"
FrameSaver: q=2.10787e+06 n=995355 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 65673 traces, 65673 summary
Retagger: tagging trace set: gauss with 60411 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 60411 traces tagged "gauss"
FrameSaver: q=1.5088e+06 n=806898 tag=gauss
wclsFrameSaver: saving 65673 traces tagged "wiener"
FrameSaver: q=1.53416e+06 n=864871 tag=wiener
05-Aug-2025 00:08:05 CEST  Closed output file "pdhd_processing_1_14359_2_1754324657_stage1_stage2.root"
05-Aug-2025 00:08:05 CEST  Closed input file "pdhd_processing_1_14359_2_1754324657_stage1.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              241.892       486.365       985.385       443.969       177.557        40     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00151025    0.00578784     0.0122608    0.00462681    0.00328125       40     
produce:fembfilter:PDHDFEMBFilter                      0.655295      0.793606      0.901779      0.858613      0.0954857       40     
produce:wclsdatahd:WireCellToolkit                      41.5127       63.6476       74.3367       67.4879       9.30867        40     
produce:gaushit:GausHitFinder                          0.467149      0.787983       1.43904      0.741702      0.191675        40     
produce:nhitsfilter:NumberOfHitsFilter                0.00027426    0.000657155   0.00296979    0.000463344   0.000649753      40     
produce:reco3d:SpacePointSolver                        0.253199       0.95015       3.19497      0.737643      0.645492        40     
produce:hitpdune:DisambigFromSpacePoints                0.20237      0.566527       1.11464      0.512754      0.212536        40     
produce:pandora:StandardPandora                         6.94767       16.7599       47.033        13.6746       8.41862        40     
produce:pandoraWriter:StandardPandora                  0.118397      0.196447      0.260286      0.190581      0.0361593       40     
produce:pandoraTrack:LArPandoraTrackCreation            1.03742       6.44057       15.746        5.74926       3.63764        40     
produce:pandoraShower:LArPandoraShowerCreation          1.11736       8.72789        25.39        6.70338       6.23402        40     
produce:pandoracalo:Calorimetry                        0.562934       2.06999       5.35536       1.78471       1.11101        40     
produce:pandoracalonosce:Calorimetry                   0.510444       2.07755       5.04409       1.81781       1.09086        40     
produce:pandorapid:Chi2ParticleID                     0.00164469    0.00326616    0.00885591    0.00242168    0.00192056       40     
produce:pandoraShowercalo:ShowerCalorimetry             1.08387       6.14974       17.5294       5.28903       3.88784        40     
produce:pandoraShowercalonosce:ShowerCalorimetry        1.0398        6.16557       15.6709       4.92024       4.09488        40     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.0970878     0.150916      0.234986      0.146131      0.0354576       40     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0395665     0.0640231     0.0955592     0.0639299     0.017308        40     
produce:emtrkmichelid:EmTrackMichelId                   14.2428       20.701        30.1495       19.8869       4.10631        40     
[art]:TriggerResults:TriggerResultInserter            2.7321e-05    5.9233e-05    9.7055e-05    5.8167e-05    1.38869e-05      40     
end_path:out1:RootOutput                               6.099e-06    1.15848e-05   2.8433e-05    1.0681e-05    4.16221e-06      40     
end_path:michelremoving:michelremoving                 0.0285934     0.164637      0.571489      0.129875      0.129359        40     
end_path:michelremovinggnocchi:michelremoving          0.0254556     0.157265      0.563879       0.12697      0.124918        40     
end_path:calibanagnocchi:CalibAnaTree                   127.543       347.861       769.312       294.869       154.814        40     
end_path:out1:RootOutput(write)                        0.587286      0.887946       1.33262      0.857931      0.197534        40     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9049.32 MB
  Peak resident set size usage (VmHWM): 6144.45 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'pdhd_processing_1_14359_2_1754324657_stage1_stage2.root', 'tfile_out': 'pdhd_processing_1_14359_2_1754324657_stage1_stage2_tfile.root', 'start_time': 1754325774.0566616, 'end_time': 1754345287.1753452}
justIN time: 2025-08-05 01:20:21 UTC       justIN version: 01.04.00