justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 20916.123@dunegpschedd01.fnal.gov

Jobsub ID20916.123@dunegpschedd01.fnal.gov
Workflow ID404
Stage ID1
User namemsiden@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-05 03:20:43
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-08-05 09:51:34
From worker nodeHostnamewn-lb-17.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-05 03:49:59
Input fileshd-protodune:np04hd_raw_run027087_0239_dataflow2_datawriter_0_20240614T165334.hdf5
JobscriptExit code0
Real time6h (21603s)
CPU time5h (21497s = 99%)
Max RSS bytes3849801728 (3671 MiB)
Outputting started2025-08-05 09:50:03
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/msiden/channel_map_test/fnal/00404/1/001/pdhd_processing_1_20916_123_1754365817_stage1_stage2.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/msiden/channel_map_test/fnal/00404/1/001/pdhd_processing_1_20916_123_1754365817_stage1_stage2_tfile.root
Finished2025-08-05 09:51:34
Saved logsjustin-logs:20916.123-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

q=1.6617e+06 n=906259 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 97289 traces, 97289 summary
Retagger: tagging trace set: gauss with 83369 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 83369 traces tagged "gauss"
FrameSaver: q=2.36734e+06 n=1160176 tag=gauss
wclsFrameSaver: saving 97289 traces tagged "wiener"
FrameSaver: q=2.73819e+06 n=1203802 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 68443 traces, 68443 summary
Retagger: tagging trace set: gauss with 59430 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 59430 traces tagged "gauss"
FrameSaver: q=1.46798e+06 n=756617 tag=gauss
wclsFrameSaver: saving 68443 traces tagged "wiener"
FrameSaver: q=1.56531e+06 n=798329 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 87585 traces, 87585 summary
Retagger: tagging trace set: gauss with 77881 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 77881 traces tagged "gauss"
FrameSaver: q=2.22368e+06 n=1083292 tag=gauss
wclsFrameSaver: saving 87585 traces tagged "wiener"
FrameSaver: q=2.28518e+06 n=1118894 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 69572 traces, 69572 summary
Retagger: tagging trace set: gauss with 62523 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 62523 traces tagged "gauss"
FrameSaver: q=1.5719e+06 n=864964 tag=gauss
wclsFrameSaver: saving 69572 traces tagged "wiener"
FrameSaver: q=1.71138e+06 n=882059 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 84005 traces, 84005 summary
Retagger: tagging trace set: gauss with 72656 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 72656 traces tagged "gauss"
FrameSaver: q=2.26552e+06 n=1083081 tag=gauss
wclsFrameSaver: saving 84005 traces tagged "wiener"
FrameSaver: q=2.36736e+06 n=1092388 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 91128 traces, 91128 summary
Retagger: tagging trace set: gauss with 76220 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 76220 traces tagged "gauss"
FrameSaver: q=2.27497e+06 n=1044568 tag=gauss
wclsFrameSaver: saving 91128 traces tagged "wiener"
FrameSaver: q=2.48456e+06 n=1071805 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 71331 traces, 71331 summary
Retagger: tagging trace set: gauss with 61856 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 61856 traces tagged "gauss"
FrameSaver: q=1.50708e+06 n=806509 tag=gauss
wclsFrameSaver: saving 71331 traces tagged "wiener"
FrameSaver: q=1.69358e+06 n=843822 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 88188 traces, 88188 summary
Retagger: tagging trace set: gauss with 74895 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 74895 traces tagged "gauss"
FrameSaver: q=2.18473e+06 n=1030642 tag=gauss
wclsFrameSaver: saving 88188 traces tagged "wiener"
FrameSaver: q=2.62313e+06 n=1079808 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 71165 traces, 71165 summary
Retagger: tagging trace set: gauss with 63404 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 63404 traces tagged "gauss"
FrameSaver: q=1.51575e+06 n=847367 tag=gauss
wclsFrameSaver: saving 71165 traces tagged "wiener"
FrameSaver: q=1.59167e+06 n=873373 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 79228 traces, 79228 summary
Retagger: tagging trace set: gauss with 66571 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 66571 traces tagged "gauss"
FrameSaver: q=1.92706e+06 n=910780 tag=gauss
wclsFrameSaver: saving 79228 traces tagged "wiener"
FrameSaver: q=2.10396e+06 n=948529 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 72754 traces, 72754 summary
Retagger: tagging trace set: gauss with 63717 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 63717 traces tagged "gauss"
FrameSaver: q=1.61658e+06 n=858365 tag=gauss
wclsFrameSaver: saving 72754 traces tagged "wiener"
FrameSaver: q=1.70003e+06 n=863705 tag=wiener
05-Aug-2025 11:50:01 CEST  Closed output file "pdhd_processing_1_20916_123_1754365817_stage1_stage2.root"
05-Aug-2025 11:50:01 CEST  Closed input file "pdhd_processing_1_20916_123_1754365817_stage1.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              224.048       509.641       1208.94       490.524       174.973        40     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000435329   0.00834357     0.101578     0.00607814     0.0154113       40     
produce:fembfilter:PDHDFEMBFilter                      0.649612      0.765543      0.903273      0.687791      0.103927        40     
produce:wclsdatahd:WireCellToolkit                      50.0985       61.2207       73.9034       58.5587       7.82334        40     
produce:gaushit:GausHitFinder                          0.477276      0.799047       1.13699      0.814701      0.176665        40     
produce:nhitsfilter:NumberOfHitsFilter                0.000301741   0.000583045   0.00130428    0.000577983   0.000159866      40     
produce:reco3d:SpacePointSolver                        0.388427       1.02739       3.27237      0.809384       0.65297        40     
produce:hitpdune:DisambigFromSpacePoints               0.268847      0.617245       1.17222      0.570402      0.205986        40     
produce:pandora:StandardPandora                         8.13402       18.1725       39.8066       15.1269       7.22208        40     
produce:pandoraWriter:StandardPandora                  0.130083       0.21391      0.333852      0.212029      0.0454817       40     
produce:pandoraTrack:LArPandoraTrackCreation            1.38567       8.30893       28.4283       7.14741       5.38243        40     
produce:pandoraShower:LArPandoraShowerCreation          1.82994       11.8946       35.2526       9.2446        8.22534        40     
produce:pandoracalo:Calorimetry                         0.80458       2.8308        8.78914       2.36474       1.69661        40     
produce:pandoracalonosce:Calorimetry                   0.758178       2.68139       8.70857       2.31345       1.63223        40     
produce:pandorapid:Chi2ParticleID                     0.00160931    0.00326679     0.0109243    0.00232454    0.00223551       40     
produce:pandoraShowercalo:ShowerCalorimetry             1.85579       7.97014       20.3338       6.29028       5.06559        40     
produce:pandoraShowercalonosce:ShowerCalorimetry        1.71306       7.86506       21.2044       6.73133       5.05796        40     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.0631253      0.15616      0.226215      0.153465      0.0370104       40     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.030831      0.0692864     0.116744      0.0665871     0.0189917       40     
produce:emtrkmichelid:EmTrackMichelId                   12.7477       22.5468       30.7177       23.0802       4.58239        40     
[art]:TriggerResults:TriggerResultInserter            2.5798e-05    6.04673e-05   0.000100099   5.9965e-05    1.60652e-05      40     
end_path:out1:RootOutput                               6.159e-06    1.11507e-05   2.8062e-05    1.0571e-05    3.58964e-06      40     
end_path:michelremoving:michelremoving                 0.0339746      0.22283      0.650255      0.181304      0.151992        40     
end_path:michelremovinggnocchi:michelremoving          0.0313387      0.21023      0.651683      0.191783      0.147985        40     
end_path:calibanagnocchi:CalibAnaTree                   134.184       359.926       982.768       342.39        153.037        40     
end_path:out1:RootOutput(write)                        0.476194      0.888862       1.2609       0.896488       0.18959        40     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6852.17 MB
  Peak resident set size usage (VmHWM): 3849.8 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'pdhd_processing_1_20916_123_1754365817_stage1_stage2.root', 'tfile_out': 'pdhd_processing_1_20916_123_1754365817_stage1_stage2_tfile.root', 'start_time': 1754366949.539767, 'end_time': 1754387402.732843}
justIN time: 2025-09-08 21:18:22 UTC       justIN version: 01.05.00