justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 14359.9@dunegpschedd02.fnal.gov

Jobsub ID14359.9@dunegpschedd02.fnal.gov
Workflow ID330
Stage ID1
User namemsiden@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-04 16:22:02
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-08-04 22:32:42
From worker nodeHostnamewn-lb-04.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-04 16:24:19
Input fileshd-protodune:np04hd_raw_run027087_0017_dataflow0_datawriter_0_20240614T145430.hdf5
JobscriptExit code0
Real time6h (22028s)
CPU time6h (21951s = 99%)
Max RSS bytes3742031872 (3568 MiB)
Outputting started2025-08-04 22:31:28
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/msiden/channel_map_test/fnal/00330/1/001/pdhd_processing_1_14359_9_1754324673_stage1_stage2.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/msiden/channel_map_test/fnal/00330/1/001/pdhd_processing_1_14359_9_1754324673_stage1_stage2_tfile.root
Finished2025-08-04 22:32:42
Saved logsjustin-logs:14359.9-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

er: q=2.75661e+06 n=1260021 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 62347 traces, 62347 summary
Retagger: tagging trace set: gauss with 58005 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 58005 traces tagged "gauss"
FrameSaver: q=1.31308e+06 n=787912 tag=gauss
wclsFrameSaver: saving 62347 traces tagged "wiener"
FrameSaver: q=1.33749e+06 n=815959 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 81051 traces, 81051 summary
Retagger: tagging trace set: gauss with 69676 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 69676 traces tagged "gauss"
FrameSaver: q=1.9054e+06 n=933757 tag=gauss
wclsFrameSaver: saving 81051 traces tagged "wiener"
FrameSaver: q=2.07423e+06 n=981115 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 103112 traces, 103112 summary
Retagger: tagging trace set: gauss with 87591 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 87591 traces tagged "gauss"
FrameSaver: q=3.77277e+06 n=1232193 tag=gauss
wclsFrameSaver: saving 103112 traces tagged "wiener"
FrameSaver: q=4.09419e+06 n=1279289 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 70651 traces, 70651 summary
Retagger: tagging trace set: gauss with 63273 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 63273 traces tagged "gauss"
FrameSaver: q=1.57778e+06 n=834182 tag=gauss
wclsFrameSaver: saving 70651 traces tagged "wiener"
FrameSaver: q=1.60845e+06 n=859039 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 91466 traces, 91466 summary
Retagger: tagging trace set: gauss with 77820 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 77820 traces tagged "gauss"
FrameSaver: q=2.32427e+06 n=1091612 tag=gauss
wclsFrameSaver: saving 91466 traces tagged "wiener"
FrameSaver: q=2.4039e+06 n=1114382 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 74580 traces, 74580 summary
Retagger: tagging trace set: gauss with 64492 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 64492 traces tagged "gauss"
FrameSaver: q=2.22738e+06 n=859523 tag=gauss
wclsFrameSaver: saving 74580 traces tagged "wiener"
FrameSaver: q=2.72405e+06 n=904079 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 95935 traces, 95935 summary
Retagger: tagging trace set: gauss with 81710 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 81710 traces tagged "gauss"
FrameSaver: q=2.25388e+06 n=1156058 tag=gauss
wclsFrameSaver: saving 95935 traces tagged "wiener"
FrameSaver: q=2.40752e+06 n=1167950 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 72046 traces, 72046 summary
Retagger: tagging trace set: gauss with 65174 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 65174 traces tagged "gauss"
FrameSaver: q=3.42161e+06 n=942175 tag=gauss
wclsFrameSaver: saving 72046 traces tagged "wiener"
FrameSaver: q=3.48524e+06 n=972660 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 79891 traces, 79891 summary
Retagger: tagging trace set: gauss with 70100 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 70100 traces tagged "gauss"
FrameSaver: q=1.63161e+06 n=891946 tag=gauss
wclsFrameSaver: saving 79891 traces tagged "wiener"
FrameSaver: q=1.80891e+06 n=953261 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 87722 traces, 87722 summary
Retagger: tagging trace set: gauss with 75175 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 75175 traces tagged "gauss"
FrameSaver: q=2.25426e+06 n=1069796 tag=gauss
wclsFrameSaver: saving 87722 traces tagged "wiener"
FrameSaver: q=2.22377e+06 n=1062717 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 71411 traces, 71411 summary
Retagger: tagging trace set: gauss with 62170 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 62170 traces tagged "gauss"
FrameSaver: q=1.68812e+06 n=827794 tag=gauss
wclsFrameSaver: saving 71411 traces tagged "wiener"
FrameSaver: q=1.87633e+06 n=870607 tag=wiener
05-Aug-2025 00:31:26 CEST  Closed output file "pdhd_processing_1_14359_9_1754324673_stage1_stage2.root"
05-Aug-2025 00:31:26 CEST  Closed input file "pdhd_processing_1_14359_9_1754324673_stage1.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              190.697       521.126       976.467       468.778       181.096        40     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000497344    0.0071134     0.0572426    0.00382909    0.00990164       40     
produce:fembfilter:PDHDFEMBFilter                      0.657591      0.822603      0.899339       0.8766       0.0941694       40     
produce:wclsdatahd:WireCellToolkit                      49.6468       63.2456       74.0408       64.1263       7.82538        40     
produce:gaushit:GausHitFinder                          0.447237      0.871821       1.38381      0.840204      0.230849        40     
produce:nhitsfilter:NumberOfHitsFilter                0.000249173   0.000632759   0.00291933    0.000517809   0.000524985      40     
produce:reco3d:SpacePointSolver                        0.194379       1.18259       3.08014       1.09559       0.64628        40     
produce:hitpdune:DisambigFromSpacePoints               0.190323      0.713494       1.52517      0.706673      0.319138        40     
produce:pandora:StandardPandora                         5.57071       23.5884       62.7215       22.0529       11.8721        40     
produce:pandoraWriter:StandardPandora                  0.121417      0.233018      0.327473      0.231618      0.0512917       40     
produce:pandoraTrack:LArPandoraTrackCreation            1.7459        8.80806       21.3029       7.23936       5.31445        40     
produce:pandoraShower:LArPandoraShowerCreation          1.73905       12.3223       35.2357       11.2885       8.20051        40     
produce:pandoracalo:Calorimetry                        0.688508       2.99078       7.48349       2.49271       1.84798        40     
produce:pandoracalonosce:Calorimetry                   0.649556       2.92946       6.66235       2.3672        1.7337         40     
produce:pandorapid:Chi2ParticleID                     0.00150548    0.00340522    0.00802949     0.0028113    0.00173508       40     
produce:pandoraShowercalo:ShowerCalorimetry             1.31996       8.93544       22.8702       8.34855       6.00081        40     
produce:pandoraShowercalonosce:ShowerCalorimetry        1.26693       8.87218       23.2645       7.6474        5.94425        40     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.0865861     0.174699      0.261143      0.166498      0.0514669       40     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0319763     0.0747177     0.121724      0.0701425     0.0228208       40     
produce:emtrkmichelid:EmTrackMichelId                   12.0038       23.4459       35.3147       23.6407       6.1496         40     
[art]:TriggerResults:TriggerResultInserter            3.0666e-05    6.12945e-05   9.4902e-05    6.1602e-05    1.2773e-05       40     
end_path:out1:RootOutput                               6.229e-06    1.2805e-05    3.0726e-05    1.1637e-05    4.67448e-06      40     
end_path:michelremoving:michelremoving                 0.0275521      0.19475      0.680003      0.153271      0.154072        40     
end_path:michelremovinggnocchi:michelremoving          0.025657      0.187657      0.696715      0.130419      0.153629        40     
end_path:calibanagnocchi:CalibAnaTree                   102.821       359.282       730.364       323.636       143.106        40     
end_path:out1:RootOutput(write)                        0.546133       1.03274       1.62312       0.93236      0.279801        40     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6839.11 MB
  Peak resident set size usage (VmHWM): 3742.03 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'pdhd_processing_1_14359_9_1754324673_stage1_stage2.root', 'tfile_out': 'pdhd_processing_1_14359_9_1754324673_stage1_stage2_tfile.root', 'start_time': 1754325791.4740927, 'end_time': 1754346687.8355098}
justIN time: 2025-08-05 01:21:09 UTC       justIN version: 01.04.00