justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 14423.17@dunegpschedd02.fnal.gov

Jobsub ID14423.17@dunegpschedd02.fnal.gov
Workflow ID404
Stage ID1
User namemsiden@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-05 03:28:43
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-08-05 08:42:22
From worker nodeHostnamewn-db-08.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-05 03:50:30
Input fileshd-protodune:np04hd_raw_run027087_0242_dataflow0_datawriter_0_20240614T165526.hdf5
JobscriptExit code0
Real time4h (17461s)
CPU time4h (17363s = 99%)
Max RSS bytes4132409344 (3940 MiB)
Outputting started2025-08-05 08:41:31
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/msiden/channel_map_test/fnal/00404/1/001/pdhd_processing_1_14423_17_1754365844_stage1_stage2.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/msiden/channel_map_test/fnal/00404/1/001/pdhd_processing_1_14423_17_1754365844_stage1_stage2_tfile.root
Finished2025-08-05 08:42:22
Saved logsjustin-logs:14423.17-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

q=1.85662e+06 n=909150 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 94215 traces, 94215 summary
Retagger: tagging trace set: gauss with 79478 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 79478 traces tagged "gauss"
FrameSaver: q=2.23051e+06 n=1044390 tag=gauss
wclsFrameSaver: saving 94215 traces tagged "wiener"
FrameSaver: q=2.52346e+06 n=1098624 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 70449 traces, 70449 summary
Retagger: tagging trace set: gauss with 63163 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 63163 traces tagged "gauss"
FrameSaver: q=1.63079e+06 n=831020 tag=gauss
wclsFrameSaver: saving 70449 traces tagged "wiener"
FrameSaver: q=1.69543e+06 n=879746 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 83649 traces, 83649 summary
Retagger: tagging trace set: gauss with 70922 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 70922 traces tagged "gauss"
FrameSaver: q=1.85277e+06 n=883559 tag=gauss
wclsFrameSaver: saving 83649 traces tagged "wiener"
FrameSaver: q=2.08801e+06 n=933471 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 70967 traces, 70967 summary
Retagger: tagging trace set: gauss with 63965 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 63965 traces tagged "gauss"
FrameSaver: q=1.61147e+06 n=905581 tag=gauss
wclsFrameSaver: saving 70967 traces tagged "wiener"
FrameSaver: q=1.65075e+06 n=914167 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 115744 traces, 115744 summary
Retagger: tagging trace set: gauss with 97948 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 97948 traces tagged "gauss"
FrameSaver: q=3.09487e+06 n=1469017 tag=gauss
wclsFrameSaver: saving 115744 traces tagged "wiener"
FrameSaver: q=3.39752e+06 n=1447596 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 83079 traces, 83079 summary
Retagger: tagging trace set: gauss with 72359 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 72359 traces tagged "gauss"
FrameSaver: q=2.07073e+06 n=1016185 tag=gauss
wclsFrameSaver: saving 83079 traces tagged "wiener"
FrameSaver: q=2.06811e+06 n=1023565 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 67369 traces, 67369 summary
Retagger: tagging trace set: gauss with 60268 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 60268 traces tagged "gauss"
FrameSaver: q=1.47801e+06 n=797950 tag=gauss
wclsFrameSaver: saving 67369 traces tagged "wiener"
FrameSaver: q=1.44939e+06 n=816344 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 74964 traces, 74964 summary
Retagger: tagging trace set: gauss with 65974 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 65974 traces tagged "gauss"
FrameSaver: q=1.85047e+06 n=887649 tag=gauss
wclsFrameSaver: saving 74964 traces tagged "wiener"
FrameSaver: q=1.81642e+06 n=903133 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 92224 traces, 92224 summary
Retagger: tagging trace set: gauss with 77970 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 77970 traces tagged "gauss"
FrameSaver: q=2.2698e+06 n=1082050 tag=gauss
wclsFrameSaver: saving 92224 traces tagged "wiener"
FrameSaver: q=2.47945e+06 n=1105208 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 86610 traces, 86610 summary
Retagger: tagging trace set: gauss with 73703 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 73703 traces tagged "gauss"
FrameSaver: q=1.94049e+06 n=950508 tag=gauss
wclsFrameSaver: saving 86610 traces tagged "wiener"
FrameSaver: q=2.17931e+06 n=1000312 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 80752 traces, 80752 summary
Retagger: tagging trace set: gauss with 70127 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 70127 traces tagged "gauss"
FrameSaver: q=1.88285e+06 n=945381 tag=gauss
wclsFrameSaver: saving 80752 traces tagged "wiener"
FrameSaver: q=2.01982e+06 n=978586 tag=wiener
05-Aug-2025 10:41:30 CEST  Closed output file "pdhd_processing_1_14423_17_1754365844_stage1_stage2.root"
05-Aug-2025 10:41:30 CEST  Closed input file "pdhd_processing_1_14423_17_1754365844_stage1.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              217.101       413.455       980.678       372.786       149.332        40     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000344527   0.00529044     0.043745     0.00326604    0.00778263       40     
produce:fembfilter:PDHDFEMBFilter                      0.651811       0.67997      0.717077       0.67941      0.0145125       40     
produce:wclsdatahd:WireCellToolkit                      40.931        65.9884       71.6597       67.7977       6.60221        40     
produce:gaushit:GausHitFinder                          0.556671      0.858363       1.34297       0.78849      0.211855        40     
produce:nhitsfilter:NumberOfHitsFilter                0.000273311   0.00050667    0.00239008    0.000421384   0.000323852      40     
produce:reco3d:SpacePointSolver                        0.158897       0.94674       2.34388      0.839385      0.543211        40     
produce:hitpdune:DisambigFromSpacePoints               0.167337      0.732001       2.25813      0.566728      0.436873        40     
produce:pandora:StandardPandora                         7.48619       20.7819       58.7585       17.3304       10.5011        40     
produce:pandoraWriter:StandardPandora                  0.129266      0.204479      0.329784      0.207368      0.0449959       40     
produce:pandoraTrack:LArPandoraTrackCreation           0.963926       4.88385       18.8734       4.07498       3.46158        40     
produce:pandoraShower:LArPandoraShowerCreation          1.11031       6.47216       25.1379       4.96076       5.31865        40     
produce:pandoracalo:Calorimetry                        0.408504       1.70284       5.96565       1.36344       1.16301        40     
produce:pandoracalonosce:Calorimetry                   0.377373       1.63871       6.5578        1.30506       1.23583        40     
produce:pandorapid:Chi2ParticleID                     0.00140911    0.00254841    0.00494348    0.00234258    0.000988036      40     
produce:pandoraShowercalo:ShowerCalorimetry            0.806009       4.81182       17.5906       4.10273       3.75118        40     
produce:pandoraShowercalonosce:ShowerCalorimetry       0.793947       4.98535       18.6983       4.31021       3.89319        40     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.087641      0.156385      0.301049      0.147372      0.0451291       40     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0303269     0.0620347     0.121094      0.0557069     0.0200907       40     
produce:emtrkmichelid:EmTrackMichelId                   16.8175       26.803        42.7435       25.6235       5.9138         40     
[art]:TriggerResults:TriggerResultInserter            1.9588e-05    4.81309e-05   7.2438e-05    5.36275e-05   1.66883e-05      40     
end_path:out1:RootOutput                               4.519e-06    9.91558e-06   2.7973e-05     9.378e-06    4.54288e-06      40     
end_path:michelremoving:michelremoving                 0.0240604      0.15873        0.579       0.103291      0.125949        40     
end_path:michelremovinggnocchi:michelremoving          0.0235785     0.156489      0.592626      0.102893      0.127729        40     
end_path:calibanagnocchi:CalibAnaTree                   104.651       269.751       704.291       249.785       117.573        40     
end_path:out1:RootOutput(write)                        0.577699      0.934401       1.53655      0.869988      0.219104        40     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6541.8 MB
  Peak resident set size usage (VmHWM): 4132.41 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'pdhd_processing_1_14423_17_1754365844_stage1_stage2.root', 'tfile_out': 'pdhd_processing_1_14423_17_1754365844_stage1_stage2_tfile.root', 'start_time': 1754366699.757412, 'end_time': 1754383291.3858294}
justIN time: 2025-08-05 23:17:27 UTC       justIN version: 01.04.00