justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253695.1@dunegpschedd02.fnal.gov

Jobsub ID253695.1@dunegpschedd02.fnal.gov
Workflow ID10823
Stage ID1
User namecalcuttj@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes6290407424 (5999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-27 04:20:28
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-11-27 06:30:26
From worker nodeHostnamewn-lb-17.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes7864320000 (7500 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-27 04:21:56
Input fileshd-protodune:np04hd_raw_run028799_0000_dataflow2_datawriter_0_20240821T040553.hdf5
JobscriptExit code0
Real time1h (7045s)
CPU time1h (6720s = 95%)
Max RSS bytes4021010432 (3834 MiB)
Outputting started2025-11-27 06:19:22
Output filesusertests:pdhd_prod_data__253695_1_1_20251127T042203Z.root
Finished2025-11-27 06:30:26
Saved logsjustin-logs:253695.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

s within desired window for channel 9551
wclsFrameSaver: no samples within desired window for channel 9551
wclsFrameSaver: no samples within desired window for channel 9552
wclsFrameSaver: no samples within desired window for channel 9552
wclsFrameSaver: no samples within desired window for channel 9553
wclsFrameSaver: no samples within desired window for channel 9554
wclsFrameSaver: no samples within desired window for channel 9555
wclsFrameSaver: no samples within desired window for channel 9557
wclsFrameSaver: no samples within desired window for channel 9562
wclsFrameSaver: no samples within desired window for channel 9563
wclsFrameSaver: no samples within desired window for channel 9564
wclsFrameSaver: no samples within desired window for channel 9565
wclsFrameSaver: no samples within desired window for channel 9566
wclsFrameSaver: no samples within desired window for channel 9566
wclsFrameSaver: no samples within desired window for channel 9567
wclsFrameSaver: no samples within desired window for channel 9567
wclsFrameSaver: no samples within desired window for channel 9567
wclsFrameSaver: no samples within desired window for channel 9568
wclsFrameSaver: no samples within desired window for channel 9568
wclsFrameSaver: no samples within desired window for channel 9569
wclsFrameSaver: no samples within desired window for channel 9569
wclsFrameSaver: no samples within desired window for channel 9569
wclsFrameSaver: no samples within desired window for channel 9569
wclsFrameSaver: no samples within desired window for channel 9570
wclsFrameSaver: no samples within desired window for channel 9570
wclsFrameSaver: no samples within desired window for channel 9570
wclsFrameSaver: no samples within desired window for channel 9570
wclsFrameSaver: no samples within desired window for channel 9571
wclsFrameSaver: no samples within desired window for channel 9571
wclsFrameSaver: no samples within desired window for channel 9571
wclsFrameSaver: no samples within desired window for channel 9572
wclsFrameSaver: no samples within desired window for channel 9572
wclsFrameSaver: no samples within desired window for channel 9572
wclsFrameSaver: no samples within desired window for channel 9573
wclsFrameSaver: no samples within desired window for channel 9573
wclsFrameSaver: no samples within desired window for channel 9574
wclsFrameSaver: no samples within desired window for channel 9575
wclsFrameSaver: no samples within desired window for channel 9575
wclsFrameSaver: no samples within desired window for channel 9575
wclsFrameSaver: no samples within desired window for channel 9576
wclsFrameSaver: no samples within desired window for channel 9683
wclsFrameSaver: no samples within desired window for channel 9691
FrameSaver: q=1.02998e+06 n=786350 tag=wiener
3735 hits in TPC 0
5547 hits in TPC 1
3606 hits in TPC 2
1794 hits in TPC 3
2945 hits in TPC 4
4458 hits in TPC 5
5632 hits in TPC 6
3324 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 80 XUs and 40 XVs -> 0 XUVs
C:0 T:1 16432 XUs and 13757 XVs -> 281 XUVs
C:0 T:2 16760 XUs and 61975 XVs -> 1478 XUVs
C:0 T:3 26 XUs and 83 XVs -> 0 XUVs
C:0 T:4 30 XUs and 34 XVs -> 0 XUVs
C:0 T:5 9660 XUs and 9261 XVs -> 279 XUVs
C:0 T:6 19246 XUs and 66366 XVs -> 2967 XUVs
C:0 T:7 21 XUs and 20 XVs -> 0 XUVs
5005 XUVs total
1422 collection wire objects
5005 potential space points
Neighbour search...
321939 tests to find 192094 neighbours
Iterating with no regularization...
Begin: 8.94793e+06
0 8.20353e+06
1 8.14426e+06
2 8.13319e+06
3 8.12955e+06
Now with regularization...
Begin: 7.76853e+06
0 7.76631e+06
no of hits closeby  5   event 234 TrkackID 0 startx, y, z 86.512 596.879 96.1534  wireno, peakt tpcno 466 301.854 5 dist 9.47371min T, max_T230.282 3762.33
no of hits closeby  2   event 234 TrkackID 11 startx, y, z -37.8117 247.684 444.328  wireno, peakt tpcno 404 4431 6 dist 0.533364min T, max_T4421 4976.7
no of hits closeby  5   event 234 TrkackID 0 startx, y, z 86.512 596.879 96.1534  wireno, peakt tpcno 466 301.854 5 dist 9.47371min T, max_T230.282 3762.33
no of hits closeby  2   event 234 TrkackID 11 startx, y, z -37.8117 247.684 444.328  wireno, peakt tpcno 404 4431 6 dist 0.533364min T, max_T4421 4976.7
27-Nov-2025 07:19:20 CET  Closed output file "pdhd_prod_data__253695_1_1_20251127T042203Z.root"
27-Nov-2025 07:19:20 CET  Closed input file "temp.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              88.2196       178.449       340.554       168.184       53.8572        30     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000698382   0.00357466    0.00930537    0.00296055    0.00249394       30     
produce:fembfilter:PDHDFEMBFilter                      0.652167      0.800228      0.897829      0.805159      0.0371275       30     
produce:wclsdatahd:WireCellToolkit                      63.0035       79.9289       89.5617       81.2245       5.16983        30     
produce:gaushit:GausHitFinder                          0.554747      0.827829       1.34153      0.792482      0.211259        30     
produce:nhitsfilter:NumberOfHitsFilter                0.000360407   0.000597989   0.00114316    0.000576245   0.00016792       30     
produce:reco3d:SpacePointSolver                        0.240013       1.08731       4.07249      0.709702      0.879717        29     
produce:hitpdune:DisambigFromSpacePoints               0.206047      0.577985       1.53547      0.444172      0.304256        29     
produce:pandora:StandardPandora                         5.50726       17.769        45.8119       13.2589       10.1132        29     
produce:pandoraWriter:StandardPandora                  0.101539       0.21266      0.369879      0.205215      0.0599485       29     
produce:pandoraTrack:LArPandoraTrackCreation            1.04977       6.20838       19.8017        4.94         4.20966        29     
produce:pandoraShower:LArPandoraShowerCreation          1.08395       9.43623       24.6471       7.31209       6.77383        29     
produce:pandoracalo:Calorimetry                        0.425392       2.26299       5.74491       1.87237       1.4277         29     
produce:pandoracalonosce:Calorimetry                   0.392391       2.21584       7.24657       1.73588       1.52783        29     
produce:pandorapid:Chi2ParticleID                     0.00168781    0.00394836    0.00727135     0.003684     0.00169007       29     
produce:pandoraShowercalo:ShowerCalorimetry            0.814739       6.38031       18.8776       4.14941       4.89393        29     
produce:pandoraShowercalonosce:ShowerCalorimetry       0.822662        6.169        18.0623       4.23894       4.5756         29     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.103914       0.17708      0.326645      0.168898      0.0535094       29     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0380923     0.0737342     0.147452      0.0693608     0.0246026       29     
produce:emtrkmichelid:EmTrackMichelId                   15.0478       24.0161       40.2493       21.9355       6.23343        29     
[art]:TriggerResults:TriggerResultInserter            4.2893e-05    7.25634e-05   0.000136824   6.7786e-05    2.02879e-05      30     
end_path:out1:RootOutput                               8.141e-06    1.36605e-05   2.5607e-05    1.26995e-05   3.60831e-06      30     
end_path:michelremoving:michelremoving                 7.988e-05     0.206728       1.08687      0.159634      0.203111        30     
end_path:michelremovinggnocchi:michelremoving         3.2127e-05      0.20132       1.11155      0.144306      0.204324        30     
end_path:calibanagnocchi:CalibAnaTree                 2.9543e-05      20.4834       58.3701       16.9568       12.6398        30     
end_path:out1:RootOutput(write)                        0.421511       0.94809       1.69687      0.892358      0.279294        30     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6943.95 MB
  Peak resident set size usage (VmHWM): 4021.01 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
stage2 lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 5346468
-rw-r--r--. 1 dune009 dune    7666565 Nov 27 07:19 jobscript.log
-rw-r--r--. 1 dune009 dune        163 Nov 27 07:19 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune  101285575 Nov 27 07:19 temp_20251127T061921_keepup_hists.root
-rw-r--r--. 1 dune009 dune       3574 Nov 27 07:19 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune     376832 Nov 27 07:19 mem.db
-rw-r--r--. 1 dune009 dune      61440 Nov 27 07:19 time.db
-rw-r--r--. 1 dune009 dune  788393726 Nov 27 07:19 pdhd_prod_data__253695_1_1_20251127T042203Z.root
-rw-r--r--. 1 dune009 dune  117781425 Nov 27 07:18 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune          0 Nov 27 05:49 debugprod.log
-rw-r--r--. 1 dune009 dune  888144573 Nov 27 05:49 pdhd_keepup_decoder.root
-rw-r--r--. 1 dune009 dune 3570643132 Nov 27 05:49 temp.root
-rw-r--r--. 1 dune009 dune         83 Nov 27 05:22 all-input-dids.txt
justIN time: 2026-02-08 05:41:21 UTC       justIN version: 01.06.00