Jobsub ID 259010.6@dunegpschedd01.fnal.gov

Jobsub ID	259010.6@dunegpschedd01.fnal.gov
Workflow ID	10823
Stage ID	1
User name	calcuttj@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	6290407424 (5999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-26 20:51:00
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc01
Last heartbeat	2025-11-27 04:30:29
From worker node	Hostname	wn-da-05.gina.surf.nl
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	7864320000 (7500 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-11-27 02:06:22
Input files	hd-protodune:np04hd_raw_run028799_0000_dataflow3_datawriter_0_20240821T040553.hdf5
Jobscript	Exit code	0
	Real time	2h (8548s)
	CPU time	1h (5788s = 67%)
	Max RSS bytes	3705393152 (3533 MiB)
Outputting started	2025-11-27 04:28:51
Output files	usertests:pdhd_prod_data__259010_6_1_20251127T020629Z.root
Finished	2025-11-27 04:30:29
Saved logs	justin-logs:259010.6-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

window for channel 9136
wclsFrameSaver: no samples within desired window for channel 9136
wclsFrameSaver: no samples within desired window for channel 9136
wclsFrameSaver: no samples within desired window for channel 9138
wclsFrameSaver: no samples within desired window for channel 9138
wclsFrameSaver: no samples within desired window for channel 9138
wclsFrameSaver: no samples within desired window for channel 9139
wclsFrameSaver: no samples within desired window for channel 9140
wclsFrameSaver: no samples within desired window for channel 9140
wclsFrameSaver: no samples within desired window for channel 9181
wclsFrameSaver: no samples within desired window for channel 9181
wclsFrameSaver: no samples within desired window for channel 9182
wclsFrameSaver: no samples within desired window for channel 9182
wclsFrameSaver: no samples within desired window for channel 9183
wclsFrameSaver: no samples within desired window for channel 9183
wclsFrameSaver: no samples within desired window for channel 9183
wclsFrameSaver: no samples within desired window for channel 9183
wclsFrameSaver: no samples within desired window for channel 9184
wclsFrameSaver: no samples within desired window for channel 9185
wclsFrameSaver: no samples within desired window for channel 9185
wclsFrameSaver: no samples within desired window for channel 9185
wclsFrameSaver: no samples within desired window for channel 9339
wclsFrameSaver: no samples within desired window for channel 9339
wclsFrameSaver: no samples within desired window for channel 9340
wclsFrameSaver: no samples within desired window for channel 9340
wclsFrameSaver: no samples within desired window for channel 9341
wclsFrameSaver: no samples within desired window for channel 9341
wclsFrameSaver: no samples within desired window for channel 9342
wclsFrameSaver: no samples within desired window for channel 9342
wclsFrameSaver: no samples within desired window for channel 9469
wclsFrameSaver: no samples within desired window for channel 9647
wclsFrameSaver: no samples within desired window for channel 9661
wclsFrameSaver: no samples within desired window for channel 9661
wclsFrameSaver: no samples within desired window for channel 9661
wclsFrameSaver: no samples within desired window for channel 9661
wclsFrameSaver: no samples within desired window for channel 9661
wclsFrameSaver: no samples within desired window for channel 9661
wclsFrameSaver: no samples within desired window for channel 9663
wclsFrameSaver: no samples within desired window for channel 9663
wclsFrameSaver: no samples within desired window for channel 9663
wclsFrameSaver: no samples within desired window for channel 9663
wclsFrameSaver: no samples within desired window for channel 9663
wclsFrameSaver: no samples within desired window for channel 9663
wclsFrameSaver: no samples within desired window for channel 9663
wclsFrameSaver: no samples within desired window for channel 9663
wclsFrameSaver: no samples within desired window for channel 9723
wclsFrameSaver: no samples within desired window for channel 9745
FrameSaver: q=1.02819e+06 n=767814 tag=wiener
2316 hits in TPC 0
6668 hits in TPC 1
5624 hits in TPC 2
4492 hits in TPC 3
3169 hits in TPC 4
5137 hits in TPC 5
2767 hits in TPC 6
1799 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 200 XUs and 244 XVs -> 8 XUVs
C:0 T:1 17186 XUs and 35582 XVs -> 1565 XUVs
C:0 T:2 20007 XUs and 26916 XVs -> 707 XUVs
C:0 T:3 278 XUs and 488 XVs -> 3 XUVs
C:0 T:4 53 XUs and 68 XVs -> 4 XUVs
C:0 T:5 18098 XUs and 42388 XVs -> 2842 XUVs
C:0 T:6 6356 XUs and 6406 XVs -> 121 XUVs
C:0 T:7 45 XUs and 36 XVs -> 9 XUVs
5259 XUVs total
2035 collection wire objects
5259 potential space points
Neighbour search...
407823 tests to find 215394 neighbours
Iterating with no regularization...
Begin: 1.11883e+07
0 9.66144e+06
1 9.55634e+06
2 9.53764e+06
3 9.53077e+06
Now with regularization...
Begin: 8.94676e+06
0 8.94104e+06
no of hits closeby  0   event 235 TrkackID 5 startx, y, z -40.156 397.552 437.754  wireno, peakt tpcno 91 4094.88 6 dist 0min T, max_T4092.82 4978.83
no of hits closeby  0   event 235 TrkackID 5 startx, y, z -40.156 397.552 437.754  wireno, peakt tpcno 91 4094.88 6 dist 0min T, max_T4092.82 4978.83
27-Nov-2025 05:28:49 CET  Closed output file "pdhd_prod_data__259010_6_1_20251127T020629Z.root"
27-Nov-2025 05:28:49 CET  Closed input file "temp.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              96.5513       154.847       262.912       142.309       45.3914        30     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0005784    0.00293018    0.00862808    0.00219371     0.0020666       30     
produce:fembfilter:PDHDFEMBFilter                      0.573177      0.605985      0.663708      0.605039      0.0188491       30     
produce:wclsdatahd:WireCellToolkit                      50.6753       79.1546       95.2525       85.7382       13.5547        30     
produce:gaushit:GausHitFinder                          0.406007      0.816137       3.61489        0.696       0.552857        30     
produce:nhitsfilter:NumberOfHitsFilter                0.000279723   0.000558212   0.00400381    0.000434933   0.000647359      30     
produce:reco3d:SpacePointSolver                         0.19758       0.79957       1.88894      0.716227       0.50542        29     
produce:hitpdune:DisambigFromSpacePoints               0.178825      0.526149       1.68045       0.41114      0.350771        29     
produce:pandora:StandardPandora                         5.39648       18.7539       69.024        13.1757       15.0999        29     
produce:pandoraWriter:StandardPandora                  0.110476      0.176005      0.263714      0.165999      0.0485513       29     
produce:pandoraTrack:LArPandoraTrackCreation           0.723184       3.28634       9.40223       2.94846       2.09825        29     
produce:pandoraShower:LArPandoraShowerCreation         0.773175       4.64464       12.1253       3.83058       3.3402         29     
produce:pandoracalo:Calorimetry                        0.381331       1.22779       3.84566      0.977353      0.814005        29     
produce:pandoracalonosce:Calorimetry                   0.346378       1.12523       3.94721      0.893568       0.79943        29     
produce:pandorapid:Chi2ParticleID                     0.00121486    0.00370062     0.0341514    0.00228438    0.00590264       29     
produce:pandoraShowercalo:ShowerCalorimetry            0.651435       3.11598       9.69459       2.45199       2.15788        29     
produce:pandoraShowercalonosce:ShowerCalorimetry       0.651427       3.03683       8.78362       2.36947       2.10248        29     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.0818021     0.137608      0.219539      0.138485      0.0358255       29     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0313842     0.0540861     0.0937908     0.0511948     0.0167578       29     
produce:emtrkmichelid:EmTrackMichelId                   13.5452       22.0966       32.9365       20.887        5.68783        29     
[art]:TriggerResults:TriggerResultInserter            1.8905e-05    5.56916e-05   8.1735e-05    5.65875e-05   1.23995e-05      30     
end_path:out1:RootOutput                               4.549e-06    1.19448e-05   4.0537e-05    1.1482e-05    5.97539e-06      30     
end_path:michelremoving:michelremoving                3.8092e-05     0.102892      0.273951      0.0878703     0.0673872       30     
end_path:michelremovinggnocchi:michelremoving         3.0068e-05     0.103753      0.283629      0.0884374     0.0666709       30     
end_path:calibanagnocchi:CalibAnaTree                 2.9977e-05      15.5596       33.9908       13.1809       9.22788        30     
end_path:out1:RootOutput(write)                        0.473742      0.800648       1.45354      0.736081      0.224167        30     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6032.31 MB
  Peak resident set size usage (VmHWM): 3705.39 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
stage2 lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 5353112
-rw-r--r--. 1 dune009 dune    5260200 Nov 27 05:28 jobscript.log
-rw-r--r--. 1 dune009 dune        163 Nov 27 05:28 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune   98278312 Nov 27 05:28 temp_20251127T042850_keepup_hists.root
-rw-r--r--. 1 dune009 dune       3574 Nov 27 05:28 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune     380928 Nov 27 05:28 mem.db
-rw-r--r--. 1 dune009 dune      61440 Nov 27 05:28 time.db
-rw-r--r--. 1 dune009 dune  784959131 Nov 27 05:28 pdhd_prod_data__259010_6_1_20251127T020629Z.root
-rw-r--r--. 1 dune009 dune  112813825 Nov 27 05:27 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune          0 Nov 27 04:10 debugprod.log
-rw-r--r--. 1 dune009 dune  908575034 Nov 27 04:10 pdhd_keepup_decoder.root
-rw-r--r--. 1 dune009 dune 3570720470 Nov 27 04:10 temp.root
-rw-r--r--. 1 dune009 dune         83 Nov 27 03:06 all-input-dids.txt