Jobsub ID 253327.0@dunegpschedd02.fnal.gov

Jobsub ID	253327.0@dunegpschedd02.fnal.gov
Workflow ID	10822
Stage ID	1
User name	calcuttj@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	6290407424 (5999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-26 20:48:57
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc01
Last heartbeat	2025-11-27 01:36:28
From worker node	Hostname	wn-db-05.gina.surf.nl
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	7864320000 (7500 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-11-27 01:29:36
Input files	hd-protodune:np04hd_raw_run028799_0000_dataflow2_datawriter_0_20240821T040553.hdf5
Jobscript	Exit code	0
	Real time	4m (289s)
	CPU time	4m (269s = 93%)
	Max RSS bytes	2492096512 (2376 MiB)
Outputting started	2025-11-27 01:34:26
Output files	usertests:pdhd_prod_data__253327_0_1_20251127T012943Z.root
Finished	2025-11-27 01:36:28
Saved logs	justin-logs:253327.0-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

el 9555
wclsFrameSaver: no samples within desired window for channel 9556
wclsFrameSaver: no samples within desired window for channel 9556
wclsFrameSaver: no samples within desired window for channel 9557
wclsFrameSaver: no samples within desired window for channel 9557
wclsFrameSaver: no samples within desired window for channel 9558
wclsFrameSaver: no samples within desired window for channel 9558
wclsFrameSaver: no samples within desired window for channel 9559
wclsFrameSaver: no samples within desired window for channel 9559
wclsFrameSaver: no samples within desired window for channel 9560
wclsFrameSaver: no samples within desired window for channel 9560
wclsFrameSaver: no samples within desired window for channel 9560
wclsFrameSaver: no samples within desired window for channel 9561
wclsFrameSaver: no samples within desired window for channel 9562
wclsFrameSaver: no samples within desired window for channel 9563
wclsFrameSaver: no samples within desired window for channel 9564
wclsFrameSaver: no samples within desired window for channel 9565
wclsFrameSaver: no samples within desired window for channel 9668
wclsFrameSaver: no samples within desired window for channel 9669
wclsFrameSaver: no samples within desired window for channel 9703
wclsFrameSaver: no samples within desired window for channel 9738
wclsFrameSaver: no samples within desired window for channel 9739
wclsFrameSaver: no samples within desired window for channel 9739
wclsFrameSaver: no samples within desired window for channel 9740
wclsFrameSaver: no samples within desired window for channel 9740
wclsFrameSaver: no samples within desired window for channel 9741
wclsFrameSaver: no samples within desired window for channel 9742
wclsFrameSaver: no samples within desired window for channel 9742
wclsFrameSaver: no samples within desired window for channel 9743
wclsFrameSaver: no samples within desired window for channel 9743
wclsFrameSaver: no samples within desired window for channel 9743
wclsFrameSaver: no samples within desired window for channel 9743
wclsFrameSaver: no samples within desired window for channel 9743
wclsFrameSaver: no samples within desired window for channel 9743
wclsFrameSaver: no samples within desired window for channel 9744
wclsFrameSaver: no samples within desired window for channel 9744
wclsFrameSaver: no samples within desired window for channel 9744
wclsFrameSaver: no samples within desired window for channel 9744
wclsFrameSaver: no samples within desired window for channel 9745
wclsFrameSaver: no samples within desired window for channel 9745
wclsFrameSaver: no samples within desired window for channel 9746
wclsFrameSaver: no samples within desired window for channel 9746
wclsFrameSaver: no samples within desired window for channel 9747
wclsFrameSaver: no samples within desired window for channel 9748
wclsFrameSaver: no samples within desired window for channel 9748
wclsFrameSaver: no samples within desired window for channel 9748
wclsFrameSaver: no samples within desired window for channel 9748
wclsFrameSaver: no samples within desired window for channel 9748
wclsFrameSaver: no samples within desired window for channel 9749
wclsFrameSaver: no samples within desired window for channel 9750
FrameSaver: q=1.7471e+06 n=1271821 tag=wiener
3007 hits in TPC 0
7215 hits in TPC 1
7098 hits in TPC 2
5538 hits in TPC 3
8365 hits in TPC 4
9590 hits in TPC 5
7618 hits in TPC 6
4705 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 117 XUs and 117 XVs -> 1 XUVs
C:0 T:1 29729 XUs and 23492 XVs -> 655 XUVs
C:0 T:2 32250 XUs and 25737 XVs -> 1728 XUVs
C:0 T:3 96 XUs and 162 XVs -> 1 XUVs
C:0 T:4 478 XUs and 442 XVs -> 17 XUVs
C:0 T:5 101605 XUs and 83839 XVs -> 4578 XUVs
C:0 T:6 30099 XUs and 31705 XVs -> 5308 XUVs
C:0 T:7 120 XUs and 117 XVs -> 2 XUVs
12290 XUVs total
2664 collection wire objects
12290 potential space points
Neighbour search...
1657166 tests to find 782978 neighbours
Iterating with no regularization...
Begin: 1.3754e+07
0 1.12951e+07
1 1.10476e+07
2 1.10016e+07
3 1.09868e+07
4 1.09802e+07
Now with regularization...
Begin: 1.02856e+07
0 1.02797e+07
27-Nov-2025 02:34:22 CET  Opened output file with pattern "pdhd_prod_data__253327_0_1_20251127T012943Z.root"
27-Nov-2025 02:34:24 CET  Closed output file "pdhd_prod_data__253327_0_1_20251127T012943Z.root"
27-Nov-2025 02:34:24 CET  Closed input file "temp.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              164.028       164.028       164.028       164.028          0            1     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00132799    0.00132799    0.00132799    0.00132799         0            1     
produce:fembfilter:PDHDFEMBFilter                      0.593479      0.593479      0.593479      0.593479          0            1     
produce:wclsdatahd:WireCellToolkit                      53.2443       53.2443       53.2443       53.2443          0            1     
produce:gaushit:GausHitFinder                           1.03591       1.03591       1.03591       1.03591          0            1     
produce:nhitsfilter:NumberOfHitsFilter                0.000712107   0.000712107   0.000712107   0.000712107        0            1     
produce:reco3d:SpacePointSolver                         1.28045       1.28045       1.28045       1.28045          0            1     
produce:hitpdune:DisambigFromSpacePoints               0.808985      0.808985      0.808985      0.808985          0            1     
produce:pandora:StandardPandora                         29.6111       29.6111       29.6111       29.6111          0            1     
produce:pandoraWriter:StandardPandora                   0.23183       0.23183       0.23183       0.23183          0            1     
produce:pandoraTrack:LArPandoraTrackCreation            2.69772       2.69772       2.69772       2.69772          0            1     
produce:pandoraShower:LArPandoraShowerCreation          3.50975       3.50975       3.50975       3.50975          0            1     
produce:pandoracalo:Calorimetry                         1.14346       1.14346       1.14346       1.14346          0            1     
produce:pandoracalonosce:Calorimetry                    1.09296       1.09296       1.09296       1.09296          0            1     
produce:pandorapid:Chi2ParticleID                      0.0696774     0.0696774     0.0696774     0.0696774         0            1     
produce:pandoraShowercalo:ShowerCalorimetry             3.01043       3.01043       3.01043       3.01043          0            1     
produce:pandoraShowercalonosce:ShowerCalorimetry        2.79057       2.79057       2.79057       2.79057          0            1     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.186384      0.186384      0.186384      0.186384          0            1     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0618532     0.0618532     0.0618532     0.0618532         0            1     
produce:emtrkmichelid:EmTrackMichelId                   32.7025       32.7025       32.7025       32.7025          0            1     
[art]:TriggerResults:TriggerResultInserter            7.1477e-05    7.1477e-05    7.1477e-05    7.1477e-05         0            1     
end_path:out1:RootOutput                              2.0138e-05    2.0138e-05    2.0138e-05    2.0138e-05         0            1     
end_path:michelremoving:michelremoving                 0.0377403     0.0377403     0.0377403     0.0377403         0            1     
end_path:michelremovinggnocchi:michelremoving          0.0340742     0.0340742     0.0340742     0.0340742         0            1     
end_path:calibanagnocchi:CalibAnaTree                   28.5831       28.5831       28.5831       28.5831          0            1     
end_path:out1:RootOutput(write)                         1.14219       1.14219       1.14219       1.14219          0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4852.31 MB
  Peak resident set size usage (VmHWM): 2492.1 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
stage2 lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 200228
-rw-r--r--. 1 dune009 dune    298402 Nov 27 02:34 jobscript.log
-rw-r--r--. 1 dune009 dune       163 Nov 27 02:34 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune   6732770 Nov 27 02:34 temp_20251127T013425_keepup_hists.root
-rw-r--r--. 1 dune009 dune      3574 Nov 27 02:34 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune     61440 Nov 27 02:34 mem.db
-rw-r--r--. 1 dune009 dune     16384 Nov 27 02:34 time.db
-rw-r--r--. 1 dune009 dune  32580017 Nov 27 02:34 pdhd_prod_data__253327_0_1_20251127T012943Z.root
-rw-r--r--. 1 dune009 dune   5844995 Nov 27 02:33 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune         0 Nov 27 02:31 debugprod.log
-rw-r--r--. 1 dune009 dune  38558450 Nov 27 02:30 pdhd_keepup_decoder.root
-rw-r--r--. 1 dune009 dune 120692703 Nov 27 02:30 temp.root
-rw-r--r--. 1 dune009 dune        83 Nov 27 02:29 all-input-dids.txt