Jobsub ID 259010.10@dunegpschedd01.fnal.gov

Jobsub ID	259010.10@dunegpschedd01.fnal.gov
Workflow ID	10823
Stage ID	1
User name	calcuttj@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	6290407424 (5999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-26 20:51:00
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc02
Last heartbeat	2025-11-27 04:48:57
From worker node	Hostname	wn-la-07.gina.surf.nl
	cpuinfo	AMD EPYC 9754 128-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	7864320000 (7500 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-11-27 02:06:17
Input files	hd-protodune:np04hd_raw_run028799_0000_dataflow7_datawriter_0_20240821T040553.hdf5
Jobscript	Exit code	0
	Real time	2h (9658s)
	CPU time	1h (6169s = 63%)
	Max RSS bytes	4535750656 (4325 MiB)
Outputting started	2025-11-27 04:47:16
Output files	usertests:pdhd_prod_data__259010_10_1_20251127T020624Z.root
Finished	2025-11-27 04:48:57
Saved logs	justin-logs:259010.10-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

sired window for channel 9328
wclsFrameSaver: no samples within desired window for channel 9328
wclsFrameSaver: no samples within desired window for channel 9328
wclsFrameSaver: no samples within desired window for channel 9328
wclsFrameSaver: no samples within desired window for channel 9329
wclsFrameSaver: no samples within desired window for channel 9329
wclsFrameSaver: no samples within desired window for channel 9329
wclsFrameSaver: no samples within desired window for channel 9330
wclsFrameSaver: no samples within desired window for channel 9330
wclsFrameSaver: no samples within desired window for channel 9331
wclsFrameSaver: no samples within desired window for channel 9331
wclsFrameSaver: no samples within desired window for channel 9331
wclsFrameSaver: no samples within desired window for channel 9331
wclsFrameSaver: no samples within desired window for channel 9331
wclsFrameSaver: no samples within desired window for channel 9332
wclsFrameSaver: no samples within desired window for channel 9332
wclsFrameSaver: no samples within desired window for channel 9332
wclsFrameSaver: no samples within desired window for channel 9333
wclsFrameSaver: no samples within desired window for channel 9333
wclsFrameSaver: no samples within desired window for channel 9333
wclsFrameSaver: no samples within desired window for channel 9336
wclsFrameSaver: no samples within desired window for channel 9428
wclsFrameSaver: no samples within desired window for channel 9429
wclsFrameSaver: no samples within desired window for channel 9430
wclsFrameSaver: no samples within desired window for channel 9586
wclsFrameSaver: no samples within desired window for channel 9586
wclsFrameSaver: no samples within desired window for channel 9586
wclsFrameSaver: no samples within desired window for channel 9587
wclsFrameSaver: no samples within desired window for channel 9587
wclsFrameSaver: no samples within desired window for channel 9587
wclsFrameSaver: no samples within desired window for channel 9588
wclsFrameSaver: no samples within desired window for channel 9588
wclsFrameSaver: no samples within desired window for channel 9588
wclsFrameSaver: no samples within desired window for channel 9588
wclsFrameSaver: no samples within desired window for channel 9589
wclsFrameSaver: no samples within desired window for channel 9589
wclsFrameSaver: no samples within desired window for channel 9590
wclsFrameSaver: no samples within desired window for channel 9590
wclsFrameSaver: no samples within desired window for channel 9591
wclsFrameSaver: no samples within desired window for channel 9592
wclsFrameSaver: no samples within desired window for channel 9599
wclsFrameSaver: no samples within desired window for channel 9683
wclsFrameSaver: no samples within desired window for channel 9683
wclsFrameSaver: no samples within desired window for channel 9684
wclsFrameSaver: no samples within desired window for channel 9684
wclsFrameSaver: no samples within desired window for channel 9685
wclsFrameSaver: no samples within desired window for channel 9685
FrameSaver: q=1.22629e+06 n=923951 tag=wiener
4202 hits in TPC 0
7702 hits in TPC 1
4705 hits in TPC 2
3796 hits in TPC 3
3964 hits in TPC 4
7295 hits in TPC 5
5732 hits in TPC 6
3767 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 447 XUs and 246 XVs -> 3 XUVs
C:0 T:1 56922 XUs and 42263 XVs -> 2232 XUVs
C:0 T:2 20658 XUs and 16654 XVs -> 517 XUVs
C:0 T:3 322 XUs and 293 XVs -> 10 XUVs
C:0 T:4 216 XUs and 287 XVs -> 5 XUVs
C:0 T:5 33111 XUs and 23645 XVs -> 532 XUVs
C:0 T:6 14951 XUs and 14343 XVs -> 406 XUVs
C:0 T:7 95 XUs and 159 XVs -> 2 XUVs
3707 XUVs total
1774 collection wire objects
3707 potential space points
Neighbour search...
76461 tests to find 43564 neighbours
Iterating with no regularization...
Begin: 1.58654e+07
0 1.48944e+07
1 1.48422e+07
2 1.48368e+07
Now with regularization...
Begin: 1.39663e+07
0 1.39647e+07
no of hits closeby  0   event 239 TrkackID 17 startx, y, z -108.578 138.406 395.356  wireno, peakt tpcno 279 1004.97 6 dist 0min T, max_T5.79102 1385.8
no of hits closeby  0   event 239 TrkackID 17 startx, y, z -108.578 138.406 395.356  wireno, peakt tpcno 279 1004.97 6 dist 0min T, max_T5.79102 1385.8
27-Nov-2025 05:47:14 CET  Closed output file "pdhd_prod_data__259010_10_1_20251127T020624Z.root"
27-Nov-2025 05:47:14 CET  Closed input file "temp.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              109.58        159.574       315.53        149.543       45.7448        30     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000437786   0.00266383    0.00806673    0.00239146    0.00199719       30     
produce:fembfilter:PDHDFEMBFilter                      0.782866      0.807303       1.10478      0.797437      0.0559833       30     
produce:wclsdatahd:WireCellToolkit                      58.6838       74.6549       82.1425       75.9731       5.95313        30     
produce:gaushit:GausHitFinder                          0.491324      0.756661       1.09217      0.764342      0.159247        30     
produce:nhitsfilter:NumberOfHitsFilter                0.000234651   0.000401529   0.00113524    0.000355262   0.00016494       30     
produce:reco3d:SpacePointSolver                        0.211307      0.778209       2.23722      0.586004      0.492062        30     
produce:hitpdune:DisambigFromSpacePoints               0.185024      0.528526       1.25995      0.503432      0.240843        30     
produce:pandora:StandardPandora                         5.37873       20.2094       81.2603       15.7355       18.7306        30     
produce:pandoraWriter:StandardPandora                  0.110557      0.180691      0.295349      0.183466      0.0470284       30     
produce:pandoraTrack:LArPandoraTrackCreation           0.976103       4.11677       15.4604       3.48753       2.90091        30     
produce:pandoraShower:LArPandoraShowerCreation          1.06072       4.99517       20.748        3.90468       3.73064        30     
produce:pandoracalo:Calorimetry                        0.519537       1.47788       4.91143       1.22367      0.934427        30     
produce:pandoracalonosce:Calorimetry                   0.466508       1.41596       4.90383       1.18448      0.935599        30     
produce:pandorapid:Chi2ParticleID                     0.00156823    0.00302915    0.00623184    0.00269929    0.00113686       30     
produce:pandoraShowercalo:ShowerCalorimetry            0.833314       3.5777        13.027        2.7624        2.4656         30     
produce:pandoraShowercalonosce:ShowerCalorimetry       0.825687       3.4736        13.6627       2.78237       2.47459        30     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.0930156     0.152946      0.248535      0.146289      0.0330112       30     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0354393     0.0597483     0.109407      0.0561817     0.016485        30     
produce:emtrkmichelid:EmTrackMichelId                   13.8594       21.2421       34.109        21.1314       4.82974        30     
[art]:TriggerResults:TriggerResultInserter            2.6439e-05    5.44529e-05   0.000141702   5.2329e-05    2.1217e-05       30     
end_path:out1:RootOutput                               6.84e-06     1.05474e-05   3.0896e-05     9.87e-06     4.57179e-06      30     
end_path:michelremoving:michelremoving                 0.0261203     0.178454      0.720868      0.173206      0.151842        30     
end_path:michelremovinggnocchi:michelremoving          0.0256953     0.178818      0.717533      0.173241      0.152067        30     
end_path:calibanagnocchi:CalibAnaTree                   7.5901        19.0527       40.3451       17.5485       9.17042        30     
end_path:out1:RootOutput(write)                        0.594617      0.945012       1.36278      0.912635      0.200743        30     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7455.16 MB
  Peak resident set size usage (VmHWM): 4535.75 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
stage2 lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 5344012
-rw-r--r--. 1 dune009 dune    6004658 Nov 27 05:47 jobscript.log
-rw-r--r--. 1 dune009 dune        163 Nov 27 05:47 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune   88800127 Nov 27 05:47 temp_20251127T044715_keepup_hists.root
-rw-r--r--. 1 dune009 dune       3574 Nov 27 05:47 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune     385024 Nov 27 05:47 mem.db
-rw-r--r--. 1 dune009 dune      61440 Nov 27 05:47 time.db
-rw-r--r--. 1 dune009 dune  785856989 Nov 27 05:47 pdhd_prod_data__259010_10_1_20251127T020624Z.root
-rw-r--r--. 1 dune009 dune  120699760 Nov 27 05:45 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune          0 Nov 27 04:26 debugprod.log
-rw-r--r--. 1 dune009 dune  901552132 Nov 27 04:26 pdhd_keepup_decoder.root
-rw-r--r--. 1 dune009 dune 3568845365 Nov 27 04:26 temp.root
-rw-r--r--. 1 dune009 dune         83 Nov 27 03:06 all-input-dids.txt