Jobsub ID 259010.5@dunegpschedd01.fnal.gov

Jobsub ID	259010.5@dunegpschedd01.fnal.gov
Workflow ID	10823
Stage ID	1
User name	calcuttj@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	6290407424 (5999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-26 20:51:00
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc01
Last heartbeat	2025-11-27 04:45:09
From worker node	Hostname	wn-la-23.gina.surf.nl
	cpuinfo	AMD EPYC 9754 128-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	7864320000 (7500 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-11-27 02:06:24
Input files	hd-protodune:np04hd_raw_run028799_0001_dataflow4_datawriter_0_20240821T040840.hdf5
Jobscript	Exit code	0
	Real time	2h (9430s)
	CPU time	1h (6304s = 66%)
	Max RSS bytes	3732758528 (3559 MiB)
Outputting started	2025-11-27 04:43:34
Output files	usertests:pdhd_prod_data__259010_5_1_20251127T020632Z.root
Finished	2025-11-27 04:45:09
Saved logs	justin-logs:259010.5-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

channel 7641
wclsFrameSaver: no samples within desired window for channel 7642
wclsFrameSaver: no samples within desired window for channel 7643
wclsFrameSaver: no samples within desired window for channel 7645
wclsFrameSaver: no samples within desired window for channel 7646
wclsFrameSaver: no samples within desired window for channel 7646
wclsFrameSaver: no samples within desired window for channel 7650
wclsFrameSaver: no samples within desired window for channel 7660
wclsFrameSaver: no samples within desired window for channel 7676
wclsFrameSaver: no samples within desired window for channel 8063
wclsFrameSaver: no samples within desired window for channel 8063
wclsFrameSaver: no samples within desired window for channel 8064
wclsFrameSaver: no samples within desired window for channel 8064
wclsFrameSaver: no samples within desired window for channel 8064
wclsFrameSaver: no samples within desired window for channel 8065
wclsFrameSaver: no samples within desired window for channel 8065
wclsFrameSaver: no samples within desired window for channel 8338
wclsFrameSaver: no samples within desired window for channel 8340
wclsFrameSaver: no samples within desired window for channel 8344
wclsFrameSaver: no samples within desired window for channel 8346
wclsFrameSaver: no samples within desired window for channel 8347
wclsFrameSaver: no samples within desired window for channel 8435
wclsFrameSaver: no samples within desired window for channel 8436
wclsFrameSaver: no samples within desired window for channel 8436
wclsFrameSaver: no samples within desired window for channel 8436
wclsFrameSaver: no samples within desired window for channel 8512
wclsFrameSaver: no samples within desired window for channel 8622
wclsFrameSaver: no samples within desired window for channel 8785
wclsFrameSaver: no samples within desired window for channel 8785
wclsFrameSaver: no samples within desired window for channel 8786
wclsFrameSaver: no samples within desired window for channel 8786
wclsFrameSaver: no samples within desired window for channel 8832
wclsFrameSaver: no samples within desired window for channel 8902
wclsFrameSaver: no samples within desired window for channel 8903
wclsFrameSaver: no samples within desired window for channel 8904
wclsFrameSaver: no samples within desired window for channel 9023
wclsFrameSaver: no samples within desired window for channel 9023
wclsFrameSaver: no samples within desired window for channel 9023
wclsFrameSaver: no samples within desired window for channel 9024
wclsFrameSaver: no samples within desired window for channel 9024
wclsFrameSaver: no samples within desired window for channel 9024
wclsFrameSaver: no samples within desired window for channel 9024
wclsFrameSaver: no samples within desired window for channel 9025
wclsFrameSaver: no samples within desired window for channel 9025
wclsFrameSaver: no samples within desired window for channel 9025
wclsFrameSaver: no samples within desired window for channel 9025
wclsFrameSaver: no samples within desired window for channel 9526
wclsFrameSaver: no samples within desired window for channel 9615
wclsFrameSaver: no samples within desired window for channel 9616
wclsFrameSaver: no samples within desired window for channel 9729
wclsFrameSaver: no samples within desired window for channel 9730
wclsFrameSaver: no samples within desired window for channel 9749
FrameSaver: q=1.22559e+06 n=858077 tag=wiener
2103 hits in TPC 0
8845 hits in TPC 1
3514 hits in TPC 2
2550 hits in TPC 3
4552 hits in TPC 4
6979 hits in TPC 5
6489 hits in TPC 6
4327 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 78 XUs and 40 XVs -> 1 XUVs
C:0 T:1 42354 XUs and 29479 XVs -> 936 XUVs
C:0 T:2 14587 XUs and 17947 XVs -> 868 XUVs
C:0 T:3 14 XUs and 20 XVs -> 4 XUVs
C:0 T:4 129 XUs and 186 XVs -> 6 XUVs
C:0 T:5 54232 XUs and 42467 XVs -> 722 XUVs
C:0 T:6 17858 XUs and 17304 XVs -> 599 XUVs
C:0 T:7 83 XUs and 82 XVs -> 2 XUVs
3138 XUVs total
1503 collection wire objects
3138 potential space points
Neighbour search...
75638 tests to find 44596 neighbours
Iterating with no regularization...
Begin: 9.3091e+06
0 8.92946e+06
1 8.90465e+06
2 8.90194e+06
Now with regularization...
Begin: 8.39933e+06
0 8.39823e+06
27-Nov-2025 05:43:32 CET  Closed output file "pdhd_prod_data__259010_5_1_20251127T020632Z.root"
27-Nov-2025 05:43:32 CET  Closed input file "temp.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              110.266       163.544       296.116       160.995       41.7095        30     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000413698   0.00249567    0.00815996    0.00238804    0.00161274       30     
produce:fembfilter:PDHDFEMBFilter                      0.786496      0.803271      0.851695      0.797356      0.0141061       30     
produce:wclsdatahd:WireCellToolkit                      60.1571       76.7913        81.02        78.498        4.96109        30     
produce:gaushit:GausHitFinder                          0.455832      0.763858       1.26899      0.736372      0.185688        30     
produce:nhitsfilter:NumberOfHitsFilter                0.000225787   0.000378568   0.000711313   0.00032492    0.000123857      30     
produce:reco3d:SpacePointSolver                        0.228733      0.894308       2.5257       0.872876      0.546046        30     
produce:hitpdune:DisambigFromSpacePoints               0.186673      0.546034       1.23579      0.525855      0.259328        30     
produce:pandora:StandardPandora                         5.14385       17.1695       54.8731       13.6678       11.2328        30     
produce:pandoraWriter:StandardPandora                  0.108642      0.196885      0.351975      0.197648      0.0530473       30     
produce:pandoraTrack:LArPandoraTrackCreation           0.933212       4.4106        17.1464       3.57307       3.12785        30     
produce:pandoraShower:LArPandoraShowerCreation          1.15168       5.65295       17.8571       5.20647       4.01773        30     
produce:pandoracalo:Calorimetry                        0.441979       1.54034       4.92054       1.29399      0.949332        30     
produce:pandoracalonosce:Calorimetry                   0.398838       1.46451       4.75787       1.20463      0.900845        30     
produce:pandorapid:Chi2ParticleID                     0.00142301    0.00459764     0.0459673    0.00297797    0.00778894       30     
produce:pandoraShowercalo:ShowerCalorimetry            0.813954       4.57077       23.2719       3.58656       4.22821        30     
produce:pandoraShowercalonosce:ShowerCalorimetry        0.8015        4.10475       18.3246       3.37503       3.47394        30     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.0811199      0.16334      0.281323      0.164084      0.046428        30     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0323439     0.0633023     0.117574      0.0591822     0.0187563       30     
produce:emtrkmichelid:EmTrackMichelId                   14.0309       22.5803       37.6268       23.011        5.38946        30     
[art]:TriggerResults:TriggerResultInserter            3.0116e-05    5.27214e-05   9.2798e-05    5.1197e-05    1.24561e-05      30     
end_path:out1:RootOutput                               6.329e-06    1.07862e-05    2.681e-05    1.00445e-05   3.63733e-06      30     
end_path:michelremoving:michelremoving                 0.0388364      0.2196        0.96825      0.138439      0.223628        30     
end_path:michelremovinggnocchi:michelremoving          0.0394133     0.222188      0.970838      0.147175      0.223803        30     
end_path:calibanagnocchi:CalibAnaTree                   6.45361       19.6056       55.7215       19.9276       10.5617        30     
end_path:out1:RootOutput(write)                        0.554507       0.96065       1.6292       0.986612      0.235819        30     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6661.89 MB
  Peak resident set size usage (VmHWM): 3732.76 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
stage2 lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 5387100
-rw-r--r--. 1 dune009 dune    6380351 Nov 27 05:43 jobscript.log
-rw-r--r--. 1 dune009 dune        163 Nov 27 05:43 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune  106164883 Nov 27 05:43 temp_20251127T044333_keepup_hists.root
-rw-r--r--. 1 dune009 dune       3574 Nov 27 05:43 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune     385024 Nov 27 05:43 mem.db
-rw-r--r--. 1 dune009 dune      61440 Nov 27 05:43 time.db
-rw-r--r--. 1 dune009 dune  790018907 Nov 27 05:43 pdhd_prod_data__259010_5_1_20251127T020632Z.root
-rw-r--r--. 1 dune009 dune  120782700 Nov 27 05:42 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune          0 Nov 27 04:21 debugprod.log
-rw-r--r--. 1 dune009 dune  926121987 Nov 27 04:20 pdhd_keepup_decoder.root
-rw-r--r--. 1 dune009 dune 3566143622 Nov 27 04:20 temp.root
-rw-r--r--. 1 dune009 dune         83 Nov 27 03:06 all-input-dids.txt