Jobsub ID 259010.1@dunegpschedd01.fnal.gov

Jobsub ID	259010.1@dunegpschedd01.fnal.gov
Workflow ID	10823
Stage ID	1
User name	calcuttj@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	6290407424 (5999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-26 20:51:00
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc02
Last heartbeat	2025-11-27 04:52:23
From worker node	Hostname	wn-la-17.gina.surf.nl
	cpuinfo	AMD EPYC 9754 128-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	7864320000 (7500 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-11-27 02:06:25
Input files	hd-protodune:np04hd_raw_run028799_0001_dataflow2_datawriter_0_20240821T040839.hdf5
Jobscript	Exit code	0
	Real time	2h (9806s)
	CPU time	1h (6739s = 68%)
	Max RSS bytes	6100570112 (5817 MiB)
Outputting started	2025-11-27 04:49:52
Output files	usertests:pdhd_prod_data__259010_1_1_20251127T020633Z.root
Finished	2025-11-27 04:52:23
Saved logs	justin-logs:259010.1-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

ples within desired window for channel 9548
wclsFrameSaver: no samples within desired window for channel 9549
wclsFrameSaver: no samples within desired window for channel 9550
wclsFrameSaver: no samples within desired window for channel 9551
wclsFrameSaver: no samples within desired window for channel 9552
wclsFrameSaver: no samples within desired window for channel 9553
wclsFrameSaver: no samples within desired window for channel 9554
wclsFrameSaver: no samples within desired window for channel 9554
wclsFrameSaver: no samples within desired window for channel 9555
wclsFrameSaver: no samples within desired window for channel 9556
wclsFrameSaver: no samples within desired window for channel 9557
wclsFrameSaver: no samples within desired window for channel 9558
wclsFrameSaver: no samples within desired window for channel 9559
wclsFrameSaver: no samples within desired window for channel 9560
wclsFrameSaver: no samples within desired window for channel 9560
wclsFrameSaver: no samples within desired window for channel 9561
wclsFrameSaver: no samples within desired window for channel 9562
wclsFrameSaver: no samples within desired window for channel 9562
wclsFrameSaver: no samples within desired window for channel 9563
wclsFrameSaver: no samples within desired window for channel 9564
wclsFrameSaver: no samples within desired window for channel 9565
wclsFrameSaver: no samples within desired window for channel 9566
wclsFrameSaver: no samples within desired window for channel 9567
wclsFrameSaver: no samples within desired window for channel 9567
wclsFrameSaver: no samples within desired window for channel 9568
wclsFrameSaver: no samples within desired window for channel 9569
wclsFrameSaver: no samples within desired window for channel 9569
wclsFrameSaver: no samples within desired window for channel 9570
wclsFrameSaver: no samples within desired window for channel 9570
wclsFrameSaver: no samples within desired window for channel 9571
wclsFrameSaver: no samples within desired window for channel 9571
wclsFrameSaver: no samples within desired window for channel 9572
wclsFrameSaver: no samples within desired window for channel 9572
wclsFrameSaver: no samples within desired window for channel 9572
wclsFrameSaver: no samples within desired window for channel 9573
wclsFrameSaver: no samples within desired window for channel 9573
wclsFrameSaver: no samples within desired window for channel 9574
wclsFrameSaver: no samples within desired window for channel 9574
wclsFrameSaver: no samples within desired window for channel 9575
wclsFrameSaver: no samples within desired window for channel 9575
wclsFrameSaver: no samples within desired window for channel 9575
wclsFrameSaver: no samples within desired window for channel 9576
wclsFrameSaver: no samples within desired window for channel 9577
wclsFrameSaver: no samples within desired window for channel 9577
wclsFrameSaver: no samples within desired window for channel 9578
wclsFrameSaver: no samples within desired window for channel 9579
wclsFrameSaver: no samples within desired window for channel 9583
wclsFrameSaver: no samples within desired window for channel 9584
wclsFrameSaver: no samples within desired window for channel 9632
FrameSaver: q=1.44986e+06 n=1059464 tag=wiener
6434 hits in TPC 0
11285 hits in TPC 1
4810 hits in TPC 2
3134 hits in TPC 3
5601 hits in TPC 4
9143 hits in TPC 5
4180 hits in TPC 6
2479 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 510 XUs and 457 XVs -> 5 XUVs
C:0 T:1 109475 XUs and 65462 XVs -> 2494 XUVs
C:0 T:2 13497 XUs and 13297 XVs -> 876 XUVs
C:0 T:3 154 XUs and 226 XVs -> 18 XUVs
C:0 T:4 342 XUs and 236 XVs -> 9 XUVs
C:0 T:5 47036 XUs and 32695 XVs -> 1080 XUVs
C:0 T:6 5592 XUs and 6533 XVs -> 549 XUVs
C:0 T:7 23 XUs and 50 XVs -> 0 XUVs
5031 XUVs total
2312 collection wire objects
5031 potential space points
Neighbour search...
124409 tests to find 71398 neighbours
Iterating with no regularization...
Begin: 1.68172e+07
0 1.59418e+07
1 1.58652e+07
2 1.58601e+07
Now with regularization...
Begin: 1.52433e+07
0 1.52417e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
27-Nov-2025 05:49:49 CET  Closed output file "pdhd_prod_data__259010_1_1_20251127T020633Z.root"
27-Nov-2025 05:49:49 CET  Closed input file "temp.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              102.093       178.535       399.095       165.279       63.2436        30     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000531763   0.00281089     0.0086378     0.0025905    0.00178779       30     
produce:fembfilter:PDHDFEMBFilter                      0.782326      0.797408      0.835931      0.797146      0.0104327       30     
produce:wclsdatahd:WireCellToolkit                      55.619        77.179        84.0761       79.7479       6.74898        30     
produce:gaushit:GausHitFinder                          0.400621      0.799123       1.31438       0.78697      0.207982        30     
produce:nhitsfilter:NumberOfHitsFilter                0.000249872   0.000368056   0.000682678   0.000327544   0.000122971      30     
produce:reco3d:SpacePointSolver                        0.215258       1.12114       3.11707      0.874876      0.756266        30     
produce:hitpdune:DisambigFromSpacePoints               0.199493      0.656843       2.50154      0.575479      0.441687        30     
produce:pandora:StandardPandora                          4.529        22.756        72.672        14.9943       17.1245        30     
produce:pandoraWriter:StandardPandora                  0.112693      0.204354      0.336719      0.196438      0.0630062       30     
produce:pandoraTrack:LArPandoraTrackCreation           0.656018       5.97693       36.5139       3.78351       6.73425        30     
produce:pandoraShower:LArPandoraShowerCreation         0.943738       7.2524        30.0147       4.5623         6.711         30     
produce:pandoracalo:Calorimetry                        0.305183       2.17791       13.8442       1.66696       2.49651        30     
produce:pandoracalonosce:Calorimetry                   0.286856       2.12862       12.7388       1.52969       2.38806        30     
produce:pandorapid:Chi2ParticleID                      0.0016691    0.00490593     0.0445953    0.00296081    0.00757168       30     
produce:pandoraShowercalo:ShowerCalorimetry            0.672369       5.32774       22.2715       3.6367        4.68809        30     
produce:pandoraShowercalonosce:ShowerCalorimetry       0.686047       5.18195       19.6496       3.69157       4.41126        30     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.0803912     0.169313      0.307018      0.154671      0.0518393       30     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0283436     0.065098      0.114639      0.0620299     0.0219463       30     
produce:emtrkmichelid:EmTrackMichelId                   11.3285       22.3264       36.6721       21.9435       5.78775        30     
[art]:TriggerResults:TriggerResultInserter            2.8583e-05    5.67362e-05    8.648e-05    5.7752e-05    1.32618e-05      30     
end_path:out1:RootOutput                               5.788e-06    1.20346e-05   4.9984e-05     1.021e-05    7.80928e-06      30     
end_path:michelremoving:michelremoving                 0.0300339     0.151073      0.627064       0.09145      0.146229        30     
end_path:michelremovinggnocchi:michelremoving          0.0276665     0.153029      0.592581      0.103777      0.141922        30     
end_path:calibanagnocchi:CalibAnaTree                   4.76235       22.2642       59.983        19.057        13.2158        30     
end_path:out1:RootOutput(write)                         0.5069        1.00709       1.63647       0.98901      0.265695        30     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8996.46 MB
  Peak resident set size usage (VmHWM): 6100.57 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
stage2 lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 5415580
-rw-r--r--. 1 dune009 dune    6267183 Nov 27 05:49 jobscript.log
-rw-r--r--. 1 dune009 dune        163 Nov 27 05:49 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune  111762609 Nov 27 05:49 temp_20251127T044950_keepup_hists.root
-rw-r--r--. 1 dune009 dune       3574 Nov 27 05:49 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune     385024 Nov 27 05:49 mem.db
-rw-r--r--. 1 dune009 dune      61440 Nov 27 05:49 time.db
-rw-r--r--. 1 dune009 dune  806210479 Nov 27 05:49 pdhd_prod_data__259010_1_1_20251127T020633Z.root
-rw-r--r--. 1 dune009 dune  126569800 Nov 27 05:47 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune          0 Nov 27 04:19 debugprod.log
-rw-r--r--. 1 dune009 dune  921510198 Nov 27 04:19 pdhd_keepup_decoder.root
-rw-r--r--. 1 dune009 dune 3572408372 Nov 27 04:19 temp.root
-rw-r--r--. 1 dune009 dune         83 Nov 27 03:06 all-input-dids.txt