Jobsub ID 253123.0@dunegpschedd02.fnal.gov
| Jobsub ID | 253123.0@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 14:40:33 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce03 |
| Last heartbeat | 2025-11-26 17:45:16 |
| From worker node | Hostname | wn73 |
| cpuinfo | AMD EPYC 7713P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 14:41:07 |
| Input files | hd-protodune:pdhd_prod_beam__243786_32_1_20251106T000520Z_gen_g4_IonScintPDExt.root_256006_2_1_20251124T034147Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (10797s) |
| CPU time | 2h (8784s = 81%) |
| Max RSS bytes | 4244738048 (4048 MiB) |
| Outputting started | 2025-11-26 17:41:06 |
| Output files | |
| Finished | 2025-11-26 17:45:16 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
eSaver: saving 84383 traces tagged "wiener"
FrameSaver: q=2.55848e+06 n=967481 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8896 XUs and 7532 XVs -> 54 XUVs
C:0 T:1 36208 XUs and 36945 XVs -> 1867 XUVs
C:0 T:2 15257 XUs and 9620 XVs -> 398 XUVs
C:0 T:3 15 XUs and 15 XVs -> 0 XUVs
C:0 T:4 28 XUs and 103 XVs -> 0 XUVs
C:0 T:5 61688 XUs and 96875 XVs -> 7508 XUVs
C:0 T:6 7172 XUs and 5485 XVs -> 703 XUVs
C:0 T:7 3 XUs and 5 XVs -> 3 XUVs
10533 XUVs total
2434 collection wire objects
10533 potential space points
Neighbour search...
1791733 tests to find 839606 neighbours
Iterating with no regularization...
Begin: 4.78196e+07
0 4.42512e+07
1 4.39829e+07
2 4.39461e+07
Now with regularization...
Begin: 4.15823e+07
0 4.15596e+07
26-Nov-2025 14:40:58 -03 Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253123_0_1_20251126T144114Z_detsim_reco1.root"
26-Nov-2025 14:40:58 -03 Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 171.13 216.451 310.936 209.619 41.7064 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.002494 0.0103117 0.0184409 0.0103997 0.00502015 10
reco:opdec:Deconvolution 1.11456 1.38986 1.58781 1.41282 0.144622 10
reco:ophitspe:OpHitFinderDeco 0.321191 0.333645 0.349175 0.333226 0.00838946 10
reco:opflash:OpFlashFinder 0.116332 0.144858 0.198864 0.140183 0.0223851 10
reco:opslicer:OpSlicer 0.361696 0.437622 0.531021 0.426267 0.0608642 10
reco:wclsdatahd:WireCellToolkit 66.302 68.5554 73.1604 68.0436 2.21555 10
reco:gaushit:GausHitFinder 1.1278 1.46189 1.78837 1.45349 0.212628 10
reco:nhitsfilter:NumberOfHitsFilter 0.000350401 0.000492239 0.000841177 0.000474625 0.000127285 10
reco:reco3d:SpacePointSolver 0.987414 1.73086 3.02642 1.6527 0.644696 10
reco:hitpdune:DisambigFromSpacePoints 0.640433 1.2788 1.98458 1.32095 0.411354 10
reco:pandora:StandardPandora 20.8236 45.873 120.073 30.1584 29.803 10
reco:pandoraWriter:StandardPandora 0.342874 0.747489 1.17599 0.736567 0.259485 10
reco:pandoraTrack:LArPandoraTrackCreation 1.54339 2.98842 4.45567 2.83417 0.879385 10
reco:pandoraShower:LArPandoraShowerCreation 1.53624 2.79767 3.49383 3.05983 0.612196 10
reco:pandoracalo:Calorimetry 0.952041 1.55958 2.45029 1.55097 0.407948 10
reco:pandoracalonosce:Calorimetry 0.923365 1.53236 2.50131 1.54891 0.475134 10
reco:pandoraShowercalo:ShowerCalorimetry 1.32542 2.79401 5.02351 2.70977 0.921852 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.0411 2.65346 4.53091 2.66348 0.850489 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0530207 0.0941333 0.130641 0.093361 0.0221321 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0538086 0.0946242 0.126437 0.0960935 0.0202195 10
reco:emtrkmichelid:EmTrackMichelId 40.3743 51.2761 61.8493 51.4137 6.64952 10
reco:crttag:SingleCRTMatchingProducer 0.00740671 0.00991432 0.0133118 0.00946739 0.00200951 10
reco:crtreco:TwoCRTMatchingProducer 0.00632793 0.00788449 0.00933761 0.00813304 0.000924136 10
reco:pandora2Track:LArPandoraTrackCreation 4.71216 7.80124 11.5067 7.98329 1.96827 10
reco:pandora2calo:ShowerCalorimetry 1.55179 2.5257 4.08806 2.41117 0.72897 10
reco:pandora2calonosce:ShowerCalorimetry 1.51222 2.47146 4.30175 2.41279 0.804629 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0517218 0.103237 0.129787 0.103733 0.0215532 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0510486 0.102976 0.129876 0.102951 0.0216111 10
reco:pandora2Shower:LArPandoraShowerCreation 1.74135 3.32596 5.93981 3.38008 1.21013 10
reco:pandora2Showercalo:ShowerCalorimetry 2.06523 3.49204 6.02359 3.49371 1.13047 10
reco:pandora2Showercalonosce:ShowerCalorimetry 2.07516 3.30856 4.43324 3.42932 0.782781 10
[art]:TriggerResults:TriggerResultInserter 2.9031e-05 4.30154e-05 0.000123074 3.5696e-05 2.6877e-05 10
end_path:out1:RootOutput 5.64e-06 9.3414e-06 3.6881e-05 6.1405e-06 9.1977e-06 10
end_path:out1:RootOutput(write) 3.67437 5.37965 6.40351 5.5584 0.831954 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6352.73 MB
Peak resident set size usage (VmHWM): 4123.25 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253123_0_1_20251126T144114Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253123_0_1_20251126T144114Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253123_0_1_20251126T144114Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253123_0_1_20251126T144114Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253123_0_1_20251126T144114Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253123_0_1_20251126T144114Z_detsim_reco1.root.temp.json ====
.:
total 6045996
-rw-r--r-- 1 nobody nobody 43789 Nov 26 14:41 jobscript.log
-rw-r--r-- 1 nobody nobody 202 Nov 26 14:41 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody 0 Nov 26 14:41 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253123_0_1_20251126T144114Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody 0 Nov 26 14:41 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253123_0_1_20251126T144114Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody 31 Nov 26 14:41 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253123_0_1_20251126T144114Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody 25 Nov 26 14:40 all-input-dids.txt
-rw-r--r-- 1 nobody nobody 519 Nov 26 14:40 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody 3574 Nov 26 14:40 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody 200704 Nov 26 14:40 mem.db
-rw-r--r-- 1 nobody nobody 36864 Nov 26 14:40 time.db
-rw-r--r-- 1 nobody nobody 6137591684 Nov 26 14:40 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253123_0_1_20251126T144114Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody 53196130 Nov 26 14:39 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody 0 Nov 26 14:06 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody 0 Nov 26 14:02 debugprod.log
-rw-r--r-- 1 nobody nobody 519 Nov 26 14:01 detsim_single_protodunehd_hist.root 