Jobsub ID 253097.4@dunegpschedd02.fnal.gov
| Jobsub ID | 253097.4@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 14:20:32 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce03 |
| Last heartbeat | 2025-11-26 17:50:50 |
| From worker node | Hostname | wn84 |
| cpuinfo | AMD EPYC 7713P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 14:21:44 |
| Input files | hd-protodune:pdhd_prod_beam__238221_31_1_20251106T051055Z_gen_g4_IonScintPDExt.root_249563_26_1_20251122T083255Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 3h (12283s) |
| CPU time | 2h (10256s = 83%) |
| Max RSS bytes | 4540534784 (4330 MiB) |
| Outputting started | 2025-11-26 17:46:28 |
| Output files | |
| Finished | 2025-11-26 17:50:50 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
> 74 XUVs
C:0 T:1 64348 XUs and 50638 XVs -> 1635 XUVs
C:0 T:2 17076 XUs and 20088 XVs -> 1151 XUVs
C:0 T:3 48 XUs and 50 XVs -> 1 XUVs
C:0 T:4 5 XUs and 12 XVs -> 4 XUVs
C:0 T:5 12988 XUs and 13445 XVs -> 790 XUVs
C:0 T:6 49429 XUs and 30447 XVs -> 1151 XUVs
C:0 T:7 21 XUs and 35 XVs -> 5 XUVs
4811 XUVs total
2576 collection wire objects
4811 potential space points
Neighbour search...
89599 tests to find 47366 neighbours
Iterating with no regularization...
Begin: 3.31117e+07
0 3.13357e+07
1 3.11948e+07
2 3.11841e+07
Now with regularization...
Begin: 2.89241e+07
0 2.89215e+07
TestBeamParticleCreationAlgorithm::SetupTestBeamVertex - Test beam particle has no initial vertex
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 14:46:18 -03 Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_4_1_20251126T142151Z_detsim_reco1.root"
26-Nov-2025 14:46:18 -03 Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 171.894 259.047 353.69 250.929 52.9591 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00165198 0.00990853 0.0172935 0.00927117 0.00568395 10
reco:opdec:Deconvolution 1.2994 1.52702 1.8008 1.54873 0.156255 10
reco:ophitspe:OpHitFinderDeco 0.244045 0.335917 0.3564 0.344726 0.0312381 10
reco:opflash:OpFlashFinder 0.117231 0.162204 0.196376 0.17379 0.0265205 10
reco:opslicer:OpSlicer 0.379441 0.507342 0.642951 0.491271 0.0861421 10
reco:wclsdatahd:WireCellToolkit 67.8566 71.1531 74.8339 70.6204 2.02262 10
reco:gaushit:GausHitFinder 0.981143 1.50658 2.25032 1.47228 0.349382 10
reco:nhitsfilter:NumberOfHitsFilter 0.000334927 0.000539692 0.00121479 0.000467158 0.000242471 10
reco:reco3d:SpacePointSolver 0.92644 2.52367 5.39487 2.11967 1.35617 10
reco:hitpdune:DisambigFromSpacePoints 0.830719 1.71198 2.74961 1.65924 0.491016 10
reco:pandora:StandardPandora 26.9327 66.3648 107.401 63.8832 29.3967 10
reco:pandoraWriter:StandardPandora 0.482372 0.826799 1.20564 0.812641 0.23246 10
reco:pandoraTrack:LArPandoraTrackCreation 1.95218 3.92831 7.44486 3.39081 1.39742 10
reco:pandoraShower:LArPandoraShowerCreation 1.38503 4.07287 7.57452 3.4763 1.86095 10
reco:pandoracalo:Calorimetry 1.02915 2.06718 4.20616 1.79704 0.892044 10
reco:pandoracalonosce:Calorimetry 1.01025 1.98066 3.94824 1.73499 0.787295 10
reco:pandoraShowercalo:ShowerCalorimetry 1.56482 4.36651 9.88913 3.34337 2.70066 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.60114 4.2301 9.69626 3.40515 2.52939 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0800474 0.11928 0.160759 0.119688 0.0238239 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0636467 0.108921 0.157422 0.114805 0.0327138 10
reco:emtrkmichelid:EmTrackMichelId 42.0024 56.1901 69.0731 55.0312 7.03711 10
reco:crttag:SingleCRTMatchingProducer 0.00569392 0.00977164 0.0130223 0.00926679 0.00212646 10
reco:crtreco:TwoCRTMatchingProducer 0.00522398 0.00849754 0.0119404 0.00819146 0.00192713 10
reco:pandora2Track:LArPandoraTrackCreation 4.31946 9.74887 16.3582 8.84919 3.39256 10
reco:pandora2calo:ShowerCalorimetry 1.24796 3.41201 7.1664 2.79541 1.71027 10
reco:pandora2calonosce:ShowerCalorimetry 1.20653 3.24461 6.20259 2.97539 1.41682 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0608966 0.114147 0.157903 0.115583 0.0270105 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.063165 0.108997 0.148276 0.11213 0.0261133 10
reco:pandora2Shower:LArPandoraShowerCreation 1.82759 4.15375 8.18248 3.50389 1.65467 10
reco:pandora2Showercalo:ShowerCalorimetry 1.83464 4.23419 8.0322 3.73598 1.75183 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.80812 4.27969 8.5784 3.56805 1.98626 10
[art]:TriggerResults:TriggerResultInserter 2.1161e-05 5.00115e-05 0.000116086 4.1162e-05 2.73192e-05 10
end_path:out1:RootOutput 4.44e-06 9.0315e-06 3.1512e-05 6.58e-06 7.60597e-06 10
end_path:out1:RootOutput(write) 4.52349 5.84762 6.96077 5.84355 0.769124 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6406.93 MB
Peak resident set size usage (VmHWM): 4180.09 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_4_1_20251126T142151Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_4_1_20251126T142151Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_4_1_20251126T142151Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_4_1_20251126T142151Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_4_1_20251126T142151Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_4_1_20251126T142151Z_detsim_reco1.root.temp.json ====
.:
total 6339616
-rw-r--r-- 1 nobody nobody 41550 Nov 26 14:46 jobscript.log
-rw-r--r-- 1 nobody nobody 203 Nov 26 14:46 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody 0 Nov 26 14:46 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_4_1_20251126T142151Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody 0 Nov 26 14:46 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_4_1_20251126T142151Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody 31 Nov 26 14:46 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_4_1_20251126T142151Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody 25 Nov 26 14:46 all-input-dids.txt
-rw-r--r-- 1 nobody nobody 519 Nov 26 14:46 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody 3574 Nov 26 14:46 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody 204800 Nov 26 14:46 mem.db
-rw-r--r-- 1 nobody nobody 36864 Nov 26 14:46 time.db
-rw-r--r-- 1 nobody nobody 6433378215 Nov 26 14:46 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_4_1_20251126T142151Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody 58076500 Nov 26 14:44 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody 0 Nov 26 14:04 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody 0 Nov 26 14:00 debugprod.log
-rw-r--r-- 1 nobody nobody 519 Nov 26 13:59 detsim_single_protodunehd_hist.root 