Jobsub ID 253097.3@dunegpschedd02.fnal.gov
| Jobsub ID | 253097.3@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 14:20:32 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce03 |
| Last heartbeat | 2025-11-26 17:50:57 |
| From worker node | Hostname | wn75 |
| cpuinfo | AMD EPYC 7713P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 14:21:47 |
| Input files | hd-protodune:pdhd_prod_beam__238576_113_1_20251106T105611Z_gen_g4_IonScintPDExt.root_255414_84_1_20251122T180805Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 3h (12280s) |
| CPU time | 2h (10348s = 84%) |
| Max RSS bytes | 4492128256 (4284 MiB) |
| Outputting started | 2025-11-26 17:46:29 |
| Output files | |
| Finished | 2025-11-26 17:50:57 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
54099e+06 n=1144761 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 6676 XUs and 6080 XVs -> 25 XUVs
C:0 T:1 17728 XUs and 14631 XVs -> 486 XUVs
C:0 T:2 19973 XUs and 17017 XVs -> 381 XUVs
C:0 T:3 18 XUs and 13 XVs -> 0 XUVs
C:0 T:4 12 XUs and 17 XVs -> 0 XUVs
C:0 T:5 23884 XUs and 21583 XVs -> 970 XUVs
C:0 T:6 10222 XUs and 11032 XVs -> 364 XUVs
C:0 T:7 6 XUs and 5 XVs -> 0 XUVs
2226 XUVs total
1679 collection wire objects
2226 potential space points
Neighbour search...
25978 tests to find 11950 neighbours
Iterating with no regularization...
Begin: 3.4618e+07
0 3.39605e+07
1 3.39402e+07
Now with regularization...
Begin: 3.19802e+07
0 3.19771e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 14:46:20 -03 Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_3_1_20251126T142154Z_detsim_reco1.root"
26-Nov-2025 14:46:20 -03 Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 180.299 228.137 363.086 214.857 52.1033 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00165001 0.0102891 0.0154158 0.0115784 0.00479806 10
reco:opdec:Deconvolution 1.33993 1.47829 1.70899 1.46269 0.124639 10
reco:ophitspe:OpHitFinderDeco 0.18469 0.326712 0.363508 0.339146 0.0490116 10
reco:opflash:OpFlashFinder 0.117342 0.147809 0.186384 0.142697 0.0223799 10
reco:opslicer:OpSlicer 0.33119 0.459029 0.63184 0.455986 0.081203 10
reco:wclsdatahd:WireCellToolkit 64.7578 68.8584 72.5329 69.155 2.48352 10
reco:gaushit:GausHitFinder 0.991645 1.53781 2.37256 1.49373 0.393284 10
reco:nhitsfilter:NumberOfHitsFilter 0.000307922 0.000500624 0.000894903 0.000444277 0.000171757 10
reco:reco3d:SpacePointSolver 0.547066 2.48932 7.71896 1.90718 2.06943 10
reco:hitpdune:DisambigFromSpacePoints 0.808482 1.45387 2.52953 1.41805 0.520849 10
reco:pandora:StandardPandora 24.0856 46.0261 115.726 40.6455 25.135 10
reco:pandoraWriter:StandardPandora 0.45177 0.676926 1.17661 0.594325 0.203843 10
reco:pandoraTrack:LArPandoraTrackCreation 1.94342 3.70476 7.42027 3.28573 1.4822 10
reco:pandoraShower:LArPandoraShowerCreation 2.11929 3.406 6.10651 2.91103 1.34302 10
reco:pandoracalo:Calorimetry 1.02594 1.87139 3.469 1.69197 0.726817 10
reco:pandoracalonosce:Calorimetry 1.25312 1.85269 2.93687 1.76523 0.523261 10
reco:pandoraShowercalo:ShowerCalorimetry 2.46965 3.43633 5.88015 2.78445 1.20095 10
reco:pandoraShowercalonosce:ShowerCalorimetry 2.36036 3.2856 5.59922 2.62427 1.17727 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0689199 0.111353 0.159417 0.112177 0.0230087 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0823603 0.112965 0.15624 0.108045 0.0208047 10
reco:emtrkmichelid:EmTrackMichelId 46.1529 53.7065 72.708 51.6175 7.84959 10
reco:crttag:SingleCRTMatchingProducer 0.00729301 0.01002 0.0141135 0.00936362 0.00240558 10
reco:crtreco:TwoCRTMatchingProducer 0.00668686 0.00932607 0.0139738 0.00842596 0.00235394 10
reco:pandora2Track:LArPandoraTrackCreation 5.21602 8.96187 15.7915 8.13624 3.27613 10
reco:pandora2calo:ShowerCalorimetry 1.7489 2.97057 5.27567 2.49957 1.21297 10
reco:pandora2calonosce:ShowerCalorimetry 1.69061 2.94437 5.04508 2.43713 1.16109 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0578041 0.108865 0.149306 0.106666 0.0232713 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0592291 0.110053 0.148292 0.111395 0.0251724 10
reco:pandora2Shower:LArPandoraShowerCreation 2.25231 3.91278 7.06314 3.33562 1.48123 10
reco:pandora2Showercalo:ShowerCalorimetry 2.53855 4.05147 6.93225 3.47983 1.54282 10
reco:pandora2Showercalonosce:ShowerCalorimetry 2.37946 3.97545 7.14701 3.42949 1.52699 10
[art]:TriggerResults:TriggerResultInserter 2.9802e-05 4.32259e-05 0.000120724 3.50715e-05 2.59845e-05 10
end_path:out1:RootOutput 5.42e-06 8.2484e-06 2.8521e-05 6.0355e-06 6.76891e-06 10
end_path:out1:RootOutput(write) 4.81156 5.96993 7.7415 5.91223 0.807939 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6431.89 MB
Peak resident set size usage (VmHWM): 4193.08 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_3_1_20251126T142154Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_3_1_20251126T142154Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_3_1_20251126T142154Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_3_1_20251126T142154Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_3_1_20251126T142154Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_3_1_20251126T142154Z_detsim_reco1.root.temp.json ====
.:
total 6284140
-rw-r--r-- 1 nobody nobody 48627 Nov 26 14:46 jobscript.log
-rw-r--r-- 1 nobody nobody 204 Nov 26 14:46 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody 0 Nov 26 14:46 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_3_1_20251126T142154Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody 0 Nov 26 14:46 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_3_1_20251126T142154Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody 31 Nov 26 14:46 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_3_1_20251126T142154Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody 25 Nov 26 14:46 all-input-dids.txt
-rw-r--r-- 1 nobody nobody 519 Nov 26 14:46 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody 3574 Nov 26 14:46 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody 204800 Nov 26 14:46 mem.db
-rw-r--r-- 1 nobody nobody 36864 Nov 26 14:46 time.db
-rw-r--r-- 1 nobody nobody 6378455234 Nov 26 14:46 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253097_3_1_20251126T142154Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody 56182410 Nov 26 14:44 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody 0 Nov 26 14:09 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody 0 Nov 26 14:05 debugprod.log
-rw-r--r-- 1 nobody nobody 519 Nov 26 14:05 detsim_single_protodunehd_hist.root 