Jobsub ID 252822.26@dunegpschedd02.fnal.gov
| Jobsub ID | 252822.26@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 08:41:16 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce04 |
| Last heartbeat | 2025-11-26 12:03:11 |
| From worker node | Hostname | wn123 |
| cpuinfo | AMD EPYC 7713P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 08:42:48 |
| Input files | hd-protodune:pdhd_prod_beam__238396_75_1_20251106T074631Z_gen_g4_IonScintPDExt.root_255826_86_1_20251123T215639Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 3h (11759s) |
| CPU time | 2h (8672s = 73%) |
| Max RSS bytes | 4500594688 (4292 MiB) |
| Outputting started | 2025-11-26 11:58:49 |
| Output files | |
| Finished | 2025-11-26 12:03:11 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
ameSaver: saving 61778 traces tagged "gauss"
FrameSaver: q=1.71024e+06 n=881418 tag=gauss
wclsFrameSaver: saving 73341 traces tagged "wiener"
FrameSaver: q=2.36548e+06 n=878987 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11742 XUs and 9054 XVs -> 179 XUVs
C:0 T:1 111200 XUs and 98094 XVs -> 4959 XUVs
C:0 T:2 8678 XUs and 8317 XVs -> 757 XUVs
C:0 T:5 12025 XUs and 10756 XVs -> 292 XUVs
C:0 T:6 11598 XUs and 5198 XVs -> 96 XUVs
6283 XUVs total
2278 collection wire objects
6283 potential space points
Neighbour search...
315247 tests to find 104042 neighbours
Iterating with no regularization...
Begin: 4.44133e+07
0 3.83788e+07
1 3.78432e+07
2 3.78083e+07
Now with regularization...
Begin: 3.64114e+07
0 3.64034e+07
26-Nov-2025 08:58:30 -03 Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252822_26_1_20251126T084254Z_detsim_reco1.root"
26-Nov-2025 08:58:30 -03 Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 169.445 204.827 255.155 197.834 23.6158 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00175282 0.00990177 0.0252476 0.00986071 0.00667774 10
reco:opdec:Deconvolution 1.08786 1.37198 1.64296 1.3843 0.163406 10
reco:ophitspe:OpHitFinderDeco 0.179865 0.31529 0.354939 0.333115 0.0478374 10
reco:opflash:OpFlashFinder 0.0838361 0.129077 0.172954 0.127107 0.028974 10
reco:opslicer:OpSlicer 0.30238 0.423765 0.56086 0.428153 0.0856858 10
reco:wclsdatahd:WireCellToolkit 69.2735 71.604 73.3493 71.7463 1.31669 10
reco:gaushit:GausHitFinder 1.18657 1.49601 1.85848 1.52871 0.212483 10
reco:nhitsfilter:NumberOfHitsFilter 0.000337936 0.000605064 0.00103491 0.000493819 0.000241755 10
reco:reco3d:SpacePointSolver 1.13581 1.33213 1.82667 1.26944 0.187339 10
reco:hitpdune:DisambigFromSpacePoints 0.686541 1.13094 1.87918 1.08981 0.351818 10
reco:pandora:StandardPandora 26.5195 36.9322 50.5254 34.7647 7.49999 10
reco:pandoraWriter:StandardPandora 0.431951 0.582375 1.05081 0.56068 0.170311 10
reco:pandoraTrack:LArPandoraTrackCreation 1.42979 2.92752 5.94763 2.76603 1.18413 10
reco:pandoraShower:LArPandoraShowerCreation 1.44051 2.72087 5.13364 2.49862 0.981598 10
reco:pandoracalo:Calorimetry 0.685125 1.49166 2.82081 1.45994 0.540864 10
reco:pandoracalonosce:Calorimetry 0.830996 1.50827 2.59162 1.53926 0.449537 10
reco:pandoraShowercalo:ShowerCalorimetry 1.35418 2.75869 4.33856 2.74496 0.734399 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.44951 2.64364 4.06395 2.58696 0.652474 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0848716 0.105182 0.12128 0.106343 0.0123933 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0884373 0.105438 0.12155 0.110379 0.0124016 10
reco:emtrkmichelid:EmTrackMichelId 39.4682 48.4152 58.5832 46.4911 6.38476 10
reco:crttag:SingleCRTMatchingProducer 0.00573522 0.00943693 0.0145481 0.00860378 0.00252751 10
reco:crtreco:TwoCRTMatchingProducer 0.00517833 0.00792313 0.0123906 0.00743238 0.00216499 10
reco:pandora2Track:LArPandoraTrackCreation 3.92252 7.0723 11.3328 6.72971 2.30739 10
reco:pandora2calo:ShowerCalorimetry 1.46642 2.32371 3.50391 2.00326 0.651358 10
reco:pandora2calonosce:ShowerCalorimetry 1.28545 2.28296 3.57196 2.1191 0.728204 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0595881 0.101412 0.131339 0.0997 0.0217672 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.064016 0.102215 0.127697 0.102145 0.0195979 10
reco:pandora2Shower:LArPandoraShowerCreation 1.75842 3.28093 5.59882 3.08973 1.08355 10
reco:pandora2Showercalo:ShowerCalorimetry 1.84486 3.35051 5.09126 3.02775 0.99923 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.86122 3.23294 4.9098 2.92458 0.932157 10
[art]:TriggerResults:TriggerResultInserter 2.8391e-05 4.51231e-05 0.000103415 3.9802e-05 2.01269e-05 10
end_path:out1:RootOutput 5.58e-06 9.0375e-06 2.9132e-05 6.836e-06 6.76433e-06 10
end_path:out1:RootOutput(write) 3.47146 4.89209 5.86054 5.03076 0.642583 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6063.08 MB
Peak resident set size usage (VmHWM): 3849.52 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252822_26_1_20251126T084254Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252822_26_1_20251126T084254Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252822_26_1_20251126T084254Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252822_26_1_20251126T084254Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252822_26_1_20251126T084254Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252822_26_1_20251126T084254Z_detsim_reco1.root.temp.json ====
.:
total 5548232
-rw-r--r-- 1 nobody nobody 40589 Nov 26 08:58 jobscript.log
-rw-r--r-- 1 nobody nobody 203 Nov 26 08:58 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody 0 Nov 26 08:58 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252822_26_1_20251126T084254Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody 0 Nov 26 08:58 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252822_26_1_20251126T084254Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody 31 Nov 26 08:58 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252822_26_1_20251126T084254Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody 25 Nov 26 08:58 all-input-dids.txt
-rw-r--r-- 1 nobody nobody 519 Nov 26 08:58 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody 3574 Nov 26 08:58 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody 204800 Nov 26 08:58 mem.db
-rw-r--r-- 1 nobody nobody 36864 Nov 26 08:58 time.db
-rw-r--r-- 1 nobody nobody 5631659724 Nov 26 08:58 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252822_26_1_20251126T084254Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody 49418400 Nov 26 08:57 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody 0 Nov 26 08:25 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody 0 Nov 26 08:21 debugprod.log
-rw-r--r-- 1 nobody nobody 519 Nov 26 08:21 detsim_single_protodunehd_hist.root 