Jobsub ID 258341.22@dunegpschedd01.fnal.gov
| Jobsub ID | 258341.22@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 05:05:06 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce04 |
| Last heartbeat | 2025-11-26 09:15:19 |
| From worker node | Hostname | wn120 |
| cpuinfo | AMD EPYC 7713P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 05:06:42 |
| Input files | hd-protodune:pdhd_prod_beam__238519_12_1_20251106T102021Z_gen_g4_IonScintPDExt.root_255523_15_1_20251122T220723Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 4h (14657s) |
| CPU time | 2h (10057s = 68%) |
| Max RSS bytes | 4177678336 (3984 MiB) |
| Outputting started | 2025-11-26 09:11:01 |
| Output files | |
| Finished | 2025-11-26 09:15:19 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
=3.30048e+06 n=1266898 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 16783 XUs and 16033 XVs -> 216 XUVs
C:0 T:1 89981 XUs and 82151 XVs -> 3203 XUVs
C:0 T:2 44981 XUs and 28721 XVs -> 884 XUVs
C:0 T:3 8 XUs and 8 XVs -> 0 XUVs
C:0 T:4 21 XUs and 39 XVs -> 0 XUVs
C:0 T:5 48245 XUs and 51734 XVs -> 3501 XUVs
C:0 T:6 14529 XUs and 10724 XVs -> 466 XUVs
8270 XUVs total
2905 collection wire objects
8270 potential space points
Neighbour search...
672960 tests to find 342602 neighbours
Iterating with no regularization...
Begin: 4.19243e+07
0 3.76984e+07
1 3.73275e+07
2 3.72924e+07
Now with regularization...
Begin: 3.52128e+07
0 3.51969e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 06:10:50 -03 Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258341_22_1_20251126T050648Z_detsim_reco1.root"
26-Nov-2025 06:10:50 -03 Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 154.213 262.642 393.108 253.704 72.5287 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00146808 0.0113727 0.0195663 0.0128703 0.00649106 10
reco:opdec:Deconvolution 1.06292 1.42259 1.64871 1.48244 0.160677 10
reco:ophitspe:OpHitFinderDeco 0.172114 0.323148 0.404094 0.333034 0.0549863 10
reco:opflash:OpFlashFinder 0.0797221 0.147741 0.212267 0.153813 0.0326214 10
reco:opslicer:OpSlicer 0.237551 0.448732 0.572585 0.456833 0.0841597 10
reco:wclsdatahd:WireCellToolkit 64.1124 72.5874 81.0202 71.7394 4.5738 10
reco:gaushit:GausHitFinder 1.11004 1.70424 2.15924 1.69749 0.333192 10
reco:nhitsfilter:NumberOfHitsFilter 0.000447154 0.000771794 0.0011443 0.000723922 0.000199098 10
reco:reco3d:SpacePointSolver 0.648194 2.16244 3.94052 2.12956 1.10284 10
reco:hitpdune:DisambigFromSpacePoints 0.592653 1.68993 2.68327 1.70503 0.709265 10
reco:pandora:StandardPandora 16.6393 59.5257 122.126 44.7525 36.3468 10
reco:pandoraWriter:StandardPandora 0.330538 0.529281 0.698397 0.579941 0.115432 10
reco:pandoraTrack:LArPandoraTrackCreation 2.01204 4.13424 7.37257 4.26726 1.44574 10
reco:pandoraShower:LArPandoraShowerCreation 1.75198 4.17846 6.70443 4.56786 1.58339 10
reco:pandoracalo:Calorimetry 1.13017 2.30075 4.09801 2.30278 0.8355 10
reco:pandoracalonosce:Calorimetry 1.14637 2.22315 3.97768 2.2195 0.799174 10
reco:pandoraShowercalo:ShowerCalorimetry 1.80795 4.22511 6.80719 4.51394 1.71063 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.74212 3.9346 6.35288 4.10283 1.58042 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0970332 0.12159 0.162835 0.118666 0.0189316 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0974452 0.12121 0.165427 0.120395 0.0198325 10
reco:emtrkmichelid:EmTrackMichelId 41.2819 63.222 89.9344 59.013 15.6097 10
reco:crttag:SingleCRTMatchingProducer 0.00595757 0.0101733 0.013943 0.0101477 0.00241774 10
reco:crtreco:TwoCRTMatchingProducer 0.00519586 0.00885771 0.0124165 0.00872537 0.00226798 10
reco:pandora2Track:LArPandoraTrackCreation 3.1969 10.4043 17.4523 10.5523 4.55825 10
reco:pandora2calo:ShowerCalorimetry 1.27394 3.90941 7.704 3.57535 1.95705 10
reco:pandora2calonosce:ShowerCalorimetry 1.35359 3.8614 7.47254 3.49954 1.89942 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0735316 0.116833 0.146399 0.125017 0.0213273 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0718692 0.117042 0.14763 0.125664 0.0217475 10
reco:pandora2Shower:LArPandoraShowerCreation 1.87801 4.16991 6.38251 4.34019 1.50458 10
reco:pandora2Showercalo:ShowerCalorimetry 1.99843 4.8215 7.32611 4.84303 1.93301 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.90208 4.63156 7.53415 4.81943 1.95759 10
[art]:TriggerResults:TriggerResultInserter 2.8421e-05 4.02561e-05 0.000109413 3.2526e-05 2.32878e-05 10
end_path:out1:RootOutput 4.99e-06 8.2534e-06 2.803e-05 6.3505e-06 6.6137e-06 10
end_path:out1:RootOutput(write) 4.18154 5.39367 8.23759 4.97261 1.24541 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6363.45 MB
Peak resident set size usage (VmHWM): 4119.58 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258341_22_1_20251126T050648Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258341_22_1_20251126T050648Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258341_22_1_20251126T050648Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258341_22_1_20251126T050648Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258341_22_1_20251126T050648Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258341_22_1_20251126T050648Z_detsim_reco1.root.temp.json ====
.:
total 6532580
-rw-r--r-- 1 nobody nobody 40914 Nov 26 06:11 jobscript.log
-rw-r--r-- 1 nobody nobody 203 Nov 26 06:11 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody 0 Nov 26 06:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258341_22_1_20251126T050648Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody 0 Nov 26 06:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258341_22_1_20251126T050648Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody 31 Nov 26 06:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258341_22_1_20251126T050648Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody 25 Nov 26 06:10 all-input-dids.txt
-rw-r--r-- 1 nobody nobody 519 Nov 26 06:10 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody 3574 Nov 26 06:10 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody 204800 Nov 26 06:10 mem.db
-rw-r--r-- 1 nobody nobody 36864 Nov 26 06:10 time.db
-rw-r--r-- 1 nobody nobody 6630169579 Nov 26 06:10 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258341_22_1_20251126T050648Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody 58882800 Nov 26 06:08 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody 0 Nov 26 05:26 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody 0 Nov 26 05:23 debugprod.log
-rw-r--r-- 1 nobody nobody 519 Nov 26 05:22 detsim_single_protodunehd_hist.root 