Jobsub ID 252789.15@dunegpschedd02.fnal.gov
| Jobsub ID | 252789.15@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 07:57:14 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce04 |
| Last heartbeat | 2025-11-26 12:15:52 |
| From worker node | Hostname | wn121.cat.cbpf.br |
| cpuinfo | AMD EPYC 7713P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 07:58:38 |
| Input files | hd-protodune:pdhd_prod_beam__238531_9_1_20251106T104602Z_gen_g4_IonScintPDExt.root_249482_33_1_20251121T232539Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 4h (15164s) |
| CPU time | 2h (10229s = 67%) |
| Max RSS bytes | 4237168640 (4040 MiB) |
| Outputting started | 2025-11-26 12:11:23 |
| Output files | |
| Finished | 2025-11-26 12:15:52 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
02568e+06 n=911751 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 6301 XUs and 5170 XVs -> 31 XUVs
C:0 T:1 27615 XUs and 20896 XVs -> 389 XUVs
C:0 T:2 31038 XUs and 44350 XVs -> 2358 XUVs
C:0 T:3 17 XUs and 11 XVs -> 3 XUVs
C:0 T:4 9 XUs and 5 XVs -> 0 XUVs
C:0 T:5 16552 XUs and 11381 XVs -> 321 XUVs
C:0 T:6 4758 XUs and 9379 XVs -> 95 XUVs
3197 XUVs total
1532 collection wire objects
3197 potential space points
Neighbour search...
78199 tests to find 47492 neighbours
Iterating with no regularization...
Begin: 1.97391e+07
0 1.82415e+07
1 1.81047e+07
2 1.80811e+07
3 1.80736e+07
Now with regularization...
Begin: 1.67103e+07
0 1.67061e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 09:11:12 -03 Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252789_15_1_20251126T075844Z_detsim_reco1.root"
26-Nov-2025 09:11:12 -03 Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 217.654 313.346 400.475 321.373 53.3043 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00205851 0.0140534 0.0236063 0.0142998 0.00741816 10
reco:opdec:Deconvolution 1.1825 1.37672 1.60154 1.38131 0.128566 10
reco:ophitspe:OpHitFinderDeco 0.32101 0.408502 0.714939 0.338358 0.136293 10
reco:opflash:OpFlashFinder 0.11367 0.140737 0.166815 0.144312 0.0201376 10
reco:opslicer:OpSlicer 0.349559 0.416847 0.469417 0.413336 0.0418518 10
reco:wclsdatahd:WireCellToolkit 80.4537 102.39 150.704 89.8051 24.8529 10
reco:gaushit:GausHitFinder 0.996794 1.51423 1.96135 1.48012 0.307136 10
reco:nhitsfilter:NumberOfHitsFilter 0.00120435 0.00231448 0.00565426 0.00184378 0.00125831 10
reco:reco3d:SpacePointSolver 1.07406 2.09374 4.67193 1.92068 1.02748 10
reco:hitpdune:DisambigFromSpacePoints 0.728555 1.37411 2.00033 1.34177 0.456241 10
reco:pandora:StandardPandora 20.7022 49.8419 97.2023 41.2264 24.1571 10
reco:pandoraWriter:StandardPandora 0.605267 0.805666 0.953218 0.81781 0.110194 10
reco:pandoraTrack:LArPandoraTrackCreation 4.58131 7.02636 9.68466 6.81627 1.71645 10
reco:pandoraShower:LArPandoraShowerCreation 2.9494 6.49622 9.94783 5.84044 2.57026 10
reco:pandoracalo:Calorimetry 2.04057 3.24056 4.97455 2.94054 0.897165 10
reco:pandoracalonosce:Calorimetry 2.0004 3.16162 5.07788 2.89839 0.885053 10
reco:pandoraShowercalo:ShowerCalorimetry 3.1839 5.75723 9.68603 4.96457 2.17659 10
reco:pandoraShowercalonosce:ShowerCalorimetry 2.77519 5.68052 9.76091 5.01066 2.10813 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0934098 0.128758 0.15619 0.134417 0.0210147 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0762906 0.125597 0.153685 0.130413 0.0244932 10
reco:emtrkmichelid:EmTrackMichelId 50.8133 73.5 99.1192 73.9865 14.8801 10
reco:crttag:SingleCRTMatchingProducer 0.00928639 0.0180616 0.0261164 0.0174574 0.0048972 10
reco:crtreco:TwoCRTMatchingProducer 0.00819146 0.015351 0.0207912 0.0146772 0.00356167 10
reco:pandora2Track:LArPandoraTrackCreation 8.73015 16.6282 31.7521 14.8201 7.27499 10
reco:pandora2calo:ShowerCalorimetry 2.22049 4.76552 10.4628 4.85002 2.19241 10
reco:pandora2calonosce:ShowerCalorimetry 2.2785 4.46467 7.5266 4.82051 1.4874 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0851062 0.118665 0.163554 0.113847 0.0222152 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0774287 0.113078 0.144368 0.115057 0.0196089 10
reco:pandora2Shower:LArPandoraShowerCreation 2.87606 6.09403 9.87877 5.94308 2.13572 10
reco:pandora2Showercalo:ShowerCalorimetry 2.74049 6.0049 11.7316 5.93158 2.5958 10
reco:pandora2Showercalonosce:ShowerCalorimetry 2.52386 5.20754 9.23914 5.03355 2.03109 10
[art]:TriggerResults:TriggerResultInserter 2.4961e-05 7.29176e-05 0.000157737 6.2737e-05 3.45578e-05 10
end_path:out1:RootOutput 6.18e-06 2.88381e-05 0.000160676 1.12355e-05 4.49093e-05 10
end_path:out1:RootOutput(write) 3.0462 4.05575 5.0735 4.03906 0.600033 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6385.34 MB
Peak resident set size usage (VmHWM): 4163.7 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252789_15_1_20251126T075844Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252789_15_1_20251126T075844Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252789_15_1_20251126T075844Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252789_15_1_20251126T075844Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252789_15_1_20251126T075844Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252789_15_1_20251126T075844Z_detsim_reco1.root.temp.json ====
.:
total 6168044
-rw-r--r-- 1 nobody nobody 42294 Nov 26 09:11 jobscript.log
-rw-r--r-- 1 nobody nobody 202 Nov 26 09:11 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody 0 Nov 26 09:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252789_15_1_20251126T075844Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody 0 Nov 26 09:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252789_15_1_20251126T075844Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody 31 Nov 26 09:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252789_15_1_20251126T075844Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody 25 Nov 26 09:11 all-input-dids.txt
-rw-r--r-- 1 nobody nobody 519 Nov 26 09:11 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody 3574 Nov 26 09:11 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody 204800 Nov 26 09:11 mem.db
-rw-r--r-- 1 nobody nobody 36864 Nov 26 09:11 time.db
-rw-r--r-- 1 nobody nobody 6262976851 Nov 26 09:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252789_15_1_20251126T075844Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody 52782310 Nov 26 09:09 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody 0 Nov 26 08:21 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody 0 Nov 26 08:15 debugprod.log
-rw-r--r-- 1 nobody nobody 519 Nov 26 08:15 detsim_single_protodunehd_hist.root 