Jobsub ID 258585.2@dunegpschedd01.fnal.gov
| Jobsub ID | 258585.2@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 10:31:25 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce04 |
| Last heartbeat | 2025-11-26 15:59:55 |
| From worker node | Hostname | wn120 |
| cpuinfo | AMD EPYC 7713P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 10:32:38 |
| Input files | hd-protodune:pdhd_prod_beam__238300_65_1_20251106T064044Z_gen_g4_IonScintPDExt.root_255550_131_1_20251123T000148Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 5h (19336s) |
| CPU time | 3h (13648s = 70%) |
| Max RSS bytes | 4196126720 (4001 MiB) |
| Outputting started | 2025-11-26 15:54:56 |
| Output files | |
| Finished | 2025-11-26 15:59:55 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
437e+06 n=1382273 tag=gauss
wclsFrameSaver: saving 102914 traces tagged "wiener"
FrameSaver: q=3.0244e+06 n=1312363 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11882 XUs and 6651 XVs -> 43 XUVs
C:0 T:1 56993 XUs and 29824 XVs -> 833 XUVs
C:0 T:2 23222 XUs and 34799 XVs -> 1064 XUVs
C:0 T:4 8 XUs and 16 XVs -> 0 XUVs
C:0 T:5 82132 XUs and 69676 XVs -> 3131 XUVs
C:0 T:6 19400 XUs and 23947 XVs -> 838 XUVs
C:0 T:7 6 XUs and 15 XVs -> 0 XUVs
5909 XUVs total
3139 collection wire objects
5909 potential space points
Neighbour search...
341643 tests to find 166630 neighbours
Iterating with no regularization...
Begin: 3.4952e+07
0 3.25351e+07
1 3.24161e+07
2 3.24002e+07
Now with regularization...
Begin: 3.04208e+07
0 3.04146e+07
26-Nov-2025 12:54:34 -03 Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258585_2_1_20251126T103247Z_detsim_reco1.root"
26-Nov-2025 12:54:34 -03 Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 358.879 573.614 891.77 557.935 130.941 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00436596 0.0151371 0.0256911 0.0172564 0.00753582 10
reco:opdec:Deconvolution 1.23905 1.8516 3.42902 1.57761 0.634867 10
reco:ophitspe:OpHitFinderDeco 0.385838 0.972706 1.59569 1.02656 0.417506 10
reco:opflash:OpFlashFinder 0.0867428 0.149163 0.250194 0.131608 0.0466691 10
reco:opslicer:OpSlicer 0.293405 0.514269 1.57172 0.382449 0.363633 10
reco:wclsdatahd:WireCellToolkit 127.734 168.741 215.664 166.206 24.7403 10
reco:gaushit:GausHitFinder 1.18212 2.56984 9.70981 1.80953 2.40908 10
reco:nhitsfilter:NumberOfHitsFilter 0.00236091 0.00546406 0.0105864 0.00513602 0.00233456 10
reco:reco3d:SpacePointSolver 1.39041 2.79808 7.63032 2.17111 1.77421 10
reco:hitpdune:DisambigFromSpacePoints 0.754055 1.48916 2.91229 1.4378 0.547045 10
reco:pandora:StandardPandora 23.7289 62.1018 109.244 56.5279 25.2211 10
reco:pandoraWriter:StandardPandora 0.713548 1.11257 1.34498 1.17839 0.211325 10
reco:pandoraTrack:LArPandoraTrackCreation 7.13609 12.3101 18.3149 11.5977 3.54896 10
reco:pandoraShower:LArPandoraShowerCreation 7.63422 17.3519 22.2821 17.9939 4.40475 10
reco:pandoracalo:Calorimetry 2.69325 5.41805 8.70914 5.37354 1.9045 10
reco:pandoracalonosce:Calorimetry 2.35057 6.32046 11.3875 5.71266 3.01418 10
reco:pandoraShowercalo:ShowerCalorimetry 3.37202 11.9631 21.6201 9.56928 6.07404 10
reco:pandoraShowercalonosce:ShowerCalorimetry 3.79386 12.8618 26.7927 9.55325 7.04108 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0900203 0.179051 0.360662 0.169695 0.0708572 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0845601 0.177235 0.388788 0.166723 0.0779892 10
reco:emtrkmichelid:EmTrackMichelId 106.848 142.51 188.984 136.286 25.3151 10
reco:crttag:SingleCRTMatchingProducer 0.014325 0.0388305 0.0787068 0.0318097 0.0174553 10
reco:crtreco:TwoCRTMatchingProducer 0.0190852 0.0379191 0.0718167 0.0300234 0.0169657 10
reco:pandora2Track:LArPandoraTrackCreation 12.8296 35.2877 46.3066 38.1754 11.4148 10
reco:pandora2calo:ShowerCalorimetry 2.33874 14.4426 62.2751 9.97364 16.4101 10
reco:pandora2calonosce:ShowerCalorimetry 2.95605 12.6145 55.8738 6.78266 14.8066 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0551588 0.199318 0.90307 0.123587 0.236863 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.052994 0.191692 0.880102 0.125797 0.232443 10
reco:pandora2Shower:LArPandoraShowerCreation 4.09443 15.9185 55.0699 11.3349 14.3562 10
reco:pandora2Showercalo:ShowerCalorimetry 4.12521 19.5379 81.7412 9.67257 22.4752 10
reco:pandora2Showercalonosce:ShowerCalorimetry 3.3952 16.8108 71.1669 9.95706 19.0139 10
[art]:TriggerResults:TriggerResultInserter 4.6332e-05 0.000107844 0.000241313 9.9023e-05 5.45195e-05 10
end_path:out1:RootOutput 1.151e-05 2.05326e-05 5.7023e-05 1.6115e-05 1.28754e-05 10
end_path:out1:RootOutput(write) 3.38588 6.55778 12.6958 5.80902 2.96797 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6035.25 MB
Peak resident set size usage (VmHWM): 3772.98 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258585_2_1_20251126T103247Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258585_2_1_20251126T103247Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258585_2_1_20251126T103247Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258585_2_1_20251126T103247Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258585_2_1_20251126T103247Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258585_2_1_20251126T103247Z_detsim_reco1.root.temp.json ====
.:
total 5851504
-rw-r--r-- 1 nobody nobody 40671 Nov 26 12:54 jobscript.log
-rw-r--r-- 1 nobody nobody 204 Nov 26 12:54 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody 0 Nov 26 12:54 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258585_2_1_20251126T103247Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody 0 Nov 26 12:54 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258585_2_1_20251126T103247Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody 31 Nov 26 12:54 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258585_2_1_20251126T103247Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody 25 Nov 26 12:54 all-input-dids.txt
-rw-r--r-- 1 nobody nobody 519 Nov 26 12:54 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody 3574 Nov 26 12:54 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody 204800 Nov 26 12:54 mem.db
-rw-r--r-- 1 nobody nobody 36864 Nov 26 12:54 time.db
-rw-r--r-- 1 nobody nobody 5937219129 Nov 26 12:54 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258585_2_1_20251126T103247Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody 54407780 Nov 26 12:44 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody 0 Nov 26 11:22 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody 0 Nov 26 11:12 debugprod.log
-rw-r--r-- 1 nobody nobody 519 Nov 26 11:11 detsim_single_protodunehd_hist.root 