Jobsub ID 258430.169@dunegpschedd01.fnal.gov
| Jobsub ID | 258430.169@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 07:19:14 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 09:26:48 |
| From worker node | Hostname | ccwcondor0551 |
| cpuinfo | AMD EPYC 7453 28-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 07:21:51 |
| Input files | hd-protodune:pdhd_prod_beam__243877_46_1_20251106T035315Z_gen_g4_IonScintPDExt.root_249557_39_1_20251122T074630Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (7452s) |
| CPU time | 1h (6774s = 90%) |
| Max RSS bytes | 6983016448 (6659 MiB) |
| Outputting started | 2025-11-26 09:26:05 |
| Output files | |
| Finished | 2025-11-26 09:26:48 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
aving 97763 traces tagged "wiener"
FrameSaver: q=3.09924e+06 n=1162437 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9500 XUs and 10254 XVs -> 95 XUVs
C:0 T:1 71469 XUs and 104817 XVs -> 2248 XUVs
C:0 T:2 45934 XUs and 35309 XVs -> 1064 XUVs
C:0 T:3 6 XUs and 5 XVs -> 0 XUVs
C:0 T:4 30 XUs and 20 XVs -> 0 XUVs
C:0 T:5 44640 XUs and 32574 XVs -> 1037 XUVs
C:0 T:6 36973 XUs and 25051 XVs -> 670 XUVs
C:0 T:7 18 XUs and 12 XVs -> 3 XUVs
5117 XUVs total
2451 collection wire objects
5117 potential space points
Neighbour search...
157545 tests to find 79890 neighbours
Iterating with no regularization...
Begin: 4.33729e+07
0 4.00974e+07
1 3.99217e+07
2 3.99087e+07
Now with regularization...
Begin: 3.86086e+07
0 3.86055e+07
26-Nov-2025 10:25:54 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_169_1_20251126T072157Z_detsim_reco1.root"
26-Nov-2025 10:25:54 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 144.066 225.752 377.849 218.773 62.0014 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00127157 0.0186929 0.0359005 0.0186215 0.0126728 10
reco:opdec:Deconvolution 0.850929 1.09494 1.37568 1.09291 0.157419 10
reco:ophitspe:OpHitFinderDeco 0.234051 0.270074 0.317076 0.260232 0.0288479 10
reco:opflash:OpFlashFinder 0.066768 0.117632 0.180354 0.106277 0.0338118 10
reco:opslicer:OpSlicer 0.248394 0.376278 0.525768 0.394214 0.0819689 10
reco:wclsdatahd:WireCellToolkit 56.2809 58.7923 63.8092 58.0374 2.30058 10
reco:gaushit:GausHitFinder 0.78157 1.18034 1.60721 1.159 0.240452 10
reco:nhitsfilter:NumberOfHitsFilter 0.0003798 0.000565615 0.000841102 0.0005614 0.000150071 10
reco:reco3d:SpacePointSolver 0.881863 1.62718 2.5739 1.54446 0.594016 10
reco:hitpdune:DisambigFromSpacePoints 0.485001 1.10666 1.82693 1.02527 0.409831 10
reco:pandora:StandardPandora 20.4166 44.023 102.122 35.8162 22.4359 10
reco:pandoraWriter:StandardPandora 0.285585 0.416915 0.558351 0.407799 0.0765094 10
reco:pandoraTrack:LArPandoraTrackCreation 2.35317 5.6634 12.42 4.69617 2.82201 10
reco:pandoraShower:LArPandoraShowerCreation 1.41194 6.05427 13.0511 6.21358 3.13321 10
reco:pandoracalo:Calorimetry 1.05717 2.84685 6.87005 2.62619 1.53268 10
reco:pandoracalonosce:Calorimetry 1.30915 2.80963 7.45102 2.21242 1.68896 10
reco:pandoraShowercalo:ShowerCalorimetry 2.28192 5.33772 11.1099 4.62776 2.53099 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.39117 4.82215 11.086 4.24531 2.44054 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0526838 0.0903931 0.113266 0.0985653 0.0177237 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0544635 0.0882185 0.115248 0.0914857 0.0160311 10
reco:emtrkmichelid:EmTrackMichelId 36.7841 47.7626 59.6772 48.8072 7.24657 10
reco:crttag:SingleCRTMatchingProducer 0.0095031 0.014199 0.0200727 0.0134948 0.00302964 10
reco:crtreco:TwoCRTMatchingProducer 0.00754044 0.012823 0.0186971 0.0129641 0.00293883 10
reco:pandora2Track:LArPandoraTrackCreation 3.98754 12.0625 24.6785 11.5149 5.38126 10
reco:pandora2calo:ShowerCalorimetry 0.799782 4.23094 10.4375 3.73874 2.50857 10
reco:pandora2calonosce:ShowerCalorimetry 0.864048 4.46656 12.1727 4.03244 2.93078 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0566574 0.0897173 0.123664 0.0863968 0.0194182 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0517039 0.0879944 0.12649 0.0860485 0.0208198 10
reco:pandora2Shower:LArPandoraShowerCreation 1.67583 6.01785 13.3555 6.29332 3.04627 10
reco:pandora2Showercalo:ShowerCalorimetry 1.2571 5.78355 15.9456 4.93656 3.90544 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.33033 5.38616 13.0775 5.11955 3.12501 10
[art]:TriggerResults:TriggerResultInserter 3.9364e-05 8.79957e-05 0.000193051 7.91835e-05 4.18362e-05 10
end_path:out1:RootOutput 1.0139e-05 1.94752e-05 5.9731e-05 1.4842e-05 1.3717e-05 10
end_path:out1:RootOutput(write) 2.07997 2.84622 3.59148 2.86128 0.388648 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6346.23 MB
Peak resident set size usage (VmHWM): 4121.41 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_169_1_20251126T072157Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_169_1_20251126T072157Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_169_1_20251126T072157Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_169_1_20251126T072157Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_169_1_20251126T072157Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_169_1_20251126T072157Z_detsim_reco1.root.temp.json ====
.:
total 6140772
-rw-r--r-- 1 dune001 lbno 41505 Nov 26 10:26 jobscript.log
-rw-r--r-- 1 dune001 lbno 203 Nov 26 10:26 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_169_1_20251126T072157Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_169_1_20251126T072157Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 10:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_169_1_20251126T072157Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 10:25 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 10:25 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 10:25 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 10:25 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 10:25 time.db
-rw-r--r-- 1 dune001 lbno 6233043024 Nov 26 10:25 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_169_1_20251126T072157Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 54789260 Nov 26 10:23 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:50 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:46 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:45 detsim_single_protodunehd_hist.root 