Jobsub ID 258490.24@dunegpschedd01.fnal.gov
| Jobsub ID | 258490.24@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 08:37:16 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce02 |
| Last heartbeat | 2025-11-26 10:10:02 |
| From worker node | Hostname | ccwcondor0028 |
| cpuinfo | AMD EPYC 9334 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 08:38:44 |
| Input files | hd-protodune:pdhd_prod_beam__238521_104_1_20251106T103343Z_gen_g4_IonScintPDExt.root_249555_21_1_20251122T073844Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (5439s) |
| CPU time | 1h (4961s = 91%) |
| Max RSS bytes | 6193823744 (5906 MiB) |
| Outputting started | 2025-11-26 10:09:24 |
| Output files | |
| Finished | 2025-11-26 10:10:02 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
51427 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8299 XUs and 5034 XVs -> 37 XUVs
C:0 T:1 45733 XUs and 38987 XVs -> 1808 XUVs
C:0 T:2 21489 XUs and 44045 XVs -> 1032 XUVs
C:0 T:3 5 XUs and 5 XVs -> 0 XUVs
C:0 T:4 3 XUs and 6 XVs -> 0 XUVs
C:0 T:5 23505 XUs and 18903 XVs -> 477 XUVs
C:0 T:6 33409 XUs and 36642 XVs -> 1419 XUVs
C:0 T:7 30 XUs and 26 XVs -> 0 XUVs
4773 XUVs total
2020 collection wire objects
4773 potential space points
Neighbour search...
255191 tests to find 125706 neighbours
Iterating with no regularization...
Begin: 3.76587e+07
0 3.49546e+07
1 3.46785e+07
2 3.46579e+07
Now with regularization...
Begin: 3.31381e+07
0 3.31329e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 11:09:16 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_24_1_20251126T083849Z_detsim_reco1.root"
26-Nov-2025 11:09:16 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 91.0819 114.856 152.469 115.539 17.1094 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00118213 0.010827 0.0315855 0.00857684 0.00844846 10
reco:opdec:Deconvolution 0.574855 0.819473 0.991503 0.830255 0.10527 10
reco:ophitspe:OpHitFinderDeco 0.181169 0.207621 0.242907 0.196791 0.0214921 10
reco:opflash:OpFlashFinder 0.0522654 0.0887093 0.129148 0.0908065 0.0203464 10
reco:opslicer:OpSlicer 0.20082 0.244465 0.319802 0.23468 0.0356828 10
reco:wclsdatahd:WireCellToolkit 46.6536 50.2585 53.3382 51.0743 2.43036 10
reco:gaushit:GausHitFinder 0.623772 0.806297 0.983974 0.795408 0.112628 10
reco:nhitsfilter:NumberOfHitsFilter 0.000149286 0.000290783 0.000548692 0.000274867 0.000109214 10
reco:reco3d:SpacePointSolver 0.426703 0.999385 2.02649 0.885345 0.471375 10
reco:hitpdune:DisambigFromSpacePoints 0.348887 0.565675 0.870063 0.55936 0.171109 10
reco:pandora:StandardPandora 11.322 19.4656 42.9563 18.3004 8.32192 10
reco:pandoraWriter:StandardPandora 0.216838 0.302855 0.393101 0.313756 0.0605577 10
reco:pandoraTrack:LArPandoraTrackCreation 1.05882 1.66342 2.45616 1.49896 0.499243 10
reco:pandoraShower:LArPandoraShowerCreation 0.880917 1.6781 3.08246 1.76258 0.705202 10
reco:pandoracalo:Calorimetry 0.541574 0.863872 1.28591 0.842191 0.23252 10
reco:pandoracalonosce:Calorimetry 0.551178 0.871194 1.29575 0.901552 0.23459 10
reco:pandoraShowercalo:ShowerCalorimetry 0.828683 1.66948 2.63993 1.62529 0.666659 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.850215 1.42827 2.15869 1.51027 0.463343 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0420714 0.0548308 0.0710415 0.0519219 0.00981368 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0418939 0.0541043 0.0701563 0.0526529 0.0086973 10
reco:emtrkmichelid:EmTrackMichelId 15.132 18.7301 24.4985 18.8883 2.80939 10
reco:crttag:SingleCRTMatchingProducer 0.00476223 0.0071356 0.00914475 0.00731798 0.00139189 10
reco:crtreco:TwoCRTMatchingProducer 0.00344098 0.00511342 0.00644082 0.00507693 0.000925221 10
reco:pandora2Track:LArPandoraTrackCreation 2.65666 4.14136 6.12172 4.46329 1.20055 10
reco:pandora2calo:ShowerCalorimetry 0.807907 1.32692 2.07903 1.36522 0.423818 10
reco:pandora2calonosce:ShowerCalorimetry 0.754326 1.27832 2.10455 1.31452 0.439493 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0435517 0.0573286 0.0755806 0.0575737 0.0105366 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0424788 0.0544546 0.0656732 0.0542695 0.00834396 10
reco:pandora2Shower:LArPandoraShowerCreation 0.93335 1.58181 2.42386 1.68376 0.48056 10
reco:pandora2Showercalo:ShowerCalorimetry 1.01444 1.69777 2.41332 1.76528 0.519808 10
reco:pandora2Showercalonosce:ShowerCalorimetry 0.945264 1.59072 2.21517 1.65624 0.429233 10
[art]:TriggerResults:TriggerResultInserter 3.0697e-05 4.88241e-05 0.00011216 4.35015e-05 2.2176e-05 10
end_path:out1:RootOutput 3.826e-06 1.0392e-05 2.8363e-05 8.7535e-06 6.23387e-06 10
end_path:out1:RootOutput(write) 1.90976 2.16206 2.4453 2.16594 0.206815 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6659.46 MB
Peak resident set size usage (VmHWM): 4096.65 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_24_1_20251126T083849Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_24_1_20251126T083849Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_24_1_20251126T083849Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_24_1_20251126T083849Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_24_1_20251126T083849Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_24_1_20251126T083849Z_detsim_reco1.root.temp.json ====
.:
total 5536640
-rw-r--r-- 1 dune001 lbno 40620 Nov 26 11:09 jobscript.log
-rw-r--r-- 1 dune001 lbno 204 Nov 26 11:09 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 11:09 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_24_1_20251126T083849Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 11:09 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_24_1_20251126T083849Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 11:09 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_24_1_20251126T083849Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 11:09 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 11:09 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 11:09 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 11:09 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 11:09 time.db
-rw-r--r-- 1 dune001 lbno 5622540916 Nov 26 11:09 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_24_1_20251126T083849Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 46666970 Nov 26 11:08 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:51 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:48 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 10:48 detsim_single_protodunehd_hist.root 