Jobsub ID 258399.15@dunegpschedd01.fnal.gov
| Jobsub ID | 258399.15@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 06:23:29 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 08:25:59 |
| From worker node | Hostname | ccwcondor0722 |
| cpuinfo | AMD EPYC 7302 16-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 06:24:20 |
| Input files | hd-protodune:pdhd_prod_beam__238267_77_1_20251106T055745Z_gen_g4_IonScintPDExt.root_255554_125_1_20251123T001226Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (7272s) |
| CPU time | 1h (6611s = 90%) |
| Max RSS bytes | 6986960896 (6663 MiB) |
| Outputting started | 2025-11-26 08:25:33 |
| Output files | |
| Finished | 2025-11-26 08:25:59 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
ss
wclsFrameSaver: saving 81554 traces tagged "wiener"
FrameSaver: q=2.55592e+06 n=994692 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12247 XUs and 10769 XVs -> 61 XUVs
C:0 T:1 79001 XUs and 80165 XVs -> 2375 XUVs
C:0 T:2 9322 XUs and 10266 XVs -> 522 XUVs
C:0 T:3 6 XUs and 11 XVs -> 1 XUVs
C:0 T:4 11 XUs and 25 XVs -> 3 XUVs
C:0 T:5 12738 XUs and 12396 XVs -> 626 XUVs
C:0 T:6 8788 XUs and 5622 XVs -> 366 XUVs
C:0 T:7 2 XUs and 3 XVs -> 0 XUVs
3954 XUVs total
1948 collection wire objects
3954 potential space points
Neighbour search...
92666 tests to find 50606 neighbours
Iterating with no regularization...
Begin: 3.27874e+07
0 3.07849e+07
1 3.06888e+07
2 3.06797e+07
Now with regularization...
Begin: 2.92419e+07
0 2.92394e+07
26-Nov-2025 09:25:27 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258399_15_1_20251126T062424Z_detsim_reco1.root"
26-Nov-2025 09:25:27 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 133.097 174.459 260.559 172.789 34.9274 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00124464 0.00739817 0.0114115 0.00858494 0.00309056 10
reco:opdec:Deconvolution 0.820673 0.989708 1.23404 0.961342 0.132011 10
reco:ophitspe:OpHitFinderDeco 0.128761 0.210347 0.279766 0.217269 0.0430024 10
reco:opflash:OpFlashFinder 0.0494944 0.0972129 0.148433 0.0965391 0.0278957 10
reco:opslicer:OpSlicer 0.147469 0.272004 0.422054 0.259552 0.0772535 10
reco:wclsdatahd:WireCellToolkit 53.3323 60.1745 66.6917 59.0997 3.90219 10
reco:gaushit:GausHitFinder 0.656272 1.05021 1.5749 0.993273 0.247432 10
reco:nhitsfilter:NumberOfHitsFilter 0.000420443 0.000551838 0.00106064 0.000494828 0.000181051 10
reco:reco3d:SpacePointSolver 0.612859 1.11864 1.93439 0.96261 0.363901 10
reco:hitpdune:DisambigFromSpacePoints 0.530486 0.952981 1.66028 0.930522 0.315766 10
reco:pandora:StandardPandora 15.0589 33.7087 61.9798 31.9118 12.9929 10
reco:pandoraWriter:StandardPandora 0.273097 0.327314 0.418853 0.313446 0.0434344 10
reco:pandoraTrack:LArPandoraTrackCreation 1.44576 2.74884 6.13761 2.2354 1.28716 10
reco:pandoraShower:LArPandoraShowerCreation 1.2659 2.33793 4.23794 2.0176 1.01934 10
reco:pandoracalo:Calorimetry 0.657347 1.38278 2.98543 1.27348 0.648762 10
reco:pandoracalonosce:Calorimetry 0.74132 1.36086 3.21375 1.13732 0.72519 10
reco:pandoraShowercalo:ShowerCalorimetry 1.15634 2.28653 3.89645 1.9905 0.996376 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.2813 2.1322 3.53491 1.88714 0.864143 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0520964 0.0754315 0.101597 0.0703381 0.0158078 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0493643 0.0751409 0.0975273 0.0727235 0.0141964 10
reco:emtrkmichelid:EmTrackMichelId 29.3074 43.0933 63.5437 42.7894 9.05936 10
reco:crttag:SingleCRTMatchingProducer 0.00485919 0.00697034 0.0107054 0.00619024 0.0020869 10
reco:crtreco:TwoCRTMatchingProducer 0.00378614 0.00564282 0.00838797 0.00499799 0.00159047 10
reco:pandora2Track:LArPandoraTrackCreation 3.57334 5.89931 11.3287 4.53239 2.52212 10
reco:pandora2calo:ShowerCalorimetry 1.16083 1.84736 3.08967 1.53533 0.713912 10
reco:pandora2calonosce:ShowerCalorimetry 1.13649 1.72012 3.26831 1.36489 0.677425 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0502273 0.0659863 0.0987971 0.0596381 0.0164877 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0494428 0.0663521 0.0945698 0.064051 0.0154257 10
reco:pandora2Shower:LArPandoraShowerCreation 1.48648 2.5382 4.55383 1.90282 1.08308 10
reco:pandora2Showercalo:ShowerCalorimetry 1.39212 2.55566 5.06435 1.94774 1.16798 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.28678 2.54434 5.20142 1.94333 1.25554 10
[art]:TriggerResults:TriggerResultInserter 2.3765e-05 5.4059e-05 0.000125266 4.00105e-05 3.6464e-05 10
end_path:out1:RootOutput 6.653e-06 1.52871e-05 5.7999e-05 1.00795e-05 1.46217e-05 10
end_path:out1:RootOutput(write) 1.9844 2.66349 3.77418 2.48292 0.547956 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6466.44 MB
Peak resident set size usage (VmHWM): 4247.2 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258399_15_1_20251126T062424Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258399_15_1_20251126T062424Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258399_15_1_20251126T062424Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258399_15_1_20251126T062424Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258399_15_1_20251126T062424Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258399_15_1_20251126T062424Z_detsim_reco1.root.temp.json ====
.:
total 6185948
-rw-r--r-- 1 dune001 lbno 40560 Nov 26 09:25 jobscript.log
-rw-r--r-- 1 dune001 lbno 204 Nov 26 09:25 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:25 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258399_15_1_20251126T062424Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:25 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258399_15_1_20251126T062424Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 09:25 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258399_15_1_20251126T062424Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 09:25 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:25 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 09:25 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 09:25 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 09:25 time.db
-rw-r--r-- 1 dune001 lbno 6280495368 Nov 26 09:25 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258399_15_1_20251126T062424Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 53602580 Nov 26 09:24 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:58 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:54 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 08:54 detsim_single_protodunehd_hist.root 