Jobsub ID 252748.60@dunegpschedd02.fnal.gov
| Jobsub ID | 252748.60@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 06:41:11 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 08:53:36 |
| From worker node | Hostname | ccwcondor0573 |
| cpuinfo | AMD EPYC 7453 28-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 06:43:38 |
| Input files | hd-protodune:pdhd_prod_beam__244163_16_1_20251106T091100Z_gen_g4_IonScintPDExt.root_249562_98_1_20251122T082531Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (7753s) |
| CPU time | 1h (7186s = 92%) |
| Max RSS bytes | 7482871808 (7136 MiB) |
| Outputting started | 2025-11-26 08:52:53 |
| Output files | |
| Finished | 2025-11-26 08:53:36 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
2.25883e+06 n=1138581 tag=gauss
wclsFrameSaver: saving 86872 traces tagged "wiener"
FrameSaver: q=2.85195e+06 n=1115893 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 7396 XUs and 8562 XVs -> 61 XUVs
C:0 T:1 47929 XUs and 52531 XVs -> 2301 XUVs
C:0 T:2 9318 XUs and 17700 XVs -> 543 XUVs
C:0 T:4 22 XUs and 23 XVs -> 0 XUVs
C:0 T:5 25119 XUs and 18883 XVs -> 729 XUVs
C:0 T:6 23937 XUs and 26676 XVs -> 1766 XUVs
C:0 T:7 17 XUs and 29 XVs -> 2 XUVs
5402 XUVs total
2444 collection wire objects
5402 potential space points
Neighbour search...
222444 tests to find 76100 neighbours
Iterating with no regularization...
Begin: 5.07481e+07
0 4.64426e+07
1 4.60035e+07
2 4.59704e+07
Now with regularization...
Begin: 4.35846e+07
0 4.35788e+07
26-Nov-2025 09:52:43 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_60_1_20251126T064348Z_detsim_reco1.root"
26-Nov-2025 09:52:43 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 120.428 222.273 370.544 212.714 68.41 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00138034 0.0130762 0.0241142 0.0112941 0.00849134 10
reco:opdec:Deconvolution 0.785602 0.981178 1.15906 1.00619 0.10974 10
reco:ophitspe:OpHitFinderDeco 0.234318 0.262657 0.327936 0.256421 0.0255005 10
reco:opflash:OpFlashFinder 0.0650446 0.100349 0.129271 0.102062 0.0181723 10
reco:opslicer:OpSlicer 0.212124 0.30431 0.437149 0.293014 0.0634736 10
reco:wclsdatahd:WireCellToolkit 54.4013 59.0157 64.7032 58.8432 2.94421 10
reco:gaushit:GausHitFinder 0.861785 1.14618 1.62065 1.1486 0.218607 10
reco:nhitsfilter:NumberOfHitsFilter 0.000333072 0.000493693 0.000756519 0.000480996 0.000118429 10
reco:reco3d:SpacePointSolver 0.73534 1.87979 3.98081 1.91288 0.953006 10
reco:hitpdune:DisambigFromSpacePoints 0.402305 1.09899 2.05753 0.972725 0.429765 10
reco:pandora:StandardPandora 14.9376 45.8812 95.9927 37.7173 25.8161 10
reco:pandoraWriter:StandardPandora 0.284768 0.419646 0.552048 0.425667 0.0876981 10
reco:pandoraTrack:LArPandoraTrackCreation 2.4293 5.3042 11.1565 4.44849 2.67347 10
reco:pandoraShower:LArPandoraShowerCreation 1.21098 4.9931 9.04323 5.1015 2.19292 10
reco:pandoracalo:Calorimetry 0.916813 2.70781 6.55563 2.1444 1.61163 10
reco:pandoracalonosce:Calorimetry 0.880604 2.67223 6.50334 2.00683 1.66081 10
reco:pandoraShowercalo:ShowerCalorimetry 1.00731 4.76485 9.79094 4.33635 2.65488 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.954877 4.53069 8.16809 3.93834 2.16068 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0533607 0.0898443 0.128934 0.0862574 0.0202112 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0575102 0.0911694 0.151006 0.086725 0.0239329 10
reco:emtrkmichelid:EmTrackMichelId 26.7594 48.0201 69.4231 46.1968 12.0471 10
reco:crttag:SingleCRTMatchingProducer 0.0107967 0.0119134 0.0144215 0.0114332 0.00112997 10
reco:crtreco:TwoCRTMatchingProducer 0.00847137 0.0106266 0.0131569 0.0105447 0.00138667 10
reco:pandora2Track:LArPandoraTrackCreation 3.29117 11.3757 23.9082 9.86826 5.37006 10
reco:pandora2calo:ShowerCalorimetry 1.06685 4.32642 11.3526 3.52576 2.89354 10
reco:pandora2calonosce:ShowerCalorimetry 1.04484 3.85093 8.96253 3.39258 2.11102 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.058942 0.0860366 0.116196 0.0846182 0.0160383 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0560428 0.0826412 0.121045 0.0770567 0.0206187 10
reco:pandora2Shower:LArPandoraShowerCreation 1.69126 5.1632 10.0848 4.64359 2.17376 10
reco:pandora2Showercalo:ShowerCalorimetry 2.33862 5.24194 10.9303 3.98983 2.84846 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.49746 4.83119 11.0503 3.74694 2.94738 10
[art]:TriggerResults:TriggerResultInserter 4.9867e-05 8.38999e-05 0.00016245 7.4835e-05 3.29431e-05 10
end_path:out1:RootOutput 7.613e-06 1.52604e-05 3.492e-05 1.38285e-05 6.96779e-06 10
end_path:out1:RootOutput(write) 1.98178 2.7695 3.56968 2.73072 0.464099 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6354.16 MB
Peak resident set size usage (VmHWM): 4114.92 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_60_1_20251126T064348Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_60_1_20251126T064348Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_60_1_20251126T064348Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_60_1_20251126T064348Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_60_1_20251126T064348Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_60_1_20251126T064348Z_detsim_reco1.root.temp.json ====
.:
total 6396532
-rw-r--r-- 1 dune001 lbno 43701 Nov 26 09:52 jobscript.log
-rw-r--r-- 1 dune001 lbno 203 Nov 26 09:52 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:52 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_60_1_20251126T064348Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:52 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_60_1_20251126T064348Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 09:52 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_60_1_20251126T064348Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 09:52 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:52 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 09:52 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 09:52 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 09:52 time.db
-rw-r--r-- 1 dune001 lbno 6494718796 Nov 26 09:52 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_60_1_20251126T064348Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 55015640 Nov 26 09:51 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:18 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:13 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:13 detsim_single_protodunehd_hist.root 