Jobsub ID 252723.7@dunegpschedd02.fnal.gov
| Jobsub ID | 252723.7@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 06:03:08 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 08:14:57 |
| From worker node | Hostname | ccwcondor0722 |
| cpuinfo | AMD EPYC 7302 16-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 06:04:38 |
| Input files | hd-protodune:pdhd_prod_beam__238519_110_1_20251106T102249Z_gen_g4_IonScintPDExt.root_249491_122_1_20251122T001115Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (7790s) |
| CPU time | 1h (7076s = 90%) |
| Max RSS bytes | 7677562880 (7321 MiB) |
| Outputting started | 2025-11-26 08:14:29 |
| Output files | |
| Finished | 2025-11-26 08:14:57 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
s"
FrameSaver: q=1.60479e+06 n=914048 tag=gauss
wclsFrameSaver: saving 72035 traces tagged "wiener"
FrameSaver: q=2.14227e+06 n=909750 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 14105 XUs and 9117 XVs -> 134 XUVs
C:0 T:1 101203 XUs and 73660 XVs -> 3511 XUVs
C:0 T:2 7596 XUs and 10959 XVs -> 468 XUVs
C:0 T:3 6 XUs and 2 XVs -> 0 XUVs
C:0 T:5 4239 XUs and 2426 XVs -> 139 XUVs
C:0 T:6 16995 XUs and 15125 XVs -> 482 XUVs
C:0 T:7 7 XUs and 6 XVs -> 0 XUVs
4734 XUVs total
2092 collection wire objects
4734 potential space points
Neighbour search...
243082 tests to find 77882 neighbours
Iterating with no regularization...
Begin: 4.19485e+07
0 3.65771e+07
1 3.60649e+07
2 3.60459e+07
Now with regularization...
Begin: 3.52657e+07
0 3.52606e+07
26-Nov-2025 09:14:23 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_7_1_20251126T060443Z_detsim_reco1.root"
26-Nov-2025 09:14:23 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 130.053 189.013 308.041 169.384 60.456 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00165006 0.0105534 0.0204705 0.0112085 0.00552655 10
reco:opdec:Deconvolution 0.722238 1.06933 1.3578 1.00811 0.185941 10
reco:ophitspe:OpHitFinderDeco 0.13477 0.221617 0.257206 0.233407 0.0340491 10
reco:opflash:OpFlashFinder 0.0512837 0.104206 0.156613 0.101996 0.0320238 10
reco:opslicer:OpSlicer 0.194721 0.337855 0.482265 0.319411 0.0918298 10
reco:wclsdatahd:WireCellToolkit 52.6113 60.2927 67.4634 60.8221 5.01237 10
reco:gaushit:GausHitFinder 0.73813 1.14387 1.87794 1.06965 0.345671 10
reco:nhitsfilter:NumberOfHitsFilter 0.000389564 0.000644942 0.00104043 0.000635078 0.000187182 10
reco:reco3d:SpacePointSolver 0.767915 1.30894 3.36092 0.916439 0.766067 10
reco:hitpdune:DisambigFromSpacePoints 0.449744 1.09745 2.13526 0.93337 0.576754 10
reco:pandora:StandardPandora 17.4217 36.8489 89.268 32.1294 20.7472 10
reco:pandoraWriter:StandardPandora 0.278285 0.380966 0.568515 0.359486 0.0968931 10
reco:pandoraTrack:LArPandoraTrackCreation 1.19392 2.71387 6.11722 2.04279 1.56612 10
reco:pandoraShower:LArPandoraShowerCreation 1.20588 3.12196 8.10387 2.24887 2.21684 10
reco:pandoracalo:Calorimetry 0.629327 1.46005 3.04339 1.08377 0.804466 10
reco:pandoracalonosce:Calorimetry 0.618486 1.41679 2.97457 1.05415 0.771323 10
reco:pandoraShowercalo:ShowerCalorimetry 1.21446 3.16269 7.59216 2.06737 2.10602 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.17747 2.89312 7.00245 1.99953 1.90353 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0487737 0.084982 0.133989 0.0799894 0.0274275 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0471648 0.0840547 0.13235 0.0774819 0.0255762 10
reco:emtrkmichelid:EmTrackMichelId 28.8989 45.2598 73.1428 40.7474 14.8093 10
reco:crttag:SingleCRTMatchingProducer 0.00558731 0.00840617 0.0103243 0.00908849 0.0016155 10
reco:crtreco:TwoCRTMatchingProducer 0.00439776 0.00695283 0.00883156 0.0073193 0.00153166 10
reco:pandora2Track:LArPandoraTrackCreation 2.85817 7.76441 17.956 6.1239 4.60074 10
reco:pandora2calo:ShowerCalorimetry 0.972668 2.62264 6.62755 1.88299 1.75983 10
reco:pandora2calonosce:ShowerCalorimetry 0.922508 2.60055 6.4318 1.84118 1.71373 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0445697 0.0853963 0.122344 0.0863642 0.0239572 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0461333 0.0843193 0.129635 0.0806852 0.0246598 10
reco:pandora2Shower:LArPandoraShowerCreation 0.996715 3.37123 8.13036 2.41522 2.18726 10
reco:pandora2Showercalo:ShowerCalorimetry 1.04336 3.28036 7.76859 2.1803 2.06857 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.04818 3.07368 7.40389 2.06028 1.96583 10
[art]:TriggerResults:TriggerResultInserter 2.4887e-05 5.37503e-05 0.000156435 4.0952e-05 3.65791e-05 10
end_path:out1:RootOutput 8.927e-06 1.44082e-05 4.2491e-05 1.1787e-05 9.50516e-06 10
end_path:out1:RootOutput(write) 2.15991 2.93655 4.07892 2.93845 0.652252 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6608.43 MB
Peak resident set size usage (VmHWM): 4389.55 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_7_1_20251126T060443Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_7_1_20251126T060443Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_7_1_20251126T060443Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_7_1_20251126T060443Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_7_1_20251126T060443Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_7_1_20251126T060443Z_detsim_reco1.root.temp.json ====
.:
total 6262172
-rw-r--r-- 1 dune001 lbno 42382 Nov 26 09:14 jobscript.log
-rw-r--r-- 1 dune001 lbno 205 Nov 26 09:14 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:14 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_7_1_20251126T060443Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:14 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_7_1_20251126T060443Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 09:14 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_7_1_20251126T060443Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 09:14 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:14 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 09:14 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 09:14 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 09:14 time.db
-rw-r--r-- 1 dune001 lbno 6354963414 Nov 26 09:14 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_7_1_20251126T060443Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 57184290 Nov 26 09:13 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:43 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:40 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 08:40 detsim_single_protodunehd_hist.root 