Jobsub ID 258490.12@dunegpschedd01.fnal.gov
| Jobsub ID | 258490.12@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 08:37:16 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 11:11:27 |
| From worker node | Hostname | ccwcondor0722 |
| cpuinfo | AMD EPYC 7302 16-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 08:38:44 |
| Input files | hd-protodune:pdhd_prod_beam__244163_20_1_20251106T091103Z_gen_g4_IonScintPDExt.root_255163_171_1_20251122T001638Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (9134s) |
| CPU time | 2h (7727s = 84%) |
| Max RSS bytes | 7906643968 (7540 MiB) |
| Outputting started | 2025-11-26 11:10:58 |
| Output files | |
| Finished | 2025-11-26 11:11:27 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
FrameSaver: q=2.48202e+06 n=1224785 tag=gauss
wclsFrameSaver: saving 93913 traces tagged "wiener"
FrameSaver: q=3.71017e+06 n=1222089 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 18330 XUs and 15887 XVs -> 510 XUVs
C:0 T:1 139862 XUs and 141486 XVs -> 7986 XUVs
C:0 T:2 26451 XUs and 40911 XVs -> 1511 XUVs
C:0 T:3 4 XUs and 10 XVs -> 0 XUVs
C:0 T:5 39609 XUs and 17272 XVs -> 292 XUVs
C:0 T:6 15639 XUs and 21031 XVs -> 1246 XUVs
11545 XUVs total
3389 collection wire objects
11545 potential space points
Neighbour search...
737743 tests to find 294806 neighbours
Iterating with no regularization...
Begin: 6.0847e+07
0 5.24679e+07
1 5.17883e+07
2 5.17009e+07
3 5.16766e+07
Now with regularization...
Begin: 4.92594e+07
0 4.92436e+07
26-Nov-2025 12:10:52 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_12_1_20251126T083848Z_detsim_reco1.root"
26-Nov-2025 12:10:52 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 135.104 202.716 263.267 207.436 39.7268 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00120739 0.0121349 0.020914 0.0105907 0.00647715 10
reco:opdec:Deconvolution 0.725428 1.0979 1.33641 1.10304 0.157242 10
reco:ophitspe:OpHitFinderDeco 0.164385 0.220259 0.239907 0.225192 0.023653 10
reco:opflash:OpFlashFinder 0.0592208 0.115542 0.187751 0.110034 0.0314546 10
reco:opslicer:OpSlicer 0.219913 0.353431 0.431736 0.366881 0.0693629 10
reco:wclsdatahd:WireCellToolkit 57.5825 60.3195 63.5023 60.0682 2.03136 10
reco:gaushit:GausHitFinder 0.741332 1.04722 1.45862 1.06206 0.193433 10
reco:nhitsfilter:NumberOfHitsFilter 0.000323691 0.000560347 0.00087025 0.000563963 0.000148207 10
reco:reco3d:SpacePointSolver 0.700634 1.73622 2.89183 1.91347 0.633416 10
reco:hitpdune:DisambigFromSpacePoints 0.434329 1.1521 1.91768 1.17395 0.421241 10
reco:pandora:StandardPandora 17.9069 43.1035 95.4033 36.419 21.287 10
reco:pandoraWriter:StandardPandora 0.27417 0.398113 0.470796 0.421485 0.0619592 10
reco:pandoraTrack:LArPandoraTrackCreation 1.59102 3.00883 4.50443 2.88459 0.788422 10
reco:pandoraShower:LArPandoraShowerCreation 1.36005 3.21333 5.79803 3.25147 1.42666 10
reco:pandoracalo:Calorimetry 0.817875 1.47295 2.32955 1.49627 0.402263 10
reco:pandoracalonosce:Calorimetry 0.815744 1.40232 2.27742 1.30314 0.412801 10
reco:pandoraShowercalo:ShowerCalorimetry 1.48874 2.81702 5.17506 2.56553 1.22758 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.35635 2.70639 4.68155 2.63795 1.12964 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0492334 0.0766207 0.121591 0.0693546 0.0257829 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0471492 0.071695 0.121433 0.065778 0.0225641 10
reco:emtrkmichelid:EmTrackMichelId 33.2624 51.0341 67.8587 53.13 9.9413 10
reco:crttag:SingleCRTMatchingProducer 0.00513914 0.00775247 0.00986969 0.00767484 0.00117685 10
reco:crtreco:TwoCRTMatchingProducer 0.00432597 0.00651031 0.00844843 0.00650606 0.000971217 10
reco:pandora2Track:LArPandoraTrackCreation 3.74877 8.16496 13.2964 8.54624 2.82327 10
reco:pandora2calo:ShowerCalorimetry 1.19467 2.73453 4.68388 2.6295 1.12042 10
reco:pandora2calonosce:ShowerCalorimetry 0.96552 2.60305 4.18771 2.72246 1.0796 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0535722 0.0839704 0.130521 0.0818162 0.0239364 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0497967 0.0791803 0.123078 0.0779001 0.0234626 10
reco:pandora2Shower:LArPandoraShowerCreation 1.63693 3.64734 7.07621 3.51775 1.6031 10
reco:pandora2Showercalo:ShowerCalorimetry 1.51078 3.40722 4.77541 3.67627 1.18373 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.44893 3.48558 6.38737 3.56024 1.4089 10
[art]:TriggerResults:TriggerResultInserter 2.8123e-05 4.98188e-05 0.000118283 4.4794e-05 2.46561e-05 10
end_path:out1:RootOutput 8.556e-06 1.25937e-05 3.3293e-05 1.0545e-05 7.03596e-06 10
end_path:out1:RootOutput(write) 2.09588 2.96833 4.08386 2.97976 0.551908 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6535.18 MB
Peak resident set size usage (VmHWM): 4320.5 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_12_1_20251126T083848Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_12_1_20251126T083848Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_12_1_20251126T083848Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_12_1_20251126T083848Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_12_1_20251126T083848Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_12_1_20251126T083848Z_detsim_reco1.root.temp.json ====
.:
total 7039612
-rw-r--r-- 1 dune001 lbno 41115 Nov 26 12:10 jobscript.log
-rw-r--r-- 1 dune001 lbno 204 Nov 26 12:10 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 12:10 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_12_1_20251126T083848Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 12:10 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_12_1_20251126T083848Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 12:10 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_12_1_20251126T083848Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 12:10 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 12:10 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 12:10 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 12:10 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 12:10 time.db
-rw-r--r-- 1 dune001 lbno 7147976263 Nov 26 12:10 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258490_12_1_20251126T083848Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 60272210 Nov 26 12:09 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 11:38 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 11:35 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 11:34 detsim_single_protodunehd_hist.root 