Jobsub ID 258403.20@dunegpschedd01.fnal.gov
| Jobsub ID | 258403.20@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 06:29:09 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 08:09:41 |
| From worker node | Hostname | ccwcondor0386 |
| cpuinfo | AMD EPYC 9334 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 06:29:54 |
| Input files | hd-protodune:pdhd_prod_beam__238305_82_1_20251106T070027Z_gen_g4_IonScintPDExt.root_255413_57_1_20251122T175717Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (5962s) |
| CPU time | 1h (5797s = 97%) |
| Max RSS bytes | 7173038080 (6840 MiB) |
| Outputting started | 2025-11-26 08:09:17 |
| Output files | |
| Finished | 2025-11-26 08:09:41 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
aces tagged "gauss"
FrameSaver: q=2.37877e+06 n=1244013 tag=gauss
wclsFrameSaver: saving 96668 traces tagged "wiener"
FrameSaver: q=2.97241e+06 n=1199726 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 14021 XUs and 13662 XVs -> 121 XUVs
C:0 T:1 61461 XUs and 62930 XVs -> 1630 XUVs
C:0 T:2 13341 XUs and 15689 XVs -> 528 XUVs
C:0 T:4 21 XUs and 30 XVs -> 0 XUVs
C:0 T:5 36225 XUs and 45377 XVs -> 607 XUVs
C:0 T:6 17485 XUs and 14479 XVs -> 494 XUVs
3380 XUVs total
2186 collection wire objects
3380 potential space points
Neighbour search...
52698 tests to find 29204 neighbours
Iterating with no regularization...
Begin: 2.92985e+07
0 2.81861e+07
1 2.81381e+07
2 2.81353e+07
Now with regularization...
Begin: 2.67758e+07
0 2.67745e+07
26-Nov-2025 09:09:10 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_20_1_20251126T062959Z_detsim_reco1.root"
26-Nov-2025 09:09:10 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 83.3515 133.251 183.382 136.277 27.7801 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00119914 0.00856566 0.0163156 0.00891251 0.00441598 10
reco:opdec:Deconvolution 0.731539 0.950845 1.25528 0.920797 0.139986 10
reco:ophitspe:OpHitFinderDeco 0.167528 0.210065 0.248875 0.212542 0.0218211 10
reco:opflash:OpFlashFinder 0.0655902 0.114346 0.198291 0.111721 0.0353482 10
reco:opslicer:OpSlicer 0.174426 0.286705 0.396828 0.28403 0.0622993 10
reco:wclsdatahd:WireCellToolkit 45.7165 47.8044 53.0518 47.5601 2.04235 10
reco:gaushit:GausHitFinder 0.567635 0.945289 1.15679 0.991532 0.170759 10
reco:nhitsfilter:NumberOfHitsFilter 0.000213503 0.000322681 0.000569603 0.000306353 9.27454e-05 10
reco:reco3d:SpacePointSolver 0.347387 1.4502 3.40885 1.38044 0.903864 10
reco:hitpdune:DisambigFromSpacePoints 0.25447 0.801318 1.35969 0.862555 0.31347 10
reco:pandora:StandardPandora 7.76624 25.9543 48.2627 23.5675 11.4804 10
reco:pandoraWriter:StandardPandora 0.182126 0.343805 0.469822 0.363534 0.0825029 10
reco:pandoraTrack:LArPandoraTrackCreation 0.617296 2.50452 4.72571 2.52559 1.02752 10
reco:pandoraShower:LArPandoraShowerCreation 0.894729 2.35409 4.08666 2.40167 0.924528 10
reco:pandoracalo:Calorimetry 0.389963 1.20697 2.09512 1.19833 0.433909 10
reco:pandoracalonosce:Calorimetry 0.387511 1.13657 2.11838 1.17547 0.446826 10
reco:pandoraShowercalo:ShowerCalorimetry 0.89481 2.09678 3.42025 1.99832 0.804682 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.848721 2.08947 4.1292 1.9346 0.989209 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0392429 0.0630193 0.0905969 0.0647907 0.0153541 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0443514 0.0655718 0.0872701 0.0655775 0.0128099 10
reco:emtrkmichelid:EmTrackMichelId 14.355 22.675 27.5516 23.3428 3.93975 10
reco:crttag:SingleCRTMatchingProducer 0.00504719 0.00940453 0.0173348 0.00906302 0.00361455 10
reco:crtreco:TwoCRTMatchingProducer 0.00361011 0.00658887 0.0122537 0.00659969 0.00234981 10
reco:pandora2Track:LArPandoraTrackCreation 2.05044 6.12606 11.3194 6.01861 2.48839 10
reco:pandora2calo:ShowerCalorimetry 0.747433 1.92474 3.50909 1.81317 0.86361 10
reco:pandora2calonosce:ShowerCalorimetry 0.690038 1.84134 2.92719 1.89289 0.72122 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0396535 0.0683384 0.0896306 0.0693056 0.0158932 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0364858 0.0667551 0.0915416 0.0685814 0.0161007 10
reco:pandora2Shower:LArPandoraShowerCreation 0.883945 2.39844 3.82656 2.4509 0.860641 10
reco:pandora2Showercalo:ShowerCalorimetry 0.89687 2.56357 4.09008 2.52588 0.983466 10
reco:pandora2Showercalonosce:ShowerCalorimetry 0.893501 2.424 3.49019 2.55034 0.857139 10
[art]:TriggerResults:TriggerResultInserter 3.6405e-05 6.18815e-05 0.000116005 5.0717e-05 2.74655e-05 10
end_path:out1:RootOutput 5.318e-06 1.15284e-05 2.8253e-05 9.895e-06 6.53536e-06 10
end_path:out1:RootOutput(write) 1.81834 2.57556 3.2592 2.72116 0.420076 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6707.74 MB
Peak resident set size usage (VmHWM): 4208.99 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_20_1_20251126T062959Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_20_1_20251126T062959Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_20_1_20251126T062959Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_20_1_20251126T062959Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_20_1_20251126T062959Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_20_1_20251126T062959Z_detsim_reco1.root.temp.json ====
.:
total 6282548
-rw-r--r-- 1 dune001 lbno 41067 Nov 26 09:09 jobscript.log
-rw-r--r-- 1 dune001 lbno 179 Nov 26 09:09 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:09 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_20_1_20251126T062959Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:09 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_20_1_20251126T062959Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 09:09 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_20_1_20251126T062959Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 09:09 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:09 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 09:09 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 09:09 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 09:09 time.db
-rw-r--r-- 1 dune001 lbno 6377289861 Nov 26 09:09 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_20_1_20251126T062959Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 55722720 Nov 26 09:08 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:47 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:45 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 08:45 detsim_single_protodunehd_hist.root 