Jobsub ID 258430.80@dunegpschedd01.fnal.gov
| Jobsub ID | 258430.80@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 07:19:14 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 09:28:40 |
| From worker node | Hostname | ccwcondor0722 |
| cpuinfo | AMD EPYC 7302 16-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 07:20:24 |
| Input files | hd-protodune:pdhd_prod_beam__243989_84_1_20251106T070120Z_gen_g4_IonScintPDExt.root_255826_30_1_20251123T215600Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (7665s) |
| CPU time | 2h (7318s = 95%) |
| Max RSS bytes | 7329468416 (6989 MiB) |
| Outputting started | 2025-11-26 09:28:11 |
| Output files | |
| Finished | 2025-11-26 09:28:40 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
and 23683 XVs -> 507 XUVs
C:0 T:1 268998 XUs and 209372 XVs -> 10467 XUVs
C:0 T:2 81008 XUs and 83875 XVs -> 2296 XUVs
C:0 T:3 19 XUs and 27 XVs -> 2 XUVs
C:0 T:4 9 XUs and 10 XVs -> 2 XUVs
C:0 T:5 45045 XUs and 27794 XVs -> 1340 XUVs
C:0 T:6 66420 XUs and 75363 XVs -> 1351 XUVs
C:0 T:7 10 XUs and 15 XVs -> 3 XUVs
15968 XUVs total
7018 collection wire objects
15968 potential space points
Neighbour search...
608124 tests to find 220552 neighbours
Iterating with no regularization...
Begin: 1.17228e+08
0 1.05883e+08
1 1.0498e+08
2 1.04915e+08
Now with regularization...
Begin: 1.01323e+08
0 1.01307e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 10:28:04 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_80_1_20251126T072030Z_detsim_reco1.root"
26-Nov-2025 10:28:04 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 107.262 212.081 397.06 185.019 79.8859 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000938299 0.00940956 0.0378706 0.00687758 0.00990295 10
reco:opdec:Deconvolution 0.871241 1.10305 1.36003 1.05078 0.143965 10
reco:ophitspe:OpHitFinderDeco 0.187984 0.227235 0.244622 0.230451 0.0146058 10
reco:opflash:OpFlashFinder 0.0665245 0.106223 0.165464 0.0955225 0.0321411 10
reco:opslicer:OpSlicer 0.197447 0.3255 0.484144 0.316128 0.0767078 10
reco:wclsdatahd:WireCellToolkit 48.3949 57.1034 64.2311 58.9195 5.67371 10
reco:gaushit:GausHitFinder 0.695214 1.11091 1.60758 1.02625 0.330796 10
reco:nhitsfilter:NumberOfHitsFilter 0.00041384 0.00061355 0.000824264 0.000609299 0.000118093 10
reco:reco3d:SpacePointSolver 0.358913 1.89216 5.08418 1.27581 1.41336 10
reco:hitpdune:DisambigFromSpacePoints 0.390313 1.34446 3.03888 1.10337 0.753607 10
reco:pandora:StandardPandora 10.5434 49.6649 138.573 39.7371 34.0807 10
reco:pandoraWriter:StandardPandora 0.249561 0.405895 0.533852 0.421302 0.0973463 10
reco:pandoraTrack:LArPandoraTrackCreation 1.14548 3.91178 8.97971 3.86986 2.30329 10
reco:pandoraShower:LArPandoraShowerCreation 0.803351 3.8013 8.93242 3.77444 2.45745 10
reco:pandoracalo:Calorimetry 0.599629 1.79899 4.16069 1.64294 1.05541 10
reco:pandoracalonosce:Calorimetry 0.599392 1.77063 4.25914 1.44594 1.08022 10
reco:pandoraShowercalo:ShowerCalorimetry 0.847591 3.39243 8.0958 3.05953 2.1884 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.820442 2.99562 6.62329 2.69682 1.79952 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0379188 0.0899576 0.130179 0.089993 0.0299215 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0380691 0.0878603 0.127056 0.0912624 0.0292919 10
reco:emtrkmichelid:EmTrackMichelId 25.9589 50.4732 76.4845 46.6082 16.3034 10
reco:crttag:SingleCRTMatchingProducer 0.00581968 0.00906585 0.0136365 0.00865325 0.00250225 10
reco:crtreco:TwoCRTMatchingProducer 0.00463251 0.00786361 0.0121848 0.00785339 0.002509 10
reco:pandora2Track:LArPandoraTrackCreation 2.44651 9.50398 21.4683 8.93852 5.761 10
reco:pandora2calo:ShowerCalorimetry 0.766599 2.98119 7.02405 2.71339 1.89497 10
reco:pandora2calonosce:ShowerCalorimetry 0.737757 2.87521 7.23396 2.19368 1.95148 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0594937 0.0905197 0.129516 0.0832056 0.0246967 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0571201 0.0889174 0.132756 0.0801585 0.026867 10
reco:pandora2Shower:LArPandoraShowerCreation 1.05928 3.91648 8.49005 3.06071 2.37557 10
reco:pandora2Showercalo:ShowerCalorimetry 1.11649 4.00178 8.39368 3.86582 2.30182 10
reco:pandora2Showercalonosce:ShowerCalorimetry 0.897466 3.6759 8.09076 3.10941 2.21151 10
[art]:TriggerResults:TriggerResultInserter 2.8083e-05 5.04373e-05 0.00014286 3.96445e-05 3.18684e-05 10
end_path:out1:RootOutput 8.005e-06 1.13625e-05 3.1439e-05 9.553e-06 6.73074e-06 10
end_path:out1:RootOutput(write) 1.98222 3.13251 4.20601 3.02765 0.725771 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6326.12 MB
Peak resident set size usage (VmHWM): 4097.88 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_80_1_20251126T072030Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_80_1_20251126T072030Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_80_1_20251126T072030Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_80_1_20251126T072030Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_80_1_20251126T072030Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_80_1_20251126T072030Z_detsim_reco1.root.temp.json ====
.:
total 6464772
-rw-r--r-- 1 dune001 lbno 42751 Nov 26 10:28 jobscript.log
-rw-r--r-- 1 dune001 lbno 204 Nov 26 10:28 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:28 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_80_1_20251126T072030Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:28 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_80_1_20251126T072030Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 10:28 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_80_1_20251126T072030Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 10:28 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 10:28 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 10:28 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 10:28 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 10:28 time.db
-rw-r--r-- 1 dune001 lbno 6556437915 Nov 26 10:28 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_80_1_20251126T072030Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 63175330 Nov 26 10:24 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:53 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:50 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:50 detsim_single_protodunehd_hist.root 