Jobsub ID 252804.0@dunegpschedd02.fnal.gov
| Jobsub ID | 252804.0@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 08:27:15 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce01 |
| Last heartbeat | 2025-11-26 10:29:58 |
| From worker node | Hostname | ccwcondor0680 |
| cpuinfo | AMD EPYC 7302 16-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 08:27:55 |
| Input files | hd-protodune:pdhd_prod_beam__238300_65_1_20251106T064044Z_gen_g4_IonScintPDExt.root_255550_131_1_20251123T000148Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (7289s) |
| CPU time | 1h (6501s = 89%) |
| Max RSS bytes | 6414848000 (6117 MiB) |
| Outputting started | 2025-11-26 10:29:25 |
| Output files | |
| Finished | 2025-11-26 10:29:58 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
ged "wiener"
FrameSaver: q=3.02479e+06 n=1312058 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11094 XUs and 6026 XVs -> 40 XUVs
C:0 T:1 54124 XUs and 28159 XVs -> 850 XUVs
C:0 T:2 22695 XUs and 35236 XVs -> 1025 XUVs
C:0 T:4 12 XUs and 182 XVs -> 0 XUVs
C:0 T:5 81929 XUs and 69991 XVs -> 3128 XUVs
C:0 T:6 19832 XUs and 23951 XVs -> 873 XUVs
C:0 T:7 6 XUs and 14 XVs -> 0 XUVs
5916 XUVs total
3171 collection wire objects
5916 potential space points
Neighbour search...
346340 tests to find 168600 neighbours
Iterating with no regularization...
Begin: 3.24263e+07
0 3.00624e+07
1 2.99317e+07
2 2.99168e+07
Now with regularization...
Begin: 2.79888e+07
0 2.79836e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 11:29:17 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_0_1_20251126T082800Z_detsim_reco1.root"
26-Nov-2025 11:29:17 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 125.167 166.314 203.916 167.159 20.5232 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.001557 0.0149995 0.0388883 0.0114274 0.011762 10
reco:opdec:Deconvolution 0.925227 1.05336 1.22233 1.0371 0.108941 10
reco:ophitspe:OpHitFinderDeco 0.217511 0.234925 0.25462 0.231724 0.0123822 10
reco:opflash:OpFlashFinder 0.0675859 0.0970222 0.140064 0.0878375 0.0257281 10
reco:opslicer:OpSlicer 0.236117 0.323174 0.407626 0.321898 0.0526749 10
reco:wclsdatahd:WireCellToolkit 49.978 53.359 60.1405 51.7137 3.54814 10
reco:gaushit:GausHitFinder 0.801448 1.11101 1.40396 1.14535 0.154624 10
reco:nhitsfilter:NumberOfHitsFilter 0.000367242 0.000535315 0.000876031 0.000512561 0.000130863 10
reco:reco3d:SpacePointSolver 0.665486 1.12595 2.24493 0.968051 0.456224 10
reco:hitpdune:DisambigFromSpacePoints 0.54402 0.941275 1.26548 0.879384 0.229335 10
reco:pandora:StandardPandora 18.5324 31.9242 44.6435 30.6737 7.96938 10
reco:pandoraWriter:StandardPandora 0.302915 0.362828 0.413781 0.36366 0.0305092 10
reco:pandoraTrack:LArPandoraTrackCreation 1.85273 2.43822 3.98971 2.29846 0.637632 10
reco:pandoraShower:LArPandoraShowerCreation 1.11195 2.38217 3.78501 2.25633 0.70361 10
reco:pandoracalo:Calorimetry 0.746108 1.27512 2.17852 1.19379 0.3686 10
reco:pandoracalonosce:Calorimetry 0.733105 1.23846 2.14159 1.14324 0.369496 10
reco:pandoraShowercalo:ShowerCalorimetry 1.19468 2.45721 4.08539 2.2249 0.772549 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.16316 2.3311 3.85702 2.11985 0.720499 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0614337 0.0870528 0.106585 0.0895714 0.0123272 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0592624 0.0843118 0.10542 0.0870474 0.0114304 10
reco:emtrkmichelid:EmTrackMichelId 30.1835 41.0283 46.3482 41.5655 4.98139 10
reco:crttag:SingleCRTMatchingProducer 0.00578716 0.00804506 0.0117254 0.00768473 0.0015057 10
reco:crtreco:TwoCRTMatchingProducer 0.00453292 0.00760186 0.0187856 0.00635314 0.00382401 10
reco:pandora2Track:LArPandoraTrackCreation 3.62566 6.61974 10.7175 6.11547 1.89273 10
reco:pandora2calo:ShowerCalorimetry 1.09502 2.14327 3.45282 1.93058 0.656105 10
reco:pandora2calonosce:ShowerCalorimetry 1.04931 2.06532 3.18366 1.88881 0.601156 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0568523 0.0849791 0.104818 0.0862361 0.0124066 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.056275 0.0831083 0.101693 0.0842449 0.0115341 10
reco:pandora2Shower:LArPandoraShowerCreation 1.4256 2.66621 4.10402 2.56634 0.699131 10
reco:pandora2Showercalo:ShowerCalorimetry 1.52114 2.84833 4.41126 2.66409 0.80438 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.53386 2.82855 4.35113 2.54922 0.776632 10
[art]:TriggerResults:TriggerResultInserter 3.2181e-05 4.92258e-05 0.00012719 3.8437e-05 2.71976e-05 10
end_path:out1:RootOutput 9.808e-06 1.56403e-05 5.6707e-05 1.1356e-05 1.37118e-05 10
end_path:out1:RootOutput(write) 2.17944 2.95027 3.35959 3.02688 0.350313 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6055.95 MB
Peak resident set size usage (VmHWM): 3838.42 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_0_1_20251126T082800Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_0_1_20251126T082800Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_0_1_20251126T082800Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_0_1_20251126T082800Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_0_1_20251126T082800Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_0_1_20251126T082800Z_detsim_reco1.root.temp.json ====
.:
total 5852932
-rw-r--r-- 1 dune001 lbno 40863 Nov 26 11:29 jobscript.log
-rw-r--r-- 1 dune001 lbno 204 Nov 26 11:29 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 11:29 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_0_1_20251126T082800Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 11:29 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_0_1_20251126T082800Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 11:29 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_0_1_20251126T082800Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 11:29 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 11:29 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 11:29 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 11:29 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 11:29 time.db
-rw-r--r-- 1 dune001 lbno 5938607263 Nov 26 11:29 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_0_1_20251126T082800Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 54483020 Nov 26 11:27 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 11:02 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:59 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 10:59 detsim_single_protodunehd_hist.root 