Jobsub ID 258506.11@dunegpschedd01.fnal.gov
| Jobsub ID | 258506.11@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 09:03:17 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce02 |
| Last heartbeat | 2025-11-26 10:29:32 |
| From worker node | Hostname | ccwcondor0371 |
| cpuinfo | AMD EPYC 9334 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 09:04:19 |
| Input files | hd-protodune:pdhd_prod_beam__244014_36_1_20251106T075542Z_gen_g4_IonScintPDExt.root_255457_105_1_20251122T200605Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (5088s) |
| CPU time | 1h (4924s = 96%) |
| Max RSS bytes | 6918221824 (6597 MiB) |
| Outputting started | 2025-11-26 10:29:08 |
| Output files | |
| Finished | 2025-11-26 10:29:32 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
000 ticks
wclsFrameSaver: saving 68031 traces tagged "gauss"
FrameSaver: q=1.86017e+06 n=1024699 tag=gauss
wclsFrameSaver: saving 82162 traces tagged "wiener"
FrameSaver: q=2.30044e+06 n=986959 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 13379 XUs and 10814 XVs -> 79 XUVs
C:0 T:1 84473 XUs and 73639 XVs -> 2762 XUVs
C:0 T:2 5541 XUs and 6658 XVs -> 298 XUVs
C:0 T:5 27729 XUs and 23267 XVs -> 590 XUVs
C:0 T:6 11469 XUs and 9018 XVs -> 503 XUVs
4232 XUVs total
2132 collection wire objects
4232 potential space points
Neighbour search...
85340 tests to find 42008 neighbours
Iterating with no regularization...
Begin: 1.97327e+07
0 1.84191e+07
1 1.83407e+07
2 1.83347e+07
Now with regularization...
Begin: 1.72224e+07
0 1.72207e+07
26-Nov-2025 11:29:03 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258506_11_1_20251126T090424Z_detsim_reco1.root"
26-Nov-2025 11:29:03 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 78.8152 108.383 143.878 105.932 18.2177 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00109097 0.00972627 0.0358356 0.00757077 0.00917565 10
reco:opdec:Deconvolution 0.496908 0.743179 0.882783 0.761923 0.11194 10
reco:ophitspe:OpHitFinderDeco 0.111741 0.154876 0.202224 0.1631 0.0350649 10
reco:opflash:OpFlashFinder 0.0427947 0.0863492 0.109148 0.0888135 0.0200447 10
reco:opslicer:OpSlicer 0.140777 0.231921 0.279368 0.232382 0.0407636 10
reco:wclsdatahd:WireCellToolkit 39.5371 42.2424 47.6908 41.2332 2.71633 10
reco:gaushit:GausHitFinder 0.469918 0.700655 1.08052 0.700844 0.190184 10
reco:nhitsfilter:NumberOfHitsFilter 0.000221736 0.000377329 0.000772429 0.000350926 0.000148557 10
reco:reco3d:SpacePointSolver 0.257271 1.18173 3.41127 0.81535 0.946933 10
reco:hitpdune:DisambigFromSpacePoints 0.373652 0.601845 1.07206 0.516145 0.212088 10
reco:pandora:StandardPandora 11.4598 18.6411 32.3029 16.812 5.54215 10
reco:pandoraWriter:StandardPandora 0.186633 0.273139 0.355642 0.279037 0.0513023 10
reco:pandoraTrack:LArPandoraTrackCreation 0.975215 2.00169 3.29543 1.93623 0.639252 10
reco:pandoraShower:LArPandoraShowerCreation 0.634945 1.70471 3.10106 1.78428 0.753057 10
reco:pandoracalo:Calorimetry 0.427828 0.99954 1.96936 0.939005 0.427615 10
reco:pandoracalonosce:Calorimetry 0.463461 0.997943 2.13207 0.942467 0.433534 10
reco:pandoraShowercalo:ShowerCalorimetry 0.595184 1.69519 3.38419 1.49044 0.822409 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.732984 1.66019 3.70191 1.49272 0.875918 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.02979 0.0487306 0.0716967 0.0476568 0.0119525 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0291555 0.0464321 0.0701585 0.0457667 0.0112918 10
reco:emtrkmichelid:EmTrackMichelId 12.6306 18.369 22.4009 18.684 3.14227 10
reco:crttag:SingleCRTMatchingProducer 0.00293698 0.00792724 0.012726 0.00731521 0.00257049 10
reco:crtreco:TwoCRTMatchingProducer 0.00208132 0.00673165 0.0122225 0.005898 0.00274488 10
reco:pandora2Track:LArPandoraTrackCreation 1.83425 4.85308 8.27632 4.16189 2.0694 10
reco:pandora2calo:ShowerCalorimetry 0.496406 1.35607 2.50701 1.23408 0.625837 10
reco:pandora2calonosce:ShowerCalorimetry 0.494194 1.46459 2.47286 1.46584 0.652979 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0238641 0.0528122 0.0812876 0.0533357 0.0166279 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0215149 0.0487304 0.0734521 0.0500855 0.0152623 10
reco:pandora2Shower:LArPandoraShowerCreation 0.725453 1.93763 3.47655 1.72324 0.886158 10
reco:pandora2Showercalo:ShowerCalorimetry 0.975859 2.08011 3.46372 1.63812 0.925718 10
reco:pandora2Showercalonosce:ShowerCalorimetry 0.984889 1.96503 3.29083 1.97828 0.795405 10
[art]:TriggerResults:TriggerResultInserter 2.5188e-05 4.40116e-05 8.1904e-05 3.95155e-05 1.45181e-05 10
end_path:out1:RootOutput 6.069e-06 9.5935e-06 2.2714e-05 7.8665e-06 4.52796e-06 10
end_path:out1:RootOutput(write) 1.5963 2.0537 2.57731 1.99332 0.338235 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6707.68 MB
Peak resident set size usage (VmHWM): 4201.48 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258506_11_1_20251126T090424Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258506_11_1_20251126T090424Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258506_11_1_20251126T090424Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258506_11_1_20251126T090424Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258506_11_1_20251126T090424Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258506_11_1_20251126T090424Z_detsim_reco1.root.temp.json ====
.:
total 6084204
-rw-r--r-- 1 dune001 lbno 40290 Nov 26 11:29 jobscript.log
-rw-r--r-- 1 dune001 lbno 188 Nov 26 11:29 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 11:29 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258506_11_1_20251126T090424Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 11:29 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258506_11_1_20251126T090424Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 11:29 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258506_11_1_20251126T090424Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 11:29 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 11:29 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 11:29 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 11:29 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 11:29 time.db
-rw-r--r-- 1 dune001 lbno 6178945688 Nov 26 11:29 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258506_11_1_20251126T090424Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 50968850 Nov 26 11:28 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 11:11 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 11:09 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 11:09 detsim_single_protodunehd_hist.root 