Jobsub ID 258430.142@dunegpschedd01.fnal.gov
| Jobsub ID | 258430.142@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 07:19:14 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 09:44:40 |
| From worker node | Hostname | ccwcondor0581 |
| cpuinfo | AMD EPYC 7453 28-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 07:21:29 |
| Input files | hd-protodune:pdhd_prod_beam__244195_119_1_20251106T094538Z_gen_g4_IonScintPDExt.root_249776_105_1_20251122T195734Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (8554s) |
| CPU time | 2h (7751s = 90%) |
| Max RSS bytes | 8148566016 (7771 MiB) |
| Outputting started | 2025-11-26 09:44:05 |
| Output files | |
| Finished | 2025-11-26 09:44:40 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
89143 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 21256 XUs and 20076 XVs -> 270 XUVs
C:0 T:1 140103 XUs and 153543 XVs -> 5279 XUVs
C:0 T:2 45136 XUs and 85783 XVs -> 6753 XUVs
C:0 T:3 7 XUs and 19 XVs -> 0 XUVs
C:0 T:4 183 XUs and 36 XVs -> 4 XUVs
C:0 T:5 44318 XUs and 33954 XVs -> 898 XUVs
C:0 T:6 58181 XUs and 59491 XVs -> 4034 XUVs
17238 XUVs total
4919 collection wire objects
17238 potential space points
Neighbour search...
1410906 tests to find 751670 neighbours
Iterating with no regularization...
Begin: 7.82153e+07
0 6.87513e+07
1 6.80007e+07
2 6.78906e+07
3 6.7854e+07
Now with regularization...
Begin: 6.43649e+07
0 6.43387e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 10:43:56 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_142_1_20251126T072138Z_detsim_reco1.root"
26-Nov-2025 10:43:56 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 160.753 237.944 348.234 247.919 56.7857 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00132408 0.0140911 0.0302053 0.0117593 0.00965441 10
reco:opdec:Deconvolution 0.894318 1.11048 1.29041 1.10569 0.134413 10
reco:ophitspe:OpHitFinderDeco 0.240063 0.264329 0.327818 0.260544 0.0242323 10
reco:opflash:OpFlashFinder 0.0979331 0.13201 0.178935 0.135621 0.0266971 10
reco:opslicer:OpSlicer 0.281969 0.368318 0.44941 0.363664 0.0507124 10
reco:wclsdatahd:WireCellToolkit 51.8146 55.3079 59.5983 54.6582 2.48869 10
reco:gaushit:GausHitFinder 0.95612 1.26026 1.56735 1.29532 0.200132 10
reco:nhitsfilter:NumberOfHitsFilter 0.000354746 0.000515901 0.000880313 0.000470238 0.00014725 10
reco:reco3d:SpacePointSolver 0.951903 2.01666 3.39475 1.92405 0.902363 10
reco:hitpdune:DisambigFromSpacePoints 0.628635 1.42101 2.52438 1.49301 0.550962 10
reco:pandora:StandardPandora 19.4822 49.2171 86.7908 43.8648 20.1037 10
reco:pandoraWriter:StandardPandora 0.332779 0.433381 0.606639 0.431053 0.0768564 10
reco:pandoraTrack:LArPandoraTrackCreation 3.18519 5.87188 10.1387 5.6953 1.86685 10
reco:pandoraShower:LArPandoraShowerCreation 3.62036 6.49176 10.292 6.48882 1.86848 10
reco:pandoracalo:Calorimetry 1.79212 3.24838 6.05158 3.1956 1.18159 10
reco:pandoracalonosce:Calorimetry 1.718 3.12818 6.2705 2.82815 1.26802 10
reco:pandoraShowercalo:ShowerCalorimetry 2.94262 6.51027 13.1929 6.16722 2.86364 10
reco:pandoraShowercalonosce:ShowerCalorimetry 2.83841 6.13097 11.6549 5.53553 2.69713 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0621856 0.0988741 0.146172 0.0933357 0.0240435 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0636542 0.0949457 0.143607 0.0887065 0.0225329 10
reco:emtrkmichelid:EmTrackMichelId 30.6434 46.3905 60.4058 48.599 9.90749 10
reco:crttag:SingleCRTMatchingProducer 0.0103533 0.0140251 0.0190764 0.0133557 0.00285283 10
reco:crtreco:TwoCRTMatchingProducer 0.00960248 0.0126212 0.0179138 0.0113621 0.00252067 10
reco:pandora2Track:LArPandoraTrackCreation 7.56188 14.0003 24.6469 13.4712 5.09296 10
reco:pandora2calo:ShowerCalorimetry 2.30222 5.2554 9.23823 4.88768 2.23327 10
reco:pandora2calonosce:ShowerCalorimetry 2.41691 5.4835 9.50804 5.54672 2.28414 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0644384 0.0923117 0.121175 0.0857416 0.0197708 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0644747 0.0897754 0.117111 0.0851411 0.0181296 10
reco:pandora2Shower:LArPandoraShowerCreation 3.22868 6.50033 10.5291 6.21924 2.19612 10
reco:pandora2Showercalo:ShowerCalorimetry 2.85618 6.71875 11.0417 6.29492 2.74838 10
reco:pandora2Showercalonosce:ShowerCalorimetry 3.0635 6.94098 11.636 6.89979 3.13777 10
[art]:TriggerResults:TriggerResultInserter 3.5046e-05 6.22027e-05 0.00010667 4.8977e-05 2.59759e-05 10
end_path:out1:RootOutput 4.097e-06 1.20044e-05 2.9736e-05 1.0054e-05 7.04743e-06 10
end_path:out1:RootOutput(write) 2.43671 3.07931 4.02689 2.78514 0.568147 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6543.63 MB
Peak resident set size usage (VmHWM): 4324.32 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_142_1_20251126T072138Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_142_1_20251126T072138Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_142_1_20251126T072138Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_142_1_20251126T072138Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_142_1_20251126T072138Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_142_1_20251126T072138Z_detsim_reco1.root.temp.json ====
.:
total 6914428
-rw-r--r-- 1 dune001 lbno 42205 Nov 26 10:44 jobscript.log
-rw-r--r-- 1 dune001 lbno 205 Nov 26 10:44 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:44 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_142_1_20251126T072138Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:44 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_142_1_20251126T072138Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 10:44 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_142_1_20251126T072138Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 10:43 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 10:43 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 10:43 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 10:43 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 10:43 time.db
-rw-r--r-- 1 dune001 lbno 7017864906 Nov 26 10:43 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_142_1_20251126T072138Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 62195560 Nov 26 10:40 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:06 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:02 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 10:02 detsim_single_protodunehd_hist.root 