Jobsub ID 252489.151@dunegpschedd02.fnal.gov
| Jobsub ID | 252489.151@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 02:27:59 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-11-26 05:06:41 |
| From worker node | Hostname | n179.dur.scotgrid.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 02:30:28 |
| Input files | hd-protodune:pdhd_prod_beam__243931_74_1_20251106T051037Z_gen_g4_IonScintPDExt.root_255792_91_1_20251123T185914Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (9349s) |
| CPU time | 1h (5277s = 56%) |
| Max RSS bytes | 6743134208 (6430 MiB) |
| Outputting started | 2025-11-26 05:06:18 |
| Output files | |
| Finished | 2025-11-26 05:06:41 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 13147 XUs and 8887 XVs -> 76 XUVs
C:0 T:1 80640 XUs and 56162 XVs -> 1751 XUVs
C:0 T:2 39696 XUs and 15357 XVs -> 337 XUVs
C:0 T:3 36 XUs and 9 XVs -> 0 XUVs
C:0 T:4 5 XUs and 7 XVs -> 1 XUVs
C:0 T:5 38002 XUs and 28746 XVs -> 1075 XUVs
C:0 T:6 22436 XUs and 25743 XVs -> 943 XUVs
C:0 T:7 24 XUs and 27 XVs -> 1 XUVs
4184 XUVs total
2316 collection wire objects
4184 potential space points
Neighbour search...
80870 tests to find 40800 neighbours
Iterating with no regularization...
Begin: 2.73759e+07
0 2.55365e+07
1 2.54116e+07
2 2.54037e+07
Now with regularization...
Begin: 2.37426e+07
0 2.37391e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 05:06:14 GMT Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_151_1_20251126T023034Z_detsim_reco1.root"
26-Nov-2025 05:06:14 GMT Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 127.551 178.305 283.26 178.493 39.084 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0010968 0.0104932 0.034961 0.00874408 0.00875558 10
reco:opdec:Deconvolution 0.975089 1.20668 1.59102 1.17336 0.16587 10
reco:ophitspe:OpHitFinderDeco 0.234666 0.249722 0.261728 0.250996 0.00846522 10
reco:opflash:OpFlashFinder 0.0873322 0.125651 0.197872 0.120986 0.0306193 10
reco:opslicer:OpSlicer 0.215789 0.301171 0.418781 0.293159 0.0609679 10
reco:wclsdatahd:WireCellToolkit 63.6226 66.9238 71.9203 66.6128 2.00079 10
reco:gaushit:GausHitFinder 0.750198 1.01357 1.45404 0.980358 0.174634 10
reco:nhitsfilter:NumberOfHitsFilter 0.000359693 0.000488308 0.000637127 0.000469749 7.63008e-05 10
reco:reco3d:SpacePointSolver 0.575017 1.33091 3.19367 1.10369 0.733295 10
reco:hitpdune:DisambigFromSpacePoints 0.458484 1.00079 2.19317 0.93131 0.429922 10
reco:pandora:StandardPandora 16.5854 30.6992 47.2971 31.0271 9.20492 10
reco:pandoraWriter:StandardPandora 0.251697 0.337476 0.472975 0.338239 0.0540615 10
reco:pandoraTrack:LArPandoraTrackCreation 2.32518 4.20284 8.16365 3.85734 1.60683 10
reco:pandoraShower:LArPandoraShowerCreation 1.64736 4.45982 11.0858 3.79959 2.38738 10
reco:pandoracalo:Calorimetry 0.928693 1.69936 3.92046 1.43394 0.810863 10
reco:pandoracalonosce:Calorimetry 0.903944 1.73144 3.66143 1.60884 0.732606 10
reco:pandoraShowercalo:ShowerCalorimetry 1.22102 3.55855 9.01608 2.9114 2.04716 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.25031 3.32516 8.80775 2.59258 1.98474 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0491461 0.0687536 0.109822 0.0650883 0.0160422 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0485151 0.0680613 0.109147 0.0640898 0.0157617 10
reco:emtrkmichelid:EmTrackMichelId 18.8613 25.0556 35.2477 25.2699 4.01741 10
reco:crttag:SingleCRTMatchingProducer 0.00909291 0.012909 0.0201103 0.012199 0.00323853 10
reco:crtreco:TwoCRTMatchingProducer 0.00550297 0.0108414 0.0177766 0.0103023 0.00354547 10
reco:pandora2Track:LArPandoraTrackCreation 4.6605 10.589 25.2251 9.26378 5.29121 10
reco:pandora2calo:ShowerCalorimetry 1.19322 2.97769 8.18809 2.48713 1.85318 10
reco:pandora2calonosce:ShowerCalorimetry 1.04773 2.81106 7.26697 2.37299 1.61522 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0484717 0.0670417 0.10499 0.0649876 0.0146882 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.047958 0.0666927 0.1047 0.0634067 0.014561 10
reco:pandora2Shower:LArPandoraShowerCreation 2.01462 4.31723 10.2853 4.03519 2.15419 10
reco:pandora2Showercalo:ShowerCalorimetry 1.52819 3.7311 9.62711 3.17198 2.1335 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.54446 3.63315 9.41891 2.95751 2.08907 10
[art]:TriggerResults:TriggerResultInserter 2.5247e-05 3.39756e-05 7.0548e-05 2.89975e-05 1.26493e-05 10
end_path:out1:RootOutput 5.577e-06 8.2464e-06 2.2512e-05 6.517e-06 4.81894e-06 10
end_path:out1:RootOutput(write) 1.96454 2.57972 3.67566 2.50857 0.411077 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 7076.69 MB
Peak resident set size usage (VmHWM): 4634.39 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_151_1_20251126T023034Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_151_1_20251126T023034Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_151_1_20251126T023034Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_151_1_20251126T023034Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_151_1_20251126T023034Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_151_1_20251126T023034Z_detsim_reco1.root.temp.json ====
.:
total 6250572
-rw-r--r--. 1 dune004 dune 41040 Nov 26 05:06 jobscript.log
-rw-r--r--. 1 dune004 dune 203 Nov 26 05:06 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 0 Nov 26 05:06 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_151_1_20251126T023034Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune 0 Nov 26 05:06 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_151_1_20251126T023034Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune 31 Nov 26 05:06 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_151_1_20251126T023034Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune 25 Nov 26 05:06 all-input-dids.txt
-rw-r--r--. 1 dune004 dune 519 Nov 26 05:06 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune 3574 Nov 26 05:06 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune 204800 Nov 26 05:06 mem.db
-rw-r--r--. 1 dune004 dune 36864 Nov 26 05:06 time.db
-rw-r--r--. 1 dune004 dune 6346469747 Nov 26 05:06 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_151_1_20251126T023034Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune 53782540 Nov 26 05:04 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune 0 Nov 26 04:37 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune 0 Nov 26 04:34 debugprod.log
-rw-r--r--. 1 dune004 dune 519 Nov 26 04:34 detsim_single_protodunehd_hist.root 