Jobsub ID 252489.147@dunegpschedd02.fnal.gov
| Jobsub ID | 252489.147@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 02:27:59 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-11-26 04:50:27 |
| From worker node | Hostname | n179.dur.scotgrid.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 02:30:41 |
| Input files | hd-protodune:pdhd_prod_beam__238287_168_1_20251106T062911Z_gen_g4_IonScintPDExt.root_255533_44_1_20251122T223939Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (8361s) |
| CPU time | 1h (4922s = 58%) |
| Max RSS bytes | 5887266816 (5614 MiB) |
| Outputting started | 2025-11-26 04:50:03 |
| Output files | |
| Finished | 2025-11-26 04:50:27 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
3 traces tagged "wiener"
FrameSaver: q=2.57289e+06 n=1016912 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9718 XUs and 8706 XVs -> 59 XUVs
C:0 T:1 59690 XUs and 50320 XVs -> 1612 XUVs
C:0 T:2 12269 XUs and 26746 XVs -> 1295 XUVs
C:0 T:3 20 XUs and 39 XVs -> 5 XUVs
C:0 T:4 17 XUs and 12 XVs -> 0 XUVs
C:0 T:5 11740 XUs and 10009 XVs -> 461 XUVs
C:0 T:6 12973 XUs and 27546 XVs -> 731 XUVs
C:0 T:7 2 XUs and 6 XVs -> 0 XUVs
4163 XUVs total
2221 collection wire objects
4163 potential space points
Neighbour search...
100637 tests to find 49432 neighbours
Iterating with no regularization...
Begin: 4.09684e+07
0 3.89055e+07
1 3.87592e+07
2 3.87413e+07
Now with regularization...
Begin: 3.67593e+07
0 3.67545e+07
26-Nov-2025 04:49:55 GMT Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_147_1_20251126T023044Z_detsim_reco1.root"
26-Nov-2025 04:49:55 GMT Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 117.756 169.759 243.128 167.907 34.8353 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00113942 0.0074261 0.0155381 0.00804284 0.00417421 10
reco:opdec:Deconvolution 0.898451 1.09746 1.28655 1.09666 0.131616 10
reco:ophitspe:OpHitFinderDeco 0.228217 0.238855 0.253923 0.237357 0.00777024 10
reco:opflash:OpFlashFinder 0.0529364 0.10182 0.147041 0.108984 0.0268776 10
reco:opslicer:OpSlicer 0.208164 0.261424 0.323237 0.25435 0.0418608 10
reco:wclsdatahd:WireCellToolkit 64.5893 68.9534 72.4911 68.401 2.71657 10
reco:gaushit:GausHitFinder 0.713272 0.976768 1.21286 0.97105 0.178614 10
reco:nhitsfilter:NumberOfHitsFilter 0.000353776 0.000474274 0.00068575 0.000453098 9.25084e-05 10
reco:reco3d:SpacePointSolver 0.409722 1.13633 2.40611 1.09254 0.536592 10
reco:hitpdune:DisambigFromSpacePoints 0.473686 0.864393 1.57286 0.86505 0.332235 10
reco:pandora:StandardPandora 10.9662 29.1677 58.671 27.0191 12.9553 10
reco:pandoraWriter:StandardPandora 0.240509 0.321361 0.409654 0.327348 0.0486658 10
reco:pandoraTrack:LArPandoraTrackCreation 2.01642 3.8959 6.01146 4.00039 1.25808 10
reco:pandoraShower:LArPandoraShowerCreation 1.18362 3.7647 6.61289 3.23577 1.83135 10
reco:pandoracalo:Calorimetry 0.91717 1.64228 2.76054 1.54664 0.505889 10
reco:pandoracalonosce:Calorimetry 0.907926 1.50877 2.41096 1.39709 0.476023 10
reco:pandoraShowercalo:ShowerCalorimetry 1.13457 2.75599 5.346 2.54499 1.21993 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.57904 2.75596 5.36978 2.36607 1.19129 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0467162 0.066947 0.0915928 0.0656564 0.0140128 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0477601 0.0668655 0.0902987 0.0669003 0.0135457 10
reco:emtrkmichelid:EmTrackMichelId 17.2689 24.1815 30.3786 24.4415 4.07093 10
reco:crttag:SingleCRTMatchingProducer 0.00763896 0.0123123 0.0151355 0.0126856 0.00259997 10
reco:crtreco:TwoCRTMatchingProducer 0.00550437 0.0107524 0.0141068 0.0109249 0.00287573 10
reco:pandora2Track:LArPandoraTrackCreation 4.23028 8.95738 15.9496 7.54238 3.84031 10
reco:pandora2calo:ShowerCalorimetry 0.953617 2.39399 4.70297 1.9129 1.20003 10
reco:pandora2calonosce:ShowerCalorimetry 0.842193 2.29356 4.30332 2.0028 1.1025 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0480205 0.0639078 0.0847079 0.0633238 0.0118646 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.04797 0.0644404 0.0854016 0.0636372 0.0117751 10
reco:pandora2Shower:LArPandoraShowerCreation 1.46175 3.54672 6.28442 3.02981 1.63292 10
reco:pandora2Showercalo:ShowerCalorimetry 1.40987 3.09284 5.56998 2.76221 1.31327 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.24548 2.97152 5.60953 2.51801 1.34575 10
[art]:TriggerResults:TriggerResultInserter 3.0223e-05 3.71528e-05 7.3453e-05 3.28505e-05 1.23845e-05 10
end_path:out1:RootOutput 5.488e-06 7.9194e-06 2.2167e-05 6.4245e-06 4.77541e-06 10
end_path:out1:RootOutput(write) 1.90804 2.44186 3.05641 2.41038 0.364878 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6698.4 MB
Peak resident set size usage (VmHWM): 4181.5 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_147_1_20251126T023044Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_147_1_20251126T023044Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_147_1_20251126T023044Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_147_1_20251126T023044Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_147_1_20251126T023044Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_147_1_20251126T023044Z_detsim_reco1.root.temp.json ====
.:
total 5701260
-rw-r--r--. 1 dune004 dune 40936 Nov 26 04:50 jobscript.log
-rw-r--r--. 1 dune004 dune 204 Nov 26 04:50 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 0 Nov 26 04:50 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_147_1_20251126T023044Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune 0 Nov 26 04:50 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_147_1_20251126T023044Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune 31 Nov 26 04:50 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_147_1_20251126T023044Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune 25 Nov 26 04:49 all-input-dids.txt
-rw-r--r--. 1 dune004 dune 519 Nov 26 04:49 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune 3574 Nov 26 04:49 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune 204800 Nov 26 04:49 mem.db
-rw-r--r--. 1 dune004 dune 36864 Nov 26 04:49 time.db
-rw-r--r--. 1 dune004 dune 5786954996 Nov 26 04:49 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_147_1_20251126T023044Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune 50809130 Nov 26 04:49 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune 0 Nov 26 04:22 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune 0 Nov 26 04:19 debugprod.log
-rw-r--r--. 1 dune004 dune 519 Nov 26 04:19 detsim_single_protodunehd_hist.root 