Jobsub ID 258332.9@dunegpschedd01.fnal.gov
| Jobsub ID | 258332.9@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 04:53:06 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce04 |
| Last heartbeat | 2025-11-26 07:59:06 |
| From worker node | Hostname | wn123 |
| cpuinfo | AMD EPYC 7713P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 04:54:09 |
| Input files | hd-protodune:pdhd_prod_beam__243778_0_1_20251105T233546Z_gen_g4_IonScintPDExt.root_255822_60_1_20251123T211855Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 3h (10839s) |
| CPU time | 2h (7675s = 70%) |
| Max RSS bytes | 4239335424 (4042 MiB) |
| Outputting started | 2025-11-26 07:54:50 |
| Output files | |
| Finished | 2025-11-26 07:59:06 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
q=2.25539e+06 n=926731 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 7885 XUs and 8598 XVs -> 63 XUVs
C:0 T:1 43680 XUs and 56411 XVs -> 1033 XUVs
C:0 T:2 8182 XUs and 40496 XVs -> 1107 XUVs
C:0 T:3 16 XUs and 71 XVs -> 0 XUVs
C:0 T:5 2344 XUs and 4565 XVs -> 218 XUVs
C:0 T:6 19632 XUs and 51577 XVs -> 3428 XUVs
C:0 T:7 6 XUs and 196 XVs -> 0 XUVs
5849 XUVs total
1541 collection wire objects
5849 potential space points
Neighbour search...
860809 tests to find 406584 neighbours
Iterating with no regularization...
Begin: 2.77275e+07
0 2.5915e+07
1 2.57597e+07
2 2.57293e+07
3 2.57192e+07
Now with regularization...
Begin: 2.46807e+07
0 2.46691e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 04:54:43 -03 Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258332_9_1_20251126T045416Z_detsim_reco1.root"
26-Nov-2025 04:54:43 -03 Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 126.831 189.991 338.102 179.94 57.0135 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00153213 0.034035 0.209273 0.00945148 0.0611189 10
reco:opdec:Deconvolution 0.886883 1.22735 1.64332 1.26945 0.22174 10
reco:ophitspe:OpHitFinderDeco 0.180751 0.302602 0.34615 0.315577 0.0427568 10
reco:opflash:OpFlashFinder 0.0591423 0.11296 0.187669 0.11448 0.0386967 10
reco:opslicer:OpSlicer 0.220728 0.374974 0.581967 0.386613 0.116889 10
reco:wclsdatahd:WireCellToolkit 62.1044 66.9184 70.7 66.5783 2.41276 10
reco:gaushit:GausHitFinder 0.845389 1.33718 2.32497 1.23755 0.414129 10
reco:nhitsfilter:NumberOfHitsFilter 0.000253152 0.000403032 0.000643221 0.000371834 0.000122194 10
reco:reco3d:SpacePointSolver 0.410864 1.33609 2.93056 1.14027 0.756147 10
reco:hitpdune:DisambigFromSpacePoints 0.317695 1.00891 2.70969 0.871492 0.650818 10
reco:pandora:StandardPandora 9.82803 34.6413 81.9639 31.1692 19.9274 10
reco:pandoraWriter:StandardPandora 0.214833 0.571472 1.06773 0.424203 0.306312 10
reco:pandoraTrack:LArPandoraTrackCreation 1.09322 2.75484 7.76339 1.88749 1.93412 10
reco:pandoraShower:LArPandoraShowerCreation 0.998261 2.42809 6.814 1.85944 1.68455 10
reco:pandoracalo:Calorimetry 0.628265 1.35444 3.08311 1.04 0.744958 10
reco:pandoracalonosce:Calorimetry 0.565765 1.30316 2.92797 1.02108 0.733079 10
reco:pandoraShowercalo:ShowerCalorimetry 1.00892 2.29428 6.68466 1.62476 1.66411 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.988165 2.22068 6.47139 1.57635 1.57228 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0570134 0.0971315 0.171543 0.0906755 0.0323857 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0573187 0.0977025 0.172791 0.0891027 0.032536 10
reco:emtrkmichelid:EmTrackMichelId 29.355 44.3315 73.4909 42.0719 12.1344 10
reco:crttag:SingleCRTMatchingProducer 0.00477517 0.00803973 0.0163461 0.00750951 0.00318131 10
reco:crtreco:TwoCRTMatchingProducer 0.00372076 0.00702234 0.0144307 0.00637194 0.00299643 10
reco:pandora2Track:LArPandoraTrackCreation 2.63081 6.7769 21.2959 4.5208 5.29215 10
reco:pandora2calo:ShowerCalorimetry 0.901849 2.31497 7.43441 1.33881 1.87837 10
reco:pandora2calonosce:ShowerCalorimetry 0.886701 2.10896 6.10893 1.34089 1.49461 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0590256 0.0961048 0.169438 0.0852417 0.0319796 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0591406 0.0967125 0.165666 0.0880707 0.0309389 10
reco:pandora2Shower:LArPandoraShowerCreation 1.11299 2.7408 7.12862 2.14612 1.6934 10
reco:pandora2Showercalo:ShowerCalorimetry 1.22474 2.91136 8.11908 2.29758 1.91778 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.1621 2.80995 7.92671 2.23651 1.9441 10
[art]:TriggerResults:TriggerResultInserter 1.9371e-05 3.69757e-05 0.000115115 3.01315e-05 2.63439e-05 10
end_path:out1:RootOutput 4.05e-06 1.04795e-05 3.2521e-05 5.2955e-06 1.04806e-05 10
end_path:out1:RootOutput(write) 3.59587 5.24844 7.48594 5.23181 1.10874 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6267.57 MB
Peak resident set size usage (VmHWM): 4048.23 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258332_9_1_20251126T045416Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258332_9_1_20251126T045416Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258332_9_1_20251126T045416Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258332_9_1_20251126T045416Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258332_9_1_20251126T045416Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258332_9_1_20251126T045416Z_detsim_reco1.root.temp.json ====
.:
total 5610240
-rw-r--r-- 1 nobody nobody 41318 Nov 26 04:54 jobscript.log
-rw-r--r-- 1 nobody nobody 202 Nov 26 04:54 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody 0 Nov 26 04:54 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258332_9_1_20251126T045416Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody 0 Nov 26 04:54 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258332_9_1_20251126T045416Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody 31 Nov 26 04:54 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258332_9_1_20251126T045416Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody 25 Nov 26 04:54 all-input-dids.txt
-rw-r--r-- 1 nobody nobody 519 Nov 26 04:54 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody 3574 Nov 26 04:54 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody 200704 Nov 26 04:54 mem.db
-rw-r--r-- 1 nobody nobody 36864 Nov 26 04:54 time.db
-rw-r--r-- 1 nobody nobody 5697132119 Nov 26 04:54 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258332_9_1_20251126T045416Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody 47440710 Nov 26 04:53 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody 0 Nov 26 04:21 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody 0 Nov 26 04:18 debugprod.log
-rw-r--r-- 1 nobody nobody 519 Nov 26 04:18 detsim_single_protodunehd_hist.root 