Jobsub ID 252698.20@dunegpschedd02.fnal.gov
| Jobsub ID | 252698.20@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 05:35:07 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce04 |
| Last heartbeat | 2025-11-26 09:06:40 |
| From worker node | Hostname | wn113 |
| cpuinfo | AMD EPYC 7713P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 05:36:37 |
| Input files | hd-protodune:pdhd_prod_beam__238442_108_1_20251106T085408Z_gen_g4_IonScintPDExt.root_255521_84_1_20251122T220524Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 3h (12345s) |
| CPU time | 2h (9151s = 74%) |
| Max RSS bytes | 4192681984 (3998 MiB) |
| Outputting started | 2025-11-26 09:02:24 |
| Output files | |
| Finished | 2025-11-26 09:06:40 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
98 tag=gauss
wclsFrameSaver: saving 115401 traces tagged "wiener"
FrameSaver: q=3.83338e+06 n=1438550 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 18894 XUs and 12898 XVs -> 180 XUVs
C:0 T:1 128547 XUs and 98446 XVs -> 3035 XUVs
C:0 T:2 74798 XUs and 75630 XVs -> 1737 XUVs
C:0 T:3 19 XUs and 13 XVs -> 1 XUVs
C:0 T:5 23391 XUs and 44165 XVs -> 653 XUVs
C:0 T:6 34557 XUs and 39141 XVs -> 1085 XUVs
C:0 T:7 6 XUs and 6 XVs -> 0 XUVs
6691 XUVs total
3552 collection wire objects
6691 potential space points
Neighbour search...
172439 tests to find 94004 neighbours
Iterating with no regularization...
Begin: 5.51275e+07
0 5.07793e+07
1 5.05475e+07
2 5.0526e+07
Now with regularization...
Begin: 4.87876e+07
0 4.87804e+07
26-Nov-2025 06:02:13 -03 Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252698_20_1_20251126T053645Z_detsim_reco1.root"
26-Nov-2025 06:02:13 -03 Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 164.887 236.356 306.95 244.231 41.1405 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00156391 0.0107933 0.0187907 0.0117129 0.00578324 10
reco:opdec:Deconvolution 1.22931 1.45518 1.61052 1.47206 0.135346 10
reco:ophitspe:OpHitFinderDeco 0.297189 0.320046 0.335584 0.320517 0.0102169 10
reco:opflash:OpFlashFinder 0.108748 0.156415 0.195835 0.155668 0.0256749 10
reco:opslicer:OpSlicer 0.33607 0.475541 0.575161 0.471194 0.0662543 10
reco:wclsdatahd:WireCellToolkit 63.4257 67.0755 69.4997 67.4023 1.61939 10
reco:gaushit:GausHitFinder 1.28016 1.66136 1.95532 1.66311 0.19678 10
reco:nhitsfilter:NumberOfHitsFilter 0.000556748 0.000751375 0.00105826 0.000722519 0.000164077 10
reco:reco3d:SpacePointSolver 1.25004 1.74567 2.25229 1.67094 0.315169 10
reco:hitpdune:DisambigFromSpacePoints 0.768108 1.60021 2.53815 1.59793 0.430021 10
reco:pandora:StandardPandora 30.3734 53.5551 87.9683 52.0714 18.5945 10
reco:pandoraWriter:StandardPandora 0.351482 0.569149 1.28042 0.509314 0.249772 10
reco:pandoraTrack:LArPandoraTrackCreation 2.349 4.03359 5.92211 4.01111 0.998549 10
reco:pandoraShower:LArPandoraShowerCreation 1.3246 3.70243 5.6482 3.98594 1.1416 10
reco:pandoracalo:Calorimetry 1.26487 2.20802 3.06864 2.16863 0.555404 10
reco:pandoracalonosce:Calorimetry 1.27842 2.19128 3.16232 2.20412 0.56587 10
reco:pandoraShowercalo:ShowerCalorimetry 1.53795 4.1852 6.32151 4.28706 1.49853 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.44276 3.90469 5.96031 4.09293 1.36066 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0733231 0.115693 0.137552 0.120201 0.0188206 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0723092 0.114618 0.135506 0.12255 0.019272 10
reco:emtrkmichelid:EmTrackMichelId 36.9627 51.8944 60.2961 52.026 6.76008 10
reco:crttag:SingleCRTMatchingProducer 0.00629074 0.00897781 0.011622 0.00899661 0.00182893 10
reco:crtreco:TwoCRTMatchingProducer 0.00558745 0.00748449 0.00975162 0.00705951 0.00149231 10
reco:pandora2Track:LArPandoraTrackCreation 4.44276 9.35817 14.3553 10.0886 2.83139 10
reco:pandora2calo:ShowerCalorimetry 1.30489 3.43106 6.32266 3.80899 1.36174 10
reco:pandora2calonosce:ShowerCalorimetry 1.28889 3.38963 5.90879 3.74262 1.26287 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0586353 0.106151 0.126984 0.115819 0.0222806 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0596231 0.107565 0.131073 0.116721 0.0223239 10
reco:pandora2Shower:LArPandoraShowerCreation 1.9785 4.13838 6.2488 4.56674 1.17767 10
reco:pandora2Showercalo:ShowerCalorimetry 2.03854 4.7328 7.22402 4.74669 1.51843 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.99626 4.76177 7.12699 5.02989 1.53755 10
[art]:TriggerResults:TriggerResultInserter 2.6611e-05 3.88362e-05 9.9843e-05 3.1746e-05 2.05507e-05 10
end_path:out1:RootOutput 5.241e-06 3.44502e-05 0.000269709 6.15e-06 7.86752e-05 10
end_path:out1:RootOutput(write) 3.55319 5.1684 6.99931 5.06078 0.927648 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6407.97 MB
Peak resident set size usage (VmHWM): 4123.07 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252698_20_1_20251126T053645Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252698_20_1_20251126T053645Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252698_20_1_20251126T053645Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252698_20_1_20251126T053645Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252698_20_1_20251126T053645Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252698_20_1_20251126T053645Z_detsim_reco1.root.temp.json ====
.:
total 6634524
-rw-r--r-- 1 nobody nobody 41331 Nov 26 06:02 jobscript.log
-rw-r--r-- 1 nobody nobody 204 Nov 26 06:02 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody 0 Nov 26 06:02 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252698_20_1_20251126T053645Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody 0 Nov 26 06:02 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252698_20_1_20251126T053645Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody 31 Nov 26 06:02 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252698_20_1_20251126T053645Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody 25 Nov 26 06:02 all-input-dids.txt
-rw-r--r-- 1 nobody nobody 519 Nov 26 06:02 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody 3574 Nov 26 06:02 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody 204800 Nov 26 06:02 mem.db
-rw-r--r-- 1 nobody nobody 36864 Nov 26 06:02 time.db
-rw-r--r-- 1 nobody nobody 6733127038 Nov 26 06:02 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252698_20_1_20251126T053645Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody 60311810 Nov 26 05:59 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody 0 Nov 26 05:24 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody 0 Nov 26 05:19 debugprod.log
-rw-r--r-- 1 nobody nobody 519 Nov 26 05:19 detsim_single_protodunehd_hist.root 