justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258684.5@dunegpschedd01.fnal.gov

Jobsub ID258684.5@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 11:40:25
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce3
Last heartbeat2025-11-26 13:08:48
From worker nodeHostnamen187.dur.scotgrid.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 11:41:30
Input fileshd-protodune:pdhd_prod_beam__238093_65_1_20251105T235618Z_gen_g4_IonScintPDExt.root_250348_35_1_20251124T034512Z_PDInt.root
JobscriptExit code0
Real time1h (5212s)
CPU time1h (4784s = 91%)
Max RSS bytes6341484544 (6047 MiB)
Outputting started2025-11-26 13:08:23
Output files
Finished2025-11-26 13:08:48
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

saving 96010 traces tagged "wiener"
FrameSaver: q=2.84147e+06 n=1179491 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11410 XUs and 10983 XVs -> 96 XUVs
C:0 T:1 63321 XUs and 51989 XVs -> 1671 XUVs
C:0 T:2 44562 XUs and 40222 XVs -> 2687 XUVs
C:0 T:3 6 XUs and 6 XVs -> 0 XUVs
C:0 T:5 21889 XUs and 32676 XVs -> 430 XUVs
C:0 T:6 56460 XUs and 31751 XVs -> 620 XUVs
5504 XUVs total
2137 collection wire objects
5504 potential space points
Neighbour search...
215978 tests to find 117434 neighbours
Iterating with no regularization...
Begin: 2.81076e+07
0 2.61537e+07
1 2.59695e+07
2 2.59474e+07
Now with regularization...
Begin: 2.48682e+07
0 2.4859e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 13:08:19 GMT  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258684_5_1_20251126T114133Z_detsim_reco1.root"
26-Nov-2025 13:08:19 GMT  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            120.807       155.045       193.721       150.545       19.5865        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000936706   0.00870976     0.017177     0.00890195    0.00441648       10     
reco:opdec:Deconvolution                             0.919913       1.15494       1.27822       1.17827      0.0984915       10     
reco:ophitspe:OpHitFinderDeco                        0.225922      0.239051      0.247554      0.239163     0.00585764       10     
reco:opflash:OpFlashFinder                           0.0702436      0.11225       0.14461       0.10706      0.0214135       10     
reco:opslicer:OpSlicer                               0.194398      0.271579      0.360256      0.271494      0.0410254       10     
reco:wclsdatahd:WireCellToolkit                       60.9433       65.5961        70.96        64.6131       3.79706        10     
reco:gaushit:GausHitFinder                           0.746603      0.942548       1.08422      0.951477      0.106536        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000364288   0.00050284    0.000596882   0.000524186   6.71789e-05      10     
reco:reco3d:SpacePointSolver                         0.425557       1.14763       1.84868       1.11927      0.449749        10     
reco:hitpdune:DisambigFromSpacePoints                0.473182      0.917101       1.86091      0.851326      0.371787        10     
reco:pandora:StandardPandora                          16.841        27.3435       44.2199       23.7923       8.62221        10     
reco:pandoraWriter:StandardPandora                   0.226879      0.322213      0.380327      0.337268      0.042461        10     
reco:pandoraTrack:LArPandoraTrackCreation             1.9402        3.09187       4.54706       2.74721      0.826831        10     
reco:pandoraShower:LArPandoraShowerCreation           1.23703       2.56852       4.75616       2.35492      0.980608        10     
reco:pandoracalo:Calorimetry                         0.839515       1.27896       1.86642       1.14618      0.361398        10     
reco:pandoracalonosce:Calorimetry                    0.803449       1.21556       1.8216        1.12429      0.324297        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.1148        2.21079       4.25804       2.13441      0.846617        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.13287       2.09978       3.8721        1.96896      0.794228        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0413173     0.0601836     0.0732125     0.061399     0.00857198       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.043692      0.0613619     0.0715148     0.0613863    0.00794353       10     
reco:emtrkmichelid:EmTrackMichelId                    18.7302       23.9673       28.3836       24.0221       2.52205        10     
reco:crttag:SingleCRTMatchingProducer               0.00848481     0.0114754     0.0128377     0.0119733    0.00133138       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00629194    0.00875875     0.0113489    0.00907078    0.00124095       10     
reco:pandora2Track:LArPandoraTrackCreation            3.69846       6.69138       9.42863       6.69251       1.65396        10     
reco:pandora2calo:ShowerCalorimetry                  0.998239       1.86625       3.22478       1.79259      0.605978        10     
reco:pandora2calonosce:ShowerCalorimetry             0.915271       1.84026       3.17499       1.71643      0.598577        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0403537     0.0589438     0.0762016     0.0608322    0.00922627       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0411724     0.0587767     0.0709073     0.0591876    0.00788816       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.43802       2.46586       3.51099       2.38451      0.642761        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.37386       2.45605       3.95352       2.47127      0.725961        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.32201       2.35215       3.7354        2.27009      0.679718        10     
[art]:TriggerResults:TriggerResultInserter          2.9765e-05    5.19231e-05   0.000106154   4.5635e-05    2.30084e-05      10     
end_path:out1:RootOutput                             6.159e-06    1.00553e-05   2.4787e-05    7.1785e-06    5.98377e-06      10     
end_path:out1:RootOutput(write)                       1.89344       2.47766       3.19492       2.5239        0.32644        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6648.25 MB
  Peak resident set size usage (VmHWM): 4136.03 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258684_5_1_20251126T114133Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258684_5_1_20251126T114133Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258684_5_1_20251126T114133Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258684_5_1_20251126T114133Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258684_5_1_20251126T114133Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258684_5_1_20251126T114133Z_detsim_reco1.root.temp.json ====
.:
total 5729024
-rw-r--r--. 1 dune004 dune      44530 Nov 26 13:08 jobscript.log
-rw-r--r--. 1 dune004 dune        185 Nov 26 13:08 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune          0 Nov 26 13:08 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258684_5_1_20251126T114133Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune          0 Nov 26 13:08 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258684_5_1_20251126T114133Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune         31 Nov 26 13:08 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258684_5_1_20251126T114133Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune         25 Nov 26 13:08 all-input-dids.txt
-rw-r--r--. 1 dune004 dune        519 Nov 26 13:08 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune       3574 Nov 26 13:08 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune     200704 Nov 26 13:08 mem.db
-rw-r--r--. 1 dune004 dune      36864 Nov 26 13:08 time.db
-rw-r--r--. 1 dune004 dune 5816701041 Nov 26 13:08 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258684_5_1_20251126T114133Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune   49489460 Nov 26 13:07 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune          0 Nov 26 12:43 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune          0 Nov 26 12:40 debugprod.log
-rw-r--r--. 1 dune004 dune        519 Nov 26 12:40 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 23:18:21 UTC       justIN version: 01.05.03