justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258439.20@dunegpschedd01.fnal.gov

Jobsub ID258439.20@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 07:33:13
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce03
Last heartbeat2025-11-26 09:37:55
From worker nodeHostnameccwcondor0551
cpuinfoAMD EPYC 7453 28-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 07:33:56
Input fileshd-protodune:pdhd_prod_beam__238230_42_1_20251106T053132Z_gen_g4_IonScintPDExt.root_250203_79_1_20251123T214757Z_PDInt.root
JobscriptExit code0
Real time2h (7396s)
CPU time1h (7136s = 96%)
Max RSS bytes8493219840 (8099 MiB)
Outputting started2025-11-26 09:37:14
Output files
Finished2025-11-26 09:37:55
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

r
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11233 XUs and 8059 XVs -> 227 XUVs
C:0 T:1 80181 XUs and 59763 XVs -> 2929 XUVs
C:0 T:2 27060 XUs and 30312 XVs -> 590 XUVs
C:0 T:3 3 XUs and 11 XVs -> 1 XUVs
C:0 T:4 30 XUs and 31 XVs -> 6 XUVs
C:0 T:5 31798 XUs and 28293 XVs -> 1393 XUVs
C:0 T:6 31046 XUs and 43822 XVs -> 3162 XUVs
C:0 T:7 4 XUs and 4 XVs -> 0 XUVs
8308 XUVs total
2551 collection wire objects
8308 potential space points
Neighbour search...
597820 tests to find 304838 neighbours
Iterating with no regularization...
Begin: 6.6933e+07
0 6.03232e+07
1 5.97301e+07
2 5.96609e+07
3 5.96459e+07
Now with regularization...
Begin: 5.67725e+07
0 5.67604e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 10:37:04 CET  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258439_20_1_20251126T073402Z_detsim_reco1.root"
26-Nov-2025 10:37:04 CET  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            144.037       214.511       336.371       182.477       61.8096        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00148167     0.0169578     0.0324411     0.0183044    0.00982069       10     
reco:opdec:Deconvolution                             0.762562       1.09722       1.43235       1.10307      0.198363        10     
reco:ophitspe:OpHitFinderDeco                        0.243538      0.268635      0.315393      0.263238      0.0205064       10     
reco:opflash:OpFlashFinder                           0.0654493     0.123217      0.201272      0.121113      0.0451995       10     
reco:opslicer:OpSlicer                               0.222323      0.377309       0.63445      0.350128      0.124444        10     
reco:wclsdatahd:WireCellToolkit                       53.4657       58.6696       63.8699       59.2865       3.62193        10     
reco:gaushit:GausHitFinder                           0.787324       1.16343       1.66616       1.1384       0.243673        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000360784   0.000641544    0.0010778    0.000597412   0.00021539       10     
reco:reco3d:SpacePointSolver                         0.859503       1.88092       3.36463       1.89223      0.745503        10     
reco:hitpdune:DisambigFromSpacePoints                0.543798       1.04281       1.99214      0.887438      0.439288        10     
reco:pandora:StandardPandora                          18.0787       39.1426       67.1525       33.8919       16.327         10     
reco:pandoraWriter:StandardPandora                    0.31128      0.415104      0.585256      0.384947      0.0808633       10     
reco:pandoraTrack:LArPandoraTrackCreation             2.62888       5.08575       8.67161       4.0954        1.90595        10     
reco:pandoraShower:LArPandoraShowerCreation           3.39417       5.57928       10.8958       4.11079       2.40122        10     
reco:pandoracalo:Calorimetry                          1.20906       2.47015       4.29718       1.97887       1.00905        10     
reco:pandoracalonosce:Calorimetry                     1.08564       2.6662        4.30046       2.28673       1.05038        10     
reco:pandoraShowercalo:ShowerCalorimetry              2.55559       5.59795       11.3722       4.42457       2.79676        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         2.39541       5.12019       12.4043       4.13414       3.06724        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0691931     0.0964917     0.139697      0.0867098     0.020139        10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0674323     0.0930516     0.128304      0.087079      0.0178755       10     
reco:emtrkmichelid:EmTrackMichelId                    28.2486       40.4438       56.1435       40.0483       8.24819        10     
reco:crttag:SingleCRTMatchingProducer               0.00920345     0.0137247     0.0193824     0.0135979     0.0033663       10     
reco:crtreco:TwoCRTMatchingProducer                  0.0074674     0.0124959     0.0184816     0.0121965    0.00397387       10     
reco:pandora2Track:LArPandoraTrackCreation            5.80704       12.0186       23.9669       8.31063       6.09251        10     
reco:pandora2calo:ShowerCalorimetry                   2.30374       4.73935       10.3395       2.78974       2.86355        10     
reco:pandora2calonosce:ShowerCalorimetry              2.16419       4.86496       12.3984       2.75663       3.28602        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0571485     0.0899948     0.144757      0.0835958     0.0261918       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0578292     0.0912146      0.13723      0.0825892     0.0271156       10     
reco:pandora2Shower:LArPandoraShowerCreation          3.09035       5.7654        11.9394       3.67762       3.10213        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.79996       6.31605       15.4228       3.9679        4.02759        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        2.77518       6.08724       15.7403       3.41471       4.19919        10     
[art]:TriggerResults:TriggerResultInserter          4.4202e-05    7.70671e-05   0.000126617   7.9318e-05    2.36326e-05      10     
end_path:out1:RootOutput                            1.1121e-05    1.68015e-05   3.6659e-05    1.40715e-05   7.10742e-06      10     
end_path:out1:RootOutput(write)                       1.83829       2.90115       4.00734       2.80285      0.617265        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7710.6 MB
  Peak resident set size usage (VmHWM): 5488.98 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258439_20_1_20251126T073402Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258439_20_1_20251126T073402Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258439_20_1_20251126T073402Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258439_20_1_20251126T073402Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258439_20_1_20251126T073402Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258439_20_1_20251126T073402Z_detsim_reco1.root.temp.json ====
.:
total 6755824
-rw-r--r-- 1 dune001 lbno      41080 Nov 26 10:37 jobscript.log
-rw-r--r-- 1 dune001 lbno        193 Nov 26 10:37 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno          0 Nov 26 10:37 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258439_20_1_20251126T073402Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno          0 Nov 26 10:37 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258439_20_1_20251126T073402Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno         31 Nov 26 10:37 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258439_20_1_20251126T073402Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno         25 Nov 26 10:37 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno        519 Nov 26 10:37 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno       3574 Nov 26 10:37 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno     204800 Nov 26 10:37 mem.db
-rw-r--r-- 1 dune001 lbno      36864 Nov 26 10:37 time.db
-rw-r--r-- 1 dune001 lbno 6863246920 Nov 26 10:37 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258439_20_1_20251126T073402Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno   54399420 Nov 26 10:35 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno          0 Nov 26 10:02 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno          0 Nov 26 09:59 debugprod.log
-rw-r--r-- 1 dune001 lbno        519 Nov 26 09:58 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 23:17:52 UTC       justIN version: 01.05.03