justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258437.9@dunegpschedd01.fnal.gov

Jobsub ID258437.9@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 07:27:12
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce03
Last heartbeat2025-11-26 08:57:03
From worker nodeHostnameccwcondor0045
cpuinfoAMD EPYC 9334 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 07:27:45
Input fileshd-protodune:pdhd_prod_beam__238322_88_1_20251106T071402Z_gen_g4_IonScintPDExt.root_255796_124_1_20251123T200015Z_PDInt.root
JobscriptExit code0
Real time1h (5317s)
CPU time1h (5190s = 97%)
Max RSS bytes6484738048 (6184 MiB)
Outputting started2025-11-26 08:56:23
Output files
Finished2025-11-26 08:57:03
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

s tagged "wiener"
FrameSaver: q=3.09616e+06 n=1146790 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8307 XUs and 7817 XVs -> 103 XUVs
C:0 T:1 92347 XUs and 85635 XVs -> 4697 XUVs
C:0 T:2 11778 XUs and 7862 XVs -> 364 XUVs
C:0 T:3 10 XUs and 11 XVs -> 0 XUVs
C:0 T:5 52206 XUs and 54591 XVs -> 1490 XUVs
C:0 T:6 6284 XUs and 8461 XVs -> 435 XUVs
C:0 T:7 7 XUs and 7 XVs -> 1 XUVs
7090 XUVs total
2685 collection wire objects
7090 potential space points
Neighbour search...
337780 tests to find 160104 neighbours
Iterating with no regularization...
Begin: 4.88555e+07
0 4.44165e+07
1 4.40695e+07
2 4.40258e+07
Now with regularization...
Begin: 4.14919e+07
0 4.14765e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 09:56:15 CET  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258437_9_1_20251126T072751Z_detsim_reco1.root"
26-Nov-2025 09:56:15 CET  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            93.2642       129.139       161.467       130.767       21.6579        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00139689    0.00975886     0.0300753    0.00935846    0.00776221       10     
reco:opdec:Deconvolution                             0.735082      0.877318       1.03054       0.88721      0.0904705       10     
reco:ophitspe:OpHitFinderDeco                        0.159123      0.204964      0.228847      0.207738      0.018437        10     
reco:opflash:OpFlashFinder                           0.0702715     0.088909      0.115702      0.092699      0.0146307       10     
reco:opslicer:OpSlicer                               0.200449      0.248517      0.286981      0.251711      0.0302637       10     
reco:wclsdatahd:WireCellToolkit                       44.768        50.3125       58.9332       49.3848       3.87042        10     
reco:gaushit:GausHitFinder                           0.669724       0.90516       1.1562       0.910628      0.158033        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000225051   0.000368517   0.000562453   0.000323911   0.000114813      10     
reco:reco3d:SpacePointSolver                          0.34029       1.33424       2.75416       1.23194      0.773164        10     
reco:hitpdune:DisambigFromSpacePoints                0.293146      0.730064       1.18687      0.786473      0.263781        10     
reco:pandora:StandardPandora                          10.8508       26.0203       58.7423       27.4115        13.63         10     
reco:pandoraWriter:StandardPandora                   0.237576      0.321684      0.389282      0.325005      0.0470925       10     
reco:pandoraTrack:LArPandoraTrackCreation             1.2788        2.29498       3.26413       2.32433      0.675958        10     
reco:pandoraShower:LArPandoraShowerCreation           1.13834       2.0381        3.33188       2.00172      0.750835        10     
reco:pandoracalo:Calorimetry                         0.630334       1.11321       1.92543       1.09966      0.377979        10     
reco:pandoracalonosce:Calorimetry                    0.638232       1.07285       1.51836       1.14983       0.27333        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.06623       1.99003       3.25148       1.99055      0.675402        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.996602       1.93812       3.84039       1.86298      0.796361        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0486642     0.0643266     0.0912764     0.0657427     0.0118301       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0464163     0.0631042     0.0908186     0.0639504     0.0115127       10     
reco:emtrkmichelid:EmTrackMichelId                    15.7262       19.9374       24.8521       20.5486       2.69062        10     
reco:crttag:SingleCRTMatchingProducer                0.0045929    0.00708468    0.00955397    0.00730876    0.00163174       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00379403    0.00549351    0.00758307    0.00530527    0.00110901       10     
reco:pandora2Track:LArPandoraTrackCreation            2.34162       5.14349       9.42826       4.87554       2.14387        10     
reco:pandora2calo:ShowerCalorimetry                  0.820662       1.67474       3.2764        1.44872      0.798408        10     
reco:pandora2calonosce:ShowerCalorimetry             0.755941       1.62924       3.12298       1.42541      0.739316        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0435766     0.0643686     0.0833529     0.0647115    0.00988063       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.044598      0.0650664     0.0814914     0.0660569     0.0100627       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.961803       2.11874       3.90236       2.01908      0.911753        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.00074       2.20637       3.87531       1.97858      0.888979        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.954793       2.11777       3.88972       1.9868       0.863385        10     
[art]:TriggerResults:TriggerResultInserter          4.2494e-05    5.53839e-05   0.00011908    4.8714e-05    2.15934e-05      10     
end_path:out1:RootOutput                             8.223e-06    1.28036e-05   3.5314e-05    1.04915e-05   7.57425e-06      10     
end_path:out1:RootOutput(write)                       1.7977        2.34767       2.82594       2.3631       0.268309        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6705.79 MB
  Peak resident set size usage (VmHWM): 4175.07 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258437_9_1_20251126T072751Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258437_9_1_20251126T072751Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258437_9_1_20251126T072751Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258437_9_1_20251126T072751Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258437_9_1_20251126T072751Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258437_9_1_20251126T072751Z_detsim_reco1.root.temp.json ====
.:
total 5715160
-rw-r--r-- 1 dune001 lbno      41003 Nov 26 09:56 jobscript.log
-rw-r--r-- 1 dune001 lbno        194 Nov 26 09:56 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno          0 Nov 26 09:56 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258437_9_1_20251126T072751Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno          0 Nov 26 09:56 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258437_9_1_20251126T072751Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno         31 Nov 26 09:56 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258437_9_1_20251126T072751Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno         25 Nov 26 09:56 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno        519 Nov 26 09:56 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno       3574 Nov 26 09:56 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno     204800 Nov 26 09:56 mem.db
-rw-r--r-- 1 dune001 lbno      36864 Nov 26 09:56 time.db
-rw-r--r-- 1 dune001 lbno 5802683910 Nov 26 09:56 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258437_9_1_20251126T072751Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno   49325450 Nov 26 09:55 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno          0 Nov 26 09:35 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno          0 Nov 26 09:33 debugprod.log
-rw-r--r-- 1 dune001 lbno        519 Nov 26 09:32 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 23:18:00 UTC       justIN version: 01.05.03