justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252775.4@dunegpschedd02.fnal.gov

Jobsub ID252775.4@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 07:39:13
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce03
Last heartbeat2025-11-26 09:27:16
From worker nodeHostnameccwcondor0386
cpuinfoAMD EPYC 9334 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 07:39:49
Input fileshd-protodune:pdhd_prod_beam__244152_94_1_20251106T090242Z_gen_g4_IonScintPDExt.root_255537_64_1_20251122T230147Z_PDInt.root
JobscriptExit code0
Real time1h (6420s)
CPU time1h (6112s = 95%)
Max RSS bytes7156641792 (6825 MiB)
Outputting started2025-11-26 09:26:50
Output files
Finished2025-11-26 09:27:16
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Saver: saving 101534 traces tagged "wiener"
FrameSaver: q=3.15088e+06 n=1215489 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11524 XUs and 10200 XVs -> 75 XUVs
C:0 T:1 61181 XUs and 67873 XVs -> 2903 XUVs
C:0 T:2 31381 XUs and 20744 XVs -> 695 XUVs
C:0 T:3 5 XUs and 37 XVs -> 0 XUVs
C:0 T:4 31 XUs and 14 XVs -> 0 XUVs
C:0 T:5 33247 XUs and 30667 XVs -> 954 XUVs
C:0 T:6 79020 XUs and 43516 XVs -> 1945 XUVs
C:0 T:7 63 XUs and 38 XVs -> 0 XUVs
6572 XUVs total
2896 collection wire objects
6572 potential space points
Neighbour search...
199214 tests to find 121164 neighbours
Iterating with no regularization...
Begin: 3.15021e+07
0 2.91413e+07
1 2.89799e+07
2 2.89504e+07
3 2.89421e+07
Now with regularization...
Begin: 2.69546e+07
0 2.695e+07
26-Nov-2025 10:26:43 CET  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252775_4_1_20251126T073953Z_detsim_reco1.root"
26-Nov-2025 10:26:43 CET  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            103.728        147.6        203.455       142.35        27.8145        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00139594    0.00952981     0.0160077     0.0106166    0.00489413       10     
reco:opdec:Deconvolution                             0.861644       1.0274        1.27464       1.02453      0.122218        10     
reco:ophitspe:OpHitFinderDeco                        0.199827      0.219609      0.254552      0.213633      0.0184151       10     
reco:opflash:OpFlashFinder                           0.0905042     0.115745      0.154744      0.109732      0.021701        10     
reco:opslicer:OpSlicer                               0.237715      0.336305        0.438       0.342372      0.0612273       10     
reco:wclsdatahd:WireCellToolkit                       45.1877       49.6241       56.9066        48.59        3.5399         10     
reco:gaushit:GausHitFinder                           0.838898       1.00067       1.38852      0.955842      0.170757        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000278682   0.000402654   0.000690456   0.000347752   0.000118906      10     
reco:reco3d:SpacePointSolver                         0.585286       1.21374       1.83386       1.17971      0.364994        10     
reco:hitpdune:DisambigFromSpacePoints                0.502766      0.804229       1.12682      0.759533       0.21786        10     
reco:pandora:StandardPandora                          13.7613       30.4673       51.472        24.4884       13.4543        10     
reco:pandoraWriter:StandardPandora                   0.280662      0.371417       0.48895      0.366448      0.0533984       10     
reco:pandoraTrack:LArPandoraTrackCreation             1.73796       3.16189       4.75759       3.16923      0.719039        10     
reco:pandoraShower:LArPandoraShowerCreation           1.59674       3.56753       6.06237       3.77693       1.19636        10     
reco:pandoracalo:Calorimetry                         0.966388       1.5564        2.60033       1.46877      0.457971        10     
reco:pandoracalonosce:Calorimetry                    0.781657       1.42057       2.5952        1.31664      0.486893        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.21663       2.95261       4.93521       2.76267       1.09165        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.5215        2.67552       4.22723       2.51299      0.916662        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.049163      0.0723521     0.0986716     0.0731495     0.0151806       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0438143     0.0676101     0.0908592     0.0694894     0.0144785       10     
reco:emtrkmichelid:EmTrackMichelId                    18.1046       22.5499       29.9411       21.6465       3.29733        10     
reco:crttag:SingleCRTMatchingProducer               0.00749054     0.0102467     0.0150528    0.00939952    0.00276147       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00565776    0.00813447     0.0161343    0.00738366    0.00293037       10     
reco:pandora2Track:LArPandoraTrackCreation            3.67544       7.62252       11.3267       7.3464        2.17353        10     
reco:pandora2calo:ShowerCalorimetry                  0.990323       2.45359       3.88474       2.28261      0.853353        10     
reco:pandora2calonosce:ShowerCalorimetry              1.02144       2.25274       3.2093        2.20575      0.672524        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0619477     0.0768667      0.08775      0.0785117    0.00737084       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0603761     0.074189      0.0927208     0.0722122    0.00990974       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.70479       3.03697       4.42489       3.16665      0.792602        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.87788       3.02747       4.71497       2.82929      0.902404        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.73903       2.94969       4.63475       2.79248      0.822091        10     
[art]:TriggerResults:TriggerResultInserter          3.1688e-05    6.37618e-05   0.000127614   6.10525e-05   2.48047e-05      10     
end_path:out1:RootOutput                             7.411e-06    1.20962e-05   3.1718e-05    1.0095e-05    6.67882e-06      10     
end_path:out1:RootOutput(write)                       2.15195       2.64494       3.50824       2.58476      0.370869        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6678.83 MB
  Peak resident set size usage (VmHWM): 4201.04 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252775_4_1_20251126T073953Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252775_4_1_20251126T073953Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252775_4_1_20251126T073953Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252775_4_1_20251126T073953Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252775_4_1_20251126T073953Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252775_4_1_20251126T073953Z_detsim_reco1.root.temp.json ====
.:
total 6419608
-rw-r--r-- 1 dune001 lbno      42018 Nov 26 10:26 jobscript.log
-rw-r--r-- 1 dune001 lbno        185 Nov 26 10:26 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno          0 Nov 26 10:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252775_4_1_20251126T073953Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno          0 Nov 26 10:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252775_4_1_20251126T073953Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno         31 Nov 26 10:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252775_4_1_20251126T073953Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno         25 Nov 26 10:26 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno        519 Nov 26 10:26 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno       3574 Nov 26 10:26 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno     204800 Nov 26 10:26 mem.db
-rw-r--r-- 1 dune001 lbno      36864 Nov 26 10:26 time.db
-rw-r--r-- 1 dune001 lbno 6516996367 Nov 26 10:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252775_4_1_20251126T073953Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno   56367100 Nov 26 10:25 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno          0 Nov 26 10:04 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno          0 Nov 26 10:00 debugprod.log
-rw-r--r-- 1 dune001 lbno        519 Nov 26 10:00 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 23:17:56 UTC       justIN version: 01.05.03