justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 38205.76@dunegpschedd02.fnal.gov

Jobsub ID38205.76@dunegpschedd02.fnal.gov
Workflow ID2486
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-11 10:56:33
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2025-09-11 12:08:22
From worker nodeHostnameb9p11p9087.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statestalled
Started2025-09-11 11:26:07
Input filesvd-protodune:np02vd_raw_run039350_2046_df-s04-d2_dw_0_20250911T080008.hdf5
Outputting started2025-09-11 12:08:22
Output files
Finished2025-09-11 12:26:56
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

wclsFrameSaver: no samples within desired window for channel 9053
wclsFrameSaver: no samples within desired window for channel 9054
wclsFrameSaver: no samples within desired window for channel 9054
wclsFrameSaver: no samples within desired window for channel 9055
wclsFrameSaver: no samples within desired window for channel 9056
wclsFrameSaver: no samples within desired window for channel 9056
wclsFrameSaver: no samples within desired window for channel 9057
wclsFrameSaver: no samples within desired window for channel 9057
wclsFrameSaver: no samples within desired window for channel 9058
wclsFrameSaver: no samples within desired window for channel 9058
wclsFrameSaver: no samples within desired window for channel 9059
wclsFrameSaver: no samples within desired window for channel 9060
wclsFrameSaver: no samples within desired window for channel 9060
wclsFrameSaver: no samples within desired window for channel 9061
wclsFrameSaver: no samples within desired window for channel 9467
wclsFrameSaver: no samples within desired window for channel 9467
wclsFrameSaver: no samples within desired window for channel 9799
wclsFrameSaver: no samples within desired window for channel 10244
wclsFrameSaver: no samples within desired window for channel 10334
wclsFrameSaver: no samples within desired window for channel 10356
wclsFrameSaver: no samples within desired window for channel 10536
wclsFrameSaver: no samples within desired window for channel 10536
wclsFrameSaver: no samples within desired window for channel 10982
wclsFrameSaver: no samples within desired window for channel 10982
wclsFrameSaver: no samples within desired window for channel 11058
wclsFrameSaver: no samples within desired window for channel 11419
wclsFrameSaver: no samples within desired window for channel 11419
wclsFrameSaver: no samples within desired window for channel 11419
wclsFrameSaver: no samples within desired window for channel 11421
wclsFrameSaver: no samples within desired window for channel 11421
wclsFrameSaver: no samples within desired window for channel 11422
wclsFrameSaver: no samples within desired window for channel 11422
wclsFrameSaver: no samples within desired window for channel 11422
wclsFrameSaver: no samples within desired window for channel 12118
wclsFrameSaver: no samples within desired window for channel 12118
wclsFrameSaver: no samples within desired window for channel 12118
wclsFrameSaver: no samples within desired window for channel 12118
FrameSaver: q=7.78895e+06 n=917577 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1260 XUs and 1092 XVs -> 72 XUVs
C:0 T:1 601 XUs and 825 XVs -> 43 XUVs
C:0 T:2 817 XUs and 1062 XVs -> 48 XUVs
C:0 T:3 1419 XUs and 1713 XVs -> 79 XUVs
C:0 T:4 2074 XUs and 3325 XVs -> 83 XUVs
C:0 T:5 190 XUs and 382 XVs -> 9 XUVs
C:0 T:6 419 XUs and 602 XVs -> 13 XUVs
C:0 T:7 862 XUs and 1434 XVs -> 44 XUVs
C:0 T:8 447 XUs and 536 XVs -> 28 XUVs
C:0 T:9 978 XUs and 1910 XVs -> 140 XUVs
C:0 T:10 448 XUs and 455 XVs -> 23 XUVs
C:0 T:11 5403 XUs and 6152 XVs -> 275 XUVs
C:0 T:12 1465 XUs and 3012 XVs -> 134 XUVs
C:0 T:13 1016 XUs and 874 XVs -> 77 XUVs
C:0 T:14 2151 XUs and 2943 XVs -> 182 XUVs
C:0 T:15 2191 XUs and 1873 XVs -> 163 XUVs
1413 XUVs total
1208 collection wire objects
1413 potential space points
Neighbour search...
10223 tests to find 4748 neighbours
Iterating with no regularization...
Begin: 4.73155e+08
0 4.60851e+08
1 4.60467e+08
Now with regularization...
Begin: 4.54729e+08
0 4.54725e+08
PcaShowerParticleBuildingAlgorithm::OpeningAngle - principal eigenvalue less than or equal to 0.
11-Sep-2025 14:08:11 CEST  Closed output file "np02vd_raw_run039350_2046_df-s04-d2_dw_0_20250911T080008_reco_stage1_20250911T120811_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        110.01        138.215       233.845       125.062       32.4032        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.0081e-05    9.40854e-05   0.000151082   8.7521e-05    2.39745e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               6.65867       7.12606       9.53562       6.90614      0.636507        18     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00563628    0.00690687    0.00978176    0.00666159    0.00101509       18     
produce:pdvddaphne:DAPHNEReaderPDVD               1.60229       2.31471       2.76556       2.44644      0.310503        18     
produce:ophit:OpHitFinder                        0.0257507     0.0393405     0.0484279     0.0419066    0.00663398       18     
produce:opflash:OpFlashFinderVerticalDrift       0.0041869    0.00941666     0.0140758    0.00884698    0.00313152       18     
produce:wclsdatavd:WireCellToolkit                50.2495       61.3643       71.4371       62.6746       6.05926        18     
produce:gaushit:GausHitFinder                    0.829409       1.19205       1.92008       1.09364      0.307044        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000198552   0.000310441   0.000483165   0.000308213   7.36205e-05      18     
produce:reco3d:SpacePointSolver                   9.05563       13.525        26.6187       11.9589       4.30661        18     
produce:hitpdune:DisambigFromSpacePoints         0.104048      0.198172      0.515013      0.137463      0.106152        18     
produce:pandora:StandardPandora                   27.2181       46.5649       119.205       39.175        24.0557        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.500332       0.93926       2.35679      0.865898      0.443618        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0156483     0.0296557     0.0531482     0.028096     0.00908924       18     
[art]:TriggerResults:TriggerResultInserter       2.018e-05    3.29264e-05   0.000108061   2.5595e-05    2.05001e-05      18     
end_path:out1:RootOutput                         4.04e-06     9.34956e-06   3.3791e-05    8.6605e-06    6.27281e-06      18     
end_path:out1:RootOutput(write)                   4.0196        4.87162       6.31959       4.6592       0.763122        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4869.73 MB
  Peak resident set size usage (VmHWM): 2906.42 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039350_2046_df-s04-d2_dw_0_20250911T080008_reco_stage1_20250911T120811_keepup.root
\tHists: np02vd_raw_run039350_2046_df-s04-d2_dw_0_20250911T080008_reco_stage1_20250911T120811_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039350_2046_df-s04-d2_dw_0_20250911T080008_reco_stage1_20250911T120811_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039350_2046_df-s04-d2_dw_0_20250911T080008_reco_stage1_20250911T120811_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757592491.0,
    "core.end_time": 1757592491.0,
    "core.events": [
      149574,
      149578,
      149582,
      149586,
      149590,
      149594,
      149598,
      149602,
      149606,
      149610,
      149614,
      149618,
      149622,
      149626,
      149630,
      149634,
      149638,
      149642
    ],
    "core.event_count": 18,
    "core.first_event_number": 149574,
    "core.last_event_number": 149642,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39350
    ],
    "core.runs_subruns": [
      3935000001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039350_2046_df-s04-d2_dw_0_20250911T080008.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039350_2046_df-s04-d2_dw_0_20250911T080008_reco_stage1_20250911T120811_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757592491.0,
    "core.end_time": 1757592491.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39350
    ],
    "core.runs_subruns": [
      3935000001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039350_2046_df-s04-d2_dw_0_20250911T080008.hdf5"
    }
  ]
}
justIN time: 2025-09-19 13:30:53 UTC       justIN version: 01.05.00