justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 38203.52@dunegpschedd02.fnal.gov

Jobsub ID38203.52@dunegpschedd02.fnal.gov
Workflow ID2486
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-11 10:54:41
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce510
Last heartbeat2025-09-11 12:01:13
From worker nodeHostnameb9p06p7146.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statestalled
Started2025-09-11 11:21:10
Input filesvd-protodune:np02vd_raw_run039350_2090_df-s04-d2_dw_0_20250911T085252.hdf5
Outputting started2025-09-11 12:01:12
Output files
Finished2025-09-11 12:26:56
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

: no samples within desired window for channel 8478
wclsFrameSaver: no samples within desired window for channel 9161
wclsFrameSaver: no samples within desired window for channel 9161
wclsFrameSaver: no samples within desired window for channel 9162
wclsFrameSaver: no samples within desired window for channel 9162
wclsFrameSaver: no samples within desired window for channel 9163
wclsFrameSaver: no samples within desired window for channel 9163
wclsFrameSaver: no samples within desired window for channel 9164
wclsFrameSaver: no samples within desired window for channel 9165
wclsFrameSaver: no samples within desired window for channel 9392
wclsFrameSaver: no samples within desired window for channel 9395
wclsFrameSaver: no samples within desired window for channel 9579
wclsFrameSaver: no samples within desired window for channel 9580
wclsFrameSaver: no samples within desired window for channel 9582
wclsFrameSaver: no samples within desired window for channel 9901
wclsFrameSaver: no samples within desired window for channel 10600
wclsFrameSaver: no samples within desired window for channel 10604
wclsFrameSaver: no samples within desired window for channel 10606
wclsFrameSaver: no samples within desired window for channel 10606
wclsFrameSaver: no samples within desired window for channel 10608
wclsFrameSaver: no samples within desired window for channel 10609
wclsFrameSaver: no samples within desired window for channel 10610
wclsFrameSaver: no samples within desired window for channel 10703
wclsFrameSaver: no samples within desired window for channel 10761
wclsFrameSaver: no samples within desired window for channel 11156
wclsFrameSaver: no samples within desired window for channel 11156
wclsFrameSaver: no samples within desired window for channel 11157
wclsFrameSaver: no samples within desired window for channel 11157
wclsFrameSaver: no samples within desired window for channel 11158
wclsFrameSaver: no samples within desired window for channel 11159
wclsFrameSaver: no samples within desired window for channel 11160
wclsFrameSaver: no samples within desired window for channel 11444
wclsFrameSaver: no samples within desired window for channel 11444
wclsFrameSaver: no samples within desired window for channel 11445
wclsFrameSaver: no samples within desired window for channel 11445
wclsFrameSaver: no samples within desired window for channel 11446
wclsFrameSaver: no samples within desired window for channel 11447
wclsFrameSaver: no samples within desired window for channel 11448
wclsFrameSaver: no samples within desired window for channel 11449
FrameSaver: q=5.32085e+06 n=596239 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 216 XUs and 212 XVs -> 6 XUVs
C:0 T:1 70 XUs and 70 XVs -> 1 XUVs
C:0 T:2 434 XUs and 539 XVs -> 24 XUVs
C:0 T:3 6 XUs and 8 XVs -> 0 XUVs
C:0 T:4 408 XUs and 637 XVs -> 20 XUVs
C:0 T:5 150 XUs and 229 XVs -> 20 XUVs
C:0 T:6 144 XUs and 173 XVs -> 2 XUVs
C:0 T:7 102 XUs and 104 XVs -> 4 XUVs
C:0 T:8 1473 XUs and 1381 XVs -> 95 XUVs
C:0 T:9 877 XUs and 1385 XVs -> 88 XUVs
C:0 T:10 1557 XUs and 1203 XVs -> 115 XUVs
C:0 T:11 934 XUs and 877 XVs -> 60 XUVs
C:0 T:12 1274 XUs and 1909 XVs -> 91 XUVs
C:0 T:13 1260 XUs and 1923 XVs -> 91 XUVs
C:0 T:14 312 XUs and 345 XVs -> 32 XUVs
C:0 T:15 2237 XUs and 3008 XVs -> 144 XUVs
793 XUVs total
708 collection wire objects
793 potential space points
Neighbour search...
5115 tests to find 2542 neighbours
Iterating with no regularization...
Begin: 3.03489e+08
0 3.01794e+08
1 3.01789e+08
Now with regularization...
Begin: 2.98014e+08
0 2.98014e+08
11-Sep-2025 14:00:49 CEST  Closed output file "np02vd_raw_run039350_2090_df-s04-d2_dw_0_20250911T085252_reco_stage1_20250911T120049_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        93.6377       130.362       157.202       132.408       19.9496        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.833e-05    0.000129416   0.000605977   0.000110051   0.000119467      18     
produce:tpcrawdecoder:PDVDTPCReader               6.59388       7.27524       7.79445       7.28914      0.311866        18     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00644825     0.0113927     0.052996     0.00745341     0.0110806       18     
produce:pdvddaphne:DAPHNEReaderPDVD               1.77848       2.39866       2.85575       2.5049       0.266676        18     
produce:ophit:OpHitFinder                        0.0338495     0.0428881     0.0484981     0.044483     0.00402333       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00588347     0.011643      0.0179063     0.0116607    0.00336312       18     
produce:wclsdatavd:WireCellToolkit                46.5459       63.4213       80.491        65.6429       9.85733        18     
produce:gaushit:GausHitFinder                    0.755703       1.16731       1.73912       1.17961       0.26274        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000205202   0.000267006   0.000365064   0.000266548   4.2069e-05       18     
produce:reco3d:SpacePointSolver                   7.75104       11.2033       16.9358       11.1377       2.49752        18     
produce:hitpdune:DisambigFromSpacePoints         0.0861859     0.168069      0.278213       0.16516      0.0561268       18     
produce:pandora:StandardPandora                   17.3918       38.2521       56.6901       36.5262       11.9496        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.431512      0.805228       1.52073      0.766899      0.261203        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0142329     0.0288705     0.0384914     0.0297626    0.00662394       18     
[art]:TriggerResults:TriggerResultInserter       1.922e-05    3.06297e-05   4.2701e-05     3.028e-05    6.13251e-06      18     
end_path:out1:RootOutput                         4.27e-06     8.29222e-06    2.071e-05     8.02e-06     3.28978e-06      18     
end_path:out1:RootOutput(write)                   4.05715       5.54513       6.42349       5.82044      0.728265        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4801.16 MB
  Peak resident set size usage (VmHWM): 2848.77 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039350_2090_df-s04-d2_dw_0_20250911T085252_reco_stage1_20250911T120049_keepup.root
\tHists: np02vd_raw_run039350_2090_df-s04-d2_dw_0_20250911T085252_reco_stage1_20250911T120049_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039350_2090_df-s04-d2_dw_0_20250911T085252_reco_stage1_20250911T120049_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039350_2090_df-s04-d2_dw_0_20250911T085252_reco_stage1_20250911T120049_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757592050.0,
    "core.end_time": 1757592050.0,
    "core.events": [
      152778,
      152782,
      152786,
      152790,
      152794,
      152798,
      152802,
      152806,
      152810,
      152814,
      152818,
      152822,
      152826,
      152830,
      152834,
      152838,
      152842,
      152846
    ],
    "core.event_count": 18,
    "core.first_event_number": 152778,
    "core.last_event_number": 152846,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39350
    ],
    "core.runs_subruns": [
      3935000001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039350_2090_df-s04-d2_dw_0_20250911T085252.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039350_2090_df-s04-d2_dw_0_20250911T085252_reco_stage1_20250911T120049_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757592050.0,
    "core.end_time": 1757592050.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39350
    ],
    "core.runs_subruns": [
      3935000001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039350_2090_df-s04-d2_dw_0_20250911T085252.hdf5"
    }
  ]
}
justIN time: 2025-09-19 06:25:34 UTC       justIN version: 01.05.00