justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 45991.127@dunegpschedd01.fnal.gov

Jobsub ID45991.127@dunegpschedd01.fnal.gov
Workflow ID2576
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-13 15:40:18
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel
Last heartbeat2025-09-13 18:15:54
From worker nodeHostnamewn-snel-026.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job stateaborted
Started2025-09-13 16:31:22
Input filesvd-protodune:np02vd_raw_run039275_0113_df-s05-d0_dw_0_20250901T183339.hdf5
Outputting started2025-09-13 18:03:28
Output filesvd-protodune-det-reco:np02vd_raw_run039275_0113_df-s05-d0_dw_0_20250901T183339_reco_stage1_20250913T180321_keepup.root
Finished2025-09-13 18:15:54
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

l 8255
wclsFrameSaver: no samples within desired window for channel 8255
wclsFrameSaver: no samples within desired window for channel 8345
wclsFrameSaver: no samples within desired window for channel 8346
wclsFrameSaver: no samples within desired window for channel 8346
wclsFrameSaver: no samples within desired window for channel 8348
wclsFrameSaver: no samples within desired window for channel 8348
wclsFrameSaver: no samples within desired window for channel 8348
wclsFrameSaver: no samples within desired window for channel 8349
wclsFrameSaver: no samples within desired window for channel 8591
wclsFrameSaver: no samples within desired window for channel 8591
wclsFrameSaver: no samples within desired window for channel 8591
wclsFrameSaver: no samples within desired window for channel 8610
wclsFrameSaver: no samples within desired window for channel 9518
wclsFrameSaver: no samples within desired window for channel 9518
wclsFrameSaver: no samples within desired window for channel 9521
wclsFrameSaver: no samples within desired window for channel 9650
wclsFrameSaver: no samples within desired window for channel 9991
wclsFrameSaver: no samples within desired window for channel 9992
wclsFrameSaver: no samples within desired window for channel 10221
wclsFrameSaver: no samples within desired window for channel 10241
wclsFrameSaver: no samples within desired window for channel 10270
wclsFrameSaver: no samples within desired window for channel 10389
wclsFrameSaver: no samples within desired window for channel 10676
wclsFrameSaver: no samples within desired window for channel 10807
wclsFrameSaver: no samples within desired window for channel 10855
wclsFrameSaver: no samples within desired window for channel 10858
wclsFrameSaver: no samples within desired window for channel 10904
wclsFrameSaver: no samples within desired window for channel 11050
wclsFrameSaver: no samples within desired window for channel 11593
wclsFrameSaver: no samples within desired window for channel 11593
wclsFrameSaver: no samples within desired window for channel 11593
wclsFrameSaver: no samples within desired window for channel 12272
wclsFrameSaver: no samples within desired window for channel 12272
wclsFrameSaver: no samples within desired window for channel 12273
wclsFrameSaver: no samples within desired window for channel 12274
wclsFrameSaver: no samples within desired window for channel 12274
wclsFrameSaver: no samples within desired window for channel 12275
FrameSaver: q=1.17932e+07 n=1370681 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 959 XUs and 1141 XVs -> 51 XUVs
C:0 T:1 2584 XUs and 2402 XVs -> 100 XUVs
C:0 T:2 2147 XUs and 2227 XVs -> 148 XUVs
C:0 T:3 3332 XUs and 3142 XVs -> 107 XUVs
C:0 T:4 247 XUs and 426 XVs -> 18 XUVs
C:0 T:5 1265 XUs and 1004 XVs -> 48 XUVs
C:0 T:6 1194 XUs and 1690 XVs -> 118 XUVs
C:0 T:7 2395 XUs and 2440 XVs -> 140 XUVs
C:0 T:8 2922 XUs and 3595 XVs -> 107 XUVs
C:0 T:9 2912 XUs and 2244 XVs -> 122 XUVs
C:0 T:10 6031 XUs and 4564 XVs -> 349 XUVs
C:0 T:11 3388 XUs and 3216 XVs -> 191 XUVs
C:0 T:12 3674 XUs and 2359 XVs -> 267 XUVs
C:0 T:13 4134 XUs and 3420 XVs -> 137 XUVs
C:0 T:14 3506 XUs and 2136 XVs -> 210 XUVs
C:0 T:15 1277 XUs and 1436 XVs -> 93 XUVs
2206 XUVs total
1770 collection wire objects
2206 potential space points
Neighbour search...
27394 tests to find 13824 neighbours
Iterating with no regularization...
Begin: 1.04922e+09
0 1.02395e+09
1 1.02257e+09
2 1.0225e+09
Now with regularization...
Begin: 1.00403e+09
0 1.00402e+09
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
13-Sep-2025 20:03:21 CEST  Closed output file "np02vd_raw_run039275_0113_df-s05-d0_dw_0_20250901T183339_reco_stage1_20250913T180321_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        212.778       304.648       639.364       280.005       95.5391        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.2368e-05    7.7584e-05    0.000188102   7.2726e-05    2.75018e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               59.5183       100.74        135.703       98.5803       18.0943        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.034658      0.0422812     0.139111      0.0349821     0.0238639       18     
produce:pdvddaphne:DAPHNEReaderPDVD               9.14572       13.9101       19.7013       14.2016       2.52595        18     
produce:ophit:OpHitFinder                        0.0642701     0.0798735     0.0944142     0.0796976    0.00858206       18     
produce:opflash:OpFlashFinderVerticalDrift       0.0127929     0.0192069     0.0265417     0.0193497    0.00401894       18     
produce:wclsdatavd:WireCellToolkit                62.5073       74.2729       110.076       70.4966       11.2304        18     
produce:gaushit:GausHitFinder                     0.71971       1.48326       2.49081       1.39645      0.396762        18     
produce:nhitsfilter:NumberOfHitsFilter          0.00018727    0.000419373   0.000624418   0.000409241   0.000102201      18     
produce:reco3d:SpacePointSolver                   6.30175       19.1659       32.9526       18.4428       6.03985        18     
produce:hitpdune:DisambigFromSpacePoints         0.0751921     0.308195      0.648121      0.293184      0.134691        18     
produce:pandora:StandardPandora                   22.1353       88.1538       359.768       65.0174       74.6312        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.384196       1.54363       4.42277       1.36913      0.856925        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0166062     0.035676      0.0466954     0.035714     0.00729675       18     
[art]:TriggerResults:TriggerResultInserter      1.3896e-05    1.80026e-05   4.5505e-05    1.60045e-05   6.95411e-06      18     
end_path:out1:RootOutput                         3.216e-06    4.36528e-06   1.8204e-05    3.5165e-06    3.36351e-06      18     
end_path:out1:RootOutput(write)                   4.35723       4.85087       5.44955       4.81033      0.248733        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7565.7 MB
  Peak resident set size usage (VmHWM): 5682.26 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039275_0113_df-s05-d0_dw_0_20250901T183339_reco_stage1_20250913T180321_keepup.root
\tHists: np02vd_raw_run039275_0113_df-s05-d0_dw_0_20250901T183339_reco_stage1_20250913T180321_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039275_0113_df-s05-d0_dw_0_20250901T183339_reco_stage1_20250913T180321_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039275_0113_df-s05-d0_dw_0_20250901T183339_reco_stage1_20250913T180321_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757786601.0,
    "core.end_time": 1757786601.0,
    "core.events": [
      40694,
      40714,
      40734,
      40754,
      40774,
      40794,
      40814,
      40834,
      40854,
      40874,
      40894,
      40914,
      40934,
      40954,
      40974,
      40994,
      41014,
      41034
    ],
    "core.event_count": 18,
    "core.first_event_number": 40694,
    "core.last_event_number": 41034,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39275
    ],
    "core.runs_subruns": [
      3927500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039275_0113_df-s05-d0_dw_0_20250901T183339.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039275_0113_df-s05-d0_dw_0_20250901T183339_reco_stage1_20250913T180321_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757786601.0,
    "core.end_time": 1757786601.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39275
    ],
    "core.runs_subruns": [
      3927500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039275_0113_df-s05-d0_dw_0_20250901T183339.hdf5"
    }
  ]
}
justIN time: 2025-09-18 19:29:29 UTC       justIN version: 01.05.00