justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40882.8@dunegpschedd02.fnal.gov

Jobsub ID40882.8@dunegpschedd02.fnal.gov
Workflow ID2745
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-17 09:03:03
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp_multicore
Last heartbeat2025-09-17 12:29:24
From worker nodeHostnamewn-snel-003.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-17 10:28:24
Input filesvd-protodune:np02vd_raw_run039279_0005_df-s05-d3_dw_0_20250902T161304.hdf5
JobscriptExit code0
Real time1h (7189s)
CPU time55m (3311s = 46%)
Max RSS bytes3468976128 (3308 MiB)
Outputting started2025-09-17 12:28:14
Output filesvd-protodune-det-reco:np02vd_raw_run039279_0005_df-s05-d3_dw_0_20250902T161304_reco_stage1_20250917T122805_keepup.root
Finished2025-09-17 12:29:24
Saved logsjustin-logs:40882.8-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] Timer: WireCell::SigProc::OmnibusSigProc : 7.25 sec
[14:26:52.240] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.92 sec
[14:26:52.240] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.77 sec
[14:26:52.240] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.77 sec
[14:26:52.240] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.75 sec
[14:26:52.240] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.75 sec
[14:26:52.240] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[14:26:52.240] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[14:26:52.241] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[14:26:52.241] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[14:26:52.241] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[14:26:52.241] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[14:26:52.241] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:26:52.241] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:26:52.241] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:26:52.241] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[14:26:52.241] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:26:52.241] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:26:52.241] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[14:26:52.241] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[14:26:52.241] I [ timer  ] Timer: Total node execution : 65.9200001321733 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 52856 traces tagged "gauss"
FrameSaver: q=8.18695e+06 n=1099606 tag=gauss
wclsFrameSaver: saving 65733 traces tagged "wiener"
FrameSaver: q=8.77865e+06 n=1052704 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1294 XUs and 1685 XVs -> 119 XUVs
C:0 T:1 1324 XUs and 1751 XVs -> 69 XUVs
C:0 T:2 576 XUs and 395 XVs -> 28 XUVs
C:0 T:3 541 XUs and 836 XVs -> 33 XUVs
C:0 T:4 576 XUs and 770 XVs -> 27 XUVs
C:0 T:5 390 XUs and 489 XVs -> 20 XUVs
C:0 T:6 1906 XUs and 2185 XVs -> 98 XUVs
C:0 T:7 1379 XUs and 1923 XVs -> 66 XUVs
C:0 T:8 9990 XUs and 24672 XVs -> 3249 XUVs
C:0 T:9 6630 XUs and 11042 XVs -> 1415 XUVs
C:0 T:10 4286 XUs and 7362 XVs -> 2018 XUVs
C:0 T:11 2574 XUs and 3130 XVs -> 591 XUVs
C:0 T:12 5049 XUs and 8827 XVs -> 1828 XUVs
C:0 T:13 4889 XUs and 6506 XVs -> 975 XUVs
C:0 T:14 2755 XUs and 3312 XVs -> 615 XUVs
C:0 T:15 1891 XUs and 2101 XVs -> 137 XUVs
11288 XUVs total
2047 collection wire objects
11288 potential space points
Neighbour search...
1446728 tests to find 632146 neighbours
Iterating with no regularization...
Begin: 2.87356e+08
0 2.53135e+08
1 2.50831e+08
2 2.50414e+08
3 2.50255e+08
Now with regularization...
Begin: 2.41217e+08
0 2.41024e+08
this->SetupTestBeamPfo(pNuPfo, pTestBeamPfo, testBeamStartVertex) return STATUS_CODE_NOT_FOUND
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArThreeDReco/LArEventBuilding/TestBeamParticleCreationAlgorithm.cc line#: 43
iter->second->Run() throw STATUS_CODE_NOT_FOUND
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0079, LArTestBeamParticleCreation, STATUS_CODE_NOT_FOUND
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
17-Sep-2025 14:28:05 CEST  Closed output file "np02vd_raw_run039279_0005_df-s05-d3_dw_0_20250902T161304_reco_stage1_20250917T122805_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        305.848       396.729       520.426       381.936       60.1147        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.382e-05    8.24533e-05   0.000197611   7.6103e-05    2.87024e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               160.682       203.861       254.632       203.189       29.4901        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.034254      0.0578064     0.342647      0.0352525     0.070169        18     
produce:pdvddaphne:DAPHNEReaderPDVD               19.7684       24.5491       31.6918       23.2891       3.39654        18     
produce:ophit:OpHitFinder                        0.0572388     0.0810192     0.0911248     0.0821543    0.00828326       18     
produce:opflash:OpFlashFinderVerticalDrift       0.012045      0.0212368     0.0382661     0.0207726    0.00561975       18     
produce:wclsdatavd:WireCellToolkit                60.5144       68.6628       85.3892       67.0443       7.13543        18     
produce:gaushit:GausHitFinder                     0.78562       1.35049       1.95551       1.32894      0.328203        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000309541   0.000729571   0.00122821     0.0007567    0.00030352       18     
produce:reco3d:SpacePointSolver                   7.62127       17.9304       28.5639       17.4073       5.66101        18     
produce:hitpdune:DisambigFromSpacePoints         0.105337      0.272312       0.52854      0.250308      0.115702        18     
produce:pandora:StandardPandora                   18.8419       74.0438       228.284       63.1147       47.1073        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.629015       1.22146       2.64942       1.12336      0.499588        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0237009     0.0336299     0.0498923     0.0314896     0.0070012       18     
[art]:TriggerResults:TriggerResultInserter      1.3195e-05    1.7906e-05    3.7551e-05     1.597e-05    5.5758e-06       18     
end_path:out1:RootOutput                         3.427e-06    5.06617e-06   1.4187e-05    4.3835e-06    2.42159e-06      18     
end_path:out1:RootOutput(write)                   4.26558       4.60583       4.97134       4.58547      0.187136        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5356.5 MB
  Peak resident set size usage (VmHWM): 3468.98 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039279_0005_df-s05-d3_dw_0_20250902T161304_reco_stage1_20250917T122805_keepup.root
\tHists: np02vd_raw_run039279_0005_df-s05-d3_dw_0_20250902T161304_reco_stage1_20250917T122805_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039279_0005_df-s05-d3_dw_0_20250902T161304_reco_stage1_20250917T122805_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039279_0005_df-s05-d3_dw_0_20250902T161304_reco_stage1_20250917T122805_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758112085.0,
    "core.end_time": 1758112085.0,
    "core.events": [
      1817,
      1837,
      1857,
      1877,
      1897,
      1917,
      1937,
      1957,
      1977,
      1997,
      2017,
      2037,
      2057,
      2077,
      2097,
      2117,
      2137,
      2157
    ],
    "core.event_count": 18,
    "core.first_event_number": 1817,
    "core.last_event_number": 2157,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39279
    ],
    "core.runs_subruns": [
      3927900001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039279_0005_df-s05-d3_dw_0_20250902T161304.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039279_0005_df-s05-d3_dw_0_20250902T161304_reco_stage1_20250917T122805_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758112085.0,
    "core.end_time": 1758112085.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39279
    ],
    "core.runs_subruns": [
      3927900001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039279_0005_df-s05-d3_dw_0_20250902T161304.hdf5"
    }
  ]
}
justIN time: 2025-09-18 19:57:29 UTC       justIN version: 01.05.00