justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 44667.158@dunegpschedd01.fnal.gov

Jobsub ID44667.158@dunegpschedd01.fnal.gov
Workflow ID2482
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-10 23:46:57
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2025-09-11 03:08:47
From worker nodeHostnameb9p07p7593.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job stateaborted
Started2025-09-11 01:55:27
Input filesvd-protodune:np02vd_raw_run039255_1140_df-s03-d2_dw_0_20250830T224216.hdf5
Outputting started2025-09-11 02:59:32
Output filesvd-protodune-det-reco:np02vd_raw_run039255_1140_df-s03-d2_dw_0_20250830T224216_reco_stage1_20250911T025920_keepup.root
Finished2025-09-11 03:08:47
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

meFanin:nfsp> EOS at call=35 with 8 
[04:57:53.299] D [  glue  ] frame sink sees EOS
[04:57:53.299] D [ pgraph ] <Pgrapher:> graph execution complete 
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.18 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.73 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.48 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.39 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.04 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.97 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.16 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.87 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.82 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.76 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.62 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[04:57:53.300] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:57:53.300] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[04:57:53.300] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[04:57:53.300] I [ timer  ] Timer: Total node execution : 65.0599988475442 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 62018 traces tagged "gauss"
FrameSaver: q=1.10262e+07 n=1287350 tag=gauss
wclsFrameSaver: saving 76524 traces tagged "wiener"
FrameSaver: q=1.17178e+07 n=1228076 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 295 XUs and 457 XVs -> 21 XUVs
C:0 T:1 996 XUs and 1457 XVs -> 63 XUVs
C:0 T:2 359 XUs and 412 XVs -> 19 XUVs
C:0 T:3 765 XUs and 1016 XVs -> 43 XUVs
C:0 T:4 984 XUs and 1062 XVs -> 58 XUVs
C:0 T:5 991 XUs and 850 XVs -> 49 XUVs
C:0 T:6 2866 XUs and 3911 XVs -> 264 XUVs
C:0 T:7 1372 XUs and 1810 XVs -> 94 XUVs
C:0 T:8 12484 XUs and 27250 XVs -> 2992 XUVs
C:0 T:9 4844 XUs and 8715 XVs -> 928 XUVs
C:0 T:10 2086 XUs and 3143 XVs -> 253 XUVs
C:0 T:11 1277 XUs and 1724 XVs -> 81 XUVs
C:0 T:12 12206 XUs and 16694 XVs -> 1026 XUVs
C:0 T:13 1990 XUs and 1383 XVs -> 91 XUVs
C:0 T:14 2139 XUs and 2298 XVs -> 118 XUVs
C:0 T:15 848 XUs and 907 XVs -> 54 XUVs
6154 XUVs total
2189 collection wire objects
6154 potential space points
Neighbour search...
418162 tests to find 184708 neighbours
Iterating with no regularization...
Begin: 1.03787e+09
0 9.20158e+08
1 9.13711e+08
2 9.12885e+08
Now with regularization...
Begin: 8.97426e+08
0 8.97172e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
11-Sep-2025 04:59:20 CEST  Closed output file "np02vd_raw_run039255_1140_df-s03-d2_dw_0_20250830T224216_reco_stage1_20250911T025920_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        132.355       211.036       322.669       200.213       56.7964        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.6681e-05    0.000105881   0.000143121   0.000108146   2.61697e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               5.84073       6.36925       6.86304       6.46476       0.25713        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0010283    0.00455203     0.0135796     0.0030209    0.00380389       18     
produce:pdvddaphne:DAPHNEReaderPDVD               3.37337       4.18213       5.34701       4.20174      0.519458        18     
produce:ophit:OpHitFinder                         0.03544      0.0479378     0.0644635     0.045241     0.00793733       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00808076     0.0132596     0.0214982     0.0116711    0.00411438       18     
produce:wclsdatavd:WireCellToolkit                59.6333       74.9693       97.5719       73.6309       9.31896        18     
produce:gaushit:GausHitFinder                     1.22622       1.6552        2.53313       1.52056      0.375258        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000243471   0.000385724   0.000680064   0.000386772   0.000101428      18     
produce:reco3d:SpacePointSolver                   10.7754       20.2551       31.2891       20.0757       5.47227        18     
produce:hitpdune:DisambigFromSpacePoints         0.148091      0.374075      0.679267       0.37196      0.137968        18     
produce:pandora:StandardPandora                   35.0931       95.8275       200.372       92.6014       46.3875        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.571576       1.61716       2.89811       1.6248       0.550122        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0154636     0.0388922     0.0590248     0.0365169     0.010104        18     
[art]:TriggerResults:TriggerResultInserter      1.8981e-05    2.93824e-05   6.1111e-05     2.809e-05    9.72821e-06      18     
end_path:out1:RootOutput                         5.19e-06     1.04329e-05    2.439e-05     8.965e-06    4.16776e-06      18     
end_path:out1:RootOutput(write)                   4.68535       5.63926       6.48798       5.69157      0.585579        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5240.43 MB
  Peak resident set size usage (VmHWM): 3289.9 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039255_1140_df-s03-d2_dw_0_20250830T224216_reco_stage1_20250911T025920_keepup.root
\tHists: np02vd_raw_run039255_1140_df-s03-d2_dw_0_20250830T224216_reco_stage1_20250911T025920_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039255_1140_df-s03-d2_dw_0_20250830T224216_reco_stage1_20250911T025920_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039255_1140_df-s03-d2_dw_0_20250830T224216_reco_stage1_20250911T025920_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757559561.0,
    "core.end_time": 1757559561.0,
    "core.events": [
      410401,
      410421,
      410441,
      410461,
      410481,
      410501,
      410521,
      410541,
      410561,
      410581,
      410601,
      410621,
      410641,
      410661,
      410681,
      410701,
      410721,
      410741
    ],
    "core.event_count": 18,
    "core.first_event_number": 410401,
    "core.last_event_number": 410741,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39255
    ],
    "core.runs_subruns": [
      3925500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039255_1140_df-s03-d2_dw_0_20250830T224216.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039255_1140_df-s03-d2_dw_0_20250830T224216_reco_stage1_20250911T025920_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757559561.0,
    "core.end_time": 1757559561.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39255
    ],
    "core.runs_subruns": [
      3925500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039255_1140_df-s03-d2_dw_0_20250830T224216.hdf5"
    }
  ]
}
justIN time: 2025-09-19 13:39:34 UTC       justIN version: 01.05.00