justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 38119.136@dunegpschedd02.fnal.gov

Jobsub ID38119.136@dunegpschedd02.fnal.gov
Workflow ID2496
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-11 10:00:32
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce510
Last heartbeat2025-09-11 12:06:27
From worker nodeHostnameb9p11p9887.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job stateaborted
Started2025-09-11 10:47:59
Input filesvd-protodune:np02vd_raw_run039273_0291_df-s03-d3_dw_0_20250831T132142.hdf5
Outputting started2025-09-11 12:00:56
Output filesvd-protodune-det-reco:np02vd_raw_run039273_0291_df-s03-d3_dw_0_20250831T132142_reco_stage1_20250911T120025_keepup.root
Finished2025-09-11 12:06:27
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

n desired window for channel 12243
wclsFrameSaver: no samples within desired window for channel 12243
wclsFrameSaver: no samples within desired window for channel 12244
wclsFrameSaver: no samples within desired window for channel 12245
wclsFrameSaver: no samples within desired window for channel 12246
wclsFrameSaver: no samples within desired window for channel 12246
wclsFrameSaver: no samples within desired window for channel 12247
wclsFrameSaver: no samples within desired window for channel 12247
wclsFrameSaver: no samples within desired window for channel 12248
wclsFrameSaver: no samples within desired window for channel 12248
wclsFrameSaver: no samples within desired window for channel 12250
wclsFrameSaver: no samples within desired window for channel 12251
wclsFrameSaver: no samples within desired window for channel 12252
wclsFrameSaver: no samples within desired window for channel 12252
wclsFrameSaver: no samples within desired window for channel 12253
wclsFrameSaver: no samples within desired window for channel 12253
wclsFrameSaver: no samples within desired window for channel 12254
wclsFrameSaver: no samples within desired window for channel 12255
wclsFrameSaver: no samples within desired window for channel 12256
wclsFrameSaver: no samples within desired window for channel 12257
wclsFrameSaver: no samples within desired window for channel 12258
wclsFrameSaver: no samples within desired window for channel 12259
wclsFrameSaver: no samples within desired window for channel 12260
wclsFrameSaver: no samples within desired window for channel 12263
wclsFrameSaver: no samples within desired window for channel 12264
wclsFrameSaver: no samples within desired window for channel 12265
wclsFrameSaver: no samples within desired window for channel 12266
wclsFrameSaver: no samples within desired window for channel 12267
wclsFrameSaver: no samples within desired window for channel 12268
wclsFrameSaver: no samples within desired window for channel 12269
wclsFrameSaver: no samples within desired window for channel 12270
wclsFrameSaver: no samples within desired window for channel 12271
wclsFrameSaver: no samples within desired window for channel 12272
wclsFrameSaver: no samples within desired window for channel 12273
wclsFrameSaver: no samples within desired window for channel 12278
FrameSaver: q=1.19529e+07 n=1172507 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 559 XUs and 501 XVs -> 24 XUVs
C:0 T:1 833 XUs and 1096 XVs -> 86 XUVs
C:0 T:2 635 XUs and 652 XVs -> 31 XUVs
C:0 T:3 1148 XUs and 1073 XVs -> 42 XUVs
C:0 T:4 334 XUs and 293 XVs -> 16 XUVs
C:0 T:5 194 XUs and 233 XVs -> 7 XUVs
C:0 T:6 680 XUs and 442 XVs -> 18 XUVs
C:0 T:7 730 XUs and 676 XVs -> 34 XUVs
C:0 T:8 19996 XUs and 41283 XVs -> 6233 XUVs
C:0 T:9 7923 XUs and 11519 XVs -> 1733 XUVs
C:0 T:10 7986 XUs and 14560 XVs -> 2628 XUVs
C:0 T:11 11354 XUs and 20640 XVs -> 3436 XUVs
C:0 T:12 17024 XUs and 36080 XVs -> 5582 XUVs
C:0 T:13 9323 XUs and 13911 XVs -> 1344 XUVs
C:0 T:14 5765 XUs and 9165 XVs -> 1222 XUVs
C:0 T:15 2774 XUs and 2707 XVs -> 225 XUVs
22661 XUVs total
2978 collection wire objects
22661 potential space points
Neighbour search...
3170333 tests to find 1360058 neighbours
Iterating with no regularization...
Begin: 5.71956e+08
0 4.71902e+08
1 4.65164e+08
2 4.63796e+08
3 4.63281e+08
4 4.63028e+08
Now with regularization...
Begin: 4.47144e+08
0 4.46818e+08
TestBeamParticleCreationAlgorithm::SetupTestBeamVertex - Test beam particle has no initial vertex
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
11-Sep-2025 14:00:25 CEST  Closed output file "np02vd_raw_run039273_0291_df-s03-d3_dw_0_20250831T132142_reco_stage1_20250911T120025_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        172.749       239.264       433.411       218.493        62.83         18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.902e-05    9.33236e-05   0.00016458    8.82005e-05   2.73132e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               6.38838       7.66353       25.2816       6.64724       4.27428        18     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00115455    0.00338821     0.0192601    0.00133818    0.00501109       18     
produce:pdvddaphne:DAPHNEReaderPDVD               3.57031       4.3031        5.05244       4.3641       0.443867        18     
produce:ophit:OpHitFinder                        0.0406146     0.0499222     0.0587218     0.0506463    0.00514267       18     
produce:opflash:OpFlashFinderVerticalDrift       0.0077978     0.0153744     0.0238049     0.0150442    0.00500152       18     
produce:wclsdatavd:WireCellToolkit                68.2675       79.003        96.9148       75.6419       8.34885        18     
produce:gaushit:GausHitFinder                     1.09627       1.77528       2.61044       1.83907      0.369343        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000286761   0.000424986   0.000603232   0.000419576   7.77607e-05      18     
produce:reco3d:SpacePointSolver                   17.1839       22.0795       32.5592       20.9946       4.39808        18     
produce:hitpdune:DisambigFromSpacePoints         0.200339      0.382357      0.624727      0.394137       0.11938        18     
produce:pandora:StandardPandora                   53.5475       116.538       304.047       98.9721       58.4455        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.905868       1.70128        2.972        1.58937      0.490534        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0265078     0.0433809     0.0590835     0.0471019     0.0114284       18     
[art]:TriggerResults:TriggerResultInserter       2.021e-05    3.76068e-05   0.000134441    2.947e-05    2.54987e-05      18     
end_path:out1:RootOutput                         6.37e-06     1.04389e-05    3.991e-05     8.335e-06    7.28052e-06      18     
end_path:out1:RootOutput(write)                   4.43021       5.65916       6.90256       5.58969      0.736662        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5198.54 MB
  Peak resident set size usage (VmHWM): 3255.68 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039273_0291_df-s03-d3_dw_0_20250831T132142_reco_stage1_20250911T120025_keepup.root
\tHists: np02vd_raw_run039273_0291_df-s03-d3_dw_0_20250831T132142_reco_stage1_20250911T120025_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039273_0291_df-s03-d3_dw_0_20250831T132142_reco_stage1_20250911T120025_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039273_0291_df-s03-d3_dw_0_20250831T132142_reco_stage1_20250911T120025_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757592026.0,
    "core.end_time": 1757592026.0,
    "core.events": [
      104769,
      104789,
      104809,
      104829,
      104849,
      104869,
      104889,
      104909,
      104929,
      104949,
      104969,
      104989,
      105009,
      105029,
      105049,
      105069,
      105089,
      105109
    ],
    "core.event_count": 18,
    "core.first_event_number": 104769,
    "core.last_event_number": 105109,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39273
    ],
    "core.runs_subruns": [
      3927300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039273_0291_df-s03-d3_dw_0_20250831T132142.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039273_0291_df-s03-d3_dw_0_20250831T132142_reco_stage1_20250911T120025_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757592026.0,
    "core.end_time": 1757592026.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39273
    ],
    "core.runs_subruns": [
      3927300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039273_0291_df-s03-d3_dw_0_20250831T132142.hdf5"
    }
  ]
}
justIN time: 2025-09-19 05:04:47 UTC       justIN version: 01.05.00