justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 39335.143@dunegpschedd02.fnal.gov

Jobsub ID39335.143@dunegpschedd02.fnal.gov
Workflow ID2576
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-13 15:34:17
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2025-09-13 18:12:58
From worker nodeHostnameb9p09p3516.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job stateaborted
Started2025-09-13 16:48:54
Input filesvd-protodune:np02vd_raw_run039275_0140_df-s03-d2_dw_0_20250901T190618.hdf5
Outputting started2025-09-13 18:06:32
Output filesvd-protodune-det-reco:np02vd_raw_run039275_0140_df-s03-d2_dw_0_20250901T190618_reco_stage1_20250913T180619_keepup.root
Finished2025-09-13 18:12:58
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ndow for channel 10764
wclsFrameSaver: no samples within desired window for channel 10833
wclsFrameSaver: no samples within desired window for channel 11100
wclsFrameSaver: no samples within desired window for channel 11119
wclsFrameSaver: no samples within desired window for channel 11142
wclsFrameSaver: no samples within desired window for channel 11475
wclsFrameSaver: no samples within desired window for channel 11475
wclsFrameSaver: no samples within desired window for channel 11479
wclsFrameSaver: no samples within desired window for channel 11480
wclsFrameSaver: no samples within desired window for channel 11480
wclsFrameSaver: no samples within desired window for channel 11480
wclsFrameSaver: no samples within desired window for channel 11481
wclsFrameSaver: no samples within desired window for channel 11481
wclsFrameSaver: no samples within desired window for channel 11482
wclsFrameSaver: no samples within desired window for channel 11482
wclsFrameSaver: no samples within desired window for channel 11482
wclsFrameSaver: no samples within desired window for channel 11488
wclsFrameSaver: no samples within desired window for channel 11586
wclsFrameSaver: no samples within desired window for channel 11587
wclsFrameSaver: no samples within desired window for channel 11587
wclsFrameSaver: no samples within desired window for channel 11588
wclsFrameSaver: no samples within desired window for channel 11588
wclsFrameSaver: no samples within desired window for channel 11589
wclsFrameSaver: no samples within desired window for channel 11589
wclsFrameSaver: no samples within desired window for channel 11590
wclsFrameSaver: no samples within desired window for channel 11590
wclsFrameSaver: no samples within desired window for channel 11849
wclsFrameSaver: no samples within desired window for channel 11849
wclsFrameSaver: no samples within desired window for channel 11850
wclsFrameSaver: no samples within desired window for channel 11850
wclsFrameSaver: no samples within desired window for channel 11851
wclsFrameSaver: no samples within desired window for channel 11851
wclsFrameSaver: no samples within desired window for channel 11852
wclsFrameSaver: no samples within desired window for channel 11852
wclsFrameSaver: no samples within desired window for channel 11853
wclsFrameSaver: no samples within desired window for channel 11854
wclsFrameSaver: no samples within desired window for channel 11898
FrameSaver: q=1.06674e+07 n=1119433 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 561 XUs and 789 XVs -> 29 XUVs
C:0 T:1 391 XUs and 472 XVs -> 27 XUVs
C:0 T:2 799 XUs and 1708 XVs -> 67 XUVs
C:0 T:3 295 XUs and 418 XVs -> 14 XUVs
C:0 T:4 1310 XUs and 2174 XVs -> 95 XUVs
C:0 T:5 1575 XUs and 2157 XVs -> 137 XUVs
C:0 T:6 651 XUs and 1626 XVs -> 79 XUVs
C:0 T:7 1389 XUs and 2102 XVs -> 54 XUVs
C:0 T:8 12070 XUs and 28760 XVs -> 4920 XUVs
C:0 T:9 10147 XUs and 14160 XVs -> 1745 XUVs
C:0 T:10 9621 XUs and 16141 XVs -> 2324 XUVs
C:0 T:11 12500 XUs and 20499 XVs -> 2464 XUVs
C:0 T:12 13464 XUs and 24134 XVs -> 3496 XUVs
C:0 T:13 5479 XUs and 13802 XVs -> 1032 XUVs
C:0 T:14 3906 XUs and 5656 XVs -> 1371 XUVs
C:0 T:15 16883 XUs and 26566 XVs -> 5191 XUVs
23045 XUVs total
3156 collection wire objects
23045 potential space points
Neighbour search...
3148841 tests to find 1325690 neighbours
Iterating with no regularization...
Begin: 6.32446e+08
0 5.34362e+08
1 5.25788e+08
2 5.24369e+08
3 5.23862e+08
Now with regularization...
Begin: 5.04854e+08
0 5.04423e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
13-Sep-2025 20:06:19 CEST  Closed output file "np02vd_raw_run039275_0140_df-s03-d2_dw_0_20250901T190618_reco_stage1_20250913T180619_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        141.941       255.625       517.121       229.435       96.118         18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.5151e-05    0.000132828   0.000346541   0.00013052    6.62849e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               6.29289       8.03362       21.8789       7.21353       3.37733        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0011853    0.00146011    0.00279662    0.00129606    0.000395024      18     
produce:pdvddaphne:DAPHNEReaderPDVD               3.87889       4.79062       5.62734       4.72963      0.612499        18     
produce:ophit:OpHitFinder                        0.057456      0.0705252     0.0839817     0.0698554    0.00678275       18     
produce:opflash:OpFlashFinderVerticalDrift       0.0087461     0.0164672     0.0230819     0.0155715    0.00419236       18     
produce:wclsdatavd:WireCellToolkit                 66.69        84.6414       109.039       84.8156       10.3761        18     
produce:gaushit:GausHitFinder                    0.952528       1.96334       3.00462       1.88268      0.545575        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000228091   0.00046077    0.000658251   0.000435086   0.000116697      18     
produce:reco3d:SpacePointSolver                   11.6575       23.7969       42.7516       22.6085       7.82676        18     
produce:hitpdune:DisambigFromSpacePoints         0.156601       0.44172      0.768748      0.429764      0.167534        18     
produce:pandora:StandardPandora                   37.3378       123.148       336.945       101.223       79.1488        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.750747       1.69719       3.6305        1.64823      0.630417        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0236827     0.0500043     0.0773478     0.0494912     0.0120603       18     
[art]:TriggerResults:TriggerResultInserter       2.174e-05    3.78769e-05    9.904e-05    3.49055e-05   1.73386e-05      18     
end_path:out1:RootOutput                         6.31e-06     1.12668e-05    3.059e-05    8.7455e-06    6.37789e-06      18     
end_path:out1:RootOutput(write)                   4.72554       6.92427       8.23589       7.06712      0.794069        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5229.3 MB
  Peak resident set size usage (VmHWM): 3299.26 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039275_0140_df-s03-d2_dw_0_20250901T190618_reco_stage1_20250913T180619_keepup.root
\tHists: np02vd_raw_run039275_0140_df-s03-d2_dw_0_20250901T190618_reco_stage1_20250913T180619_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039275_0140_df-s03-d2_dw_0_20250901T190618_reco_stage1_20250913T180619_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039275_0140_df-s03-d2_dw_0_20250901T190618_reco_stage1_20250913T180619_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757786780.0,
    "core.end_time": 1757786780.0,
    "core.events": [
      50408,
      50428,
      50448,
      50468,
      50488,
      50508,
      50528,
      50548,
      50568,
      50588,
      50608,
      50628,
      50648,
      50668,
      50688,
      50708,
      50728,
      50748
    ],
    "core.event_count": 18,
    "core.first_event_number": 50408,
    "core.last_event_number": 50748,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39275
    ],
    "core.runs_subruns": [
      3927500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039275_0140_df-s03-d2_dw_0_20250901T190618.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039275_0140_df-s03-d2_dw_0_20250901T190618_reco_stage1_20250913T180619_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757786780.0,
    "core.end_time": 1757786780.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39275
    ],
    "core.runs_subruns": [
      3927500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039275_0140_df-s03-d2_dw_0_20250901T190618.hdf5"
    }
  ]
}
justIN time: 2025-09-19 13:39:21 UTC       justIN version: 01.05.00