justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 44681.55@dunegpschedd01.fnal.gov

Jobsub ID44681.55@dunegpschedd01.fnal.gov
Workflow ID2483
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-11 00:28:01
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce508
Last heartbeat2025-09-11 03:08:51
From worker nodeHostnameb9p08p1116.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job stateaborted
Started2025-09-11 02:06:17
Input filesvd-protodune:np02vd_raw_run039255_1374_df-s04-d1_dw_0_20250831T012408.hdf5
Outputting started2025-09-11 03:00:58
Output files
Finished2025-09-11 03:08:51
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

samples within desired window for channel 11515
wclsFrameSaver: no samples within desired window for channel 11516
wclsFrameSaver: no samples within desired window for channel 11516
wclsFrameSaver: no samples within desired window for channel 11517
wclsFrameSaver: no samples within desired window for channel 11517
wclsFrameSaver: no samples within desired window for channel 11518
wclsFrameSaver: no samples within desired window for channel 11518
wclsFrameSaver: no samples within desired window for channel 11519
wclsFrameSaver: no samples within desired window for channel 11519
wclsFrameSaver: no samples within desired window for channel 11520
wclsFrameSaver: no samples within desired window for channel 11521
wclsFrameSaver: no samples within desired window for channel 11521
wclsFrameSaver: no samples within desired window for channel 11522
wclsFrameSaver: no samples within desired window for channel 11523
wclsFrameSaver: no samples within desired window for channel 11523
wclsFrameSaver: no samples within desired window for channel 11619
wclsFrameSaver: no samples within desired window for channel 11734
wclsFrameSaver: no samples within desired window for channel 11737
wclsFrameSaver: no samples within desired window for channel 11737
wclsFrameSaver: no samples within desired window for channel 11738
wclsFrameSaver: no samples within desired window for channel 11739
wclsFrameSaver: no samples within desired window for channel 11739
wclsFrameSaver: no samples within desired window for channel 11739
wclsFrameSaver: no samples within desired window for channel 11757
wclsFrameSaver: no samples within desired window for channel 11758
wclsFrameSaver: no samples within desired window for channel 11758
wclsFrameSaver: no samples within desired window for channel 11759
wclsFrameSaver: no samples within desired window for channel 11891
wclsFrameSaver: no samples within desired window for channel 11891
wclsFrameSaver: no samples within desired window for channel 11892
wclsFrameSaver: no samples within desired window for channel 11897
wclsFrameSaver: no samples within desired window for channel 11935
wclsFrameSaver: no samples within desired window for channel 11935
wclsFrameSaver: no samples within desired window for channel 11936
wclsFrameSaver: no samples within desired window for channel 11937
wclsFrameSaver: no samples within desired window for channel 11959
wclsFrameSaver: no samples within desired window for channel 12027
wclsFrameSaver: no samples within desired window for channel 12028
FrameSaver: q=1.26349e+07 n=1244118 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 606 XUs and 801 XVs -> 33 XUVs
C:0 T:1 1149 XUs and 1354 XVs -> 48 XUVs
C:0 T:2 853 XUs and 693 XVs -> 41 XUVs
C:0 T:3 379 XUs and 537 XVs -> 25 XUVs
C:0 T:4 295 XUs and 354 XVs -> 18 XUVs
C:0 T:5 1863 XUs and 1545 XVs -> 118 XUVs
C:0 T:6 1388 XUs and 1952 XVs -> 55 XUVs
C:0 T:7 279 XUs and 362 XVs -> 27 XUVs
C:0 T:8 6936 XUs and 19582 XVs -> 2027 XUVs
C:0 T:9 5299 XUs and 10415 XVs -> 860 XUVs
C:0 T:10 7109 XUs and 8284 XVs -> 767 XUVs
C:0 T:11 1791 XUs and 2727 XVs -> 253 XUVs
C:0 T:12 14209 XUs and 19323 XVs -> 1490 XUVs
C:0 T:13 3946 XUs and 4166 XVs -> 268 XUVs
C:0 T:14 22698 XUs and 26387 XVs -> 2016 XUVs
C:0 T:15 747 XUs and 866 XVs -> 69 XUVs
8115 XUVs total
2887 collection wire objects
8115 potential space points
Neighbour search...
526307 tests to find 229994 neighbours
Iterating with no regularization...
Begin: 1.81232e+09
0 1.65268e+09
1 1.64236e+09
2 1.64091e+09
Now with regularization...
Begin: 1.61475e+09
0 1.61429e+09
11-Sep-2025 05:00:37 CEST  Closed output file "np02vd_raw_run039255_1374_df-s04-d1_dw_0_20250831T012408_reco_stage1_20250911T030037_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        109.992       179.292       289.75         179.4        40.8854        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.139e-05    9.22332e-05   0.000135391   9.5021e-05    2.70738e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               6.53164       7.39171       7.93335       7.43614      0.333152        18     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00114095    0.00221491    0.00596434    0.00154147    0.00142148       18     
produce:pdvddaphne:DAPHNEReaderPDVD               2.75056       4.04959       5.09728       4.07487      0.563656        18     
produce:ophit:OpHitFinder                        0.0298429     0.0459958     0.060373      0.0448575    0.00722722       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00569881     0.0121179     0.0196836     0.0103331    0.00429075       18     
produce:wclsdatavd:WireCellToolkit                50.002        62.1099        71.82        61.2851       5.58869        18     
produce:gaushit:GausHitFinder                    0.880803       1.38031       2.43776       1.31862      0.341703        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000269953   0.000365528   0.000491155   0.000363409   6.72669e-05      18     
produce:reco3d:SpacePointSolver                   10.3617       18.6928       30.2841       19.0783       4.35173        18     
produce:hitpdune:DisambigFromSpacePoints         0.131711      0.294366       0.61632      0.278558      0.108917        18     
produce:pandora:StandardPandora                   30.6671       78.8383       169.492       81.6797       33.2885        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.570725       1.48185       2.59435       1.40454      0.522724        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0191144     0.0309408     0.0497307     0.0291809    0.00731874       18     
[art]:TriggerResults:TriggerResultInserter       1.52e-05     2.47802e-05   6.5401e-05    2.19455e-05   1.09097e-05      18     
end_path:out1:RootOutput                         4.52e-06     1.06862e-05    2.309e-05     9.435e-06    4.23145e-06      18     
end_path:out1:RootOutput(write)                   4.23506       4.92033       6.4229        4.71173      0.619075        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5501.01 MB
  Peak resident set size usage (VmHWM): 3524.83 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039255_1374_df-s04-d1_dw_0_20250831T012408_reco_stage1_20250911T030037_keepup.root
\tHists: np02vd_raw_run039255_1374_df-s04-d1_dw_0_20250831T012408_reco_stage1_20250911T030038_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039255_1374_df-s04-d1_dw_0_20250831T012408_reco_stage1_20250911T030037_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039255_1374_df-s04-d1_dw_0_20250831T012408_reco_stage1_20250911T030037_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757559638.0,
    "core.end_time": 1757559638.0,
    "core.events": [
      495464,
      495484,
      495504,
      495524,
      495544,
      495564,
      495584,
      495604,
      495624,
      495644,
      495664,
      495684,
      495704,
      495724,
      495744,
      495764,
      495784,
      495804
    ],
    "core.event_count": 18,
    "core.first_event_number": 495464,
    "core.last_event_number": 495804,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39255
    ],
    "core.runs_subruns": [
      3925500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039255_1374_df-s04-d1_dw_0_20250831T012408.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039255_1374_df-s04-d1_dw_0_20250831T012408_reco_stage1_20250911T030038_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757559638.0,
    "core.end_time": 1757559638.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39255
    ],
    "core.runs_subruns": [
      3925500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039255_1374_df-s04-d1_dw_0_20250831T012408.hdf5"
    }
  ]
}
justIN time: 2025-09-19 13:38:14 UTC       justIN version: 01.05.00