justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 39334.163@dunegpschedd02.fnal.gov

Jobsub ID39334.163@dunegpschedd02.fnal.gov
Workflow ID2576
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-13 15:32:17
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2025-09-13 18:14:43
From worker nodeHostnameb9p16p7012.cern.ch
cpuinfoIntel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statestalled
Started2025-09-13 16:46:29
Input filesvd-protodune:np02vd_raw_run039275_0133_df-s03-d1_dw_0_20250901T185756.hdf5
Outputting started2025-09-13 18:14:43
Output files
Finished2025-09-13 18:57:41
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

rameSaver: no samples within desired window for channel 7145
wclsFrameSaver: no samples within desired window for channel 7145
wclsFrameSaver: no samples within desired window for channel 7176
wclsFrameSaver: no samples within desired window for channel 7310
wclsFrameSaver: no samples within desired window for channel 7634
wclsFrameSaver: no samples within desired window for channel 7812
wclsFrameSaver: no samples within desired window for channel 7833
wclsFrameSaver: no samples within desired window for channel 7868
wclsFrameSaver: no samples within desired window for channel 7884
wclsFrameSaver: no samples within desired window for channel 8224
wclsFrameSaver: no samples within desired window for channel 9844
wclsFrameSaver: no samples within desired window for channel 11072
wclsFrameSaver: no samples within desired window for channel 11087
FrameSaver: q=9.01706e+06 n=1159859 tag=gauss
wclsFrameSaver: saving 70893 traces tagged "wiener"
wclsFrameSaver: no samples within desired window for channel 6244
wclsFrameSaver: no samples within desired window for channel 6408
wclsFrameSaver: no samples within desired window for channel 6454
wclsFrameSaver: no samples within desired window for channel 6744
wclsFrameSaver: no samples within desired window for channel 6835
wclsFrameSaver: no samples within desired window for channel 7135
wclsFrameSaver: no samples within desired window for channel 7143
wclsFrameSaver: no samples within desired window for channel 7143
wclsFrameSaver: no samples within desired window for channel 7144
wclsFrameSaver: no samples within desired window for channel 7144
wclsFrameSaver: no samples within desired window for channel 7145
wclsFrameSaver: no samples within desired window for channel 7145
wclsFrameSaver: no samples within desired window for channel 7145
wclsFrameSaver: no samples within desired window for channel 7176
wclsFrameSaver: no samples within desired window for channel 7310
wclsFrameSaver: no samples within desired window for channel 7634
wclsFrameSaver: no samples within desired window for channel 7812
wclsFrameSaver: no samples within desired window for channel 7833
wclsFrameSaver: no samples within desired window for channel 7868
wclsFrameSaver: no samples within desired window for channel 7884
wclsFrameSaver: no samples within desired window for channel 8224
wclsFrameSaver: no samples within desired window for channel 9844
wclsFrameSaver: no samples within desired window for channel 11072
wclsFrameSaver: no samples within desired window for channel 11087
FrameSaver: q=9.6268e+06 n=1108237 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 970 XUs and 852 XVs -> 56 XUVs
C:0 T:1 434 XUs and 517 XVs -> 22 XUVs
C:0 T:2 491 XUs and 501 XVs -> 21 XUVs
C:0 T:3 2416 XUs and 2888 XVs -> 96 XUVs
C:0 T:4 844 XUs and 819 XVs -> 52 XUVs
C:0 T:5 940 XUs and 1055 XVs -> 75 XUVs
C:0 T:6 1170 XUs and 1116 XVs -> 35 XUVs
C:0 T:7 2426 XUs and 2164 XVs -> 74 XUVs
C:0 T:8 16761 XUs and 43468 XVs -> 5585 XUVs
C:0 T:9 830 XUs and 687 XVs -> 66 XUVs
C:0 T:10 1712 XUs and 2443 XVs -> 249 XUVs
C:0 T:11 1018 XUs and 1143 XVs -> 46 XUVs
C:0 T:12 2414 XUs and 2100 XVs -> 213 XUVs
C:0 T:13 801 XUs and 1038 XVs -> 132 XUVs
C:0 T:14 5774 XUs and 8287 XVs -> 838 XUVs
C:0 T:15 13364 XUs and 14071 XVs -> 2403 XUVs
9963 XUVs total
1902 collection wire objects
9963 potential space points
Neighbour search...
1187491 tests to find 526990 neighbours
Iterating with no regularization...
Begin: 6.56718e+08
0 6.09082e+08
1 6.04386e+08
2 6.03111e+08
3 6.0262e+08
Now with regularization...
Begin: 5.88645e+08
0 5.88352e+08
13-Sep-2025 20:14:30 CEST  Closed output file "np02vd_raw_run039275_0133_df-s03-d1_dw_0_20250901T185756_reco_stage1_20250913T181430_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        188.196       291.187       475.853       286.031       70.4181        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.9717e-05    9.95863e-05   0.000194096   9.14905e-05   3.61681e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               7.47433       8.74008       11.9479       8.60778       1.05682        18     
produce:triggerrawdecoder:PDVDTriggerReader4    0.000997416   0.00200403    0.00726204    0.00132856    0.00180407       18     
produce:pdvddaphne:DAPHNEReaderPDVD               8.29692       9.62009       11.674        9.50385      0.807574        18     
produce:ophit:OpHitFinder                        0.0560548     0.0734842     0.120201      0.0643427     0.0176525       18     
produce:opflash:OpFlashFinderVerticalDrift       0.0101338     0.0195558     0.0306941     0.0189819    0.00496928       18     
produce:wclsdatavd:WireCellToolkit                99.8333       111.155       123.648       110.42        6.65598        18     
produce:gaushit:GausHitFinder                     1.1998        2.24029       3.92886       2.09624       0.58629        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000361455   0.000580723   0.000762328   0.000624913   0.000126248      18     
produce:reco3d:SpacePointSolver                   11.2103       22.6924       33.0323       22.8041       6.48448        18     
produce:hitpdune:DisambigFromSpacePoints         0.185756      0.431682      0.718895      0.436202      0.143976        18     
produce:pandora:StandardPandora                   43.2277       127.189       296.357       125.926       62.6679        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.921284       2.42506       4.2894        2.44894      0.988309        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.023442      0.0448849     0.0834409     0.0430465     0.0158196       18     
[art]:TriggerResults:TriggerResultInserter      2.3617e-05     3.611e-05    0.000104783   3.2205e-05    1.7501e-05       18     
end_path:out1:RootOutput                         7.335e-06    1.06152e-05   3.5707e-05     8.725e-06    6.29715e-06      18     
end_path:out1:RootOutput(write)                   5.64882       6.49183       10.1308       6.29338       0.96859        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5062.15 MB
  Peak resident set size usage (VmHWM): 3114.16 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039275_0133_df-s03-d1_dw_0_20250901T185756_reco_stage1_20250913T181430_keepup.root
\tHists: np02vd_raw_run039275_0133_df-s03-d1_dw_0_20250901T185756_reco_stage1_20250913T181430_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039275_0133_df-s03-d1_dw_0_20250901T185756_reco_stage1_20250913T181430_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039275_0133_df-s03-d1_dw_0_20250901T185756_reco_stage1_20250913T181430_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757787271.0,
    "core.end_time": 1757787271.0,
    "core.events": [
      47887,
      47907,
      47927,
      47947,
      47967,
      47987,
      48007,
      48027,
      48047,
      48067,
      48087,
      48107,
      48127,
      48147,
      48167,
      48187,
      48207,
      48227
    ],
    "core.event_count": 18,
    "core.first_event_number": 47887,
    "core.last_event_number": 48227,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39275
    ],
    "core.runs_subruns": [
      3927500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039275_0133_df-s03-d1_dw_0_20250901T185756.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039275_0133_df-s03-d1_dw_0_20250901T185756_reco_stage1_20250913T181430_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757787271.0,
    "core.end_time": 1757787271.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39275
    ],
    "core.runs_subruns": [
      3927500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039275_0133_df-s03-d1_dw_0_20250901T185756.hdf5"
    }
  ]
}
justIN time: 2025-09-19 13:36:56 UTC       justIN version: 01.05.00