justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 37932.123@dunegpschedd02.fnal.gov

Jobsub ID37932.123@dunegpschedd02.fnal.gov
Workflow ID2484
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-11 00:26:06
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce508
Last heartbeat2025-09-11 03:07:02
From worker nodeHostnameb9p07p6748.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job stateaborted
Started2025-09-11 01:21:21
Input filesvd-protodune:np02vd_raw_run039255_1597_df-s03-d0_dw_0_20250831T041236.hdf5
Outputting started2025-09-11 02:51:21
Output files
Finished2025-09-11 03:07:02
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

mples within desired window for channel 9468
wclsFrameSaver: no samples within desired window for channel 9469
wclsFrameSaver: no samples within desired window for channel 9470
wclsFrameSaver: no samples within desired window for channel 9470
wclsFrameSaver: no samples within desired window for channel 9471
wclsFrameSaver: no samples within desired window for channel 9867
wclsFrameSaver: no samples within desired window for channel 10007
wclsFrameSaver: no samples within desired window for channel 10008
wclsFrameSaver: no samples within desired window for channel 10009
wclsFrameSaver: no samples within desired window for channel 10009
wclsFrameSaver: no samples within desired window for channel 10208
wclsFrameSaver: no samples within desired window for channel 10416
wclsFrameSaver: no samples within desired window for channel 10417
wclsFrameSaver: no samples within desired window for channel 10716
wclsFrameSaver: no samples within desired window for channel 10813
wclsFrameSaver: no samples within desired window for channel 10912
wclsFrameSaver: no samples within desired window for channel 10914
wclsFrameSaver: no samples within desired window for channel 10915
wclsFrameSaver: no samples within desired window for channel 10915
wclsFrameSaver: no samples within desired window for channel 10941
wclsFrameSaver: no samples within desired window for channel 11291
wclsFrameSaver: no samples within desired window for channel 11291
wclsFrameSaver: no samples within desired window for channel 11292
wclsFrameSaver: no samples within desired window for channel 11293
wclsFrameSaver: no samples within desired window for channel 11563
wclsFrameSaver: no samples within desired window for channel 11565
wclsFrameSaver: no samples within desired window for channel 11566
wclsFrameSaver: no samples within desired window for channel 11567
wclsFrameSaver: no samples within desired window for channel 11568
wclsFrameSaver: no samples within desired window for channel 11569
wclsFrameSaver: no samples within desired window for channel 11569
wclsFrameSaver: no samples within desired window for channel 11570
wclsFrameSaver: no samples within desired window for channel 11570
wclsFrameSaver: no samples within desired window for channel 11571
wclsFrameSaver: no samples within desired window for channel 11571
wclsFrameSaver: no samples within desired window for channel 12208
wclsFrameSaver: no samples within desired window for channel 12208
wclsFrameSaver: no samples within desired window for channel 12211
FrameSaver: q=1.37933e+07 n=1395882 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1371 XUs and 1394 XVs -> 41 XUVs
C:0 T:1 660 XUs and 481 XVs -> 15 XUVs
C:0 T:2 74 XUs and 79 XVs -> 5 XUVs
C:0 T:3 1130 XUs and 1460 XVs -> 71 XUVs
C:0 T:4 1731 XUs and 2300 XVs -> 59 XUVs
C:0 T:5 2914 XUs and 2789 XVs -> 240 XUVs
C:0 T:6 411 XUs and 586 XVs -> 17 XUVs
C:0 T:7 80 XUs and 182 XVs -> 2 XUVs
C:0 T:8 28061 XUs and 54859 XVs -> 7768 XUVs
C:0 T:9 11833 XUs and 21213 XVs -> 2875 XUVs
C:0 T:10 13588 XUs and 28164 XVs -> 6633 XUVs
C:0 T:11 14937 XUs and 27168 XVs -> 1275 XUVs
C:0 T:12 7112 XUs and 7051 XVs -> 564 XUVs
C:0 T:13 3261 XUs and 4790 XVs -> 803 XUVs
C:0 T:14 9662 XUs and 11208 XVs -> 1068 XUVs
C:0 T:15 3205 XUs and 2507 XVs -> 413 XUVs
21849 XUVs total
3190 collection wire objects
21849 potential space points
Neighbour search...
3372591 tests to find 1313186 neighbours
Iterating with no regularization...
Begin: 2.54633e+09
0 2.28392e+09
1 2.26386e+09
2 2.2628e+09
Now with regularization...
Begin: 2.23513e+09
0 2.23465e+09
11-Sep-2025 04:51:08 CEST  Closed output file "np02vd_raw_run039255_1597_df-s03-d0_dw_0_20250831T041236_reco_stage1_20250911T025108_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        182.551       296.872       562.31        268.582       91.1928        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      7.7911e-05    0.000130145   0.000414784   0.000113346   7.29107e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               7.13224       8.90616       22.6575       7.52691       3.60244        18     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00128129    0.00352684     0.0192449    0.00162426    0.00491575       18     
produce:pdvddaphne:DAPHNEReaderPDVD               4.04158       4.95085       6.18651       4.91653      0.561421        18     
produce:ophit:OpHitFinder                        0.050183      0.0588463     0.0688627     0.0581611    0.00536149       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00921596     0.0170126     0.0231802     0.0170252    0.00367328       18     
produce:wclsdatavd:WireCellToolkit                73.2289       89.5711       118.558       87.0063       9.95883        18     
produce:gaushit:GausHitFinder                     1.60574       2.19315       2.89118       2.19494      0.373786        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000281663   0.000485716   0.000861609   0.000462015   0.000115518      18     
produce:reco3d:SpacePointSolver                   15.1071       28.3642       48.6493       28.4505       7.71165        18     
produce:hitpdune:DisambigFromSpacePoints         0.330384      0.530504      0.894297      0.502473      0.157632        18     
produce:pandora:StandardPandora                   63.8096       153.096       352.328       134.019       73.9786        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.805997       2.28949       3.9258        2.27568      0.799577        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0362975     0.0561684     0.0778298     0.0568657     0.011071        18     
[art]:TriggerResults:TriggerResultInserter       2.996e-05    4.17182e-05   7.6621e-05    3.6925e-05    1.2377e-05       18     
end_path:out1:RootOutput                         5.44e-06     1.04951e-05    3.027e-05      9.5e-06     4.98353e-06      18     
end_path:out1:RootOutput(write)                   5.30793       6.7837        7.82163       6.98905      0.745356        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5654.53 MB
  Peak resident set size usage (VmHWM): 3696.19 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039255_1597_df-s03-d0_dw_0_20250831T041236_reco_stage1_20250911T025108_keepup.root
\tHists: np02vd_raw_run039255_1597_df-s03-d0_dw_0_20250831T041236_reco_stage1_20250911T025108_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039255_1597_df-s03-d0_dw_0_20250831T041236_reco_stage1_20250911T025108_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039255_1597_df-s03-d0_dw_0_20250831T041236_reco_stage1_20250911T025108_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757559069.0,
    "core.end_time": 1757559069.0,
    "core.events": [
      575879,
      575899,
      575919,
      575939,
      575959,
      575979,
      575999,
      576019,
      576039,
      576059,
      576079,
      576099,
      576119,
      576139,
      576159,
      576179,
      576199,
      576219
    ],
    "core.event_count": 18,
    "core.first_event_number": 575879,
    "core.last_event_number": 576219,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39255
    ],
    "core.runs_subruns": [
      3925500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039255_1597_df-s03-d0_dw_0_20250831T041236.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039255_1597_df-s03-d0_dw_0_20250831T041236_reco_stage1_20250911T025108_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757559069.0,
    "core.end_time": 1757559069.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39255
    ],
    "core.runs_subruns": [
      3925500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039255_1597_df-s03-d0_dw_0_20250831T041236.hdf5"
    }
  ]
}
justIN time: 2025-09-19 13:39:38 UTC       justIN version: 01.05.00