justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 44823.149@dunegpschedd01.fnal.gov

Jobsub ID44823.149@dunegpschedd01.fnal.gov
Workflow ID2495
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-11 08:46:29
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce505
Last heartbeat2025-09-11 12:06:34
From worker nodeHostnameb9p04p1577.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job stateaborted
Started2025-09-11 10:37:59
Input filesvd-protodune:np02vd_raw_run039273_0061_df-s03-d1_dw_0_20250831T103147.hdf5
Outputting started2025-09-11 12:04:20
Output filesvd-protodune-det-reco:np02vd_raw_run039273_0061_df-s03-d1_dw_0_20250831T103147_reco_stage1_20250911T120410_keepup.root
Finished2025-09-11 12:06:34
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

for channel 11455
wclsFrameSaver: no samples within desired window for channel 11456
wclsFrameSaver: no samples within desired window for channel 11456
wclsFrameSaver: no samples within desired window for channel 11457
wclsFrameSaver: no samples within desired window for channel 11457
wclsFrameSaver: no samples within desired window for channel 11490
wclsFrameSaver: no samples within desired window for channel 11490
wclsFrameSaver: no samples within desired window for channel 11535
wclsFrameSaver: no samples within desired window for channel 11572
wclsFrameSaver: no samples within desired window for channel 11573
wclsFrameSaver: no samples within desired window for channel 11574
wclsFrameSaver: no samples within desired window for channel 11574
wclsFrameSaver: no samples within desired window for channel 11575
wclsFrameSaver: no samples within desired window for channel 11576
wclsFrameSaver: no samples within desired window for channel 11576
wclsFrameSaver: no samples within desired window for channel 11748
wclsFrameSaver: no samples within desired window for channel 11748
wclsFrameSaver: no samples within desired window for channel 11749
wclsFrameSaver: no samples within desired window for channel 11749
wclsFrameSaver: no samples within desired window for channel 11750
wclsFrameSaver: no samples within desired window for channel 11751
wclsFrameSaver: no samples within desired window for channel 11751
wclsFrameSaver: no samples within desired window for channel 11751
wclsFrameSaver: no samples within desired window for channel 11752
wclsFrameSaver: no samples within desired window for channel 11755
wclsFrameSaver: no samples within desired window for channel 12086
wclsFrameSaver: no samples within desired window for channel 12087
wclsFrameSaver: no samples within desired window for channel 12087
wclsFrameSaver: no samples within desired window for channel 12088
wclsFrameSaver: no samples within desired window for channel 12088
wclsFrameSaver: no samples within desired window for channel 12088
wclsFrameSaver: no samples within desired window for channel 12089
wclsFrameSaver: no samples within desired window for channel 12089
wclsFrameSaver: no samples within desired window for channel 12089
wclsFrameSaver: no samples within desired window for channel 12090
wclsFrameSaver: no samples within desired window for channel 12090
wclsFrameSaver: no samples within desired window for channel 12091
wclsFrameSaver: no samples within desired window for channel 12092
wclsFrameSaver: no samples within desired window for channel 12093
FrameSaver: q=9.13806e+06 n=960768 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 211 XUs and 249 XVs -> 12 XUVs
C:0 T:1 195 XUs and 186 XVs -> 12 XUVs
C:0 T:2 244 XUs and 326 XVs -> 9 XUVs
C:0 T:3 746 XUs and 773 XVs -> 37 XUVs
C:0 T:4 560 XUs and 762 XVs -> 42 XUVs
C:0 T:5 819 XUs and 744 XVs -> 51 XUVs
C:0 T:6 603 XUs and 660 XVs -> 24 XUVs
C:0 T:7 491 XUs and 674 XVs -> 23 XUVs
C:0 T:8 17706 XUs and 32757 XVs -> 4170 XUVs
C:0 T:9 1639 XUs and 2976 XVs -> 187 XUVs
C:0 T:10 701 XUs and 1328 XVs -> 51 XUVs
C:0 T:11 769 XUs and 900 XVs -> 85 XUVs
C:0 T:12 1361 XUs and 1283 XVs -> 111 XUVs
C:0 T:13 6508 XUs and 5214 XVs -> 445 XUVs
C:0 T:14 3153 XUs and 3217 XVs -> 296 XUVs
C:0 T:15 1608 XUs and 2172 XVs -> 122 XUVs
5677 XUVs total
1500 collection wire objects
5677 potential space points
Neighbour search...
667895 tests to find 260018 neighbours
Iterating with no regularization...
Begin: 6.61726e+08
0 6.09659e+08
1 6.06428e+08
2 6.05942e+08
Now with regularization...
Begin: 5.93782e+08
0 5.93496e+08
11-Sep-2025 14:04:10 CEST  Closed output file "np02vd_raw_run039273_0061_df-s03-d1_dw_0_20250831T103147_reco_stage1_20250911T120410_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        141.453       285.036       1428.72       211.86        283.899        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.002e-05    8.5017e-05    0.000183541   7.41005e-05   3.03704e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               6.14962       6.45661       7.08581       6.42828      0.211491        18     
produce:triggerrawdecoder:PDVDTriggerReader4    0.000812434   0.00387635     0.0191406    0.00124997    0.00547063       18     
produce:pdvddaphne:DAPHNEReaderPDVD               3.31048       4.1526        4.98959       4.24122      0.481269        18     
produce:ophit:OpHitFinder                        0.0341642     0.0484853     0.0589087     0.0493247     0.0065434       18     
produce:opflash:OpFlashFinderVerticalDrift       0.0072827     0.0128585     0.0201797     0.0123588    0.00348919       18     
produce:wclsdatavd:WireCellToolkit                60.7726       72.3836       85.7355       71.4126       6.6465         18     
produce:gaushit:GausHitFinder                     1.08449       1.50794       2.45672       1.46207      0.346049        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000223451   0.000359426   0.000524223   0.000371702   7.90156e-05      18     
produce:reco3d:SpacePointSolver                   13.7184       21.4658       34.1887       21.3812       6.1759         18     
produce:hitpdune:DisambigFromSpacePoints         0.189278      0.377735      0.599296      0.365466      0.147503        18     
produce:pandora:StandardPandora                   43.1404       171.933       1288.2        98.3232       275.664        18     
produce:pandoraTrack:LArPandoraTrackCreation      0.86267       1.47434       2.38156       1.29203      0.519764        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0212744     0.0365214     0.0676197     0.0322961     0.0122116       18     
[art]:TriggerResults:TriggerResultInserter       1.777e-05    2.98617e-05    7.961e-05    2.51005e-05   1.44323e-05      18     
end_path:out1:RootOutput                         5.61e-06     1.24558e-05    3.629e-05     7.24e-06     1.04549e-05      18     
end_path:out1:RootOutput(write)                   4.29258       5.1452        6.62624       4.7798        0.79525        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5218.5 MB
  Peak resident set size usage (VmHWM): 3279.54 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039273_0061_df-s03-d1_dw_0_20250831T103147_reco_stage1_20250911T120410_keepup.root
\tHists: np02vd_raw_run039273_0061_df-s03-d1_dw_0_20250831T103147_reco_stage1_20250911T120410_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039273_0061_df-s03-d1_dw_0_20250831T103147_reco_stage1_20250911T120410_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039273_0061_df-s03-d1_dw_0_20250831T103147_reco_stage1_20250911T120410_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757592250.0,
    "core.end_time": 1757592250.0,
    "core.events": [
      21967,
      21987,
      22007,
      22027,
      22047,
      22067,
      22087,
      22107,
      22127,
      22147,
      22167,
      22187,
      22207,
      22227,
      22247,
      22267,
      22287,
      22307
    ],
    "core.event_count": 18,
    "core.first_event_number": 21967,
    "core.last_event_number": 22307,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39273
    ],
    "core.runs_subruns": [
      3927300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039273_0061_df-s03-d1_dw_0_20250831T103147.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039273_0061_df-s03-d1_dw_0_20250831T103147_reco_stage1_20250911T120410_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757592250.0,
    "core.end_time": 1757592250.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39273
    ],
    "core.runs_subruns": [
      3927300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039273_0061_df-s03-d1_dw_0_20250831T103147.hdf5"
    }
  ]
}
justIN time: 2025-09-19 07:33:38 UTC       justIN version: 01.05.00