justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.88@dunegpschedd01.fnal.gov

Jobsub ID303647.88@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-03 23:33:41
From worker nodeHostnameb9p06p2461.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:24
Input filesvd-protodune:np02vd_raw_run042421_0156_df-s04-d0_dw_0_20260203T145521.hdf5
JobscriptExit code0
Real time43m (2627s)
CPU time41m (2510s = 95%)
Max RSS bytes2622959616 (2501 MiB)
Outputting started2026-02-03 23:31:12
Output files
Finished2026-02-03 23:33:41
Saved logsjustin-logs:303647.88-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ccd30
Caught exception in looking up space point ptr... 0x251d15f0, 0x5f8ccf10
Caught exception in looking up space point ptr... 0x38179ee0, 0x3817a390
Caught exception in looking up space point ptr... 0x3817a390, 0x38179ee0
Caught exception in looking up space point ptr... 0x426aa6a0, 0x45c683d0
Caught exception in looking up space point ptr... 0x45c683d0, 0x426aa6a0
Caught exception in looking up space point ptr... 0x15e28c20, 0x15e28db0
Caught exception in looking up space point ptr... 0x15e28db0, 0x15e28c20
Caught exception in looking up space point ptr... 0x420b8a90, 0x38128730
Caught exception in looking up space point ptr... 0x38128730, 0x420b8a90
Caught exception in looking up space point ptr... 0x46cac2f0, 0x46cac160
Caught exception in looking up space point ptr... 0x46cac160, 0x46cac2f0
Caught exception in looking up space point ptr... 0x46cac2f0, 0x46cac480
Caught exception in looking up space point ptr... 0x46cac480, 0x46cac2f0
Caught exception in looking up space point ptr... 0x462333c0, 0x46232e60
Caught exception in looking up space point ptr... 0x46232e60, 0x462333c0
Caught exception in looking up space point ptr... 0x390316b0, 0x60c076a0
Caught exception in looking up space point ptr... 0x60c076a0, 0x5300ea30
Caught exception in looking up space point ptr... 0x38cf06c0, 0x38cf1330
Caught exception in looking up space point ptr... 0x38cf1330, 0x38cf06c0
Caught exception in looking up space point ptr... 0x46cb82d0, 0x48ae74a0
Caught exception in looking up space point ptr... 0x48ae74a0, 0x46cb8790
Caught exception in looking up space point ptr... 0x3410b920, 0x29817cf0
Caught exception in looking up space point ptr... 0x29817cf0, 0x3add4a40
Caught exception in looking up space point ptr... 0x32ece630, 0x32ece8f0
Caught exception in looking up space point ptr... 0x32ece8f0, 0x32ecb0e0
Caught exception in looking up space point ptr... 0x2c0705b0, 0x427f96f0
Caught exception in looking up space point ptr... 0x427f96f0, 0x2c0705b0
Caught exception in looking up space point ptr... 0x47868a40, 0x47868940
Caught exception in looking up space point ptr... 0x47868940, 0x47868a40
Caught exception in looking up space point ptr... 0xd0298c0, 0xd029410
Caught exception in looking up space point ptr... 0xd029410, 0xd0298c0
Caught exception in looking up space point ptr... 0x297f8550, 0x297f82e0
Caught exception in looking up space point ptr... 0x297f82e0, 0x297f8550
Caught exception in looking up space point ptr... 0x49bb7fd0, 0x8563750
Caught exception in looking up space point ptr... 0x8563750, 0x49bb7fd0
Caught exception in looking up space point ptr... 0x2a7235f0, 0x37242390
Caught exception in looking up space point ptr... 0x37242390, 0x2a7235f0
Caught exception in looking up space point ptr... 0x3acaa3c0, 0x3aca94f0
Caught exception in looking up space point ptr... 0x3aca94f0, 0x38422540
Caught exception in looking up space point ptr... 0xd811630, 0x14918eb0
Caught exception in looking up space point ptr... 0x14918eb0, 0x521e8060
Caught exception in looking up space point ptr... 0x5067f270, 0x2611ab50
Caught exception in looking up space point ptr... 0x2611ab50, 0x5067f270
Caught exception in looking up space point ptr... 0x5caaf2e0, 0x3646e000
Caught exception in looking up space point ptr... 0x3646e000, 0x5caaf2e0
Caught exception in looking up space point ptr... 0x5da5ffa0, 0x38ca1e10
Caught exception in looking up space point ptr... 0x38ca1e10, 0x5da60170
Caught exception in looking up space point ptr... 0x42556dd0, 0x42556ab0
Caught exception in looking up space point ptr... 0x42556ab0, 0x42556dd0
Caught exception in looking up space point ptr... 0x42556dd0, 0x42556920
Caught exception in looking up space point ptr... 0x42556920, 0x42556dd0
Caught exception in looking up space point ptr... 0xae8a680, 0xae89790
Caught exception in looking up space point ptr... 0xae89790, 0x2c8dba10
Caught exception in looking up space point ptr... 0x47867f40, 0x47867d40
Caught exception in looking up space point ptr... 0x47867d40, 0x47867f40
Caught exception in looking up space point ptr... 0x37f5ab00, 0x37f5a970
Caught exception in looking up space point ptr... 0x37f5a970, 0x37f5ab00
Caught exception in looking up space point ptr... 0x2a72cef0, 0x41f57fe0
Caught exception in looking up space point ptr... 0x41f57fe0, 0x2a72cef0
Caught exception in looking up space point ptr... 0x2a72cef0, 0x41f57cc0
Caught exception in looking up space point ptr... 0x41f57cc0, 0x2a72cef0
Caught exception in looking up space point ptr... 0x1494dd50, 0xf661a30
Caught exception in looking up space point ptr... 0xf661a30, 0x1494dd50
Caught exception in looking up space point ptr... 0x3f061680, 0x3f061800
Caught exception in looking up space point ptr... 0x3f061800, 0x3f061680
++++>>>> total num hits: 76390, num free: 72225
04-Feb-2026 00:31:03 CET  Closed output file "np02vd_raw_run042421_0156_df-s04-d0_dw_0_20260203T145521_reco_stage0_20260203T233103_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         77.5312       94.833        110.991       94.911        9.3783         27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.419e-05    0.000100474   0.000316323   8.0021e-05    5.52683e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.42068       4.82085       5.52341       4.78113      0.208488        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00992932     0.0207182     0.0502052     0.0142422     0.0124011       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000404924   0.000565646    0.0009726    0.000512636   0.000129932      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00145495    0.00201157    0.00489568    0.00180743    0.000640905      27     
produce:beamevent:BeamEvent                      6.1371e-05    0.000137259    0.0010865    9.7811e-05    0.000188152      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.28355       2.95133       3.29067       3.04838      0.271751        27     
produce:ophit:OpHitFinder                         0.0319016     0.0388368     0.0458113     0.0383533    0.00387384       27     
produce:wclsdatavd:WireCellToolkit                 57.7735       68.2444       75.0303       69.5108       4.58016        27     
produce:gaushit:GausHitFinder                     0.544878       1.00073       1.64467      0.909279      0.289865        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000142631   0.000278555   0.000748688   0.000249773   0.000115231      27     
produce:reco3d:SpacePointSolver                    5.8374        11.6136       21.5552       11.537        4.6212         27     
produce:hitpdune:DisambigFromSpacePoints          0.0512128     0.114086      0.235793      0.103368      0.0453319       27     
produce:cluster3d:Cluster3D                       0.989504       2.00802       5.03321       1.77123      0.948391        27     
[art]:TriggerResults:TriggerResultInserter        2.462e-05    3.92324e-05   0.000147282   3.3451e-05    2.44477e-05      27     
end_path:out1:RootOutput                          4.05e-06     1.02968e-05    3.704e-05     9.12e-06     7.29173e-06      27     
end_path:out1:RootOutput(write)                    2.93499       3.99486       4.93148       4.11621      0.582294        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4578.92 MB
  Peak resident set size usage (VmHWM): 2622.96 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0156_df-s04-d0_dw_0_20260203T145521_reco_stage0_20260203T233103_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0156_df-s04-d0_dw_0_20260203T145521_reco_stage0_20260203T233103_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0156_df-s04-d0_dw_0_20260203T145521_reco_stage0_20260203T233103_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158857.0,
    "core.end_time": 1770161464.0,
    "core.events": [
      25342,
      25348,
      25354,
      25360,
      25366,
      25372,
      25378,
      25384,
      25390,
      25396,
      25402,
      25408,
      25414,
      25420,
      25426,
      25432,
      25438,
      25444,
      25450,
      25456,
      25462,
      25468,
      25474,
      25480,
      25486,
      25492,
      25498
    ],
    "core.event_count": 27,
    "core.first_event_number": 25342,
    "core.last_event_number": 25498,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0156_df-s04-d0_dw_0_20260203T145521.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:46 UTC       justIN version: 01.06.00