justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.94@dunegpschedd01.fnal.gov

Jobsub ID303653.94@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce503
Last heartbeat2026-02-04 00:32:29
From worker nodeHostnameb9p05p0564.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:41:24
Input filesvd-protodune:np02vd_raw_run042421_0072_df-s04-d1_dw_0_20260203T140959.hdf5
JobscriptExit code0
Real time48m (2902s)
CPU time46m (2787s = 96%)
Max RSS bytes2753196032 (2625 MiB)
Outputting started2026-02-04 00:29:48
Output files
Finished2026-02-04 00:32:29
Saved logsjustin-logs:303653.94-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Caught exception in looking up space point ptr... 0x5891be00, 0x1086ecc0
Caught exception in looking up space point ptr... 0x1086ecc0, 0x335d2b90
Caught exception in looking up space point ptr... 0x50be4060, 0x50bf6170
Caught exception in looking up space point ptr... 0x50bd39e0, 0x50be4060
Caught exception in looking up space point ptr... 0x16714980, 0x167147f0
Caught exception in looking up space point ptr... 0x167147f0, 0x16714660
Caught exception in looking up space point ptr... 0x566913f0, 0x3c83fc30
Caught exception in looking up space point ptr... 0x3c83fc30, 0x3c8319a0
Caught exception in looking up space point ptr... 0x37721a80, 0x37710dc0
Caught exception in looking up space point ptr... 0x37710dc0, 0x377218f0
Caught exception in looking up space point ptr... 0x3c827bd0, 0x5bfc2b00
Caught exception in looking up space point ptr... 0x3c835cd0, 0x3c827bd0
Caught exception in looking up space point ptr... 0x5c1e8e10, 0x29301310
Caught exception in looking up space point ptr... 0x29301310, 0x5c1e8e10
Caught exception in looking up space point ptr... 0x2fc64110, 0x4e1209e0
Caught exception in looking up space point ptr... 0x2fc642a0, 0x2fc64110
Caught exception in looking up space point ptr... 0x3484b590, 0x3484b400
Caught exception in looking up space point ptr... 0x3484b400, 0x5d9574b0
Caught exception in looking up space point ptr... 0x29404210, 0x2fd02040
Caught exception in looking up space point ptr... 0x294043a0, 0x29404210
Caught exception in looking up space point ptr... 0x4f995a10, 0x56e1e6c0
Caught exception in looking up space point ptr... 0x2322bef0, 0x4f995a10
Caught exception in looking up space point ptr... 0x43e7b4f0, 0x638ea4b0
Caught exception in looking up space point ptr... 0x43e7b680, 0x43e7b4f0
Caught exception in looking up space point ptr... 0x41706ae0, 0x12a991f0
Caught exception in looking up space point ptr... 0x41706c70, 0x41706ae0
Caught exception in looking up space point ptr... 0x4bce7090, 0x509e0450
Caught exception in looking up space point ptr... 0x509e0450, 0x4bf44f40
Caught exception in looking up space point ptr... 0x56132830, 0x561248f0
Caught exception in looking up space point ptr... 0x561329c0, 0x56132830
Caught exception in looking up space point ptr... 0x4f671fe0, 0x47a98c70
Caught exception in looking up space point ptr... 0x47a98c70, 0x517b84a0
Caught exception in looking up space point ptr... 0x59b0c3d0, 0x59b10b50
Caught exception in looking up space point ptr... 0x59b10b50, 0x3eda2bc0
Caught exception in looking up space point ptr... 0x41922060, 0x4190fdc0
Caught exception in looking up space point ptr... 0x4f043a10, 0x41922060
Caught exception in looking up space point ptr... 0x137feee0, 0xe617600
Caught exception in looking up space point ptr... 0xaad57d0, 0x137feee0
Caught exception in looking up space point ptr... 0x6010c6c0, 0x349a1340
Caught exception in looking up space point ptr... 0x349a1340, 0x141da7b0
Caught exception in looking up space point ptr... 0x141da7b0, 0x5c102650
Caught exception in looking up space point ptr... 0x5c102650, 0x18dceb00
Caught exception in looking up space point ptr... 0x18dceb00, 0x6010c6c0
Caught exception in looking up space point ptr... 0x15b74780, 0x61e1fc60
Caught exception in looking up space point ptr... 0x61e1fc60, 0x61e122e0
Caught exception in looking up space point ptr... 0x336b8140, 0x432acfe0
Caught exception in looking up space point ptr... 0x432acfe0, 0xd0ce190
Caught exception in looking up space point ptr... 0x3cda44d0, 0x34a95270
Caught exception in looking up space point ptr... 0x34a95270, 0x34a94d40
Caught exception in looking up space point ptr... 0x41821310, 0x6284fe40
Caught exception in looking up space point ptr... 0x6284fe40, 0xa1b0e10
Caught exception in looking up space point ptr... 0x56ca85c0, 0x5cd07e30
Caught exception in looking up space point ptr... 0x34c3b040, 0x56ca85c0
Caught exception in looking up space point ptr... 0x2e563f50, 0x42d9ed20
Caught exception in looking up space point ptr... 0x42d9ed20, 0x2e563e50
Caught exception in looking up space point ptr... 0xe62b7f0, 0x3a668890
Caught exception in looking up space point ptr... 0x3a668890, 0xe62b650
Caught exception in looking up space point ptr... 0x37767ac0, 0x479a4f60
Caught exception in looking up space point ptr... 0x479a4f60, 0x37767ac0
Caught exception in looking up space point ptr... 0x49cfe980, 0xfda9ea0
Caught exception in looking up space point ptr... 0xfda9ea0, 0x49cfe980
Caught exception in looking up space point ptr... 0x39bd79f0, 0x33042860
Caught exception in looking up space point ptr... 0x33042860, 0x39bd79f0
Caught exception in looking up space point ptr... 0x5f8461f0, 0x5f8460d0
Caught exception in looking up space point ptr... 0x5f8460d0, 0x5f8461f0
++++>>>> total num hits: 285526, num free: 278624
04-Feb-2026 01:29:39 CET  Closed output file "np02vd_raw_run042421_0072_df-s04-d1_dw_0_20260203T140959_reco_stage0_20260204T002939_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         74.9636       105.553       124.541       113.639       15.2059        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.7051e-05    9.40149e-05   0.000161792   8.8981e-05    3.3153e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                3.92587       4.75389       7.23051       4.62181      0.658439        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0101249     0.0237658     0.127562      0.0139201     0.0244754       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000409065   0.000547618   0.00100262    0.000503056   0.000131178      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00152377    0.00185602    0.00225334    0.00182421     0.0002087       27     
produce:beamevent:BeamEvent                      7.4041e-05    0.000153156   0.000965972   0.000114842   0.000164307      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.03662       2.73964       3.43537       2.72934      0.390055        27     
produce:ophit:OpHitFinder                         0.0221891     0.0352438     0.0485222     0.0364061    0.00675559       27     
produce:wclsdatavd:WireCellToolkit                 53.0272       60.4615       73.0535       59.4511       4.18997        27     
produce:gaushit:GausHitFinder                     0.522322       1.09455       2.00989       1.11578      0.291231        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000149692   0.000265982   0.000483486   0.000262143   6.79714e-05      27     
produce:reco3d:SpacePointSolver                    3.72718       8.04329       16.8817       7.77451       2.96271        27     
produce:hitpdune:DisambigFromSpacePoints          0.0402841     0.153655      0.279404      0.146094      0.0531065       27     
produce:cluster3d:Cluster3D                       0.615876       24.325        38.593        30.3785       13.805         27     
[art]:TriggerResults:TriggerResultInserter       1.6271e-05    3.00416e-05   7.7841e-05     2.643e-05    1.16432e-05      27     
end_path:out1:RootOutput                          3.49e-06     7.12044e-06   2.4261e-05     6.72e-06     3.72363e-06      27     
end_path:out1:RootOutput(write)                    2.8009        3.89701       5.67469       3.77868      0.664171        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4753.31 MB
  Peak resident set size usage (VmHWM): 2753.2 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0072_df-s04-d1_dw_0_20260203T140959_reco_stage0_20260204T002939_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0072_df-s04-d1_dw_0_20260203T140959_reco_stage0_20260204T002939_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0072_df-s04-d1_dw_0_20260203T140959_reco_stage0_20260204T002939_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162093.0,
    "core.end_time": 1770164981.0,
    "core.events": [
      11729,
      11735,
      11741,
      11747,
      11753,
      11759,
      11765,
      11771,
      11777,
      11783,
      11789,
      11795,
      11801,
      11807,
      11813,
      11819,
      11825,
      11831,
      11837,
      11843,
      11849,
      11855,
      11861,
      11867,
      11873,
      11879,
      11885
    ],
    "core.event_count": 27,
    "core.first_event_number": 11729,
    "core.last_event_number": 11885,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0072_df-s04-d1_dw_0_20260203T140959.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:11 UTC       justIN version: 01.06.00