justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.93@dunegpschedd02.fnal.gov

Jobsub ID297921.93@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce509
Last heartbeat2026-02-04 01:04:22
From worker nodeHostnameb9p04p3844.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:54:03
Input filesvd-protodune:np02vd_raw_run042433_0363_df-s02-d1_dw_0_20260203T210005.hdf5
JobscriptExit code0
Real time1h (4072s)
CPU time1h (3843s = 94%)
Max RSS bytes3401900032 (3244 MiB)
Outputting started2026-02-04 01:01:56
Output files
Finished2026-02-04 01:04:22
Saved logsjustin-logs:297921.93-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nt ptr... 0x53e5f7c0, 0x88a8cc0
Caught exception in looking up space point ptr... 0x159ff750, 0x71dd31d0
Caught exception in looking up space point ptr... 0x71dd31d0, 0x4664a210
Caught exception in looking up space point ptr... 0x42027c80, 0x42027af0
Caught exception in looking up space point ptr... 0x42027af0, 0x42027c80
Caught exception in looking up space point ptr... 0x10049400, 0x77e6570
Caught exception in looking up space point ptr... 0x77e6570, 0x10049400
Caught exception in looking up space point ptr... 0x10049400, 0x3b142eb0
Caught exception in looking up space point ptr... 0x3b142eb0, 0x10049400
Caught exception in looking up space point ptr... 0x67383ee0, 0x67381dd0
Caught exception in looking up space point ptr... 0x67381dd0, 0x67383ee0
Caught exception in looking up space point ptr... 0x67383ee0, 0xa60cd70
Caught exception in looking up space point ptr... 0xa60cd70, 0x67383ee0
Caught exception in looking up space point ptr... 0x53e28280, 0x53e278b0
Caught exception in looking up space point ptr... 0x53e278b0, 0x53e28280
Caught exception in looking up space point ptr... 0x53e28280, 0x53e29110
Caught exception in looking up space point ptr... 0x53e29110, 0x53e28280
Caught exception in looking up space point ptr... 0x3727a0a0, 0x3727a1d0
Caught exception in looking up space point ptr... 0x3727a1d0, 0x37278ed0
Caught exception in looking up space point ptr... 0x12be3570, 0x4a74c1c0
Caught exception in looking up space point ptr... 0x4a74c1c0, 0x12be3570
Caught exception in looking up space point ptr... 0x6739d350, 0x673a0110
Caught exception in looking up space point ptr... 0x673a0110, 0x6739a370
Caught exception in looking up space point ptr... 0x31a90730, 0x31a909d0
Caught exception in looking up space point ptr... 0x31a909d0, 0x31a93cf0
Caught exception in looking up space point ptr... 0x6e5bdfb0, 0x6e5d26b0
Caught exception in looking up space point ptr... 0x6e5d26b0, 0x6e5bdfb0
Caught exception in looking up space point ptr... 0x4df99580, 0x38e6c950
Caught exception in looking up space point ptr... 0x38e6c950, 0x4df99580
Caught exception in looking up space point ptr... 0x194cabb0, 0x194ca9b0
Caught exception in looking up space point ptr... 0x194ca9b0, 0x194cabb0
Caught exception in looking up space point ptr... 0x194cabb0, 0x194caab0
Caught exception in looking up space point ptr... 0x194caab0, 0x194cabb0
Caught exception in looking up space point ptr... 0x194cabb0, 0xe617c70
Caught exception in looking up space point ptr... 0xe617c70, 0x194cabb0
Caught exception in looking up space point ptr... 0x66665110, 0x66664f80
Caught exception in looking up space point ptr... 0x66664f80, 0x66665110
Caught exception in looking up space point ptr... 0x618ab320, 0x618ab0c0
Caught exception in looking up space point ptr... 0x618ab0c0, 0x618ab6b0
Caught exception in looking up space point ptr... 0x14d7bfe0, 0x14d6fdf0
Caught exception in looking up space point ptr... 0x14d6fdf0, 0x14d7bfe0
Caught exception in looking up space point ptr... 0x7a0c5910, 0x7a0c51f0
Caught exception in looking up space point ptr... 0x7a0c51f0, 0x7a0c77f0
Caught exception in looking up space point ptr... 0x15e8eeb0, 0x10200720
Caught exception in looking up space point ptr... 0x10200720, 0x67336d40
Caught exception in looking up space point ptr... 0x181194c0, 0x2af2def0
Caught exception in looking up space point ptr... 0x2af2def0, 0x181194c0
Caught exception in looking up space point ptr... 0x3a5bc130, 0x3a5b9a40
Caught exception in looking up space point ptr... 0x3a5b9a40, 0x3a5bc130
Caught exception in looking up space point ptr... 0x132b9f00, 0x132ba050
Caught exception in looking up space point ptr... 0x132ba050, 0x132ba1a0
Caught exception in looking up space point ptr... 0x3b19f320, 0x3b5295e0
Caught exception in looking up space point ptr... 0x3b5295e0, 0x3b19f320
Caught exception in looking up space point ptr... 0x5313a1d0, 0x5313a4f0
Caught exception in looking up space point ptr... 0x5313a4f0, 0x5313a1d0
Caught exception in looking up space point ptr... 0x113797d0, 0xa60d910
Caught exception in looking up space point ptr... 0xa60d910, 0x113797d0
Caught exception in looking up space point ptr... 0x5fd38a30, 0x5fd38d50
Caught exception in looking up space point ptr... 0x5fd38d50, 0x5fd38bc0
Caught exception in looking up space point ptr... 0x3cd55db0, 0x19c81ae0
Caught exception in looking up space point ptr... 0x3cd55c20, 0x3cd55db0
Caught exception in looking up space point ptr... 0x79e25560, 0x62f05b90
Caught exception in looking up space point ptr... 0x62f05b90, 0x79e25560
Caught exception in looking up space point ptr... 0x699fa1c0, 0x699fa360
Caught exception in looking up space point ptr... 0x699fa360, 0x699fa1c0
Caught exception in looking up space point ptr... 0x38576d90, 0x385776a0
Caught exception in looking up space point ptr... 0x385776a0, 0x38576d90
++++>>>> total num hits: 134552, num free: 129885
04-Feb-2026 02:01:47 CET  Closed output file "np02vd_raw_run042433_0363_df-s02-d1_dw_0_20260203T210005_reco_stage0_20260204T010147_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         131.214       210.463       276.803       228.939       53.7481        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.7121e-05    0.000139389   0.000439467   0.000108092   9.21052e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                8.25758       12.3485       17.8429       12.9155       2.60481        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0125312     0.122811      0.514767      0.0975291     0.109765        19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000483358    0.011962      0.125501     0.000878923    0.029509        19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000474797   0.000890161    0.002197      0.0006697    0.000439148      19     
produce:beamevent:BeamEvent                      8.2601e-05    0.000214115   0.000794262   0.000161392   0.000169247      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.32179       4.15317       6.64756       3.80986       1.34985        19     
produce:ophit:OpHitFinder                         0.0307883     0.0568325     0.170747       0.04967      0.0310022       19     
produce:wclsdatavd:WireCellToolkit                 91.6501       152.864       205.136       168.677       40.9127        19     
produce:gaushit:GausHitFinder                     0.889817       1.89247       3.32037       1.8531       0.594876        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000299555   0.000453954   0.000789992   0.000454217   0.000125898      19     
produce:reco3d:SpacePointSolver                    14.0516       27.8786       44.472        25.3255       9.9451         19     
produce:hitpdune:DisambigFromSpacePoints          0.113969      0.255405      0.549692      0.206239      0.118127        19     
produce:cluster3d:Cluster3D                        1.52046       3.52002       6.58501       3.46864       1.41498        19     
[art]:TriggerResults:TriggerResultInserter        2.31e-05     4.67207e-05   8.7232e-05    3.9341e-05    1.84274e-05      19     
end_path:out1:RootOutput                          3.13e-06     9.26284e-06    2.016e-05      8.7e-06     4.97279e-06      19     
end_path:out1:RootOutput(write)                    4.35255       7.31099       10.5871       7.53339       1.87071        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5435.56 MB
  Peak resident set size usage (VmHWM): 3401.9 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0363_df-s02-d1_dw_0_20260203T210005_reco_stage0_20260204T010147_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0363_df-s02-d1_dw_0_20260203T210005_reco_stage0_20260204T010147_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0363_df-s02-d1_dw_0_20260203T210005_reco_stage0_20260204T010147_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162860.0,
    "core.end_time": 1770166909.0,
    "core.events": [
      41389,
      41395,
      41401,
      41407,
      41413,
      41419,
      41425,
      41431,
      41437,
      41443,
      41449,
      41455,
      41461,
      41467,
      41473,
      41479,
      41485,
      41491,
      41497
    ],
    "core.event_count": 19,
    "core.first_event_number": 41389,
    "core.last_event_number": 41497,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0363_df-s02-d1_dw_0_20260203T210005.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:59 UTC       justIN version: 01.06.00