justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.83@dunegpschedd02.fnal.gov

Jobsub ID297921.83@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-04 00:48:25
From worker nodeHostnameb9p08p6650.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:52:53
Input filesvd-protodune:np02vd_raw_run042433_0010_df-s03-d0_dw_0_20260203T184554.hdf5
JobscriptExit code0
Real time53m (3188s)
CPU time51m (3061s = 96%)
Max RSS bytes3429351424 (3270 MiB)
Outputting started2026-02-04 00:46:01
Output files
Finished2026-02-04 00:48:25
Saved logsjustin-logs:297921.83-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ception in looking up space point ptr... 0x49f99000, 0x49f99190
Caught exception in looking up space point ptr... 0x12ea82d0, 0x12ea8420
Caught exception in looking up space point ptr... 0x12ea8420, 0x12ea86c0
Caught exception in looking up space point ptr... 0xac34e60, 0xac35110
Caught exception in looking up space point ptr... 0xac35110, 0xac34fe0
Caught exception in looking up space point ptr... 0x51e4fc60, 0x51e50670
Caught exception in looking up space point ptr... 0x51e50670, 0x51e4fc60
Caught exception in looking up space point ptr... 0x3b38eaf0, 0x3b38e7d0
Caught exception in looking up space point ptr... 0x3b38e7d0, 0x3b38eaf0
Caught exception in looking up space point ptr... 0x3bc80d90, 0x837ec520
Caught exception in looking up space point ptr... 0x837ec520, 0x3bc80d90
Caught exception in looking up space point ptr... 0xd8fe960, 0x4fede5b0
Caught exception in looking up space point ptr... 0x4fede5b0, 0xd8fe640
Caught exception in looking up space point ptr... 0x49e2ea70, 0x49e3dce0
Caught exception in looking up space point ptr... 0x49e3dce0, 0x49e35f60
Caught exception in looking up space point ptr... 0x3136a2f0, 0x2b265390
Caught exception in looking up space point ptr... 0x2b265390, 0x3136a2f0
Caught exception in looking up space point ptr... 0x35a405c0, 0x35a3b100
Caught exception in looking up space point ptr... 0x35a3b100, 0x35a3fcf0
Caught exception in looking up space point ptr... 0x11084d80, 0x63e50cb0
Caught exception in looking up space point ptr... 0x63e50cb0, 0x110849f0
Caught exception in looking up space point ptr... 0x2b348830, 0x2b3486a0
Caught exception in looking up space point ptr... 0x2b3486a0, 0x2b348830
Caught exception in looking up space point ptr... 0x2f7fc8d0, 0x537fd860
Caught exception in looking up space point ptr... 0x537fd860, 0x2f7fc8d0
Caught exception in looking up space point ptr... 0xde48f30, 0xde486a0
Caught exception in looking up space point ptr... 0xde486a0, 0xde48f30
Caught exception in looking up space point ptr... 0x49e5fad0, 0x358db9e0
Caught exception in looking up space point ptr... 0x358db9e0, 0x49e5fad0
Caught exception in looking up space point ptr... 0x3c0fdb90, 0x3c0fda00
Caught exception in looking up space point ptr... 0x3c0fda00, 0x3c0fd900
Caught exception in looking up space point ptr... 0x5ab43320, 0x3f18b430
Caught exception in looking up space point ptr... 0x3f18b430, 0x605dda20
Caught exception in looking up space point ptr... 0x39d1a690, 0x39d1a1e0
Caught exception in looking up space point ptr... 0x3a3dbc10, 0x39d1a690
Caught exception in looking up space point ptr... 0x12a22be0, 0x12a22a50
Caught exception in looking up space point ptr... 0x12a22a50, 0xde3ada0
Caught exception in looking up space point ptr... 0x6825af90, 0x4a93dfb0
Caught exception in looking up space point ptr... 0x4a93dfb0, 0x3c4d8c10
Caught exception in looking up space point ptr... 0x3c0529b0, 0xb100470
Caught exception in looking up space point ptr... 0xb100470, 0x6f143770
Caught exception in looking up space point ptr... 0x39d04f70, 0x99ded70
Caught exception in looking up space point ptr... 0x99e2190, 0x39d04f70
Caught exception in looking up space point ptr... 0x63950ec0, 0x126bd960
Caught exception in looking up space point ptr... 0x126bd960, 0x63950ec0
Caught exception in looking up space point ptr... 0x63d907f0, 0x63d6a8e0
Caught exception in looking up space point ptr... 0x63d6a8e0, 0x63d907f0
Caught exception in looking up space point ptr... 0x39d0f840, 0x49ece020
Caught exception in looking up space point ptr... 0x49ece020, 0x39d0f840
Caught exception in looking up space point ptr... 0x61d2b730, 0x61d2ba50
Caught exception in looking up space point ptr... 0x61d2ba50, 0x61d2bbe0
Caught exception in looking up space point ptr... 0x2f811260, 0x2f811710
Caught exception in looking up space point ptr... 0x2f811710, 0x2f811260
Caught exception in looking up space point ptr... 0x56093c90, 0x56093df0
Caught exception in looking up space point ptr... 0x56093df0, 0x50ec3b60
Caught exception in looking up space point ptr... 0x314f37b0, 0x6c27f890
Caught exception in looking up space point ptr... 0x6c27f890, 0x34310c20
Caught exception in looking up space point ptr... 0x3401a120, 0x12ea7760
Caught exception in looking up space point ptr... 0x12ea7760, 0x3401a120
Caught exception in looking up space point ptr... 0x51c54aa0, 0x3deca3d0
Caught exception in looking up space point ptr... 0x3deca3d0, 0x51c54aa0
Caught exception in looking up space point ptr... 0x38774390, 0x38774270
Caught exception in looking up space point ptr... 0x38774270, 0x38774390
Caught exception in looking up space point ptr... 0x4a316be0, 0x5c72ac00
Caught exception in looking up space point ptr... 0x5c72ac00, 0x4a314420
Caught exception in looking up space point ptr... 0xa1062a0, 0xa106e00
Caught exception in looking up space point ptr... 0xa106e00, 0xa106f00
++++>>>> total num hits: 101985, num free: 97056
04-Feb-2026 01:45:53 CET  Closed output file "np02vd_raw_run042433_0010_df-s03-d0_dw_0_20260203T184554_reco_stage0_20260204T004553_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         136.682       164.304       195.684       160.565       17.4796        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.298e-05    0.00011973    0.000268993   9.8711e-05    5.21425e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.97455       7.61608       8.1668        7.60717      0.297007        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0118566     0.0193972     0.0641093     0.014633      0.0119637       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000466185   0.000613219    0.001053     0.000573337   0.000156386      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000432675   0.000572659   0.00102696    0.000526266   0.000124962      19     
produce:beamevent:BeamEvent                      6.2471e-05    0.000154243   0.000987492   0.000103751   0.000199169      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.49444       3.00789       3.47176       3.02739      0.233789        19     
produce:ophit:OpHitFinder                         0.0281824     0.0388946     0.0465675     0.0387763    0.00438316       19     
produce:wclsdatavd:WireCellToolkit                 98.6768       117.876       141.202       113.996       11.7641        19     
produce:gaushit:GausHitFinder                      1.11522       1.59208       2.70434       1.59913      0.398069        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000301964   0.000447948   0.000749349   0.000386215   0.000130669      19     
produce:reco3d:SpacePointSolver                    14.9494       24.8452       36.9064       24.7939       5.99219        19     
produce:hitpdune:DisambigFromSpacePoints           0.12924      0.245679      0.378596      0.252881      0.0739502       19     
produce:cluster3d:Cluster3D                        1.54495       3.34373       5.73438       3.25344       1.15627        19     
[art]:TriggerResults:TriggerResultInserter        2.577e-05    4.24078e-05   0.000151552   3.5071e-05    2.6912e-05       19     
end_path:out1:RootOutput                          4.71e-06     8.37268e-06    3.18e-05       6.4e-06     5.84325e-06      19     
end_path:out1:RootOutput(write)                    4.46147       5.68128       7.14681       5.71493      0.885154        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5412.33 MB
  Peak resident set size usage (VmHWM): 3429.35 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0010_df-s03-d0_dw_0_20260203T184554_reco_stage0_20260204T004553_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0010_df-s03-d0_dw_0_20260203T184554_reco_stage0_20260204T004553_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0010_df-s03-d0_dw_0_20260203T184554_reco_stage0_20260204T004553_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162786.0,
    "core.end_time": 1770165954.0,
    "core.events": [
      1148,
      1154,
      1160,
      1166,
      1172,
      1178,
      1184,
      1190,
      1196,
      1202,
      1208,
      1214,
      1220,
      1226,
      1232,
      1238,
      1244,
      1250,
      1256
    ],
    "core.event_count": 19,
    "core.first_event_number": 1148,
    "core.last_event_number": 1256,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0010_df-s03-d0_dw_0_20260203T184554.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:54 UTC       justIN version: 01.06.00