justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.89@dunegpschedd01.fnal.gov

Jobsub ID303654.89@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2026-02-04 01:00:28
From worker nodeHostnameb9p30p1409.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:05:33
Input filesvd-protodune:np02vd_raw_run042433_0034_df-s04-d1_dw_0_20260203T185502.hdf5
JobscriptExit code0
Real time52m (3150s)
CPU time50m (3046s = 96%)
Max RSS bytes3427336192 (3268 MiB)
Outputting started2026-02-04 00:58:04
Output files
Finished2026-02-04 01:00:28
Saved logsjustin-logs:303654.89-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

king up space point ptr... 0x37156540, 0x5da06150
Caught exception in looking up space point ptr... 0x371563c0, 0x37156540
Caught exception in looking up space point ptr... 0x46e62db0, 0x1a4fd970
Caught exception in looking up space point ptr... 0x3cc7ef20, 0x46e62db0
Caught exception in looking up space point ptr... 0x604ed8d0, 0x604ed370
Caught exception in looking up space point ptr... 0x604ed370, 0x604ec250
Caught exception in looking up space point ptr... 0x680f0b90, 0x6480c710
Caught exception in looking up space point ptr... 0x6480c710, 0x680f0b90
Caught exception in looking up space point ptr... 0x43edba00, 0x6d7ee120
Caught exception in looking up space point ptr... 0x6d7ee120, 0x6d7ee220
Caught exception in looking up space point ptr... 0x2f466550, 0x51de43a0
Caught exception in looking up space point ptr... 0x51de43a0, 0x2f466550
Caught exception in looking up space point ptr... 0x37956200, 0x38258e50
Caught exception in looking up space point ptr... 0x38258e50, 0x37956200
Caught exception in looking up space point ptr... 0x2c54f3f0, 0x2c54f710
Caught exception in looking up space point ptr... 0x2c54f8a0, 0x64090120
Caught exception in looking up space point ptr... 0x64090120, 0x64090810
Caught exception in looking up space point ptr... 0x64090810, 0x2c54f3f0
Caught exception in looking up space point ptr... 0x16edc130, 0x56f51460
Caught exception in looking up space point ptr... 0x682006b0, 0x16edc130
Caught exception in looking up space point ptr... 0x3c6dd480, 0x3c6ddf20
Caught exception in looking up space point ptr... 0x3c6ddf20, 0x3c6dcd90
Caught exception in looking up space point ptr... 0x3cc27410, 0x3b7c08e0
Caught exception in looking up space point ptr... 0x3b7c08e0, 0x3cc27410
Caught exception in looking up space point ptr... 0x3c6b7b90, 0x3c6b7a00
Caught exception in looking up space point ptr... 0x3c6b7a00, 0x3c6b7b90
Caught exception in looking up space point ptr... 0x818c6c10, 0x404b44f0
Caught exception in looking up space point ptr... 0x404b44f0, 0x333731d0
Caught exception in looking up space point ptr... 0x66f4bd70, 0x66f4b590
Caught exception in looking up space point ptr... 0x66f4b590, 0x66f4bd70
Caught exception in looking up space point ptr... 0x7a943ed0, 0x7a943b10
Caught exception in looking up space point ptr... 0x7a943b10, 0x7a943cf0
Caught exception in looking up space point ptr... 0x41b4cb20, 0x818db8b0
Caught exception in looking up space point ptr... 0x818db8b0, 0x41b4cb20
Caught exception in looking up space point ptr... 0x7a72cdd0, 0x3ad76bf0
Caught exception in looking up space point ptr... 0x3ad76bf0, 0x7a72cdd0
Caught exception in looking up space point ptr... 0xb317b90, 0x6d7b6070
Caught exception in looking up space point ptr... 0x6d7b6070, 0x6d7b5ee0
Caught exception in looking up space point ptr... 0x6d7b5ee0, 0xb318030
Caught exception in looking up space point ptr... 0x6243dc70, 0x62437fb0
Caught exception in looking up space point ptr... 0x62437fb0, 0x6243dc70
Caught exception in looking up space point ptr... 0x853683c0, 0x6bcc03c0
Caught exception in looking up space point ptr... 0x6bcc03c0, 0x853683c0
Caught exception in looking up space point ptr... 0x647d4680, 0x85a69150
Caught exception in looking up space point ptr... 0x85a69150, 0x647d4680
Caught exception in looking up space point ptr... 0xe5b9480, 0x16e2c980
Caught exception in looking up space point ptr... 0x16e2c980, 0xe5b9480
Caught exception in looking up space point ptr... 0x3c30d490, 0x3c30c960
Caught exception in looking up space point ptr... 0x3c30c960, 0x3c30d490
Caught exception in looking up space point ptr... 0x41b44700, 0x41b44890
Caught exception in looking up space point ptr... 0x41b44890, 0x41b44700
Caught exception in looking up space point ptr... 0x5816f4f0, 0x7f0304a0
Caught exception in looking up space point ptr... 0x7f0304a0, 0x3b94f130
Caught exception in looking up space point ptr... 0x35dd8d70, 0x35dd8be0
Caught exception in looking up space point ptr... 0x35dd8be0, 0x35dd8d70
Caught exception in looking up space point ptr... 0x818c36e0, 0x818c0750
Caught exception in looking up space point ptr... 0x818c0750, 0x818c36e0
Caught exception in looking up space point ptr... 0x818c36e0, 0x818c3ce0
Caught exception in looking up space point ptr... 0x818c3ce0, 0x818c36e0
Caught exception in looking up space point ptr... 0x647de240, 0x647c89e0
Caught exception in looking up space point ptr... 0x647c89e0, 0x647de240
Caught exception in looking up space point ptr... 0x3859ba70, 0x385af220
Caught exception in looking up space point ptr... 0x385b4020, 0x3859ba70
Caught exception in looking up space point ptr... 0x17e57290, 0x17e57600
Caught exception in looking up space point ptr... 0x17e57600, 0x17e57290
Caught exception in looking up space point ptr... 0x35d2f4b0, 0x3c0060f0
Caught exception in looking up space point ptr... 0x3c0060f0, 0x35d2f4b0
++++>>>> total num hits: 89410, num free: 85270
04-Feb-2026 01:57:55 CET  Closed output file "np02vd_raw_run042433_0034_df-s04-d1_dw_0_20260203T185502_reco_stage0_20260204T005755_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         127.748       161.568       183.452       165.38        15.7893        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.9971e-05    9.80943e-05   0.000272674   7.6151e-05    5.41864e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                4.78359       5.79044       8.2032        5.76853      0.861141        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0104558      0.01403      0.0479432     0.0119791    0.00809533       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000405577   0.000651679   0.00245194    0.000537049   0.000441138      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000372847   0.000515623   0.00106413    0.000460658   0.000154122      19     
produce:beamevent:BeamEvent                      6.2532e-05    0.000151218   0.00104995    0.000101592   0.000212633      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.11698       2.75776       3.43131       2.76227      0.298269        19     
produce:ophit:OpHitFinder                         0.024164      0.0359253     0.0436845     0.0367762    0.00552155       19     
produce:wclsdatavd:WireCellToolkit                 98.0689       118.789       132.337       122.448       9.47355        19     
produce:gaushit:GausHitFinder                     0.976484       1.54995       2.12727       1.52618      0.294758        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000305175   0.000487079   0.00061985    0.000489959   9.11156e-05      19     
produce:reco3d:SpacePointSolver                    12.594        23.3871       31.9781       24.4684       6.20579        19     
produce:hitpdune:DisambigFromSpacePoints          0.150456      0.232745      0.323409      0.235598      0.045838        19     
produce:cluster3d:Cluster3D                        1.63698       3.25922       5.71622       3.01505       1.02047        19     
[art]:TriggerResults:TriggerResultInserter       2.0471e-05    3.40843e-05   8.9811e-05    2.9411e-05    1.52147e-05      19     
end_path:out1:RootOutput                          3.351e-06    8.03711e-06    2.42e-05      7.03e-06     5.28133e-06      19     
end_path:out1:RootOutput(write)                    4.62262       5.72164       7.03696       5.46741      0.792086        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5354.86 MB
  Peak resident set size usage (VmHWM): 3427.34 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0034_df-s04-d1_dw_0_20260203T185502_reco_stage0_20260204T005755_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0034_df-s04-d1_dw_0_20260203T185502_reco_stage0_20260204T005755_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0034_df-s04-d1_dw_0_20260203T185502_reco_stage0_20260204T005755_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163548.0,
    "core.end_time": 1770166677.0,
    "core.events": [
      3887,
      3893,
      3899,
      3905,
      3911,
      3917,
      3923,
      3929,
      3935,
      3941,
      3947,
      3953,
      3959,
      3965,
      3971,
      3977,
      3983,
      3989,
      3995
    ],
    "core.event_count": 19,
    "core.first_event_number": 3887,
    "core.last_event_number": 3995,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0034_df-s04-d1_dw_0_20260203T185502.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:42:38 UTC       justIN version: 01.06.00