justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.132@dunegpschedd01.fnal.gov

Jobsub ID303652.132@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce506
Last heartbeat2026-02-04 00:14:55
From worker nodeHostnameb9p06p2822.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:30:05
Input filesvd-protodune:np02vd_raw_run042433_0437_df-s02-d0_dw_0_20260203T212813.hdf5
JobscriptExit code0
Real time42m (2542s)
CPU time40m (2437s = 95%)
Max RSS bytes3371261952 (3215 MiB)
Outputting started2026-02-04 00:12:29
Output files
Finished2026-02-04 00:14:55
Saved logsjustin-logs:303652.132-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

point ptr... 0x589453f0, 0xd7e1ef0
Caught exception in looking up space point ptr... 0xd7e1ef0, 0x589453f0
Caught exception in looking up space point ptr... 0x39c34bd0, 0x39c35250
Caught exception in looking up space point ptr... 0x39c35250, 0x39c34da0
Caught exception in looking up space point ptr... 0x39c34da0, 0x39c34ca0
Caught exception in looking up space point ptr... 0x2a065270, 0x33d05a90
Caught exception in looking up space point ptr... 0x33d05a90, 0x3627da60
Caught exception in looking up space point ptr... 0x13f770e0, 0x60b3a5c0
Caught exception in looking up space point ptr... 0x60b3a5c0, 0x38c370e0
Caught exception in looking up space point ptr... 0x5c94cfe0, 0x60c0cc60
Caught exception in looking up space point ptr... 0x60c0cc60, 0x5b9733c0
Caught exception in looking up space point ptr... 0x5b9733c0, 0x10dee480
Caught exception in looking up space point ptr... 0x10dee480, 0x700be930
Caught exception in looking up space point ptr... 0x536f7ba0, 0x6f3795d0
Caught exception in looking up space point ptr... 0x6f3795d0, 0x536f7ba0
Caught exception in looking up space point ptr... 0x462ece60, 0x461ba6c0
Caught exception in looking up space point ptr... 0x462e6980, 0x462ece60
Caught exception in looking up space point ptr... 0x45b7f3a0, 0x45b7f2d0
Caught exception in looking up space point ptr... 0x45b7f2d0, 0x45b7f3a0
Caught exception in looking up space point ptr... 0x1ab07de0, 0x1ab0b640
Caught exception in looking up space point ptr... 0x1ab0b640, 0x1ab065c0
Caught exception in looking up space point ptr... 0xb6f9da0, 0x6cd46b90
Caught exception in looking up space point ptr... 0x6cd46b90, 0xb6f9da0
Caught exception in looking up space point ptr... 0x63e14bf0, 0x63e148b0
Caught exception in looking up space point ptr... 0x63e148b0, 0x63e14bf0
Caught exception in looking up space point ptr... 0x15463480, 0x15461820
Caught exception in looking up space point ptr... 0x15461820, 0x15463480
Caught exception in looking up space point ptr... 0x15463480, 0x15488130
Caught exception in looking up space point ptr... 0x15488130, 0x15463480
Caught exception in looking up space point ptr... 0x4bc84240, 0x5c538740
Caught exception in looking up space point ptr... 0x5c538740, 0x7f6e8d90
Caught exception in looking up space point ptr... 0x638213f0, 0xb462f50
Caught exception in looking up space point ptr... 0xb462f50, 0xb46e990
Caught exception in looking up space point ptr... 0x52aea450, 0x52ae9fb0
Caught exception in looking up space point ptr... 0x52ae9fb0, 0x52aea200
Caught exception in looking up space point ptr... 0x1ab0a960, 0x1aae3150
Caught exception in looking up space point ptr... 0x1aae3150, 0x1aada980
Caught exception in looking up space point ptr... 0x346942c0, 0x3468ea00
Caught exception in looking up space point ptr... 0x3468ea00, 0x34692a00
Caught exception in looking up space point ptr... 0x4914bdd0, 0x2e3f66c0
Caught exception in looking up space point ptr... 0x2e3f66c0, 0x4914bdd0
Caught exception in looking up space point ptr... 0x60e482d0, 0x60e48140
Caught exception in looking up space point ptr... 0x60e48140, 0x60e482d0
Caught exception in looking up space point ptr... 0x5424d7f0, 0x5803d960
Caught exception in looking up space point ptr... 0x5803d960, 0x5424d7f0
Caught exception in looking up space point ptr... 0x5424d7f0, 0x5424b450
Caught exception in looking up space point ptr... 0x5424b450, 0x5424d7f0
Caught exception in looking up space point ptr... 0x5424d7f0, 0x5424b5e0
Caught exception in looking up space point ptr... 0x5424b5e0, 0x5424d7f0
Caught exception in looking up space point ptr... 0x1aae8870, 0x63dedbf0
Caught exception in looking up space point ptr... 0x63e086f0, 0x1aae8870
Caught exception in looking up space point ptr... 0xb45af20, 0xe115dc0
Caught exception in looking up space point ptr... 0xe115dc0, 0xb45af20
Caught exception in looking up space point ptr... 0x588ed3c0, 0x588e4ef0
Caught exception in looking up space point ptr... 0x588e4ef0, 0x588e3810
Caught exception in looking up space point ptr... 0x1a1e5340, 0x2b614320
Caught exception in looking up space point ptr... 0x2b614320, 0x2b6141d0
Caught exception in looking up space point ptr... 0x588d5c70, 0x588cd960
Caught exception in looking up space point ptr... 0x588cd960, 0x588d5c70
Caught exception in looking up space point ptr... 0x3b099cd0, 0x3b099be0
Caught exception in looking up space point ptr... 0x3b099be0, 0x3b099cd0
Caught exception in looking up space point ptr... 0x588e32c0, 0x2a06e9e0
Caught exception in looking up space point ptr... 0x2a06e9e0, 0x588e32c0
Caught exception in looking up space point ptr... 0x58c12ee0, 0x588d7de0
Caught exception in looking up space point ptr... 0x588d7de0, 0x4631e420
Caught exception in looking up space point ptr... 0x18098fd0, 0x1808adb0
Caught exception in looking up space point ptr... 0x1808adb0, 0x78a9af60
++++>>>> total num hits: 99383, num free: 94911
04-Feb-2026 01:12:20 CET  Closed output file "np02vd_raw_run042433_0437_df-s02-d0_dw_0_20260203T212813_reco_stage0_20260204T001220_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         116.156       130.366       149.615       129.351       9.94219        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.046e-05    9.23588e-05   0.000185191   7.4791e-05    4.29218e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.41998       7.3258        12.9997       7.03969       1.37479        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00942664     0.0245117     0.110586      0.012873      0.0256693       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000402632   0.000523597   0.000816243   0.000483192   0.000114182      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000369912   0.000501414   0.000936724   0.000465212   0.000133212      19     
produce:beamevent:BeamEvent                       5.648e-05    0.000132481   0.00103906     7.653e-05    0.000214524      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.16182       2.71034       3.12013       2.75718       0.27698        19     
produce:ophit:OpHitFinder                         0.0287078     0.0360123     0.0413388     0.0361231    0.00388554       19     
produce:wclsdatavd:WireCellToolkit                 87.8534       96.8662       109.456       95.2253       6.86569        19     
produce:gaushit:GausHitFinder                     0.824784       1.21497       2.09427       1.1088       0.352358        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000237091   0.000310712   0.000474092   0.000291711   5.91755e-05      19     
produce:reco3d:SpacePointSolver                    8.60652       15.1187       21.1723       16.2735       3.4035         19     
produce:hitpdune:DisambigFromSpacePoints          0.0830659      0.16583      0.311085      0.145384      0.053708        19     
produce:cluster3d:Cluster3D                        1.11938       2.25126       3.74872       2.05423      0.730815        19     
[art]:TriggerResults:TriggerResultInserter        2.147e-05    2.8809e-05    0.00010332     2.399e-05    1.7722e-05       19     
end_path:out1:RootOutput                          4.59e-06     6.83737e-06    3.195e-05     5.36e-06     5.95929e-06      19     
end_path:out1:RootOutput(write)                    4.15976       4.62313       6.28521       4.51348      0.487303        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5366.83 MB
  Peak resident set size usage (VmHWM): 3371.26 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0437_df-s02-d0_dw_0_20260203T212813_reco_stage0_20260204T001220_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0437_df-s02-d0_dw_0_20260203T212813_reco_stage0_20260204T001220_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0437_df-s02-d0_dw_0_20260203T212813_reco_stage0_20260204T001220_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161415.0,
    "core.end_time": 1770163941.0,
    "core.events": [
      49824,
      49830,
      49836,
      49842,
      49848,
      49854,
      49860,
      49866,
      49872,
      49878,
      49884,
      49890,
      49896,
      49902,
      49908,
      49914,
      49920,
      49926,
      49932
    ],
    "core.event_count": 19,
    "core.first_event_number": 49824,
    "core.last_event_number": 49932,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0437_df-s02-d0_dw_0_20260203T212813.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:58:02 UTC       justIN version: 01.06.00