justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.43@dunegpschedd02.fnal.gov

Jobsub ID297923.43@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce513
Last heartbeat2026-02-04 01:33:14
From worker nodeHostnameb9p17p7474.cern.ch
cpuinfoIntel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:59:27
Input filesvd-protodune:np02vd_raw_run042433_0075_df-s03-d1_dw_0_20260203T191038.hdf5
JobscriptExit code0
Real time1h (5472s)
CPU time1h (5346s = 97%)
Max RSS bytes3398926336 (3241 MiB)
Outputting started2026-02-04 01:30:42
Output files
Finished2026-02-04 01:33:14
Saved logsjustin-logs:297923.43-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

king up space point ptr... 0x741e5770, 0x5e0651f0
Caught exception in looking up space point ptr... 0x1390efc0, 0x829d5440
Caught exception in looking up space point ptr... 0x829d5440, 0x1390efc0
Caught exception in looking up space point ptr... 0x53be0a60, 0x53bde650
Caught exception in looking up space point ptr... 0x53bde650, 0x53be0a60
Caught exception in looking up space point ptr... 0x322b1d30, 0x322bf230
Caught exception in looking up space point ptr... 0x322bf230, 0xb0e0430
Caught exception in looking up space point ptr... 0x7fec0950, 0x1a7da1b0
Caught exception in looking up space point ptr... 0x1a7da1b0, 0x1a7be820
Caught exception in looking up space point ptr... 0x37023ad0, 0x37023c00
Caught exception in looking up space point ptr... 0x37023c00, 0x370234e0
Caught exception in looking up space point ptr... 0x3dbe41b0, 0x3dbf11c0
Caught exception in looking up space point ptr... 0x3dbf11c0, 0x519f3bc0
Caught exception in looking up space point ptr... 0x58701bd0, 0x58701aa0
Caught exception in looking up space point ptr... 0x58701aa0, 0x58701bd0
Caught exception in looking up space point ptr... 0x42783d50, 0x42784580
Caught exception in looking up space point ptr... 0x42784580, 0x42783d50
Caught exception in looking up space point ptr... 0x42783d50, 0xffd3b40
Caught exception in looking up space point ptr... 0xffd3b40, 0x42783d50
Caught exception in looking up space point ptr... 0x3dbdd360, 0x1273ebc0
Caught exception in looking up space point ptr... 0x1273ebc0, 0x3dbdd260
Caught exception in looking up space point ptr... 0x175c8950, 0x6f59c8f0
Caught exception in looking up space point ptr... 0x6f59c8f0, 0x37139aa0
Caught exception in looking up space point ptr... 0x3cd70060, 0x3cd6fba0
Caught exception in looking up space point ptr... 0x3cd6fba0, 0x3cd6df20
Caught exception in looking up space point ptr... 0x1a7c9150, 0x1a7c8ea0
Caught exception in looking up space point ptr... 0x1a7c8ea0, 0x1a7c9150
Caught exception in looking up space point ptr... 0xa0513d0, 0x2b4583f0
Caught exception in looking up space point ptr... 0x2b4583f0, 0x7face5e0
Caught exception in looking up space point ptr... 0x42fd00e0, 0x42fd4540
Caught exception in looking up space point ptr... 0x42fd4540, 0x4c7f58f0
Caught exception in looking up space point ptr... 0x81df8ad0, 0x84feaaf0
Caught exception in looking up space point ptr... 0x84feaaf0, 0x81df8ad0
Caught exception in looking up space point ptr... 0x81df8ad0, 0x84feac80
Caught exception in looking up space point ptr... 0x84feac80, 0x81df8ad0
Caught exception in looking up space point ptr... 0x151e0440, 0x151d7670
Caught exception in looking up space point ptr... 0x1582e900, 0x151e0440
Caught exception in looking up space point ptr... 0x183b9380, 0x2b51b6a0
Caught exception in looking up space point ptr... 0x2b51b6a0, 0x183b9380
Caught exception in looking up space point ptr... 0x5e067e20, 0x5e067b80
Caught exception in looking up space point ptr... 0x5e067b80, 0x5e067e20
Caught exception in looking up space point ptr... 0x16581590, 0x16581900
Caught exception in looking up space point ptr... 0x16581900, 0x16581770
Caught exception in looking up space point ptr... 0x322ca240, 0x322ca060
Caught exception in looking up space point ptr... 0x322ca060, 0x322ca240
Caught exception in looking up space point ptr... 0x37a10860, 0x62c5b4e0
Caught exception in looking up space point ptr... 0x62c5b4e0, 0x62c6a2a0
Caught exception in looking up space point ptr... 0xd5c78d0, 0x828838c0
Caught exception in looking up space point ptr... 0x828838c0, 0x33849930
Caught exception in looking up space point ptr... 0x15975f10, 0x159741b0
Caught exception in looking up space point ptr... 0x159741b0, 0x15975f10
Caught exception in looking up space point ptr... 0x15a35b90, 0x15a356d0
Caught exception in looking up space point ptr... 0x15a356d0, 0x15a35b90
Caught exception in looking up space point ptr... 0x7fee4e40, 0x7fee6210
Caught exception in looking up space point ptr... 0x7fee6210, 0x7fee4e40
Caught exception in looking up space point ptr... 0x4124fe00, 0x41572560
Caught exception in looking up space point ptr... 0x41572560, 0x397a3c90
Caught exception in looking up space point ptr... 0x366a4110, 0x366a2c90
Caught exception in looking up space point ptr... 0x366a2c90, 0x366a4110
Caught exception in looking up space point ptr... 0x62c5ec10, 0x802123c0
Caught exception in looking up space point ptr... 0x802123c0, 0x62c5ec10
Caught exception in looking up space point ptr... 0x5e04dcd0, 0x876ad1b0
Caught exception in looking up space point ptr... 0x876ad1b0, 0x519ffde0
Caught exception in looking up space point ptr... 0x519ffde0, 0x51a20d90
Caught exception in looking up space point ptr... 0x51a20d90, 0x51a140a0
Caught exception in looking up space point ptr... 0x3ce5a050, 0x3ce59800
Caught exception in looking up space point ptr... 0x3ce59800, 0x605892d0
++++>>>> total num hits: 104965, num free: 99793
04-Feb-2026 02:30:29 CET  Closed output file "np02vd_raw_run042433_0075_df-s03-d1_dw_0_20260203T191038_reco_stage0_20260204T013029_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         227.593       282.228       343.997       268.884       33.574         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.6542e-05    0.000135453   0.000481238   0.000103729   9.50005e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                9.80091       10.4312       11.8956        10.37       0.481618        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0112843     0.0289121     0.075798      0.0155795      0.02198        19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000600856   0.000855276   0.00159703    0.000808986   0.000199985      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000540296   0.000806503   0.00163279    0.000767112   0.000221139      19     
produce:beamevent:BeamEvent                      0.000114063   0.000236638    0.001615     0.000157933   0.000325682      19     
produce:pdvddaphne:DAPHNEReaderPDVD                6.83775       7.35375       8.35561       7.31187      0.375542        19     
produce:ophit:OpHitFinder                         0.0616751     0.0751063     0.0884086     0.0750313    0.00633135       19     
produce:wclsdatavd:WireCellToolkit                 178.303       218.113       270.694       213.495       28.8502        19     
produce:gaushit:GausHitFinder                      1.5888        2.53497       4.50054       2.3278       0.732777        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000504815   0.000801656   0.00141136    0.000738713   0.000240682      19     
produce:reco3d:SpacePointSolver                    19.1292       29.8983       47.2461       28.3658       7.6322         19     
produce:hitpdune:DisambigFromSpacePoints           0.23334      0.442156      0.898467      0.374864      0.174087        19     
produce:cluster3d:Cluster3D                        1.79752       5.65063       26.7215       3.7666        5.55613        19     
[art]:TriggerResults:TriggerResultInserter        2.897e-05    4.79841e-05   0.000133579   4.3903e-05    2.12095e-05      19     
end_path:out1:RootOutput                          6.609e-06    0.000116005   0.00199901     9.589e-06    0.000443871      19     
end_path:out1:RootOutput(write)                    6.10747       7.61548       10.2748       7.26677       1.27008        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5417.73 MB
  Peak resident set size usage (VmHWM): 3398.93 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0075_df-s03-d1_dw_0_20260203T191038_reco_stage0_20260204T013029_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0075_df-s03-d1_dw_0_20260203T191038_reco_stage0_20260204T013029_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0075_df-s03-d1_dw_0_20260203T191038_reco_stage0_20260204T013029_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163186.0,
    "core.end_time": 1770168632.0,
    "core.events": [
      8559,
      8565,
      8571,
      8577,
      8583,
      8589,
      8595,
      8601,
      8607,
      8613,
      8619,
      8625,
      8631,
      8637,
      8643,
      8649,
      8655,
      8661,
      8667
    ],
    "core.event_count": 19,
    "core.first_event_number": 8559,
    "core.last_event_number": 8667,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0075_df-s03-d1_dw_0_20260203T191038.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:58 UTC       justIN version: 01.06.00