justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.132@dunegpschedd01.fnal.gov

Jobsub ID303647.132@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-03 23:50:05
From worker nodeHostnameb9p08p6650.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:24
Input filesvd-protodune:np02vd_raw_run042421_0076_df-s03-d1_dw_0_20260203T141209.hdf5
JobscriptExit code0
Real time59m (3588s)
CPU time57m (3459s = 96%)
Max RSS bytes2725191680 (2598 MiB)
Outputting started2026-02-03 23:47:14
Output files
Finished2026-02-03 23:50:05
Saved logsjustin-logs:303647.132-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0
Caught exception in looking up space point ptr... 0x3859f970, 0x3859fb00
Caught exception in looking up space point ptr... 0xfbd83c0, 0x52940790
Caught exception in looking up space point ptr... 0x52940790, 0xfbd83c0
Caught exception in looking up space point ptr... 0x4bd9f270, 0x4760bcc0
Caught exception in looking up space point ptr... 0x4760bcc0, 0x4a9d9320
Caught exception in looking up space point ptr... 0x31f2ae20, 0x31f186d0
Caught exception in looking up space point ptr... 0x2e711960, 0x31f2ae20
Caught exception in looking up space point ptr... 0x25f1b130, 0x12ae82d0
Caught exception in looking up space point ptr... 0x12ae82d0, 0x2116d020
Caught exception in looking up space point ptr... 0x2116d020, 0x25f1afa0
Caught exception in looking up space point ptr... 0x25f1afa0, 0x25f1b130
Caught exception in looking up space point ptr... 0x239b85f0, 0x239a6030
Caught exception in looking up space point ptr... 0x4b862e20, 0x239b85f0
Caught exception in looking up space point ptr... 0x3e99c740, 0x450a5f20
Caught exception in looking up space point ptr... 0x4a9a17a0, 0x3e99c740
Caught exception in looking up space point ptr... 0x4fdb65b0, 0x10a80e80
Caught exception in looking up space point ptr... 0x10a80e80, 0x4bfa13c0
Caught exception in looking up space point ptr... 0x3abdb9e0, 0x2772e440
Caught exception in looking up space point ptr... 0x3abdbb70, 0x3abdb9e0
Caught exception in looking up space point ptr... 0x2baf8d80, 0x2bae71f0
Caught exception in looking up space point ptr... 0x2bae71f0, 0x3da2b2a0
Caught exception in looking up space point ptr... 0x65d75330, 0x4e73a080
Caught exception in looking up space point ptr... 0x65d754c0, 0x65d75330
Caught exception in looking up space point ptr... 0x3c1f1710, 0x414bc0a0
Caught exception in looking up space point ptr... 0x414bc0a0, 0x414a9f90
Caught exception in looking up space point ptr... 0x4b4cddc0, 0x308c2980
Caught exception in looking up space point ptr... 0x308c2980, 0x4b4cdc30
Caught exception in looking up space point ptr... 0x2a27cf40, 0xc3e2720
Caught exception in looking up space point ptr... 0xc3e2720, 0x2a27ccb0
Caught exception in looking up space point ptr... 0x4a89d4e0, 0x618bc030
Caught exception in looking up space point ptr... 0x618bbd20, 0x4a89d4e0
Caught exception in looking up space point ptr... 0x28be6870, 0x28be6720
Caught exception in looking up space point ptr... 0x28be6720, 0x28be6620
Caught exception in looking up space point ptr... 0x3eb0f820, 0x10dc2690
Caught exception in looking up space point ptr... 0x10dc2690, 0x65cf76e0
Caught exception in looking up space point ptr... 0x1106e4b0, 0x1072e240
Caught exception in looking up space point ptr... 0x1072e240, 0x1106e4b0
Caught exception in looking up space point ptr... 0x35678470, 0x35678650
Caught exception in looking up space point ptr... 0x35678650, 0x35678830
Caught exception in looking up space point ptr... 0x24b444c0, 0x2a502ea0
Caught exception in looking up space point ptr... 0x2a502ea0, 0x56f02ca0
Caught exception in looking up space point ptr... 0x622ac170, 0x4b58f0d0
Caught exception in looking up space point ptr... 0x4b58f0d0, 0x622ac170
Caught exception in looking up space point ptr... 0x471f1bc0, 0xfb943d0
Caught exception in looking up space point ptr... 0x471d5a10, 0x471f1bc0
Caught exception in looking up space point ptr... 0x511aec70, 0x57e05850
Caught exception in looking up space point ptr... 0x57e05850, 0x10990630
Caught exception in looking up space point ptr... 0x2c20f410, 0x2c2164c0
Caught exception in looking up space point ptr... 0x2c2164c0, 0x2c20f410
Caught exception in looking up space point ptr... 0x60119650, 0x60119500
Caught exception in looking up space point ptr... 0x60119500, 0x60119650
Caught exception in looking up space point ptr... 0x12fb0a50, 0x12fa6010
Caught exception in looking up space point ptr... 0x12fa6010, 0x5e0cdc40
Caught exception in looking up space point ptr... 0xc0c8970, 0x62bc2770
Caught exception in looking up space point ptr... 0x52ad9a90, 0xc0c8970
Caught exception in looking up space point ptr... 0x237645c0, 0x52949990
Caught exception in looking up space point ptr... 0x52949990, 0x237645c0
Caught exception in looking up space point ptr... 0x35660ea0, 0x35660fa0
Caught exception in looking up space point ptr... 0x35660fa0, 0x35660ea0
Caught exception in looking up space point ptr... 0x3f2bd990, 0x455219a0
Caught exception in looking up space point ptr... 0x455219a0, 0x3f2bd990
Caught exception in looking up space point ptr... 0x93627c0, 0x32030130
Caught exception in looking up space point ptr... 0x28137f30, 0x93627c0
Caught exception in looking up space point ptr... 0x5a670bb0, 0x508d7a60
Caught exception in looking up space point ptr... 0x508d7a60, 0x5a670bb0
++++>>>> total num hits: 311597, num free: 304317
04-Feb-2026 00:47:03 CET  Closed output file "np02vd_raw_run042421_0076_df-s03-d1_dw_0_20260203T141209_reco_stage0_20260203T234703_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         95.5766       130.249       155.419       135.621       21.3324        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.515e-05    0.000106972   0.000215722   9.2511e-05    4.29087e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.10483       4.44788       4.68757       4.4357       0.140602        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0109551     0.0162811     0.0353324     0.0137511    0.00641973       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000444985   0.000590012   0.000892501   0.000566446   0.000120435      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0016404    0.00209526    0.00346705    0.00195453    0.000420935      27     
produce:beamevent:BeamEvent                      6.6821e-05    0.000142289   0.00110637    9.4201e-05    0.000193515      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.33098       2.96845       3.34257       2.98627      0.201504        27     
produce:ophit:OpHitFinder                         0.029354      0.0399971     0.0538239     0.0400499    0.00537148       27     
produce:wclsdatavd:WireCellToolkit                 67.4224       76.1275       85.4257       75.7317       4.45322        27     
produce:gaushit:GausHitFinder                     0.746382       1.38422       2.1593        1.38233      0.363647        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000231403   0.000339593   0.000574346   0.000339174   8.50882e-05      27     
produce:reco3d:SpacePointSolver                    4.95226       9.65979       15.8588       9.8251        2.55543        27     
produce:hitpdune:DisambigFromSpacePoints          0.0670416     0.186438      0.411179      0.185627      0.0821858       27     
produce:cluster3d:Cluster3D                        1.02121       30.4412       51.3718       38.8783       18.995         27     
[art]:TriggerResults:TriggerResultInserter        2.602e-05    4.01104e-05   0.000108961    3.651e-05    1.48852e-05      27     
end_path:out1:RootOutput                          4.51e-06     1.00671e-05    3.52e-05      8.52e-06     5.84431e-06      27     
end_path:out1:RootOutput(write)                    3.28451       4.94973       6.05671       4.99989      0.703581        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4688.9 MB
  Peak resident set size usage (VmHWM): 2725.19 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0076_df-s03-d1_dw_0_20260203T141209_reco_stage0_20260203T234703_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0076_df-s03-d1_dw_0_20260203T141209_reco_stage0_20260203T234703_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0076_df-s03-d1_dw_0_20260203T141209_reco_stage0_20260203T234703_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158857.0,
    "core.end_time": 1770162425.0,
    "core.events": [
      12375,
      12381,
      12387,
      12393,
      12399,
      12405,
      12411,
      12417,
      12423,
      12429,
      12435,
      12441,
      12447,
      12453,
      12459,
      12465,
      12471,
      12477,
      12483,
      12489,
      12495,
      12501,
      12507,
      12513,
      12519,
      12525,
      12531
    ],
    "core.event_count": 27,
    "core.first_event_number": 12375,
    "core.last_event_number": 12531,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0076_df-s03-d1_dw_0_20260203T141209.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:08:24 UTC       justIN version: 01.06.00