Lmod is automatically replacing "intel/19.1.1" with "gcc/9.1.0". Due to MODULEPATH changes, the following have been reloaded: 1) gsl/2.6 The following have been reloaded with a version change: 1) python3/3.7.0 => python3/3.8.2 Currently Loaded Modules: 1) git/2.24.1 5) hwloc/1.11.12 9) gsl/2.6 13) fftw3/3.3.8 2) autotools/1.2 6) xalt/2.10.34 10) python3/3.8.2 3) cmake/3.20.3 7) TACC 11) hdf5/1.10.4 4) pmix/3.1.4 8) gcc/9.1.0 12) impi/19.0.9 Current wall time Sun Mar 13 17:02:52 CDT 2022 TACC: Starting up job 4152887 TACC: Starting parallel tasks... Shared memory islands host a minimum of 8 and a maximum of 8 MPI ranks. We shall use 12 MPI ranks in total for assisting one-sided communication (1 per shared memory node). ___ __ ____ ___ ____ ____ __ / __) /__\ ( _ \ / __)( ___)(_ _)___ /. | ( (_-. /(__)\ )(_) )( (_-. )__) )( (___)(_ _) \___/(__)(__)(____/ \___/(____) (__) (_) This is Gadget, version 4.0. Git commit 30019281bed6dcfb8d018c09095aeaa1e6ff5042, Wed Aug 11 12:22:28 2021 +0200 Code was compiled with the following compiler and flags: mpicxx -std=c++11 -Wwrite-strings -Wredundant-decls -Woverloaded-virtual -Wcast-qual -Wcast-align -Wpointer-arith -Wmissing-declarations -g -Wall -W -O3 -march=native -I/opt/apps/gcc9_1/hdf5/1.10.4/x86_64/include -I/opt/apps/gcc9_1/gsl/2.6/include -I/opt/apps/gcc9_1/impi19_0/fftw3/3.3.8/include -Ibuild -Isrc Code was compiled with the following settings: DOUBLEPRECISION=1 FOF FOF_GROUP_MIN_LEN=32 FOF_LINKLENGTH=0.2 FOF_PRIMARY_LINK_TYPES=2 FOF_SECONDARY_LINK_TYPES=1+16+32 GADGET2_HEADER IDS_64BIT NTYPES=6 PERIODIC PMGRID=128 SELFGRAVITY SUBFIND Running on 84 MPI tasks. BEGRUN: Size of particle structure 136 [bytes] BEGRUN: Size of sph particle structure 192 [bytes] BEGRUN: Size of gravity tree node 104 [bytes] BEGRUN: Size of neighbour tree node 192 [bytes] BEGRUN: Size of subfind auxiliary data 104 [bytes] ------------------------------------------------------------------------------------------------------------------------- AvailMem: Largest = 189686.95 Mb (on task= 42), Smallest = 189001.95 Mb (on task= 70), Average = 189319.00 Mb Total Mem: Largest = 191386.89 Mb (on task= 0), Smallest = 191386.89 Mb (on task= 70), Average = 191386.89 Mb Committed_AS: Largest = 2384.94 Mb (on task= 70), Smallest = 1699.95 Mb (on task= 42), Average = 2067.89 Mb SwapTotal: Largest = 0.00 Mb (on task= 0), Smallest = 0.00 Mb (on task= 0), Average = 0.00 Mb SwapFree: Largest = 0.00 Mb (on task= 0), Smallest = 0.00 Mb (on task= 0), Average = 0.00 Mb AllocMem: Largest = 2384.94 Mb (on task= 70), Smallest = 1699.95 Mb (on task= 42), Average = 2067.89 Mb avail /dev/shm: Largest = 95568.51 Mb (on task= 7), Smallest = 95568.49 Mb (on task= 42), Average = 95568.50 Mb ------------------------------------------------------------------------------------------------------------------------- Task=0 has the maximum commited memory and is host: c206-004.frontera.tacc.utexas.edu ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Obtaining parameters from file 'param.txt': InitCondFile lcdm_gas_littleendian.dat OutputDir output SnapshotFileBase snap OutputListFilename outputs_lcdm_gas.txt ICFormat 3 SnapFormat 3 TimeLimitCPU 180000 CpuTimeBetRestartFile 7200 MaxMemSize 10000 TimeBegin 0.0909091 TimeMax 1 ComovingIntegrationOn 1 Omega0 0.2814 OmegaLambda 0.7186 OmegaBaryon 0.0464 HubbleParam 0.697 Hubble 0.1 BoxSize 400000 OutputListOn 0 TimeBetSnapshot 1.83842 TimeOfFirstSnapshot 0.047619 TimeBetStatistics 0.05 NumFilesPerSnapshot 1 MaxFilesWithConcurrentIO 1 ErrTolIntAccuracy 0.012 CourantFac 0.15 MaxSizeTimestep 0.025 MinSizeTimestep 0 TypeOfOpeningCriterion 1 ErrTolTheta 0.9 ErrTolThetaMax 1 ErrTolForceAcc 0.0025 TopNodeFactor 2.5 ActivePartFracForNewDomainDecomp 0.01 DesNumNgb 64 MaxNumNgbDeviation 3 UnitLength_in_cm 3.08568e+21 UnitMass_in_g 1.989e+43 UnitVelocity_in_cm_per_s 100000 GravityConstantInternal 0 DesLinkNgb 20 SofteningComovingClass0 1.5 SofteningComovingClass1 1.5 SofteningComovingClass2 1.5 SofteningComovingClass3 1.5 SofteningComovingClass4 1.5 SofteningComovingClass5 1.5 SofteningMaxPhysClass0 1.5 SofteningMaxPhysClass1 1.5 SofteningMaxPhysClass2 1.5 SofteningMaxPhysClass3 1.5 SofteningMaxPhysClass4 1.5 SofteningMaxPhysClass5 1.5 SofteningClassOfPartType0 0 SofteningClassOfPartType1 0 SofteningClassOfPartType2 0 SofteningClassOfPartType3 0 SofteningClassOfPartType4 0 SofteningClassOfPartType5 0 ArtBulkViscConst 1 MinEgySpec 0 InitGasTemp 1000 MALLOC: Allocation of shared memory took 14.1658 sec BEGRUN: Hubble (internal units) = 0.1 BEGRUN: h = 0.697 BEGRUN: G (internal units) = 43018.7 BEGRUN: UnitMass_in_g = 1.989e+43 BEGRUN: UnitLenth_in_cm = 3.08568e+21 BEGRUN: UnitTime_in_s = 3.08568e+16 BEGRUN: UnitVelocity_in_cm_per_s = 100000 BEGRUN: UnitDensity_in_cgs = 6.76991e-22 BEGRUN: UnitEnergy_in_cgs = 1.989e+53 READIC: filenr=0, 'output/snap_271.hdf5' contains: READIC: Type 0 (gas): 47695076 (tot= 47695076) masstab= 0 READIC: Type 1: 49327276 (tot= 49327276) masstab= 0 READIC: Type 2: 0 (tot= 0) masstab= 0 READIC: Type 3: 0 (tot= 0) masstab= 0 READIC: Type 4: 37172426 (tot= 37172426) masstab= 0 READIC: Type 5: 2590 (tot= 2590) masstab= 0 READIC: Reading file `output/snap_271.hdf5' on task=0 and distribute it to 0 to 83. READIC: reading block 0 (Coordinates)... READIC: reading block 1 (Velocities)... READIC: reading block 2 (ParticleIDs)... READIC: reading block 3 (Masses)... READIC: reading block 4 (InternalEnergy)... READIC: reading block 5 (Density)... Dataset Density not present for particle type 0, using zero. READIC: reading block 6 (SmoothingLength)... Dataset SmoothingLength not present for particle type 0, using zero. READIC: reading done. Took 4.66117 sec, total size 6325.01 MB, corresponds to effective I/O rate of 1356.96 MB/sec READIC: Total number of particles : 134197368 INIT: Testing ID uniqueness... INIT: success. took=0.258983 sec DOMAIN: Begin domain decomposition (sync-point 0). DOMAIN: Sum=4 TotalCost=4 NumTimeBinsToBeBalanced=1 MultipleDomains=4 DOMAIN: Increasing TopNodeAllocFactor=0.08 new value=0.104 DOMAIN: Increasing TopNodeAllocFactor=0.104 new value=0.1352 DOMAIN: Increasing TopNodeAllocFactor=0.1352 new value=0.17576 DOMAIN: Increasing TopNodeAllocFactor=0.17576 new value=0.228488 DOMAIN: Increasing TopNodeAllocFactor=0.228488 new value=0.297034 DOMAIN: Increasing TopNodeAllocFactor=0.297034 new value=0.386145 DOMAIN: Increasing TopNodeAllocFactor=0.386145 new value=0.501988 DOMAIN: Increasing TopNodeAllocFactor=0.501988 new value=0.652585 DOMAIN: Increasing TopNodeAllocFactor=0.652585 new value=0.84836 DOMAIN: Increasing TopNodeAllocFactor=0.84836 new value=1.10287 DOMAIN: Increasing TopNodeAllocFactor=1.10287 new value=1.43373 DOMAIN: Increasing TopNodeAllocFactor=1.43373 new value=1.86385 DOMAIN: Increasing TopNodeAllocFactor=1.86385 new value=2.423 DOMAIN: Increasing TopNodeAllocFactor=2.423 new value=3.1499 DOMAIN: Increasing TopNodeAllocFactor=3.1499 new value=4.09487 DOMAIN: Increasing TopNodeAllocFactor=4.09487 new value=5.32333 DOMAIN: Increasing TopNodeAllocFactor=5.32333 new value=6.92033 DOMAIN: Increasing TopNodeAllocFactor=6.92033 new value=8.99643 DOMAIN: Increasing TopNodeAllocFactor=8.99643 new value=11.6954 DOMAIN: Increasing TopNodeAllocFactor=11.6954 new value=15.204 DOMAIN: Increasing TopNodeAllocFactor=15.204 new value=19.7652 DOMAIN: NTopleaves=12412, determination of top-level tree involved 15 iterations and took 2.25306 sec DOMAIN: we are going to try at most 411 different settings for combining the domains on tasks=84, nnodes=12 DOMAIN: total_cost=4 total_load=2 DOMAIN: total_cost=4 total_load=2 DOMAIN: best solution found after 2 iterations by task=80 for nextra=1, reaching maximum imbalance of 1.14831|1.1485 DOMAIN: combining multiple-domains took 0.048699 sec DOMAIN: exchange of 86502292 particles DOMAIN: particle exchange done. (took 0.715369 sec) DOMAIN: domain decomposition done. (took in total 3.04326 sec) PEANO: Begin Peano-Hilbert order... PEANO: done, took 0.343928 sec. INIT: Converting u -> entropy All.cf_a3inv=432.081 FOF: Begin to compute FoF group catalogue... (presently allocated=824.217 MB) FOF: Comoving linking length: 11.3647 FOFTREE: Ngb-tree construction done. took 0.661553 sec =113361 NTopnodes=14185 NTopleaves=12412 FOF: Start linking particles (presently allocated=900.035 MB) FOF: linking of small cells took 0.00613045 sec FOF: local links done (took 1.31293 sec, avg-work=1.11766, imbalance=1.17134). FOF: Marked=1167125 out of the 49327276 primaries which are linked FOF: begin linking across processors (presently allocated=908.001 MB) FOF: have done 10924 cross links (processed 1167125, took 0.113146 sec) FOF: have done 7258 cross links (processed 744202, took 0.108175 sec) FOF: have done 2793 cross links (processed 342396, took 0.0969942 sec) FOF: have done 1072 cross links (processed 236956, took 0.0948451 sec) FOF: have done 394 cross links (processed 109746, took 0.0866764 sec) FOF: have done 111 cross links (processed 2049, took 0.0172073 sec) FOF: have done 34 cross links (processed 482, took 0.0169553 sec) FOF: have done 14 cross links (processed 148, took 0.0166879 sec) FOF: have done 8 cross links (processed 49, took 0.0166575 sec) FOF: have done 6 cross links (processed 34, took 0.0166312 sec) FOF: have done 2 cross links (processed 27, took 0.0165797 sec) FOF: have done 1 cross links (processed 6, took 0.0167406 sec) FOF: have done 0 cross links (processed 2, took 0.016688 sec) FOF: Local groups found. FOF: primary group finding took = 1.9975 sec FOF: Start finding nearest dm-particle (presently allocated=900.035 MB) FOF: fof-nearest iteration 1: need to repeat for 39101736 particles. (took = 2.35763 sec) FOF: fof-nearest iteration 2: need to repeat for 25059015 particles. (took = 0.781495 sec) FOF: fof-nearest iteration 3: need to repeat for 11452984 particles. (took = 0.570783 sec) FOF: fof-nearest iteration 4: need to repeat for 2248959 particles. (took = 0.359851 sec) FOF: fof-nearest iteration 5: need to repeat for 822347 particles. (took = 0.0789208 sec) FOF: fof-nearest iteration 6: need to repeat for 792687 particles. (took = 0.00663462 sec) FOF: done finding nearest dm-particle FOF: attaching gas and star particles to nearest dm particles took = 4.18224 sec FOF: compiling local group data and catalogue took = 0.11205 sec FOF: Total number of FOF groups with at least 32 particles: 811 FOF: Largest FOF group has 1715263 particles. FOF: Total number of particles in FOF groups: 133405710 FOF: group properties are now allocated.. (presently allocated=848.378 MB) FOF: computation of group properties took = 0.0302158 sec FOF: start assigning group numbers FOF: Assigning of group numbers took = 0.0570338 sec FOF: Finished computing FoF groups. Complete work took 7.10743 sec (presently allocated=824.22 MB) SUBFIND: We now execute a parallel version of SUBFIND. TREE: Full tree construction for all particles. (presently allocated=836.733 MB) GRAVTREE: Tree construction done. took 6.3406 sec =278488 NTopnodes=14185 NTopleaves=12412 tree-build-scalability=0.949671 SUBFIND: finding total densities around all particles SUBFIND: ngb iteration 1: need to repeat for 129967148 particles. (took 72.0559 sec) SUBFIND: ngb iteration 2: need to repeat for 124292248 particles. (took 80.971 sec) SUBFIND: ngb iteration 3: need to repeat for 114157611 particles. (took 102.159 sec) SUBFIND: ngb iteration 4: need to repeat for 95496196 particles. (took 121.681 sec) SUBFIND: ngb iteration 5: need to repeat for 72379240 particles. (took 111.413 sec) SUBFIND: ngb iteration 6: need to repeat for 54528080 particles. (took 87.8812 sec) SUBFIND: ngb iteration 7: need to repeat for 43672228 particles. (took 77.4271 sec) SUBFIND: ngb iteration 8: need to repeat for 37338140 particles. (took 64.4948 sec) SUBFIND: ngb iteration 9: need to repeat for 33150591 particles. (took 50.9886 sec) SUBFIND: ngb iteration 10: need to repeat for 29464215 particles. (took 35.6219 sec) SUBFIND: ngb iteration 11: need to repeat for 25574006 particles. (took 23.8534 sec) SUBFIND: ngb iteration 12: need to repeat for 21301654 particles. (took 11.7117 sec) SUBFIND: ngb iteration 13: need to repeat for 16728670 particles. (took 3.87983 sec) SUBFIND: ngb iteration 14: need to repeat for 12193250 particles. (took 2.50376 sec) SUBFIND: ngb iteration 15: need to repeat for 7950451 particles. (took 1.86392 sec) SUBFIND: ngb iteration 16: need to repeat for 4408415 particles. (took 1.22157 sec) SUBFIND: ngb iteration 17: need to repeat for 1988084 particles. (took 0.69177 sec) SUBFIND: ngb iteration 18: need to repeat for 730593 particles. (took 0.321443 sec) SUBFIND: ngb iteration 19: need to repeat for 227229 particles. (took 0.123043 sec) SUBFIND: ngb iteration 20: need to repeat for 60823 particles. (took 0.0407217 sec) SUBFIND: ngb iteration 21: need to repeat for 14317 particles. (took 0.0201853 sec) SUBFIND: ngb iteration 22: need to repeat for 3303 particles. (took 0.0143671 sec) SUBFIND: ngb iteration 23: need to repeat for 1002 particles. (took 0.010251 sec) SUBFIND: ngb iteration 24: need to repeat for 454 particles. (took 0.00685589 sec) SUBFIND: ngb iteration 25: need to repeat for 300 particles. (took 0.00424718 sec) SUBFIND: ngb iteration 26: need to repeat for 232 particles. (took 0.00261451 sec) SUBFIND: ngb iteration 27: need to repeat for 216 particles. (took 0.00183333 sec) SUBFIND: ngb iteration 28: need to repeat for 199 particles. (took 0.00139278 sec) SUBFIND: ngb iteration 29: need to repeat for 149 particles. (took 0.001174 sec) SUBFIND: ngb iteration 30: need to repeat for 109 particles. (took 0.000938656 sec) SUBFIND: ngb iteration 31: need to repeat for 55 particles. (took 0.000689948 sec) SUBFIND: ngb iteration 32: need to repeat for 7 particles. (took 0.000376537 sec) SUBFIND: iteration to correct primary neighbor count and density estimate took 850.99 sec SUBFIND: Number of FOF halos treated with collective SubFind algorithm = 0 SUBFIND: Number of processors used in different partitions for the collective SubFind code = 0 SUBFIND: (The adopted size-limit for the collective algorithm was 134197368 particles, for threshold size factor 0.6) SUBFIND: The other 811 FOF halos are treated in parallel with serial code SUBFIND: subfind_distribute_groups() took 0.000212546 sec SUBFIND: particle balance=1.14697 ALLOCATE: Changing to MaxPartSph = 1746406 ALLOCATE: Changing to MaxPart = 5147463 SUBFIND: subfind_exchange() took 1.95781 sec SUBFIND: particle balance for processing=1.93321 SUBFIND: subdomain decomposition took 1.07749 sec SUBFIND: serial subfind subdomain decomposition took 1.06951 sec