RUS New site

Advanced search

[ New messages · Forum rules · Members ]
Page 1 of 41234»
Forum » SpaceEngine » Mods and Addons » SDSS Galaxy collection 2.0 (Updated Sep. 5 2016) (101 million galaxies)
SDSS Galaxy collection 2.0 (Updated Sep. 5 2016)
ComCypherDate: Saturday, 13.12.2014, 05:53 | Message # 1
Space Tourist
Group: SE team
United States
Messages: 21
Status: Offline

Region of the Coma Cluster


Presenting the SDSS Galaxy collection. This addon contains 101,640,491 galaxies derived from the Sloan Digital Sky Survey, Phase IV Data Release 13 (http://www.sdss.org/dr13). Galaxy Zoo data (http://www.galaxyzoo.org) was used whenever available to determine the Hubble Tuning Fork classification of each galaxy. In addition, galaxy sizes, distances, luminosities, and orientations are all computed from the source data.

Current version: 2.0 (Sep. 5 2016)
SDSSGalaxyCatalog_GZ_2_0.pak (*Recommended*) (29 MB): Contains a single CSV catalog file containing 639,875 galaxies. These are the galaxies which include Galaxy Zoo data for classification, and tend to be the largest and clearest galaxies in the SDSS dataset.

SDSSGalaxyCatalogs_2_0.pak (*Experimental*) (4.5 GB): Contains 204 CSV catalog files, each containing 500 thousand galaxies. Due to potential limitations with loading large numbers of galaxies, this collection is for experimental use only.

To install, simply copy the PAK file to Space Engine's addons folder. Once in-game, you can search for SDSS galaxies by searching for objects beginning with "SDSS".


Data coverage


---Known issues---
  • The number of galaxies that can be loaded at one time may be limited by the software and your PC, which may be noticed when attempting to load more than a couple million galaxies.

  • Since Galaxy Zoo data is very sparse (only available for less than 1% of the galaxies), for most of the galaxies in the complete collection I had to assign a classification randomly based on known distributions of galaxy types.

  • Because the SDSS data coverage does not include the entire sky and is sharply defined, it may look odd when traveling in intergalactic space if "auto increase galaxies magnitude" is enabled. I recommend disabling that setting when using this mod.

  • Not an issue per se, but the number of galaxies in this mod is about 100 million less than what exists in the SDSS database. This is due to data cleansing that I needed to perform to remove galaxies with incomplete or faulty data.

  • In addition to millions of minor galaxies, the SDSS database also contains "famous" galaxies which can be found in other catalogs such as NGC. However, SE already includes many of those galaxies in its galaxies10k.sc file. This means that duplicates may be observed in-game where the SDSS galaxy overlaps the default SE galaxy. I haven't made any effort to identify or remove these duplicate galaxies in the current release.

    ---Information for developers---
    This is an open mod so I'm providing all of the information that was used to process the data, in case anyone wants to extend or improve upon it themselves.

    SDSS-IV-DR13-GalaxySourceData.zip (3.7 GB / 18.7 GB uncompressed): This zip contains a single large CSV file which is the output of the SQL resultset on the SDSS database.
    SDSSGalaxyQuery.sql: The SQL query that was used to pull the required columns from the database.
    SDSSGalaxyWriter.py: The Python script that was used to transform the source data into SE catalog files.

    SDSS navigation tool: This is a handy viewer for visually inspecting the galaxies in the SDSS database.
    CasJobs workbench: It is possible to create an account here and run queries against the SDSS SQL Server database, although the amount of data that can be downloaded may be limited (a member of the science team assisted me with the full extraction).


    600,000+ galaxies (magnitude artificially increased)


    Edited by ComCypher - Monday, 05.09.2016, 07:15
  •  
    DeathStarDate: Saturday, 13.12.2014, 12:16 | Message # 2
    Pioneer
    Group: Users
    Croatia
    Messages: 515
    Status: Offline
    Wow.... just .... wow. Just to let you know(you probably do already) there have been changes to the catalog loading system in 0.9.7.2 so SpaceEngineer said that it might be possible to load the entire SDSS database in a reasonable time. Maybe you should contact him so that he can try and include this catalog.
     
    SpaceEngineerDate: Saturday, 13.12.2014, 14:21 | Message # 3
    Author of Space Engine
    Group: Administrators
    Russian Federation
    Messages: 4799
    Status: Offline
    ComCypher, thanks for such a great addon!

    The first mistake: comment symbols is SE are "//", not "\\". Version 0.972 have an error check, and it immediately wrote an errors in the log.

    Second, you can significantly reduce the file size (and loading time) be deleting unnecessary spaces between the parameter and its value (you may leave a single tabulation), remove unnecessary leading "+" and trailing ".0", between opening and closing round bracket in the quaternion, also in the quaternion you can remove comas (they are not necessarily) and reduce accuracy to 6-7 digits. Example:

    Code

    Galaxy    "SDSS 1237648722848776804"
    {
      Type    "Irr"
      RA    13.1323
      Dec    1.17823
      Dist    243708336
      Radius    2276.75089
      AbsMagn    -20.64
      Quat    (0.2530524 0.9486663 -0.1538515 -0.1110235)
    }


    This edited entry have 186 characters, compared to 237 of original entry is almost 30% win.

    Also, if a galaxy have no orientation data, you may skip the Quat parameter - SE will generate it procedurally.

    In future I will implement a binary file format for such huge catalogs, with loading of the tiles on-the-fly, like planetary textures (or generation of procedural galaxies). But for now the text catalog should be compressed as more as possible.

    Anyway, SE 0.971 could not load all 53 catalogs for a obvious cause: their total size is 23 Gb, but SE can't load more than 4 Gb because it is a 32-bit app. Even if it would be a 64-bit app, I think only few people would have computers capable to load 23 Gb into RAM. Current 0.972 implementation even can't load a single catalog, because of limitation in the std::vector class (it is used in new accelerated script loader). So this great addon should wait implementing of the binary tiled catalog. Now I am forced to implement it for the next 0.973 version smile

    I have a suggestion now. What if limit the catalog size by 500,000 galaxies, selecting them by some principle (the most big, the most luminous, belonged to certain survey, belonged to a certain "2d" slice, etc). SE could load 500k galaxies (I just tested it - perfect performance, no problems in the map mode). This "filtered" catalog could be used with 0.971 and 0.972 until I implement the binary catalog.

    Regarding performance. You should use a parameters to set up the ocetree size and depth. Put something like this in the beginning of the catalog script:

    Code

    // Octree parameters
    OctreeSize    ( 1.0e9 1.0e9 1.0e9 )
    OctreePos    ( 2.0e7 3.0e8 2.0e8 )
    OctreeDepth    5


    OctreeSize is s span of the octree in parsecs. it should be choosen in that way, so 95% of galaxies would be inside it. The rest 5% (25k gakaxies for a 500k catalog) would be placed in a special resized octree nodes. This would get the best performance.

    OctreePos is a location of the center of the octree. It should not be zero, otherwise the MilkyWay could be clip out at some camera positions. Ie is should be located somewhere near the Milky Way, but in the intergalactic space.

    OctreeDepth is a depth level of the octree. The good value for a 500k catalog is 5-6.

    Note that SE takes octree parameters only from the first file, specified in the GalaxiesCatalogs in the universe.cfg. So your huge addons should be specified *before* the galaxies10k.sc, otherwise SE will use parameters for a small octree form the 10k catalog, resulting in extremely poor performance.

    You should experiment with these parameters to find a best performance. Disable procedural galaxies and all stars, planets, clusters and nebulae (so only catalog galaxies left). Disable VSync in the display menu to see the real FPS. Find several test locations and save them in the F6 menu. Locations could be inside the Milky Way, somewhere inside the catalog, and few gigaparsecs outside it. Measure the FPS in those locations at certain galaxy magnitude level (F7 menu). Then change the octree parameters in the script, restart SE and measure again.





     
    ComCypherDate: Saturday, 13.12.2014, 15:13 | Message # 4
    Space Tourist
    Group: SE team
    United States
    Messages: 21
    Status: Offline
    Those are great suggestions SpaceEngineer, thanks (I'm particularly embarrassed by the incorrect comment slashes surprised ). I will definitely get those things fixed and optimized in the next release.

    I also like your idea to create an additional "light" catalog of 500k galaxies. I can probably create a few based on size, luminosity, or Galaxy Zoo. Back to work cool .
     
    SalvoDate: Saturday, 13.12.2014, 20:40 | Message # 5
    Star Engineer
    Group: Local Moderators
    Italy
    Messages: 1400
    Status: Offline
    If my connection was better, I would definitely download this biggrin

    Good job!





    The universe is not required to be in perfect harmony with human ambition.

    CPU: Intel Core i7 4770 GPU: ASUS Radeon R9 270 RAM: 8 GBs

    (still don't know why everyone is doing this...)
     
    ComCypherDate: Saturday, 20.12.2014, 01:34 | Message # 6
    Space Tourist
    Group: SE team
    United States
    Messages: 21
    Status: Offline
    Changelog: 1.1
  • Created a "light" catalog containing 665,781 galaxies with Galaxy Zoo data, which should look and perform well on most PCs.
  • Octree settings have been computed and included in each catalog file which should help improve performance.
  • Reduced the number of galaxies per catalog from 2 million to 500 thousand to reduce the memory footprint of each file.
  • Corrected orientation calculations for elliptical galaxies.
  • Various catalog formatting fixes and optimizations to reduce file size. The size of the uncompressed distribution is now about 20% smaller than before.
  •  
    SpaceEngineerDate: Monday, 22.12.2014, 09:43 | Message # 7
    Author of Space Engine
    Group: Administrators
    Russian Federation
    Messages: 4799
    Status: Offline
    When 0.972 will be released, you can make a pak files for this addon, and distribute it in that form. Pak files are zip archives, compression for text files is usually very good, so this catalogs will have a huge benefit from that.

    Update: I just copied the SDSSGalaxyCatalog_GZ_1_1.zip into catalogs/galaxies folder and renamed it into .pak, and it worked)





     
    isdebeslDate: Tuesday, 23.12.2014, 00:40 | Message # 8
    Space Pilot
    Group: Users
    Indonesia
    Messages: 87
    Status: Offline
    the link doesn't work
     
    ComCypherDate: Tuesday, 23.12.2014, 00:57 | Message # 9
    Space Tourist
    Group: SE team
    United States
    Messages: 21
    Status: Offline
    Quote SpaceEngineer ()
    When 0.972 will be released, you can make a pak files for this addon, and distribute it in that form.

    Yes indeed, already on my to-do list smile . As soon as 0.972 is released I'll put up a pak link and update the instructions.

    Quote isdebesl ()
    the link doesn't work

    Can you elaborate on which link is giving you trouble? I just tried both and they seemed fine.


    Edited by ComCypher - Tuesday, 23.12.2014, 00:58
     
    isdebeslDate: Tuesday, 23.12.2014, 02:00 | Message # 10
    Space Pilot
    Group: Users
    Indonesia
    Messages: 87
    Status: Offline
    I'm clicking the recommended one, it load very slow (not because slow connection), and its said webpage is not available, im check the short url checker, and the result is this

    Added (23.12.2014, 01:00)
    ---------------------------------------------
    plz halp

     
    ComCypherDate: Tuesday, 23.12.2014, 03:03 | Message # 11
    Space Tourist
    Group: SE team
    United States
    Messages: 21
    Status: Offline
    I'm not sure if that URL checker is reliable. Here is the actual long link, give this one a try:

    *EDIT: Redundant link removed.*


    Edited by ComCypher - Tuesday, 23.12.2014, 19:14
     
    isdebeslDate: Tuesday, 23.12.2014, 11:37 | Message # 12
    Space Pilot
    Group: Users
    Indonesia
    Messages: 87
    Status: Offline
    The link's work! thx, actually i'm download it at firefox, it doesn't work on google chrome, idk why, but thx alot!

    Added (23.12.2014, 10:37)
    ---------------------------------------------
    Oh ya, i have a suggestion, what if SDSS Stars collection?? it would be awesome and make this game more fun to explore!! how 'bout that?

     
    ComCypherDate: Tuesday, 23.12.2014, 19:12 | Message # 13
    Space Tourist
    Group: SE team
    United States
    Messages: 21
    Status: Offline
    Glad to hear that. Also I have thought about processing the SDSS stars, but obviously stars have very different properties from galaxies so I would have to develop a brand new process to handle those. I haven't done much research it that yet but I might get around to it at some point.
     
    SpaceEngineerDate: Tuesday, 23.12.2014, 22:10 | Message # 14
    Author of Space Engine
    Group: Administrators
    Russian Federation
    Messages: 4799
    Status: Offline
    I can point on a obvious issue, that can make impossible to use SDSS stars catalog in SE. Are they have a distance measured? I guess no.




     
    isdebeslDate: Wednesday, 24.12.2014, 03:52 | Message # 15
    Space Pilot
    Group: Users
    Indonesia
    Messages: 87
    Status: Offline
    Oh, sorry, i didn't know that

    Added (24.12.2014, 02:52)
    ---------------------------------------------
    Quasars: http://das.sdss.org/va/qso_properties_dr7/data/catalogs/
    Stripe 82 Standard Star Catalog: http://www.astro.washington.edu/users....82.html
    UCAC3 Catalog (maybe? idk): http://sourceforge.net/projects/skychart/files/2-catalogs/Stars/UCAC3/

     
    Forum » SpaceEngine » Mods and Addons » SDSS Galaxy collection 2.0 (Updated Sep. 5 2016) (101 million galaxies)
    Page 1 of 41234»
    Search: