Optimizing loading data from array

[kravataf]

How do I…

the loading fully works… the only issue is the optimization (probably since im looping through every child of the array) the main issue of optimization isnt even the rendering (from what i tested but rather the events)

What is the expected result

i was hoping it wouldnt run this poorly since i already tried optimizing few things… (for example checking to remove children of array if there was a block placed/deleted, got changed into 3 events so it would “sort” through the blocks instead of checking each one…)

What is the actual result

900 blocks, no data = 165fps (thats what i have it capped on). 800 blocks, data of 100 deleled block = 90 - 105fps (ye its really bad)

Related screenshots

(the last screenshot is related to the actual result)

[kravataf]

im guessing the biggest reason for the lag is actually the first event. (for every child of worlddata array) since if you have like 100 children there it will probably lag no matter how much i optimize the other events since im looping over them :T im going to try to find a way to check without looping through every child

[ZeroX4]

Are you making some sort of object culling right there?
And i see it is for 3D

Anyway

1 huge trick that helps is mother freaking timer

I can go around 40k of tiles before i start to notice fps drop on my hardware

YET Before you read below be aware
1 - i suck with arrays
2 - i was not able to solve it cause lack of gdevelop knowledge
3 - idk how to make chunk system (i know how it should work but again lack the knowledge)

1 - I have was told that i can store arrays into storage which are not in use into storage and delete such arrays
2 - i know how to use storage idk how to manage arrays
3 - i know how to create grid and display it with shape painter dynamically yet idk how to give each square cell some ID to determine where and how it should be saved
Imagine creating separate arrays depending on your position so you do not have one long array but many small ones (and now combine it with storage)
4 - timer alone will help you A LOT you do not need to spam everything each frame it can be delayed
5 - all of the above i just have idea what could be done yet i have no idea how to do it so if you ask me any question i can ask you same question and we both will provide same answer

Chunk system alone would solve a lot of problems
Where on other hand storing arrays into storage would solve even more

And option B

Got 18$?

Cubic World - a premium game template for GDevelop. | GDevelop

[kravataf]

“Are you making some sort of object culling right there?” not yet, i think it wont make that much difference because if im running around without destroying/placing blocks im getting between 130 - 165fps (i have fps capped at 165) soo yeah. (im still planning to implement it later though)

“1 huge trick that helps is mother freaking timer” i mean i guess its better than checking every single frame… im going to try that!!

“Chunk system alone would solve a lot of problems” i sort of have a chunk system already and it did increase the performance by a lot but the main problem now is that the check for loading worlddata is checking each one every frame, each child also has 3 (or 4 if its worlddata.created children) in itself soo yeah… im going to try that timer somehow or just make it now run every frame or smth

“Got 18$?” im trying to make it from scratch and not use templates or stuff like that

[kravataf]

so yeahh i have a chunk system but as you can see the amount of blocks doesnt really matter on the performance that much its mainly the amount of checks being done each frame

[kravataf]

@ZeroX4 loool theres actually no way!! i added 1 timer for each and now its at my capped fps?

(i tried to recreate the last screenshot) so uhh thanks im going to test more to see at what point does it start to lag again but yeah

edit: nvm theres problem with loading… i need to change where the timer is but yeah massive performance boost

[ZeroX4]

Teach me how to do chunk system and i can sit with you and try to figure out how to optimize it even further

BUT in your case biggest problem is list of content

You know why it exist?
So when you have manual for vacuum cleaner
You do not read whole manual to find instructions how to clean it
BUT can just check on which pages are cleaning instructions
And read only them

And that is what you need
YOU DO NOT NEED to read whole array over and over
You only need part of it
And only way to do it is either save arrays into storage
OR create chunks by ID
For example if you would split your map into grid (i know its 3D i am simplifying)
And whole top row would be ID 1 and row below it be ID 2
BUT for you furthest visible row is with ID 50
Then what is the point into checking blocks in arrays that are anything below ID 50?

So what if you would have Structure which would have Arrays with chunks stored by X and Y ID of chunk
And now you do not check each block X and Y
You only check X and Y ID of chunk
Where you would have like 500 chunks to check
SO this vs 10 000 checks you can already see difference

And inside that arrays would be all blocks saved in that chunk
BUT that is something i would need to sit upon and try to deduce how to od
So again teach me how to do chunks i can try and sit with you to deduce how to make it per chunk ID

[kravataf]

@ZeroX4
“Teach me how to do chunk system and i can sit with you and try to figure out how to optimize it even further” its not really a “chunk” system but ill send screenshots of how i did it

“YOU DO NOT NEED to read whole array over and over
You only need part of it” ye i didnt figure out how to do that but that would also increase the performance by a lot (probably)

the “chunk” system:

[ZeroX4]

How it would increase performance?
I would say quite opposite
Look you have folder with 10 000 images
NOW you need to browse all of them to because they are called image1 image2 image3 … image10000

NOW what if they would be called EnemyIdle1 EnemyIdle2 PlayerIdle1 PlayerIdle2 and so go on

So just to find player idle animation you would not need to check every single file does it match what you need
But go to files that start with letter P
Now imagine you have 100000000 files in that one folder
Opening it alone would lag your pc even
And what if you would make sub folders there?
Enemies Player NPC Plants Backgrounds Animals and so go on
Instead of 100000000 files right of the bat you would have like 20 folders
And inside these folders A LOT less files per folder
And opening that would not lag your pc at all

Now imagine applying same BS to variables where each subfolders are just new arrays
So you check folder = what you need and if it does then you check content of that folder
And not every single file in parent folder

Mother of fak i look at your screenshot and i understand even less
BUT i see you did figure “generated” trick
I was about to tell you that you should have chunks that were not modified by player should be deleted and can be generated again when needed
Cause in their case there is no point in holding their data in array
So you do not generate huge array for no reason

Ok i go study what you made there
THX

[kravataf]

“I was about to tell you that you should have chunks that were not modified by player should be deleted and can be generated again when needed
Cause in their case there is no point in holding their data in array” i actually dont store any chunk data only changes made to the world like worlddata.deleted is basically list of every position where block was deleted. when you go into a “chunk” where that position is it deletes the block… the folder thingy would be great based on individual chunks but im not sure how i could exactly do that… anyways to make it more clear how it works im going to add screenshot of the worlddata array.

also “How it would increase performance?” if this was referring to me replying to the not looping through entire array i meant that it would be better to filter through it instead of looping… not sure how to exactly do that though

“Mother of fak i look at your screenshot and i understand even less” i know its probably spaghetti code but it checks if theres 0 blocks in the “chunk” and if == true then create chunk. if blocks found outside the view distance then delete, the generated variable is basiclly when block created it does stuff like setting z position, blocktype, etc. and the for i loop was js code which defined where the chunks should be placed or smth like that… i wrote it long time ago so i dont remember 100% but yeah…

so its like this… theres the “main var” worlddata which has children (created, deleted and seed which is irrelevant but ye) and then those children (created, deleted) contain changes made to the world and NOT every block just the changes soo not stuff like terrain…

this second screenshot is basically what data is stored… if you for example wanted to get the x of the 1 block placed in this world you would do worlddata.created[0][0] and for y worlddata.created[0][1].
0 = x
1 = y
2 = z
3 = blocktype

worlddata deletd only stores 0 - 2 since it doesnt need the blocktype…

not sure what else to add here…

[ZeroX4]

You see problem for me is not with your logic
Problem for me is I SUCK WITH ARRAYS
That is why i am able to only give you logic advice and not actual events how to do it

Like i can explain to you how your phone is detecting exact place where you touch the screen (electric short is sent trough largest point touching screen so center of place where you make touch) BUT that does not mean i could build you a touch screen

Other thing you are generating your world
I only need to store it and load it on 2D grid i would split into chunks
Like i said i know how to make grid with shape painter like dynamic one generated from few values
YET idk how to give each grid cel some number and or X and Y pos and use that in array to reference it by its position

But don’t worry one day i will figure it out
Screenshot from you will for sure help me understand something
So lets forget it for now

And about arrays and performance even looping from whole array

FOR EXAMPLE
Right now you are looping trough whole array to check if something is somewhere
So basically you check X and Y pos of something (In your case most likely also Z)
So you have loop trough 3 variables to check if they match with what you need

But to simplify imagine its 2D and let’s use my example

Look i check 2 things IF array X and Y is in some range (basically viewport + padding)
If it should be in that range then i create it
Simple to understand
NOW i am looping trough X and Y of every single object that is stored in my array
I cannot check some range of childs i need to loop trough whole damn array

So i have one main folder with a lot files to check
What if i could make in my main folder many sub folders and loop trough them instead?
I do not need to check every single object all the time
I only need to check if chunk that is in range i care to have objects
Is equal to some X and Y
So instead of checking 2000 vars for X and Y
I could check FOR EXAMPLE 40 sub folders or 40 childs for their X and Y and if their X and Y match what i need then check what is inside of them and inside would be X and Y of all blocks that such chunk should have

For example

Imagine i have 2000 stones here that is like x2 so 4000 child vars to check
What if i could do this

As you see i have here as most top child Chunk and its number next to it
With its X and Y 1st is at 0 0 2nd is at 200 0
So my chunks are 200 pixels wide (well height would be the same)
BUT under it you have multiple child vars representing each object such chunk holds
So instead of looping trough all 8 objects
I would loop trough 4 objects if i would only check X and Y range of single chunk

Imagine if in one chunk i would store 500 objects
Imagine how many chunks i could have before it would start to lag

Now you get the idea?

YET i do not know how to do it with arrays or does it even possible via events

[v9um]

I have took a look at the screenshots and I don’t understand what are they supposed to do. I’m assuming that these events should run to delete a block and or add a block. If this assumption is correct then why are you running this check every frame instead run it whenever the player presses a certain control like for example left click

If my assumption was false and this event should run every frame or should run continuously, then why not group each event and give them a name. Then run the performance profiler and see which group takes more time to process.

[kravataf]

well basically it loops through the entire array so it can delete and place every block that was saved in the array worlddata (.deleted or .created)… so yeah i guess the main reason for lag is the looping, so as @ZeroX4 suggested im going to try to save the worlddata.created (and deleted) into chunk arrays which are going to the children of the main array

[ZeroX4]

Imagine if you could VIRTUALLY split your your world into chunks
Now imagine if you could decide that chunks that are far away from player be saved to storage and deleted from array
And then loaded back from storage and added to array when chunk is close to player
That would make you have only 1 array yet think of it like zip games you do not play but you can always un zip them when you need them

I saw someone on discord having game with cubes like you and it was running smooth
I ask how he pulled it off and he told me just that and show tons of files that held child vars
YET he did not explain to me how to do it

Other thing is
IDK if this can help in anything but

GDevelop OpenWorld Example Using Static Chunking by danogster

Actual project you can download
Yet my knowledge is to poor to understand it

[kravataf]

“Now imagine if you could decide that chunks that are far away from player be saved to storage and deleted from array” are you talking about like the changes (that i save) or the entire chunk?

also im going to check out the example you linked

edit: after checking out the example (im not 100% sure) but it just seems like “over engineered” chunk system, because its not even infinite and when checking with debugger you can see that he only stores the letter (“a” for example since hes loading them from external layouts which i cant do… so yeah)

[v9um]

Can you please explain more what does this loop has to do? Also can you please try executing this loop only if certain keys or mouse buttons are pressed.

I had about the same situation and the fix was just simply not to run the loop and only run it when the left button is pressed.

[kravataf]

“Can you please explain more what does this loop has to do?” okay. im going to explain the worlddata.deleted but worlddata.created is pretty much the same thing… the main event is for each child in array so its basically going through the entire array and checking each child for every event below, 1: checks if a block was placed at the child pos == true then delete it from the array since theres no point in having it here. 2: checks if the child of the array is in the viewdistance, if true then it checks if theres a block with the same position as the child… and yeah thats pretty much it.

[v9um]

Don’t tell me how it works.
Tell me its purpose.
Is it a chunk system?
Is it a system for removing blocks and or adding them?

I’m referring to this system:

@kravataf

Related screenshots

[ZeroX4]

ANY CHUNK that is not needed should not exist in array at all
I mean NOT NEEDED CURRENTLY

Look imagine this

Green is chunk in which player is
Red are chunks which player see
Blue are chunks player do not see but need to exist in array
NOW all gray chunks does not need to exist cause of distance from player so you would store them in array
BUT instead if you would store them in storage then you could remove them from array
So you would not have that much to loop trough
And i mean whatever chunk was modified or not by player

Now player moves

Green rectangle indicates that player gonna move down and you gonna remove from array purple chunks and store then into storage
Where you gonna load from storage yellow chunks into array

So you will still dynamically expand your array BUT shiet stopper here is that you gonna also dynamically shrink your array by storing purple chunks into storage and then removing them from array

THAT is how you optimize your array
Array should be only a buffer where you should store your info elsewhere which is not run trough every single loop of array

Think of it as sorting pencils by color
You do not take whole box and try to dig trough all pencils
You only take bunch of pencils from box then sort them manually and then grab another bunch

[v9um]

Good idea, but wouldn’t that slowdown and cause MANY WRITE processes, which wears down SSDs?

Reading doesn’t affect the SSD, but writing too frequently and too much affects the age of the SSD negatively.