Throughout the past two weeks time and time again I thought I was so close to getting the neural network to pick up on the patterns I wanted it to pick up on but there is still much work to be done.
The idea for my project entails making a neural network that can take some sort of shape as an image (so an array of pixel values) and place onto such image shapes that follow some specific rules (they can only be attached to one another via specific attachment points). This would then be used in conjunction with physical representations of the shapes in cloth form that can be strung together to create an ever-changing generative garment with endless possibilities for how it is used.
I began my journey to reaching this goal with the idea that maybe an image as an input would be too large and the next best thing would be the border of the shape that I was trying to make the neural network associate with the target shape x, y, and rotation values.
I had claude whip up a quick Shape class that consisted of two shapes that had attachment points on each corner, some rudimentary snap attachment functionality and most importantly the ability to relay their exact x, y, and rotation values.
Initially I used an array of p1 to p2 values to represent the different crude lines that were being used to represent the blob.
I also tried a variation of this method using ellipses that were drawn continuously to create a large array of x and y coordinates that would (I hoped) give a general pattern for where the blob’s outline was.
Initial attempt using line data (point 1 x and y to point 2 x and y) in conjunction with shape position. As can be seen by the example of the output from the trained model I cant quite say it was doing what I wanted it to but at the same time it was not terrible (just that the shapes were not connecting properly)
Using ellipses instead of lines. Not the best results, not the worst either considering i had trained this particular model with about 5 examples (it took a really long time to manually hand palce all the shapes
I then decided that, despite not having properly trained a single neural network up to this point (whether because I was feeding it <5 examples or just because the code was shoddy), I was going to switch gears and go a little further towards achieving my real ambitions of using a body segmentation model’s alpha mask output as the intput to train the neural netwrok to place shapes. Reflecting on it now this decision was probably not the best and I will highlight why in the next examples I show from my experimentation.
I did not jump straight into the deep end, beginning with pre-made “masks” that were once again drawn using ellipses but this time I was submitting the entire canvas as an array of grayscale rgb values as an input and hoping that the x,y, and rotation target output patterns would be picked up on by the model.
Not always super consistent but not terrible either
Initial prediction results were semi-promising (especially considering I once again was training all of these neural networks with about 10 examples each.
I got a little carried away with the fact that these results were promising and decided that there was no time to waste on things that were not yet to the scope of my final ambitions. SO I began implementing the infamous body segmentation alpha value mask.
I got to work and, with the help of claude, managed to integrate the Body Segmentation from ml5.js (BodyPix model to be specific) and thought that having it set to “person” was my compromise for attempting to do something that I kind of knew was a little daring. I was having some issue with an error during training that was saying my inputs were not normalized correctly (all either 0 or 1) and so I downsampled the mask and made each cell be a ratio of how many cells were detecting a person out of the total in that cell (This somewhat helped as the error still appeared every once and a while if I did not train it with enough examples).
When I first ran the training I was more than dissapointed, i was shocked. It did not pick up anything. And this is where I entered no training high loss limbo. It was quite painful indeed to see my little child i was attempting to teach and have learn not pick up on pretty much anything, despite my well curated examples I provided it.
And dont worry i tested it too and the predictions were worse than i could have imagined
Around this time I began suspecting that my average of 10 training examples per training were really not going to cut it, and in hopes that this was the true issue behind all my despair I enlisted claude once again to write me an algorithm to automatically place the shapes in simple configurations inside where the mask values were equal to 0.
The algorithm allowed me to supply my neural network with upwards of a 100 examples to train it on and finally i got some training results that really did make me proud
At this point after all the ups and downs my model seems to be training correctly but for some reason I cant quite get it to actually predict anything well. I hope this issue has something to do with how I am segmenting the prediction results and re-applying them to the canvas but I am not quite sure yet, that is a mystery yet to be solved.
Latest code: https://editor.p5js.org/FabriGu/sketches/4bQB_KfXR
All my various iterations of training and testing codes can be found here. Claude is responsible for the overcomplications in many parts of it and at the same time I could not have gotten so far in so little time and have pivoted so often without consulting it. https://editor.p5js.org/FabriGu/collections/zsVMivS2L
PHYSICAL ASPECTS